Top 10 myths and illusions - SOCO Norge AS...2017/09/28 · company 1: started with 1:8 ratio after...
Transcript of Top 10 myths and illusions - SOCO Norge AS...2017/09/28 · company 1: started with 1:8 ratio after...
© Lloyd Roden Illusions120215
illusions-1
1
Dispelling Testing’s Top Ten Myths and Illusions
2
Contents
What are myths and illusions? The top ten myths and illusions
Taking action
© Lloyd Roden Illusions120215
illusions-2
What is a myth?
true or false? eating lots of carrots helps you see in the dark? eating the crusts of bread makes your hair curl? eating fish gives you brains? you must drink 2 litres of water a day? chocolate causes acne? watching TV too much gives you square eyes? stepping on a rusty nail causes tetanus feed a cold, stave a fever
3
a popular belief or story (fiction or half-truth) that has become associated with a person, institution, or occurrence,
especially one considered to illustrate a cultural ideal. A made up story that seems believable.
half truth
half truth
false
false false
false
false half truth
What is an illusion?
4
An erroneous perception of reality. A misleading image presented as a visual stimulus. Something we see which is
not there, or want to believe is there.
© Lloyd Roden Illusions120215
illusions-4
7
Contents
What are myths and illusions ? The top ten myths and illusions
Taking action
#1: test estimation involves effort and resource
fact: estimation is based on 2 aspects
effort + people = schedule half truth for testing:
effort + people + quality = schedule we must also estimate the number and severity of the bugs we are going to find to make our estimates meaningful (a direct correlation)
WARNING: be careful of Brook’s law – 9 women cannot have a baby in 1 month! Some activities cannot be accelerated by adding more resource
8
© Lloyd Roden Illusions120215
illusions-5
9
Estimate, monitor and report quality
how to estimate quality guess use historical data (previous projects, defect density) use models
what to monitor and report actual versus expected trend analysis and forecasting correlation between graphs
0
10
20
30
40
50
1 2 3 4 5 6 7 8 9 10 0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10
Cycle #
Actual bugs
Masked rate
Observed bugs
Fix Rate
Fixed bugs
Insert rate
Inserted bugs
Remaining bugs
1 10 30% 7 80% 6 20% 1 5
2 5 30% 4 80% 3 20% 1 3
3 3 0% 3 100% 3 0% 0 0
4 0 full regression test
Count the black dots!
10
© Lloyd Roden Illusions120215
illusions-6
#2: large numbers are better than small numbers
some large numbers billion 1,000,000,000 trillion 1,000,000,000,000 quadrillion 1,000,000,000,000,000 quintillion 1,000,000,000,000,000,000
large numbers tend to become meaningless without the context £1,698,100,000,000
UK Debt - Office of National Statistics Jan 2017 managers want lots of information
often unsure as to the purpose often wanting large numbers
but meaningless without the context 11
when your boss asks for all that
unnecessary information
the only difference between a rut and a grave is the
dimensions
Challenge the information that is requested
we provide endless information for our managers number of tests run number of tests passed/failed number of scripts automated number of defects raised/fixed
we provide this information on a regular basis... but what is it used for? who is analyzing the data? is it really needed?
is there a better way? different information educating managers regards test
12
…the larger the number the better it is
© Lloyd Roden Illusions120215
illusions-7
Is the ball closer to the top or bottom?
13
#3: authors know their subject
signs that they don’t plagiarism rather than uniqueness generalization rather than specifics vagueness rather than detail avoidance rather than openness nervousness rather than confidence
14
authors of what? requirement specifications design specifications code test documentation technical books presentations
they might write about it…but do they know their subject?
© Lloyd Roden Illusions120215
illusions-8
If in doubt…challenge!
study the body language use walkthroughs rather
than other review types ask questions
15
remember there is no such thing as a stupid question…only a stupid answer
anyone who produces something for the benefit of others has the right to be challenged
#4: test (development) managers will always be needed
16
1980s 1990s
2000s
prediction for 2020s
the need to change
© Lloyd Roden Illusions120215
illusions-9
Test (and development) managers need to change
traditional Test Manager fulfills 2 main roles: organizing and leading the test teams planning, monitoring and controlling what gets tested
these roles are often covered by: the Scrum Master the self-organizing Scrum teams
how can they change? become more strategic adapt to different roles
scrum masters enhance their skills
writing utilities and tools for more efficient testing data collection, statistical analysis &process improvement testing of the product!
17
#5: anyone can test
facts: anyone can run tests
this depends on the detail of the script many organisations encourage different people to test
business, developers, customer service and customers
but… this doesn’t mean they are good testers
and neither does a certificate make you a good tester!
18
following a recipe doesn’t make me a chef
© Lloyd Roden Illusions120215
illusions-10
“An excellent tester this one is”
19
a “jedi” tester will… let themselves be distracted by anomalies
and new ideas encourage variability, avoid repeating the
same test twice use their confusion as a resource know their limits and get to know/use
subject matter experts adapt testing to fit the context develop resources and tools train their minds to be curious, cautious
and critical take notes and report results in a useful
and concise way
#6: outsourcing/offshoring is cost effective
remote teams fall into 4 categories
short-sited view disturbing case studies
company 1: started with 1:8 ratio after 2 years 1:3 ratio company 2: internal offshore, after 3 years spent more in sending UK staff to Mumbai company 3: found external/offshore team wrote 10,000 test cases in excel with just a name change for each test case company 4: in 2016 out-source IT department to India. Major problems resulted in £500 million payout to customers
20
cheaper labour = cost saving
internal external off-shore on-shore
outsourced…external to the company
© Lloyd Roden Illusions120215
illusions-11
The need for a longer-term view
balancing costs and risks
risks with outsourced/off-shore teams quality of the work they produce overhead of hand-holding/knowledge transfer only do what they are told to do communication and cultural differences staff churn/loss of skill and knowledge management overhead failure to deliver security and IPR issues getting trapped 21
risks costs
test the testers and
put in penalty clauses
Is the brim of the hat smaller than the height?
© Lloyd Roden Illusions120215
illusions-12
#7: software quality is achieved through testing
fact: testing can find bugs removing bugs improves software quality
myth: conclusion: testing improves software quality
removing bugs is a development activity
23
Testing found 5 bugs
net result: quality is still the same
Development fixed 5 bugs
net result: quality is improved
testing together with
development improves the quality of the
software: good bug
reports are key to success
Testing assesses software quality
24
Low software quality High
we think we are here
but we are really here
Low
te
st q
ualit
y H
igh
To have justified confidence in the
quality of the software we MUST have
justified confidence in the
quality of our tests
© Lloyd Roden Illusions120215
illusions-13
#8: detailed scripts better than exploratory testing
A
B 25
the problem with detailed scripts
Dangers of detailed scripts
reasons to write detailed scripts describes detailed tests for inexperienced testers automation provide an audit trail
risks boring to write…boring to run more time spent writing than running tests maintenance nightmare false confidence in the quality of the
software your best testers will probably leave
26
empirical evidence shows that exploratory testing finds 13 times more defects than scripted testing
© Lloyd Roden Illusions120215
illusions-14
#9: we must automate everything for efficiency and effectiveness
fact: automation can provide increased efficiency IF…
automation effort < manual effort myth: automation is effective at finding bugs how good are your regression tests?
automated the easy tests automated the happy path continued to add to the regression tests
but who is testing the tests? a project idea...
run your regression tests error seed your system with 100 bugs run your regression tests again count how many of the bugs have been found
27
Disturbing case studies...
company 1: project spent most of test effort in automation,
testers became automation specialists…and lost their testing skills. Software quality was poor.
company 2: automation team so “locked” into providing an
automated solution, spent 1 week of effort. Manual tester took 1 hour to do the same work.
company 3: outsourced team had automated those 10,000
test cases all doing the same thing ! all they achieved was “faster rubbish”
company 4 we want to automate 90% of our tests by the end
of 2017 28
© Lloyd Roden Illusions120215
illusions-15
# 10: software testing is a co-dependent behaviour
fact: testing is dependent on many aspects in a project good quality requirements good quality environments good quality code
myth: testing is given... good quality requirements good quality environments good quality code
29
The term codependency was coined to describe an unhealthy pattern of coping with life as a result of being involved with someone who is
chemically dependent.
Testing as a co-dependent behaviour
their inappropriate behaviour
testing’s codependent response
Formal requirements for the software were either not written or are inadequate in scope and/or content.
We “do our best” by testing whatever we can.
The time allotted to testing in the project schedule is woefully inadequate.
We “do our best” by testing as much as we can.
Adequate testing process and/or tools are not provided for the testers.
We “do our best” by testing however we can.
The quality of the software was not as good as we anticipated
We “do our best” by testing as much as we can
30
© Lloyd Roden Illusions120215
illusions-16
31
Contents
What are myths and illusions ? The top ten myths and illusions
Taking action
Why do we believe in myths or allow ourselves to be illuded?
don’t know any different – seems plausible myths are “passed down” through generations some myths are based on partial truth
they are often believed by many people easier to swim with the current than against it source is often trusted (your grandma is always right!)
out of sight, out of mind our attention can be directed to other things
32
© Lloyd Roden Illusions120215
illusions-17
What can be done?
train the eye watch for the symptoms recognise early warning signs
educate the mind learn how myths and illusions are
formed on projects, gather facts take action
it is vital that we provide the right information at the right time to dispel the myths and illusions
set yourself goals and achieve them John, Susan, Nora and George
33
34
Summary
myths and illusions are everywhere on our projects
my top ten myths and illusions maybe present in your organisations
educate your mind and train your eye to recognise myths and illusions when they appear and dispel them as soon as you can