Exploratory Testing Explained and Experienced
-
Upload
maaret-pyhaejaervi -
Category
Spiritual
-
view
1.466 -
download
3
description
Transcript of Exploratory Testing Explained and Experienced
Exploratory Testing –Explained and Experienced
Maaret PyhäjärviEmail: <[email protected]> | Gsm: 040-8233777
"The purpose of testing is largely to force everyone to see the detailed implications of their vague daydreams." -- James Bach Turku Agile Day
2011(2011-03-17)
Erkki Pöyhönen & Maaret Pyhäjärvi Attribution (Tekijä mainittava) http://creativecommons.org/licenses/by/3.0/
About the Speaker: Experiences in testing and being a tester on varied basis
• Started with testing -95: localization testing for greek version of an English software system
• Subcontractor-side on various projects: moving from localizations to functional testing
• Tried being in a developer role after thinking testers are not respected: the problem is not per role
• Teaching testing at HUT, giving public presentations and courses on testing
• Researcher in testing, consulting various organizations, teaching, being a test manager and a tester, with and without test automation
• At F-Secure with product business 2005 – 2008 • Moved to insurance pension sector, first with subcontractor side,
now on customer side (Ilmarinen) as test manager / test specialist. • Doing testing-related training on the side of a normal day-to-day
job to get to meet brilliant testers in Finland.
TESTER AND A WALKING TESTING DICTIONARYAGILE PROJECTS @ F-SECURE
PLAN-DRIVEN PROJECTS @ ILMARINEN
3
Realizations about Nature of Testing
2014
16395±2
”Testing mathematics” Extra Short Curriculum
A kids game ”20 questions teaching something relevant about testing. With two optional strategies to the game: 1) listing questions beforehand at a time you know the least or 2) asking questions one by one learning from the answer to the question already asked. Why in testing we continuously play with the foolish strategy?
Common advice in testing is to think the expected results beforehand – in avoidance of blindness. Testing targets the unknown problems. Why do we invest a significant portion of limited project budgets actually to minimize the probability of finding the unexpected problems?
Human memory is limited – ”human envelope”. We need documentation as support, especially when working under pressure. Documentation is huge expense, which only passes small portion of information. Why not create the documentation at the time we understand the needs best – towards the end of project.
2014
16395±2
The same may not be the same if you change any variable. Testing is not about covering all of it against a specification, but thinking about ”is there a problem, is it relevant”.
• http://viscog.beckman.illinois.edu/flashmovie/15.php• http://www.dothetest.co.uk/whodunnit.html
5
Exploratory Testing:Better tests, better testers!
• An approach to software testing– Emphasized freedom and responsibility
of an individual in a process where continuous optimization of value of information is important
• “Any testing process that involves simultaneous learning, test design, and execution.” – James Bach & Cem Kaner
• Disciplined, planned and controlled testing that emphasizes continuous learning
• Research in Finland, Itkonen et al. 2007– No significant difference in results for
preplanned test cases and exploratory testing– More false alarms with test cases– Comparing overall effort: significantly more in
test-case based testing
Unknown territory
Test-related learning
Design of new tests
Test executionResult interpretation
6
There’s a Difference!• A comment I hear often on the ET course I
give:– ”I’ve always been doing this just did not give
it a name”• I must emphasize that:
– I require more out of exploratory testing than just ”going where you feel like while testing”
• Ask yourself:– Can you defend your choices of strategy and
tactics?– Can you explain what you’ve done in words
that don’t just explain numbers of bugs found?
– How do you know if you’re done or not?
”My testing would be unsystematic ad
hoc testing if I could not tell the story of my tests, remember what
I’ve tested or what my strategy was, or relate that back to
my mission” –James Bach
7
The Scale of Tester’s Freedom
Pure scripted
Vague scripts
Fragment test cases (scenarios) Charters
Role-based sandboxes
Freestyle
”To what extent am I in control of the test and from where did the idea originate?”
8
Two Sides of Exploratory Testing
For the tester For the manager
FreedomFun
Flexibility
Professionalism
Respect
Value
Controllability
Reliability
Visibility
9
Framework of Management
”A day’s work”
Vision (“Sandbox”) Current Charter
Other Charters Details
Bug Reports
Perception of quality and coverage
Quality ReportDebriefing
Tester
Test Manager
PastResultsObstaclesOutlookFeelings
?#
xCharter backlog of the future testing
Out of budget
Next in importanc
e!
#, ?, x, +20:20:60
Session sheets of the past testing
Idea of exploration
Metrics summary
Coaching
10
The Pieces in Management Framework
• A disciplined tester replanning on various levels• Session with charter that provides a report • Classification of information created as metrics• Prioritizing of what test idea comes next• Supporting reporting by debriefing• Supporting skills development by coaching• Creating a combined judgement of quality by
quality reporting
Test automation / tools in ET
• Any form of acquiring quality-related information fits into exploratory testing
• It’s not manual, it’s brain-engaged – and for making that happen, you need to be smart with automation and tools!
• Example: you might not at first know what you’re looking for...
Search "<ns1:Koodi>" (443 hits in 169 files) in VE Y:\ELLU\LAPA\Testaus\Kokonaiseläketurvan-ote\Ansaintatiedot-VE\HaeAnsaintatiedot-VE__ 20110307 14-22-15.xml (4 hits) Line 1313: <ns1:Koodi>YL130I</ns1:Koodi> Line 1317: <ns1:Koodi>LAPA_172_011</ns1:Koodi> Line 1321: <ns1:Koodi>67</ns1:Koodi> Line 1324: <ns1:Koodi>67</ns1:Koodi> A lot of text cut away from here…
Potentially Useful Tools for Exploratory Testing
• SnapzPro– http://www.ambrosiasw.com/utilities/snapzprox/
• HyperSnap – http://www.hyperionics.com/hsdx/
• TestExplorer – http://www.sirius-sqa.com/
• BBTestAssistant – http://www.bbsoftware.co.uk/
• Spector – http://www.spector.com/spectresoft.html
• Cygwin – http://www.cygwin.com/
• UnixUtils – http://unxutils.sourceforge.net/
• Regular Expressions– http://www.ilovejackdaniels.com/cheat-sheets/regular-
expressions-cheat-sheet/
• Process Monitor– http://www.microsoft.com/technet/sysinternals/
processesandthreads/processmonitor.mspx
• Process Explorer– http://www.microsoft.com/technet/sysinternals/
processesandthreads/processexplorer.mspx
• Holodeck– http://www.securityinnovation.com/holodeck/
index.shtml• perlclip
– http://www.satisfice.com/tools.shtml• DataGraph
– http://www.visualdatatools.com/DataPlot/index.html• Vermont High Test
– http://www.vtsoft.com/• Firebug
– http://www.getfirebug.com/• iMacros
– http://www.iopus.com/imacros/firefox/• JMeter
– http://jakarta.apache.org/jmeter/• VirtualPC
– http://www.microsoft.com/windows/products/winfamily/virtualpc/default.mspx
• VMWare – http://www.vmware.com/
• Parallels – http://www.parallels.com/
Test Ideas / Quick-and-Dirty
Download the full 2-page Cheat Sheet with ideas from Elisabeth Hendrickson, James Lyndsay, and Dale Emery on Qualitytree.com
(Exploratory) Testing Dynamics Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing Dynamics.
v.2.2. 2009
Evolving work
products
Skills and tactics
Testing polarities
Test strategy
”A set ofconsiderations designed to
help you test robustly or evaluate someone else’s
testing.”
” To develop ideas or search a complex space quickly yet thoroughly, not only must you look at theworld from many points of view and perform many kinds of activities”
”Exploratory testing spirals upward toward a complete and professional set of test
artifacts”
” …skills that comprise professional and cost effective exploration of technology. Each is distinctly observable and learnable, and each is necessary to exploratory work.”
Exploration SkillsSource: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing Dynamics.
v.2.2. 2009
Self-management
Developing ideas
Examining product
DoneTo DoIssuesCoverage
All sources available Best use of time – effective and efficient work
Making modelsTool support – creative solutionsRisk-based testing – scientific approachKeeping one’s eyes open
To End This With: Exploratory Testing Strenghts and Blind Spots
Strengths• Value-focused: stakeholders, risks,
good-enough quality and knowledge of quality
• Leverages each tester’s strengths • Makes testers better continuously• Enables adapting to changes and
surprises• Reveals information that was not
available without learning while using the system
• Avoid non-valuable duplicative work in testing through focus on results
• High bug finding rates
Blind Spots• Requires calendar time for
learning with the software: the less we know, the more we’re likely to miss
• Limited by each tester’s weaknesses if not mitigated by overlapping assignments based on individual’s competencies
• Skilled and disciplined work – can be learned with mentoring, not transferred through documentation
WHY do testers look both ways when crossing a one-way street?
• It may be one-way on this corner, two ways on that corner.
• You may only *think* it's a one-way street.• Someone might be backing up• Ears don't warn you when a bicycle is coming the
wrong way.• They realize people can be fooled by preconceptions. • When helping someone, you've extra duty to seek
improbables.
Source: Various tweets by testers in 2010.