GSJGD Exploratory Testing: Real world rapid testing in ... · Testing for customer value is...
Transcript of GSJGD Exploratory Testing: Real world rapid testing in ... · Testing for customer value is...
Exploratory Testing: Real world rapid testing in Agile teams
Ravi Shanker Principal Program Manager, Microsoft Corporation
GSJGD
Agenda
Focusing on customer value through the lifecycle
Exploratory testing – why care?
Debunking exploratory testing myths
The Agile Workflow
Customer value – where are you?
Customer value in backlog item
Tester = customer proxy
Is the backlog item definitive customer value?
Is this the best customer value to deliver this sprint?
Exploratory testing
Centered on customer value rather than spec compliance
Find bugs fast and early without overhead
Tests working software!
Exploratory testing – myths
1. Exploratory testing is ad-hoc, random testing
2. There is no way to measure exploratory testing
3. Bugs found through exploratory testing are hard to repro
What are you DOING?
Method in the madness – “Testing Tours”
Guidebook tour
Back alley tour
Supermodel tour
You can make up your own…
Dogfood IDC NAR
Sample tours – how we did it
The Reflex tour – tests responsiveness of the IDE
The Idiot tour – tests F1 inline help on each element
The Palatial tour – tests our multi monitors experience
The Footprint tour – tests virtual memory usage
The Budget tour – tests experience on a low end machine
Dogfood IDC NAR
Food for thought…
For the application that you are working on, build a “tour” that you think will be useful for exploratory testing.
Dogfood IDC NAR
Where are we?
Measuring exploratory testing
Right metrics are like a fingerprint of your team: no silver bullet
Distinguish between app quality metrics and metrics that direct exploratory testing
Metrics – how we did it
Code coverage across tours
Story complexity to bugs ratio
Session duration for stories
Session roll up per tour
Heat maps – how we did it
Food for thought…
For the project that you are working on, think of what metrics are most meaningful to direct your exploratory testing
Dogfood IDC NAR
Finding reproducible bugs
Bug filing with exploratory testing
3 components of time spent
during exploratory testing
(courtesy Jon Bach)
• S = Setup time
• T = Testing time
• B = Bug filing time
S
T
B
S
T
B
“Cut the BS and get more T !!”
Rich bugs – how we did it
Automatic repro steps
Screenshots
Desktop recording
System information
Intellitrace logs
Test documentation – how we did it
Test cases automatically created while doing exploratory testing
Can save artifacts from exploration like expected image
With each sprint, regression suites are automatically ready
Conclusion
Testing for customer value is paramount throughout the product lifecycle
Exploratory testing done with right tools enables you to:
Define and vector testing as appropriate for your app
Communicate your testing goals and progress to the team
Test more frequently and file comprehensive bugs faster
Go on…be a fearless explorer!
Q&A