Dare to Explore…

Discover ET!

Outline •  What Exploratory Testing (ET) is and isn’t •  Dissecting Exploratory Testing – Learning – Designing – Executing

•  Fitting it within the Agile Context •  Adoption Experience •  Q & A

What does Exploratory Testing mean to YOU?

Testing Pyramid

Integration Tests

Unit Tests

Developer-centric Are we building the code right?

Business-centric Are we building the right code?

Adaptation of Mike Cohn's test automation pyramid

Exploratory Testing

GUI Tests

Acceptance Tests (API layer)

veri

fica

tion

What is Exploratory Testing?

Test-related learning

Test design

Test execution

Test result interpretation

Exploratory Testing is a mindset

Embrace continuous improvement of test quality

Enabling choice, not constraining it

Tests are experiments for learning

Push the boundaries, don’t execute routine checks

Embrace continuous learning

Exploratory Testing… IS NOT IS Doing random stuff to see what happens

A rigorous investigative practice

Repetitive Adaptive

Replacement for predesigned testing

Complimentary to conventional techniques

About producing comprehensive test cases for later use

Creative & more free-style

Why ET?

Two sides of Testing

Tested Checked

(scripted manual + automated)

Explored

“No matter how many tests we write, no matter how many cases we execute, we always find the most serious bugs when we go off the script” Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson

Checking

•  Designing scripts with expected results upfront

•  Verifies what was working still works

•  Well documented and repeatable

Exploring

•  Learning, designing and executing at the same time

•  Tests the system capabilities, boundaries

•  Can be sparsely documented and hard to repeat

Exploration with a general mission without specific step-by-step instructions Effectiveness relies on the tester’s knowledge, skills, and experience Simultaneously learning about the software under test while designing and executing tests, using feedback from the last test to inform the next.

Characteristics of ET

ET is a fit where… •  Need for manual testing approach that’s adaptable

because the SUT changes rapidly

•  Requirements change often, and the specifications are vague or incomplete

•  Writing detailed test scripts for a manual test effort doesn’t make sense

•  Need to provide rapid feedback on a new product or feature

Key testing problem

• We must test a product whose purpose is emerging and/or changing

•  Coded without fully specified requirements upfront

 

Agile fit?

ET: Nuts & Bolts

The Flow…

•  Focus the exploration

•  Learn about the system/features

•  Design tests with interesting variations

•  Attack and report

Unscripted doesn’t mean unprepared  

Test charters bring Focus

Charters define the area(s) we are testing and the kind of vulnerabilities we’re looking for -  Features -  Suspected risky areas of the system -  Coupling points

Meandering without direction/purpose is getting lost

 

Sample Charter

•  Target: What are you exploring? Feature, component, interface etc. •  Resources: Data set, technique, configuration etc. •  Information: What are you hoping to find? Is it security

vulnerability, performance, capability, usability etc.

Explore <target> With <resources> To discover <information>

Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson

Explore the Foo Form Fee Payment functionality using active accounts in the Demo environment to discover how the payment interface responds.

Template Example

Learn the general shape

What does the system do?

Input, Store, Compute, Output

“I know that I do not know” ~ Socrates

 

Design tests with interesting variations

The narrower the view, the wider the ignorance

 

•  Varying sequences of actions and varying timing

•  Test error handling by making required resources

unavailable or locked, or to break connections

•  Inventing personae to generate extreme scenarios

•  ...

Use heuristics to guide test design on the fly

 •  Test Heuristics Cheat Sheet: (Elisabeth Hendrickson, James Lyndsay, and Dale Emery)

http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

•  How To Break Software: “17 Attacks” (Alan A. Jorgensen, James A. Whittaker)  

 

Ideas for Executing & Recording All code is guilty, until proven innocent.

•  Explore Early, Explore Often •  Time-box the session •  Line up the team •  Surfacing potential problems/risks quickly is the goal •  Work in pairs or groups, whenever possible •  Be distracted by anomalies and new ideas •  Avoid test repetition •  Document your findings in an appropriate way (low

overhead is key)  

Running ET Sessions

Moving Quality Concerns

Upstream

Previous State Bug bash and IV&V executed just before the release

Target State Using a whole team approach to exploring early and often, we hope to surface and address risks sooner, and that can lead to better software, fewer surprises, and greater confidence

Approach

•  Synchronized Cadence across 8 Agile teams

•  Test Charter Autonomy •  Time-boxed Exploratory Sessions

-  Explore & Record Observations -  Summarize and Report

Current  Enterprise    Comprised  of  many  disparate  data  sources  and  systems,  which  leads  to:  

Inconsistent  Data  Data  is  defined,  captured  and  interpreted  inconsistently  throughout  the  Enterprise  

Ad-­‐Hoc  and  Near  Real-­‐Time  Analysis  Limita<ons  Limited  ability  to  conduct  ad-­‐hoc  and  near  real-­‐>me  analysis,  delaying  >mely    decision-­‐making  

Cross-­‐Func<onal  Domain  Limita<ons  Inability  to  provide  a  cross-­‐func>onal  view  across  data  domains  easily  or  completely  

Lack  of  Agility  in  Response  to    Evolving  Business  Model  Lack  of  agility  to  adjust  data  procurement  and  delivery  in  response  to  a  rapidly  evolving  business  model  

Lack  of  Data  Across  the  Enterprise  Data  gaps  exist  due  to  limited  understanding  of  current  and  an>cipated  data  usage  and  applica>on  across  the  Enterprise  

Notes From the Trenches Challenges and solutions transitioning a traditional test team into an exploratory test team on a large, heavily interfaced system.

I’ve a feeling we’re not in Kansas anymore Challenge: Changing a mindset Solution: Framework, cadence, training, discussions, and metrics

Why do you want access to my system? Challenge: You need “visas” to explore Solution: You need allies, and strong ones

Yet another build!!!? Challenge: Continuous Delivery means continuous deployments Solution: Test in parallel with and after production deployments

What is truth? Challenge: Acceptance criteria vs. Desired Outcome Solution: Find someone who cares

Boldly going where no one has gone before Challenge: Indigenous population may be indigent Solution: “Marketing” brochure and chocolate chip cookies

Concluding thoughts

Benefits

Unleashes tester’s creativity and skill Ability to perform focused testing without comprehensive documentation Rapid flow of feedback to the team

Based  on  the  summary  by  Itkonen  and  Rau>ainen  

Challenges

Managing test coverage can be difficult Quality of testing highly dependent on expertise and skill of tester Repeatability of found defects challenging at times

Summary

•  Surface and address risks sooner •  Can help identify the big bugs •  Adaptive •  Complementary to scripted testing •  Agile fit

30  

Raj Indugula raj.indugula@lithespeed.com  

John Hughes

jhughes@bstonetech.com  

Resources and References

•  Bach, James. “What Is Exploratory Testing?” •  Marick, Brian. “A Survey of Exploratory Testing” •  Hendrickson, Elisabeth. “Explore It!” •  Jorgensen, Alan. “How to Break Software (with examples)” •  Kaner, Cam, Tinkham, Andy. “Learning Styles and Exploratory

Testing” •  Bolton, Michael, et al. “Exploratory Testing Dynamics” •  Itkonen, Juha. “Do test cases really matter? An experiment

comparing test case based and exploratory testing”

32