MQ PM Tutorial
4/29/13 1:00PM
The Craft of Bug Investigation
Presented by:
Jon Bach
eBay, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Jon Bach
Jon Bach works for eBay (San Jose) as a QA director for the Buyer Experience team. Formerly a manager of corporate intellect and senior test consultant at Quardev, Jon has been in testing since 1995 with experience that includes managing teams at Microsoft, HP, and LexisNexis. The co-inventor (with his brother James) of session-based test management, Jon is an award-winning speaker on test management and exploratory testing. He’s mostly known for his “half-baked” ideas about how to inspire innovation and create a learning culture for test teams. Find Jon on Facebook and Twitter @jbtestpilot. View his presentations at quardev.com/articles and his blog at jonbox.wordpress.com.
The Craft of (black box)Bug Investigation
Jon Bach
QE Director, eBay
STAR East 2013
eBay Inc. confidential
Preamble
These next few hours are designed with exercises that lead you through ways of thinking that help you narrow down a bug.
We’ll discuss techniques for bug isolation, reproducing intermittent bugs, and skills to help you know when you’ve gone as far as is reasonable before handing it off.
eBay Inc. confidential
A morning of investigation
Psychic Reader
Checkout Checking Out?
Art Show
Mysterious Spheres
OutofBounds
Gettysbug Address
eBay Inc. confidential
CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA
Product Elements
StructureFunction
DataPlatform
OperationsTime
General Test Techniques
Function testingDomain testingStress testingFlow testing
Scenario testingClaims testingUser testingRisk testing
Automatic testing
Project Environment
CustomersInformation
Developer relationsTeam
Equipment & toolsScheduleTest Items
Deliverables
Quality Criteria
CapabilityReliabilityUsabilitySecurity
ScalabilityPerformanceInstallabilityCompatibility
Ways we find bugs
} } } } }
Development Criteria
SupportabilityTestability
MaintainabilityPortability
Localizability
eBay Inc. confidential
How *well* do we find bugs?
Some Investigation Skills and Tactics
Modeling
Resourcing
Questioning
Recording
Reporting
Investigation is a mindset using this skillset.
Chartering
Observing
Manipulating
Collaboration
Generating/Elaborating
Overproduction/Abandonment
Abandonment/Recovery
Refocusing
Alternating
Branching/Backtracking
Conjecturing
“MR.Q COMC GOARABC R&R?”
eBay Inc. confidential
How well do we investigate them?
Good enough quality bar
Further time and effort
is crucial.
Further time and effort
is a waste of resources.
Unacceptable Quality
Too Much Quality
floating line
Bug Report
Bug Report
eBay Inc. confidential
Abductive Inference
1. Collect data.
2. Find several explanations that account for the data.
3. Find more data that is either consistent or inconsistent with explanations.
4. Choose the best explanation that accounts for the important data, or keep searching.
Abductive inference means finding the best explanation for a set of data.
Jump to conjectures, not conclusions
eBay Inc. confidential 8
This is a dip in bids we saw in Production.
What questions do you have to troubleshoot this?
Is Checkout Checking Out?
eBay Inc. confidential 9
Topics for investigation (FORCOSTTTIP)
• Feature – Do we know what happens when people bid?
• Origin When did this start happening?
• Recent – Did we just push code to Production?
• Consistency / Correlation What happened yesterday, tomorrow? Are there other graphs and metrics?
• Overlap – Is there any other process happening?
• Security – Is it malicious? Intentional?
• Terminology What kinds of “bids”? US, UK, International?
• ThirdParty / Silent Partner – What’s not in the room? What sort of information have we NOT heard from?
• Truth is out there – does someone already know about this?
• Instrumentation – is the graph wrong? Can I get more data?
• Prediction – can we test a conjecture with a prediction
eBay Inc. confidential
Exercise: Psychic Reader
eBay Inc. confidential
Lessons
You can be psychic, too
You may be clue-less, for now
Try “defocusing”
Are you just following orders?
eBay Inc. confidential
Exercise: Mysterious Spheres
eBay Inc. confidential
Lessons
If it fails, whose fault is it?
Assumptions need nourishment
Oracles are in “control”
“Elsewhere”, is a useful thought
eBay Inc. confidential
Exercise: Out-of-Bounds
eBay Inc. confidential
Lessons
Be a “model” tester
Stagehands work behind curtains
Outside data is not inside data
Computers are our slaves
eBay Inc. confidential
Exercise: Art Show
What is my Operating Rule?
What do you notice?
What (crazy) test haven’t you tried?
eBay Inc. confidential
Lessons
“You’re so one dimensional”
Jump to *conjectures*
Yes, I *do* look FAT in this. (M vs. O)
Breaking “rules”!
eBay Inc. confidential
Exercise: Gettysbug Address
Does it work?
What is the hidden feature?
What story does the data tell?
eBay Inc. confidential
Lessons
The Rumble Strip heuristic
Missing a mission?
Be “thoughtless” (random is good)
Defocus: do *anything*, *differently*
eBay Inc. confidential
How much investigation?
The answer must be “Yes” to all four criteria,but ask “for whom, what, and when?”
1) Sufficient benefits
2) No critical problems
3) The benefits outweigh the problems
4) In the present situation, and all things considered, improvement would be more harmful than helpful
eBay Inc. confidential
Intermittent Bugs: Observation
Link
Bad observationIrrelevant observation
Bad memoryMisattribution
MisrepresentationUnreliable oracle
Unreliable communication
eBay Inc. confidential
Intermittent Bugs: System
Purposeful change, and then back to originalAccidental changePlatform changeFlaky hardware
Trespassing systemExecutable corruption
Component competition
eBay Inc. confidential
Intermittent Bugs: Machine State
Frozen conditionalImproper Initialization
Resource denialProgressive data corruptionProgressive destabilization
OverflowOccasional functions
Different mode or option setting
eBay Inc. confidential
Intermittent Bugs: Input
AccidentalSecret boundaries or conditions
Different profileGhost input (alternative source)Simultaneous action as others
Compromised inputTiming issues
Crazy Combinations
eBay Inc. confidential
Intermittent Bugs: You?
You may not be aware of...
…variables of influence…sources of distortion in your observations
…available tools that might help…boundaries and their characteristics
…the system's missing / extra functions…complex / competing algorithms
eBay Inc. confidential
A few (free) investigation tools
Httpwatch displays log of requests and responses
Firebug – inspect HTML
JavaScript debugger
WAVE– accessibility errors
eBay Inc. confidential
More info
• ET Dynamics:
http://www.satisfice.com/articles/etdynamics.pdf
• Test Heuristics and Planning
http://www.satisfice.com
• ContextDriven Software Testing
http://groups.yahoo.com/group/softwaretesting
• Center for Software Testing Education and Research
http://www.testingeducation.org/BBST
• Books related to Exploratory Testing skills and tactics
http://www.testingreflections.com/node/view/3190
• Scenario testing examples– http://www.testingeducation.org/a/scenario2.pdf
Top Related