rial Analysis Workshop
Transcript of rial Analysis Workshop
-
8/14/2019 rial Analysis Workshop
1/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Bj RollisonTest Architect
Microsoft
http://www.TestingMentor.comhttp://blogs.msdn.com/imtesty
-
8/14/2019 rial Analysis Workshop
2/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Define the combinatorial testing problemLearn coverage array basics
State advantages of combinatorial analysisIdentify common problemsOvercome common problems by learning to usethe right tool in the right context
Decompose complex feature setsDesign effective model filesLearn custom features of PICT
Discover how combinatorial analysis is a best
practice when used in the appropriate situation
-
8/14/2019 rial Analysis Workshop
3/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Design an extremely small subset of tests fromall possibilitiesEvaluate important attributes and capabilitiesof the application under testIdentify issues and other potential risks
Exercise large portions of the code to reduce riskProvide accurate information to the decisionmakers for informed risk-assessment
Within a limited amount of timeProvides a high degree of confidence for theentire team
-
8/14/2019 rial Analysis Workshop
4/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Parameters with multipleinterdependent variablesare likely to cause bugs!Testing problem...
24 parameters
2 5 variable statesper parameter > 500,000,000,000
combinations1 test/ms > 3300 years
So, how would you choose thetests?
-
8/14/2019 rial Analysis Workshop
5/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Testing the combinations of variable stateswhen :
Parameters are directlyinterdependent
Parameters aresemi-coupled
Parameter input is unordered
-
8/14/2019 rial Analysis Workshop
6/53Copyright 2007 2008 Bj Rollison All rights reserved.
Functional testing technique in which the tester:systematically analyze parameterinteraction in a complex feature setin order to methodically select a finite subsetof tests from all possible combinations
Uses a mathematical approach based onhistorical failure indicators and empirical faultmodels
Helps reduce overall risk byHigh defect detection effectivenessHigh levels of structural coverage
-
8/14/2019 rial Analysis Workshop
7/53Copyright 2007 2008 Bj Rollison All rights reserved.
Random evaluation approachesBest guess
Experience, customer data, intuition, luckRandom selectionRandomly selected from all combinations
Systematic evaluation approachesEach choice (EC)Base choice (BC)Orthogonal arrays (OA)Coverage arrays (CA)
Exhaustive (AC)
-
8/14/2019 rial Analysis Workshop
8/53Copyright 2007 2008 Bj Rollison All rights reserved.
Finding OA is non-trivialUnrealistic expectationsequal number of variablesper parameter Redundant tuples in output
Test B I U S Col.5 Col.6 Col.7
1 unck unck unck unck 1 1 1
2 unck unck unck chk 2 2 2
3 unck chk chk unck 1 2 24 unck chk chk chk 2 1 1
5 chk unck chk unck 2 1 2
6 chk unck chk chk 1 2 1
7 chk chk unck unck 2 2 1
8 chk chk unck chk 1 1 2
L8 Orthogonal Array
Taguchi Array Selector http://www.freequality.org
-
8/14/2019 rial Analysis Workshop
9/53Copyright 2007 2008 Bj Rollison All rights reserved.
Pairs of parameters(BI, BU, BS, IU, IS, US)Every variable combinationfor each pair of parametersMultiple pair combinationsper testMinimizes redundancy
Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Bold yes yes yes yes no no no yes no no yes yes no yes no noItalic no yes yes yes yes yes yes yes no no no no no no yes no
Underline no no yes yes no yes yes no yes yes yes yes no no no no
Strikethrough no no no yes no no yes yes no yes no yes yes yes yes no
-
8/14/2019 rial Analysis Workshop
10/53
-
8/14/2019 rial Analysis Workshop
11/53Copyright 2007 2008 Bj Rollison All rights reserved.
Install PICT from www.pairwise.org/tools.aspCreate shortcut to the installation directory
Launch Notepad as editor, use file format asfollows
Open console windowCommand to run PICT tool is:
Use Excel to view output
Bold: Check, UncheckItalic: Check, UncheckStrikethrough: Check, UncheckUnderline: Check, Uncheck
[Pict_Path]:\PICT [path_to_file.txt] > [path_to_output.xls]
-
8/14/2019 rial Analysis Workshop
12/53Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Create a model file of theNumbers format feature
Steps:Decompose the parametersIdentify interesting variable
statesCreate a PICT model fileGenerate and analyze output
Skills:Equivalence class partitioningof variable states
Basic PICT usage
-
8/14/2019 rial Analysis Workshop
13/53Copyright 2007 2008 Bj Rollison All rights reserved.
Increased defect detection effectiveness withfewer number of testsIncreased code coverage with less number of testsProven results in complex and critical systems
Easy to incorporate into data-driven automationOutput almost instantaneousGreater creativity over the design of the testIncreased efficiency !
-
8/14/2019 rial Analysis Workshop
14/53Copyright 2007 2008 Bj Rollison All rights reserved.
Pair-wise found 98 percent of all detectablefaults seeded in five applications
All faults in four applications
Evaluation of Combination Strategies for Test Case Selection (Grindal, et. al.)
Pairwise
-
8/14/2019 rial Analysis Workshop
15/53Copyright 2007 2008 Bj Rollison All rights reserved.
In this study we found no significant differencein the FDE of n-way and random combinatorialtest suites. the result is not unexpected .Tests selectedrandomly fromall possiblecombinations# of tests equalto n-way # of tests
(Patrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopu)
-
8/14/2019 rial Analysis Workshop
16/53Copyright 2007 2008 Bj Rollison All rights reserved.
" a set of 29 pairwise tests gave 90% blockcoverage for the UNIX sort command. We alsocompared pairwise testing with random inputtesting and found that pairwise testing gavebetter coverage .
The AETG System: An Approach to Testing Based on Combinatorial Design (Cohen, et. al.)
-
8/14/2019 rial Analysis Workshop
17/53Copyright 2007 2008 Bj Rollison All rights reserved.
Attrib.exe path + 6 optional argv.
Findstr.exe str + 19 optional argv. (421,200 EC tests)
Total # of blocks = 483 Default test suite Exhaustive coverage Pair-wise coverage
Number of test cases 9 972 13
Blocks covered 358 370 370
Code coverage 74% 77% 77%
Functions not covered 15 15 15
Total # of blocks = 1317 Hand-crafted tests N=2 coverage N=3 coverage N=4 coverage
Number of test cases 236 136 800 3533
Blocks covered 960 979 994 1006
Code coverage 73% 74% 75% 76%
Functions not covered 11 11 10 10
-
8/14/2019 rial Analysis Workshop
18/53Copyright 2007 2008 Bj Rollison All rights reserved.
Regression test suite time reduced(104 man-days to 32 hours)Test design and implementation reduced(44 man-months to 4 man-months)Reduced OS configuration testing 50% with zerobug bounce (ZBB)Increase in automated test development adoptingdata-driven output of PICT
(@ 30 tests/week to 235 tests/week)3 show-stopper defects detected prior to releasesaved @ $375,000 (each QFE @ $125K)
-
8/14/2019 rial Analysis Workshop
19/53
-
8/14/2019 rial Analysis Workshop
20/53Copyright 2007 2008 Bj Rollison All rights reserved.
The combinatorialfault model does notapply if
Parameters areindependent
Mathematical formulasVar Param1 + Var Param2 = Var ParamOut
Sequential operationsOrdered input
-
8/14/2019 rial Analysis Workshop
21/53Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Decompose the dependentor semi-coupled
parameters and their variable statesSteps:
Decompose a featureCreate a PICT model fileGenerate and analyze output
This is important becauseParameter interactionincreases defect probabilityMisuse can provide falseconfidence
-
8/14/2019 rial Analysis Workshop
22/53Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Identify dependentparameters
Steps:Decompose theparameters
Identify dependent or semi-coupledparameters
Skills:Feature decomposition
-
8/14/2019 rial Analysis Workshop
23/53Copyright 2007 2008 Bj Rollison All rights reserved.
Misuse of technique (wrongfault model)
Unfamiliarity with thetechnique or feature
Right values not tested
Highly probable values gettoo little attention
Complex interactionsmissed
-
8/14/2019 rial Analysis Workshop
24/53Copyright 2007 2008 Bj Rollison All rights reserved.
Interdependent parametersFonts
BoldItalicSemi-coupled parameters
Color (Black, white, red, green, blue, yellow)Size (1 1638, including half sizes)StrikethroughUnderline
Total number of tests = Cartesian product of variables = 1,257,600
tests (assuming all font size values are tested)
-
8/14/2019 rial Analysis Workshop
25/53
Copyright 2007 2008 Bj Rollison All rights reserved.
File format (for PICT)param : var 1, var 2, var 3
Use equivalence classes!
1 - 9 10 - 16 17 - 72 73 - 1638 1.5 1637.5Example:
-
8/14/2019 rial Analysis Workshop
26/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Decompose the Replace feature of Wordpad for interdependent or semi-coupled parameters
Steps:Decompose a featureCreate a PICT model file
Generate outputSkills:
Feature decomposition
Variable equivalence class partitioningBasic PICT model design
-
8/14/2019 rial Analysis Workshop
27/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Misuse of technique (wrongfault model)
Unfamiliarity with thetechnique or feature
Right values not tested
Highly probable values gettoo little attention
Complex interactionsmissed
-
8/14/2019 rial Analysis Workshop
28/53
Identify mutuallyexclusive variablecombinations
Analyze outputfor happy path
Analyze output
for failureindicators
Copyright 2008 Bj Rollison. All rights reserved.
-
8/14/2019 rial Analysis Workshop
29/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Conditional orinvariantrelationships
Exclude mutuallyexclusive variable states;often increases total
number of tests
Dont change output!Not only is it foolish, butis completely negatesthe output of the tool!
-
8/14/2019 rial Analysis Workshop
30/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Modify model file to handle conditional or invariantconstraints of variable state combinations
Steps:Decompose a featureCreate a PICT model file
Generate and analyze outputModify model file usingconditional or invariant constraint syntax
Skills:Verification of model outputUse simple script language to modify model for
conditional or invariant variable constraints
-
8/14/2019 rial Analysis Workshop
31/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Misuse of technique (wrongfault model)
Unfamiliarity with thetechnique or feature
Right values not tested
Highly probable valuesget too little attention
Complex interactionsmissed
-
8/14/2019 rial Analysis Workshop
32/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Weighting variables,Variable preference Give increased weighted
value to important
variables; MAY increaseprobability of occurrence
-
8/14/2019 rial Analysis Workshop
33/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Weight variables toincrease probability
Steps:Decompose featureand create model file
Generate and analyzeoutputModify model withweights to important variable statesRegenerate output and compare
Skills:Certain variable states are more important that othersbut, ALL n-combinations should be tested first!
-
8/14/2019 rial Analysis Workshop
34/53
-
8/14/2019 rial Analysis Workshop
35/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Seed variables toguarantee output
Steps:Decompose featureand create model file
Generate and analyzeoutputCreate seeded input file
Regenerate output and compareSkills:Certain variable states are more representative of realusage or customer requirements
-
8/14/2019 rial Analysis Workshop
36/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Misuse of technique (wrongfault model)
Unfamiliarity with thetechnique or feature
Right values not tested
Highly probable values gettoo little attention
Miss complex interactions
-
8/14/2019 rial Analysis Workshop
37/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Sub-ModelsBundle specific parameters into groups
Each groups gets own combinatory ordersIncreases thoroughness of tested combinationsSyntax
{, ,, ... } @
Combinatory order cannot exceed number of its parameters
-
8/14/2019 rial Analysis Workshop
38/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Randomize output toincrease breadth oftesting and minimize
probability of error due tomissed combinations
PICT/r: n switch
-
8/14/2019 rial Analysis Workshop
39/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Majority (> 50%) of combinatorial defects aresimple pair interactions
Randomizing pair-wise combinations with PICT may improve t=2 detection effectiveness*
Subtle defects still being discovered at t=6
Effects are cumulativeNumber of tests increase
exponentially,but dont stop atpair-wise (t=2) 0%
20%40%60%80%
100%
t=2t=3 t=4 t=5 t=6
-
8/14/2019 rial Analysis Workshop
40/53
Increase order of n-wisecombinations for greater
depth of testing andminimize probability oferror due to complex
interactions
PICT/o: n switch 3-wise = 120 tests2-wise = 33 tests
-
8/14/2019 rial Analysis Workshop
41/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Objective:Create a model file
Steps:
Decompose a featureCreate a PICT model fileGenerate and analyze
outputModify as necessaryRegenerate output and
compareSkills:Certain variable state combinations must berevalidateddepending on their criticality
-
8/14/2019 rial Analysis Workshop
42/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Misuse of techniqueDomain knowledge beyond the user interface
Unfamiliarity with technique or toolTraining, in-depth technical knowledge,use the right tool for the right job!
Right values not testedIn-depth system knowledge, stochastic dataHighly probable values get too little attention
Weighted variables, include seeded inputsComplex interactions are missed
Random output, increase n-wise combinations
-
8/14/2019 rial Analysis Workshop
43/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Setup/upgrade testing
-
8/14/2019 rial Analysis Workshop
44/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Interoperability testing
-
8/14/2019 rial Analysis Workshop
45/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Configuration testing
-
8/14/2019 rial Analysis Workshop
46/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Game testing
-
8/14/2019 rial Analysis Workshop
47/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Black-box API testing
-
8/14/2019 rial Analysis Workshop
48/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Utility/command line testing
-
8/14/2019 rial Analysis Workshop
49/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Compiler testing
-
8/14/2019 rial Analysis Workshop
50/53
Copyright 2007 2008 Bj Rollison All rights reserved.
Best Practice compared to other approaches for testing complex parameter interactionsProvides critical information earlier
High probability of early defect detectionHigh structural coverage; t=6 exhaustive
Increases depth and breadth of test coverageEasy to integrate into automation (data-driven)But,
Requires in-depth technical knowledgeMust verify output its only a tool!
Dont forget Beizers Pesticide Paradox!
-
8/14/2019 rial Analysis Workshop
51/53
Copyright 2007 2008 Bj Rollison All rights reserved.
http://[email protected]
http://hwtams.com
-
8/14/2019 rial Analysis Workshop
52/53
Copyright 2007 2008 Bj Rollison All rights reserved.
http://www.pairwise.orgPairwise Testing in the Real World: PracticalExtensions to Test-Case ScenariosJacek Czerwonka(http://msdn2.microsoft.com/en-us/testing/cc150619.aspx?topic=306121)
Comparing the Fault Detection Effectiveness of N-
way and Random Test SuitesPatrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopuhttp://portal.acm.org/citation.cfm?doid=1159733.1159742
Software and Hardware Testing UsingCombinatorial Covering SuitesAlan Hartman, IBM Haifa Research Laboratoryhttp://www.haifa.il.ibm.com/projects/verification/mdt/papers/CombinatorialProblemsinSWTesting-finalDraft180703.pdf
-
8/14/2019 rial Analysis Workshop
53/53