1 Automating the Generation of Mutation Tests Mike Papadakis and Nicos Malevris Department of...
-
Upload
morgan-wallace -
Category
Documents
-
view
214 -
download
0
Transcript of 1 Automating the Generation of Mutation Tests Mike Papadakis and Nicos Malevris Department of...
1
Automating the Generation of Mutation Tests
Mike Papadakis and Nicos Malevris
Department of Informatics Athens University of Economics and Business
Test Data Generation Approaches
Symbolic execution Select set of paths Produce a system of algebraic
constraints Solve and generate test cases
Challenges (Symbolic execution) Infeasible paths Complex-unhandled expressions Availability of source code
2
Test Data Generation Approaches
Search based approaches Definition of program input Dynamic program execution Fitness function
o (efficiency and effectiveness)
Challenges Handling of dynamic program inputs Require a high number of executions Handling of specific cases (e.g. flag
problem) Effective fitness function
3
Test Data Generation Approaches
Dynamic Symbolic Execution Simultaneously perform actual and
symbolic executiono Simplify the process based on concrete program
execution Produce a system of algebraic
constraints Solve and generate test cases
Challenges Infeasible paths Scalability issues
4
5
In order to kill a mutant, tests must
Stage 1: Reach the mutant
Stage 2: infect the program state
Stage 3: propagate the infected state
Killing Mutants
6
Reach the mutant (Reachability condition) Infect program state (necessity condition)
Program execution must cause a discrepancy in the program state
E.g. Orig: a > b, Mut = a ≥ b → a > b ≠ a ≥ b o a == b where Orig = false, Mut = true
Propagate (sufficiency condition) Heuristic approaches
o explore path space, measure of mutant’s impact
Joint satisfaction (Reach && Infect && Propagate)
Killing Mutants
7
Killing Mutants (path selection)
Symbolic Execution Reach
o Use paths from program input to mutant Infect
o Use necessity constraints to infect program stateo Original Statement ≠ Mutated Statement
Propagateo Explore the path space (from mutant to program output)
Path selection Strategy
8
Enhanced Test Model (Enhanced Control Flow Graph) Represent mutants in the model Guide the selection of paths and the produced
constraints based on the Shortest path strategy
Killing Mutants (path selection)
9
Killing Mutants (path selection)
Shortest Path Strategy, the k-value An efficient path selection strategy. (eliminates
the effects of infeasible paths) Select shortest paths from program input (S) to
the mutant statement (m)
Heuristic approach to weak mutation Strong mutation can be tackled incrementally
xmxSkkm ),()()(
),(),( )()()( FxmxxmxS trkm
10
Killing Mutants Dynamic approaches
Dynamic approaches Gain the required information from actual
program execution Require high number of program execution
Mutation Testing Need to unify the runtime information of the
original and mutant programs Need to include mutant neccesity and
sufficiency conditions into program structure Need for efficiency (high cost)
11
Mutant Schemata Mutant schemata-Parameterized mutants
Technique for efficiently producing mutants Embeds all mutants into one meta-mutant
program Use of global parameter to specify the
mutants Example original meta-mutant a > b + c → RelationalGT(a, ArithmeticPlus(M(b),
M(c)))
M(x) mutates variable x, e.g. abs(x)Arithmetic(x, y) mutates arithmetic operator, e.g. x
- yRelational(x, y) mutates relational operator, e.g. x ≥
y
12
Mutant Schemata (approach I) Weak mutation
Include the mutant evaluation inside schematic functionso Original expression ≠ Mutant expressiono Original statement ≠ Mutant statement
After evaluation continue with the original execution (schematic function returns the original expression)
Able to execute all mutants with one execution run
Embeds all mutant’s killable condition into program structure (mutant evaluation → branches)
Mutant is killedMutant is Alive
Mutant Schemata
NodeN-1
NodeN
NodeN+1
NodeN+2
NodeN-1
NodeN
NodeN+1
NodeN+2
NodeN_M[1]
NodeN_M[2]
NodeN_M[n-1]
NodeN_M[n]
...
Proposed approach (Mutants to branches) Mutants M[1]…M[n] on node N. Mutant
EvaluationBranches
13
14
Mutant Schemata (approach II) Strong mutation
In Same lines as Weak o Include the mutant evaluation inside schematic functions
After evaluation continue with the mutant execution (schematic function returns the mutant expression)
Incremental approach, from weak to strong Requires multiple execution runs
Record original and mutant’s execution paths Evaluate strongly killed mutants
15
Killing Mutants (DSE)
Use the shortest path heuristic Negate the condition that will lead closer to mutant
node (reach the mutant)
Weakly kill the mutant Fulfill the mutant necessity condition Make the mutant evaluation Branch true
Negate the produced conditions after the mutant Start from the mutant program Explores the mutant program path space
16
Target Mutant
DSE-Example Scenario
17
Target Mutant
DSE-Example Scenario
18
Target Mutant
DSE-Example Scenario
19
Target Mutant
DSE-Example Scenario
20
Target Mutant
Mutant is alive
Infect Mutant
DSE-Example Scenario
21
Target Mutant
Mutant is alive
Propagate
DSE-Example Scenario
22
Target Mutant
Mutant is alive
DSE-Example Scenario
23
Target Mutant
Mutant Killed
DSE-Example Scenario
24
DSE-Necessity condition
Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=15, b=2, k=0Conditions: b≥0 && a+b>k && …Negates: b≥0 → b < 0 Produces: a=15, b=-10, k=0
25
DSE-Necessity condition
Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level conditionTest: a=15, b=-10, k=0
Conditions: b<0 && a+b>k && a+abs(b)>k && …Negates: a+abs(b)>k → b<0 && a+b>k && a+abs(b)≤k →
InfeasibleNegates: a+b>k → b<0 && a+b≤kProduces: a=-15, b=-10, k=0
26
DSE-Necessity condition
Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level conditionTest: a=-15, b=-10, k=0
Conditions: b<0 && a+b≤k && a+abs(b)≤k && …
Negates: a+abs(b)>k → b<0 && a+b≤k && a+abs(b)>k Produces: a=5, b=-10, k=0Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)
27
Killing Mutants (Search Based)
Weak-Strong mutation Measure the closeness of reaching a
mutant Measure branch distance of mutant
brancheso Closeness of weakly killing the targeted mutant (mutant
necessity condition) Use simplified necessity fitness for
improved performance Sufficiency condition can be
approximated by exploring the path space or based on mutants impact
28
Fitness function Approach level
Closeness of executing a targeted mutant Calculated by the number of control
dependent nodes missed Branch Distance
Closeness of flipping a specific branch
Mutation Distance Closeness of weakly killing the targeted
mutant𝑓𝑖𝑡𝑛𝑒𝑠𝑠= 2 ∗ 𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ 𝑙𝑒𝑣𝑒𝑙+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑ሺ𝐵𝑟𝑎𝑛𝑐ℎ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒ሻ + 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑ሺ𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒ሻ
29
Mutant Fitness (example)Operator Original expression Mutant Fitness
Relational a > b
a >= b: abs(a-b)a < b: ka <= b: 0
a != b: abs(a-b+k)a == b: abs(a-b)true: abs(a-b)false: abs(a-b+k)
Arithmetic a + ba - b:ka * b:ka / b:k
a % b:ka:kb:k
Absolute aabs(a):abs(a+k) -abs (a):abs(a)
0:abs(a)
Logical
a && b
a||b:min[Tfit(a)+Ffit(b), Ffit(a)+Tfit(b)]
a:Tfit(a)+Ffit(b)
b:Ffit(a)+Tfit(b)true:min [Ffit(a), Ffit(b)]false:Tfit(a)+Tfit(b)
a || b
a&&b:min[Tfit(a)+Ffit(b), Ffit(a)+Tfit(b)]
a:Ffit(a)+Tfit(b)
b:Tfit(a)+Ffit(b)true:Ffit(a)+Ffit(b)false:min[Tfit(a), Tfit(b)]
30
Path explosion problem
Original program 8 program paths
Mutant program 8 program paths per
mutant Schematic program
Mutant evaluation decisions
8 * 2number of mutants paths Introduction of fake
paths Same of them may be
useful for higher order mutation
Mutants
Mutants
Mutants
Mutants
31
Handling of path explosion Path selection strategy, (symbolic
execution) Use weights on the Enhanced Control Flow
Graph Shortest paths include only the original paths
Dynamic Symbolic Execution Use the mutant parameters as local values Dynamic production of constraints eliminates
the fake paths. Paths are produced dynamically Search based approaches
Approach level contains only the control depended nodeso Ignores the path space
32
Case Studies
Dynamic Symbolic Execution (Strong mutation) Program units: Tritype, Remainder Siemens Programs: Schedule, Tcas, Replace ABS, AOR and ROR operators
Search based (Weak mutation results) Hill climbing approach (AVM) Maximum 10 attempts per mutant Program units: Tritype, Triangle, Remainder,
Calendar ABS, AOR, ROR and LCR operators
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
Program½ Iterations All Iterations
Produced TestsKilled
Mutants Solver Calls
No. Mut. Executions
Killed Mutants
Solver CallsNo. Mut.
Executions
Tritype 214 1012 500 216 1588 514 90
Remainder 234 2749 435 235 5498 741 268
Replace 514 14740 19400 520 30238 80471 8927
Tcas 136 1733 3729 137 3705 4720 422
Schedule 94 714 293 94 949 303 301
DSE Study-Results Strong mutation results
Replace 86.5%, Tcas 100%, Schedule 91% (compared with the accompanied test suite),
Remainder 97.5 and Tritype 96.8% (all killable mutants)
39
Search Based Study-Results
40
Search Based Study-Results
41
Search Based Study-Results
42
Search Based Study-Results
43
Search Based Study-Results
Test Objects
No. of Killed Mutants Mutation ScoreTime (Sec)
Random Metallaxis Random Metallaxis
Tritype 168 245 66.4% 96.8% 42
Triangle 246 275 89.5% 100% 70
Remainder 238 242 98.3% 100% 402
Cal 237 252 93.3% 99.2% 44
Weak mutation results
Conclusion
Mutation based test case generation Lack of attempts First steps using dynamic, state of the art
techniques
Dynamic approaches can be adopted to perform mutation Search based approaches
o Mutation distance leads to improved effectiveness Dynamic Symbolic Execution
o Quite effective 44
Conclusion
Test data generation approaches and tools can be effectively extended to mutation Practical Technique (schemata)
Future directions Efficient handling of infeasible paths Handling of the “flag” problem Efficient handling of equivalent
mutants Hybrid approaches
45
Thank you for your attention…
Questions ?
Contact
Mike Papadakis [email protected]
Nicos Malevris [email protected]
46
47
References
Mike Papadakis and Nicos Malevris. "Automatic Mutation Test Case Generation Via Dynamic Symbolic Execution", in 21st International Symposium on Software Reliability Engineering (ISSRE'10), San Jose, California, USA, November 2010.
Mike Papadakis and Nicos Malevris. “Metallaxis an Automated Framework for Weak Mutation", Technical Report, http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf.
Mike Papadakis, Nicos Malevris and Maria Kallia. "Towards Automating the Generation of Mutation Tests", in Proceedings of the 5th International Workshop on Automation of Software Test (AST'10), Cape Town, South Africa, May 2010, pp. 111-118.
Mike Papadakis and Nicos Malevris. “An Effective Path Selection Strategy for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC'09), Penang, Malaysia, December 2009, pp. 422-429.