Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection
description
Transcript of Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection
![Page 1: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/1.jpg)
Dynamic Analysis of Algebraic Structure to Optimize Test Generation
and Test Case Selection
Anthony J H Simons and Wenwen Zhao
![Page 2: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/2.jpg)
Overview
Lazy Systematic Unit Testing JWalk testing concept and methodology
The JWalk 1.0 toolset JWalkTester, JWalkUtility, JWalkEditor, etc.
Dynamic analysis and pruning extending earlier work to full algebraic analysis
Comparison and evaluation measure path pruning, before and after test result prediction, before and after
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
![Page 3: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/3.jpg)
Motivation
State of the art in agile testing Test-driven development is good, but… …no specification to inform the selection of tests …manual test-sets are fallible (missing, redundant cases) …reusing saved tests for conformance testing is fallible –
state partitions hide paths, faults (Simons, 2005) Lazy systematic testing method: the insight
Complete testing requires a specification (even in XP!) Infer an up-to-date specification from a code prototype Let tools handle systematic test generation and coverage Let the programmer focus on novel/unpredicted results
![Page 4: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/4.jpg)
Lazy Systematic Unit Testing
Lazy Specification late inference of a specification from evolving code semi-automatic, by static and dynamic analysis of code
with limited user interaction specification evolves in step with modified code
Systematic Testing bounded exhaustive testing, up to the specification emphasis on completeness, conformance, correctness
properties after testing, repeatable test quality
http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing
![Page 5: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/5.jpg)
JWalk 1.0 Toolset
JWalk Tester JWalk Utility JWalk Editor
JWalk Marker JWalk Grapher JWalk SOAR
![Page 6: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/6.jpg)
JWalk Tester
Lazy systematic unit testing for Java static analysis - extracts the public API
of a compiled Java class protocol walk (all paths) – explores, validates all interleaved
methods to a given path depth algebra walk (memory states) – explores, validates all
observations on all mutator-method sequences state walk (high-level states) – explores, validates n-switch
transition cover for all high-level states
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
Try me
![Page 7: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/7.jpg)
Baseline Approaches
Breadth-first generation all constructors and all interleaved methods (eg JCrasher,
DSD-Crasher, Jov) generate-and-filter (eg Rostra, Java Pathfinder) by state
equivalence class
Computational cost exponential growth, memory issues, wasteful over-
generation, even if filtering is later applied
#paths = Σc.mk, for k = 0..n
Key: c = #constructors, m = #methods, k = depth
![Page 8: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/8.jpg)
Dynamic Pruning
Interleaved analysis generate-and-evaluate, pruning active paths on the fly (eg
JWalk, Randoop) remove redundant prefix paths after each test cycle, don’t
bother to expand in next cycle Increasing sophistication
prune prefix paths ending in exceptions (fail again) JWalk, Randoop (2007)
and prefixes ending in algebraic observers (unchanged) JWalk 0.8 (2007)
and prefixes ending in algebraic transformers (reentrant) JWalk 1.0 (2009)
![Page 9: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/9.jpg)
Prune Exceptions…
newpush
top
pop
pushtop
poppush
top
pop
pushtop
pop
Key: novel state
exception
top
poptop
pop
top
pop
top
poptop
pop
push push
push
push
Prune error-prefixes (JWalk0.8, Randoop)
![Page 10: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/10.jpg)
…and Observers
newpush
top
pop
pushtop
poppush
top
pop
pushtop
pop
Key: novel state
exception
unchanged state
pushtop
pop
pushtop
pop
Prune error- and observer-prefixes (JWalk0.8)
![Page 11: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/11.jpg)
Algebraic Pruning
newpush
top
pop
poptop
pushpush
top
pop
pushtop
pop
Key: novel state
exception
unchanged state
reentrant state
Prune error-, observer- and transformer-prefixes (JWalk1.0)
![Page 12: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/12.jpg)
What is the Same State?
Some earlier approaches distinguish observers, mutators by signature (Rostra) intrusive state equality predicate methods (ASTOOT) external (partial) state equality predicates (Rostra) subsumption of execution traces in JVM (Pathfinder)
Some algebraic approaches shallow, deep equality under all observers (TACCLE)
but assumes observations are also comparable very costly to compute from first principles
serialise object states and hash (Henkel & Diwan) but not all objects are serialisable no control over depth of comparison
![Page 13: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/13.jpg)
Smart State Inspection
Reflection-and-hash extract state vector from objects compute hash code for each field order-sensitive combination hash code
Proper depth control shallow or deep equality settings, to chosen depth hash on pointer, or recursively invoke algorithm
Fast state comparison each test evaluation stores posterior state code fast comparison with preceding, or all prior states possible to detect unchanged, or reentrant states
![Page 14: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/14.jpg)
Pruning: Stack
Stack baseline except. observ. algebr.
0 1 1 1 1
1 7 7 7 7
2 43 31 13 13
3 259 139 25 19
4 1555 667 43 25
5 9331 3391 79 31
Pruned: 9,300 redundant pathsRetained: 31 significant paths (best 0.33%)
Table 1: Cumulative paths explored after each test cycle
![Page 15: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/15.jpg)
Pruning: Reservable Book
ResBook baseline except. observ. algebr.
0 1 1 1 1
1 9 9 9 9
2 73 73 25 25
3 585 561 49 33
4 4681 4185 97 41
5 37449 memex 169 41
Pruned: 37,408 redundant pathsRetained: 41 significant paths (best 0.12%)
Table 2: Cumulative paths explored after each test cycle
![Page 16: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/16.jpg)
Test Result Prediction
Semi-automatic validation the user confirms or rejects key results these constitute a test oracle, used in prediction eventually > 90% test outcomes predicted
JWalk test result prediction rules eg: predict repeat failure
new().pop().push(e) == new().pop()
eg: predict same state target.size().push(e) == target.push(e)
eg: predict same result target.push(e).pop().size() == target.size()
Try me
![Page 17: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/17.jpg)
Kinds of Prediction
Strong prediction From known results, guarantee further
outcomes in the same equivalence class eg: observer prefixes empirically checked before making any
inference, unchanged state is guaranteed target.push(e).size().top() == target.push(e).top()
Weak prediction From known facts, guess further outcomes; an incorrect
guess will be revealed in the next cycle eg: methods with void type usually return no result, but may
raise an exception target.pop() predicted to have no result target.pop().size() == -1 reveals an error
![Page 18: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/18.jpg)
Test Confirmation – JWalk 0.8
newpush
top
pop
pushtop
poppush
top
pop
pushtop
pop
Key: confirm result
confirm error
predicted result
pushtop
pop
pushtop
pop
Confirm all observations, errors on all state-modifying paths
![Page 19: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/19.jpg)
Test Confirmation – JWalk 1.0
newpush
top
pop
pop
top
pushpush
top
pop
pushtop
pop
Confirm all observations, errors on all primitive algebraic constructions
Key: confirm result
confirm error
predicted result
![Page 20: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/20.jpg)
Confirmations: Stack
Stack v0.8 alg v0.8 pro v1.0 alg v1.0 pro
0 1 - 1 -
1 5 - 5 -
2 4 - 4 -
3 9 - 4 -
4 12 - 4 +4
5 26 - 4 +8
Total 57 57 22 34
Table 3: Confirmations per test cycle (new oracle)
JWalk 0.8: trained oracle after 57 confirmationsJWalk 1.0: trained oracle after 34 confirmations
![Page 21: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/21.jpg)
Confirmations: Reservable Book
ResBook v0.8 alg v0.8 pro v1.0 alg v1.0 pro
0 1 - 1 -
1 2 - 2 -
2 8 - 8 -
3 12 - 6 -
4 30 - 6 +20
5 40 memex - memex
Total 93 93 23 43
Table 4: Confirmations per test cycle (inherited oracle)
JWalk 0.8: trained oracle after 93 confirmationsJWalk 1.0: trained oracle after 43 confirmations
![Page 22: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/22.jpg)
Why Residual Confirmations?
Prediction based on state equality from state equivalence:
target.push(e).pop() == target
predict identical observations: target.push(e).pop().size() == target.size()
Novel states occur in longer protocols JWalk has deterministic argument synthesis:
elements generated in order: e1, e2, … en
algebraic reduction yields a novel state: target.push(e1).pop().push(e2) == target.push(e2) target.push(e2) != target.push(e1) from the oracle
![Page 23: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/23.jpg)
Conclusions …
Test path pruning algebraic analysis effective at eliminating redundant paths absolutely necessary when testing classes with large APIs
java.lang.Character: c = 1, m = 78; d3 base = 480,715 paths; alg = 79 paths, stable after 1 cycle
java.lang.String: c = 13, m = 64; d3 base = 54,093 paths; alg = 845 paths, stable after 1 cycle
More test automation presents user with the ideal mimimal test-set for judgement user only has to confirm all errors and observations on all
primitive algebraic constructions
![Page 24: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/24.jpg)
Conclusions
Faster state exploration algebra-walking finds the leaves of the algebra-tree faster state-walking discovers high-level states faster, by growing
only primitive state-modifying paths can afford to search to greater test depths
Test result prediction algebraic anaylsis improves predictive power as expected but oracle must also have the reduction (and may not) future idea: axiom generalisation? (Henkel & Diwan)
![Page 25: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/25.jpg)
Thank You!
And thanks also to: Wenwen Zhao – hashing on states for comparison Mihai-Gabriel Glont – prototype UI for JWalkTester Arne-Michael Toersel – case studies for JWalk
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
Let’s go JWalking!
![Page 26: Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection](https://reader036.fdocuments.us/reader036/viewer/2022062809/5681599e550346895dc6e9a9/html5/thumbnails/26.jpg)
Confirmations: Library Book
LibBook v0.8 alg v0.8 pro v1.0 alg v1.0 pro
0 1 - 1 -
1 2 - 2 -
2 3 - 3 -
3 2 - - -
4 3 - - +3
5 2 - - -
Total 13 13 6 9
Table 5: Confirmations per test cycle (new oracle)
JWalk 0.8: depth-5 oracle after 13 confirmationsJWalk 1.0: depth-5 oracle after 9 confirmations