Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test...
Transcript of Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test...
![Page 1: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/1.jpg)
Advanced Test Coverage CriteriaSpecify and Measure, Cover and Unmask
Nikolai Kosmatov
joint work with Sebastien Bardin, Omar Chebaro, Mickael Delahaye,Michael Marcozzi, Mike Papadakis, Virgile Prevosto. . .
CEA, List, Software Safety and Security LabParis-Saclay, France
TestCon 2019, Moscow, April 3, 2019
Nikolai Kosmatov Advanced Test Coverage Criteria 1/ 55
![Page 2: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/2.jpg)
Context: White-Box Testing
Nikolai Kosmatov Advanced Test Coverage Criteria 2/ 55
Testing process
Generate a test input
Run it and check forerrors
Estimate coverage: ifenough stop, else loop
![Page 3: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/3.jpg)
Context: White-Box Testing
Framework: white-box software testing process
Automate test suite generation & coverage measure
Coverage criterion = objectives to be fulfilled by the test suite
Criterion guides automation
Can be part of industrial normative requirements
Nikolai Kosmatov Advanced Test Coverage Criteria 3/ 55
![Page 4: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/4.jpg)
Coverage criteria in white-box testing
Variety and sophistication gap between literature and testing tools
Literature:
28 various white-box criteria inthe Ammann & Offutt book
Nikolai Kosmatov Advanced Test Coverage Criteria 4/ 55
![Page 5: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/5.jpg)
Coverage criteria in white-box testing
Tools:
Criteria seen as very dissimilar bases for automation
Restricted to small subsets of criteria
Extension is complex and costly
Global goal: bridge the gap between criteria and testing tools
Nikolai Kosmatov Advanced Test Coverage Criteria 5/ 55
Tool name BBC FC DC CC DCC GACC MCDC MCC BP Other
Gcov X X X 0/19
Bullseye X X 0/19
Parasoft X X X X X X 0/19
Semantic Designs X X 0/19
Testwell CTC++ X X X X 0/19
![Page 6: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/6.jpg)
Main ingredients of the talk:
Labels: a generic specification mechanism for coverage criteria◮ can easily encode a large class of criteria◮ a semantic view, with a formal treatment
DSE⋆: an efficient test generation technique for labels◮ an optimized version of DSE (Dynamic Symbolic Execution)◮ no exponential blowup of the search space
LUncov: an efficient technique for detection of infeasible objectives◮ based on existing static analysis techniques
LTest: an all-in-one testing toolset◮ on top of Frama-C and PathCrawler
HTOL: Hyperlabel Specification Language, extension of labels◮ capable to encode almost all common criteria including MCDC
[Bardin et al., ICST 2014, TAP 2014, ICST 2015][Marcozzi et al., ICST 2017 (res.), ICST 2017 (tool), ICSE 2018]
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 55
![Page 7: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/7.jpg)
Main ingredients of the talk:
Labels: a generic specification mechanism for coverage criteria◮ can easily encode a large class of criteria◮ a semantic view, with a formal treatment
DSE⋆: an efficient test generation technique for labels◮ an optimized version of DSE (Dynamic Symbolic Execution)◮ no exponential blowup of the search space
LUncov: an efficient technique for detection of infeasible objectives◮ based on existing static analysis techniques
LTest: an all-in-one testing toolset◮ on top of Frama-C and PathCrawler
HTOL: Hyperlabel Specification Language, extension of labels◮ capable to encode almost all common criteria including MCDC
[Bardin et al., ICST 2014, TAP 2014, ICST 2015][Marcozzi et al., ICST 2017 (res.), ICST 2017 (tool), ICSE 2018]
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 55
Reminder: Goals
Specify and Measure, Cover and Unmask
![Page 8: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/8.jpg)
Main ingredients of the talk:
Labels: a generic specification mechanism for coverage criteria◮ can easily encode a large class of criteria◮ a semantic view, with a formal treatment
DSE⋆: an efficient test generation technique for labels◮ an optimized version of DSE (Dynamic Symbolic Execution)◮ no exponential blowup of the search space
LUncov: an efficient technique for detection of infeasible objectives◮ based on existing static analysis techniques
LTest: an all-in-one testing toolset◮ on top of Frama-C and PathCrawler
HTOL: Hyperlabel Specification Language, extension of labels◮ capable to encode almost all common criteria including MCDC
[Bardin et al., ICST 2014, TAP 2014, ICST 2015][Marcozzi et al., ICST 2017 (res.), ICST 2017 (tool), ICSE 2018]
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 55
Reminder: Goals
Specify and Measure, Cover and Unmask
Specify and Measure,
Cover
and Unmask
![Page 9: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/9.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 55
![Page 10: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/10.jpg)
Labels and the notion of simulation (1/2)
Basic definitions Example:
Given a program P , a label l is a pair(loc , ϕ), where:
ϕ is a well-defined predicate atlocation loc in P
ϕ contains no side-effects
statement_1;
// l1: x==y
// l2: !(x==y)
if (x==y && a<b)
{...};
statement_3;
Nikolai Kosmatov Advanced Test Coverage Criteria 8/ 55
![Page 11: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/11.jpg)
Labels and the notion of simulation (2/2)
Basic definitions Example:
a test datum t covers l if P(t)reaches loc and satisfies ϕ
new criterion LC label coverage:requires to cover the labels
statement_1;
// l1: x==y
// l2: !(x==y)
if (x==y && a<b)
{...};
statement_3;
a criterion C can be simulated by LC if for any P , after adding“appropriate” labels in P , TS covers C ⇔ TS covers LC.
Nikolai Kosmatov Advanced Test Coverage Criteria 9/ 55
![Page 12: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/12.jpg)
Simulation of coverage criteria by labels: CC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
// l1: x==y
// l2: !(x==y)
// l3: a<b
// l4: !(a<b)
if (x==y && a<b)
{...};
statement_3;
Condition Coverage (CC)
Nikolai Kosmatov Advanced Test Coverage Criteria 10/ 55
![Page 13: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/13.jpg)
Simulation of coverage criteria by labels: DC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
//l1: x==y && a<b
//l2: !(x==y && a<b)
if (x==y && a<b)
{...};
statement_3;
Decision Coverage (DC)
Nikolai Kosmatov Advanced Test Coverage Criteria 11/ 55
![Page 14: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/14.jpg)
Simulation of coverage criteria by labels: MCC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
// l1: x==y && a<b
// l2: x==y && a>=b
// l3: x!=y && a<b
// l4: x!=y && a>=b
if (x==y && a<b)
{...};
statement_3;
Multiple-Condition Coverage (MCC)
Nikolai Kosmatov Advanced Test Coverage Criteria 12/ 55
![Page 15: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/15.jpg)
Simulation of coverage criteria by labels: FC
int f1() {
code1;
}
int f2() {
code2;
}
−−−−−→
int f1() {
// l1: true
code1;
}
int f2() {
// l2: true
code2;
}
Function Coverage (FC)
Nikolai Kosmatov Advanced Test Coverage Criteria 13/ 55
![Page 16: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/16.jpg)
Simulation results
Theorem
The following coverage criteria can be simulated by LC: IC, DC,FC, CC, MCC, Input Domain Partition, Run-Time Errors.
Theorem
For any finite set O of side-effect free mutation operators, weakmutations WMO can be simulated by LC.
Nikolai Kosmatov Advanced Test Coverage Criteria 14/ 55
![Page 17: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/17.jpg)
Measuring the coverage of a test suite
Labels already enjoy a simple and efficient algorithm forcoverage measurement
Given a test suite TS and a program P◮ instrument P with checks for labels (P ′)◮ run every t ∈ TS on P ′, record covered labels◮ time cost: ≤ |TS | ·maxt∈TS(P
′(t))
Works also for weak mutations, whereas the standardalgorithm for strong mutations is more costly:◮ create the set of mutants M◮ time cost: ≤ |TS | · |M| ·maxm∈M,t∈TS(m(t))
Nikolai Kosmatov Advanced Test Coverage Criteria 15/ 55
![Page 18: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/18.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 16/ 55
![Page 19: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/19.jpg)
The LTest toolset for labels
Nikolai Kosmatov Advanced Test Coverage Criteria 17/ 55
LTest is implemented on top of Frama-C
Frama-C is a toolset for analysis of C programs
◮ an extensible, open-source, plugin-orientedplatform
◮ offers value analysis (VA), weakest precondition(WP), specification language ACSL,...
LTest is open-source except test generation◮ based on the PathCrawler test generation tool
![Page 20: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/20.jpg)
The LTest toolset for labels
Nikolai Kosmatov Advanced Test Coverage Criteria 18/ 55
![Page 21: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/21.jpg)
The LTest toolset for labels
Nikolai Kosmatov Advanced Test Coverage Criteria 18/ 55
A large set of supported criteria
all treated in a unified way
rather easy to add new ones
![Page 22: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/22.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 19/ 55
![Page 23: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/23.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Nikolai Kosmatov Advanced Test Coverage Criteria 20/ 55
![Page 24: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/24.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Symbolic Execution [King 70’s]
consider a program P on input v, and a given path σ
a path predicate ϕσ for σ is a formula s.t. for any input vv satisfies ϕσ ⇔ P(v) follows σ
old idea, recently renewed interest [requires powerful solvers]
Nikolai Kosmatov Advanced Test Coverage Criteria 20/ 55
![Page 25: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/25.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Symbolic Execution [King 70’s]
consider a program P on input v, and a given path σ
a path predicate ϕσ for σ is a formula s.t. for any input vv satisfies ϕσ ⇔ P(v) follows σ
old idea, recently renewed interest [requires powerful solvers]
Dynamic Symbolic Execution [Korel+, Williams+, Godefroid+]
interleaves dynamic and symbolic executions
drives the search towards feasible paths for free
gives hints for relevant under-approximations
Nikolai Kosmatov Advanced Test Coverage Criteria 20/ 55
![Page 26: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/26.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 27: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/27.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 28: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/28.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 29: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/29.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 30: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/30.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 31: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/31.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 32: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/32.jpg)
Dynamic Symbolic Execution (2)
input: a program P
output: a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 55
![Page 33: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/33.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 55
![Page 34: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/34.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”× lack of support for many coverage criteria
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 55
![Page 35: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/35.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”× lack of support for many coverage criteria
Challenge: extend DSE to a large class of coverage criteria
well-known problem
recent efforts in this direction through instrumentation[Active Testing, Mutation DSE, Augmented DSE]
limitations:
◮ exponential explosion of the search space [APex: 272x avg]◮ very implementation-centric mechanisms◮ unclear expressiveness
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 55
![Page 36: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/36.jpg)
Direct instrumentation P′[APex, Mutation DSE]
Covering label l ⇔ Covering branch True
Nikolai Kosmatov Advanced Test Coverage Criteria 23/ 55
![Page 37: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/37.jpg)
Direct instrumentation P′[APex, Mutation DSE]
Covering label l ⇔ Covering branch True
X sound & complete instrumentation w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 23/ 55
![Page 38: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/38.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 24/ 55
![Page 39: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/39.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 24/ 55
Non-tightness 1
× P ′ has exponentially more pathsthan P
![Page 40: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/40.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 24/ 55
Non-tightness 1
× P ′ has exponentially more pathsthan P
Non-tightness 2
× Paths in P ′ too complex◮ at each label, require to cover
p or to cover ¬p◮ π′ covers up to N labels
![Page 41: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/41.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 24/ 55
X sound & complete instrumentation w.r.t. LC
× dramatic overhead [theory & practice]
![Page 42: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/42.jpg)
Our approach
The DSE⋆ algorithm
Tight instrumentation P⋆: totally prevents “complexification”
Iterative Label Deletion: discards some redundant paths
Both techniques can be implemented in a black-box manner
Nikolai Kosmatov Advanced Test Coverage Criteria 25/ 55
![Page 43: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/43.jpg)
DSE⋆: Tight Instrumentation P⋆
Covering label l ⇔ Covering exit(0)
Nikolai Kosmatov Advanced Test Coverage Criteria 26/ 55
![Page 44: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/44.jpg)
DSE⋆: Tight Instrumentation P⋆
Covering label l ⇔ Covering exit(0)
X sound & complete instrumentation w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 26/ 55
![Page 45: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/45.jpg)
DSE⋆: Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 55
![Page 46: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/46.jpg)
DSE⋆: Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 55
![Page 47: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/47.jpg)
DSE⋆: Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 55
![Page 48: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/48.jpg)
DSE⋆: Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 55
Tightness
X P⋆ has (only) linearly more paths than P
X paths in P⋆ are simple: covers ≤ 1 label
![Page 49: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/49.jpg)
DSE⋆: Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 55
X sound & complete instrumentation w.r.t. LC
X no complexification of the search space
![Page 50: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/50.jpg)
DSE⋆: Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 55
![Page 51: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/51.jpg)
DSE⋆: Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Solution: Iterative Label Deletion
keep a covered/uncovered status for each label
symbolic execution ignores paths ending in a covered label
dynamic execution updates the status [truly requires DSE]
Implementation
symbolic part: a slight modification of P⋆
dynamic part: a slight modification of P ′
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 55
![Page 52: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/52.jpg)
DSE⋆: Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Solution: Iterative Label Deletion
keep a covered/uncovered status for each label
symbolic execution ignores paths ending in a covered label
dynamic execution updates the status [truly requires DSE]
Implementation
symbolic part: a slight modification of P⋆
dynamic part: a slight modification of P ′
Iterative Label Deletion is relatively complete w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 55
![Page 53: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/53.jpg)
DSE⋆: Iterative Label Deletion (2)
Nikolai Kosmatov Advanced Test Coverage Criteria 29/ 55
![Page 54: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/54.jpg)
DSE⋆: Iterative Label Deletion (3)
Nikolai Kosmatov Advanced Test Coverage Criteria 30/ 55
![Page 55: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/55.jpg)
Summary
The DSE⋆ algorithm
Tight instrumentation P⋆: totally prevents “complexification”
Iterative Label Deletion: discards some redundant paths
Both techniques can be implemented in black-box
Nikolai Kosmatov Advanced Test Coverage Criteria 31/ 55
![Page 56: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/56.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 32/ 55
![Page 57: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/57.jpg)
Uncoverable test objectives in testing
The enemy: Uncoverable test objectives
waste generation effort, imprecise coverage ratios
reason: structural coverage criteria are ... structural
detecting uncoverable test objectives is undecidable
Recognized as a hard and important issue in testing
no practical solution
not so much work (compared to test gen.)
real pain (e.g. aeronautics, mutation testing)
Nikolai Kosmatov Advanced Test Coverage Criteria 33/ 55
![Page 58: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/58.jpg)
Detection goals
Automatic detection of uncoverable test objectives
a sound method
applicable to a large class of coverage criteria
strong detection power, reasonable speed
rely as much as possible on existing verification methods:
Observation:
Label (loc , p) is uncover-able
⇔Assertion assert (¬p);at location loc is valid
Nikolai Kosmatov Advanced Test Coverage Criteria 34/ 55
![Page 59: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/59.jpg)
Focus: checking assertion validity
Forward abstract interpretation, or Value Analysis (VA)[state approximation]
◮ compute an invariant of the program◮ then, analyze all assertions (labels) in one run
◮ global but limited reasoning
Weakest precondition calculus (WP) [goal-oriented]
◮ perform a dedicated check for each assertion◮ a single check usually easier, but many of them
◮ local but precise reasoning
Nikolai Kosmatov Advanced Test Coverage Criteria 35/ 55
![Page 60: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/60.jpg)
Example: program with two uncoverable labels
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
// l1: res == 0
// l2: res == 2
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 36/ 55
![Page 61: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/61.jpg)
Example: program with two valid assertions
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
//@ assert res != 0
//@ assert res != 2
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 37/ 55
![Page 62: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/62.jpg)
Example: program with two valid assertions
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
//@ assert res != 0 // both VA and WP fail
//@ assert res != 2 // detected as valid
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 37/ 55
![Page 63: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/63.jpg)
LUncov Methodology: Combine VA ⊕ WP
Goal: get the best of the two worlds
Idea: VA passes to WP the global information that WP needs
Which information, and how to transfer it?
VA computes variable domains
WP naturally takes into account assumptions (assume)
Proposed solution:
VA exports computed variable domains in the form ofWP-assumptions
Nikolai Kosmatov Advanced Test Coverage Criteria 38/ 55
![Page 64: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/64.jpg)
Example: alone, both VA and WP fail
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
//@ assert res != 0 // both VA and WP fail
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 39/ 55
![Page 65: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/65.jpg)
Example: combination VA⊕WP succeeds
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
//@ assume 0 <= a <= 20
//@ assume 0 <= x <= 1000 // VA inserts domains...
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
//@ assert res != 0
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 40/ 55
![Page 66: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/66.jpg)
Example: combination VA⊕WP succeeds
int main() {
int a = nondet (0 .. 20);
int x = nondet (0 .. 1000);
return g(x,a);
}
int g(int x, int a) {
//@ assume 0 <= a <= 20
//@ assume 0 <= x <= 1000 // VA inserts domains...
int res;
if(x+a >= x)
res = 1; // the only possible outcome
else
res = 0;
//@ assert res != 0 // ... and WP succeeds!
return res;
}
Nikolai Kosmatov Advanced Test Coverage Criteria 40/ 55
![Page 67: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/67.jpg)
LUncov: Results and Experiments
automatic, sound and generic method
new combination of existing verification techniques
experiments for 12 programs and 3 criteria (CC, MCC, WM):◮ strong detection power (95%),◮ reasonable detection speed (≤ 1s/obj.),◮ test generation speedup (3.8x in average),◮ more accurate coverage ratios (99.2% instead of 91.1% in
average, 91.6% instead of 61.5% minimum)
[Bardin et al. ICST 2014, TAP 2014, ICST 2015]
Nikolai Kosmatov Advanced Test Coverage Criteria 41/ 55
![Page 68: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/68.jpg)
Detecting polluting objectives
Most recent work [Marcozzi et al. ICSE 2018]
other sources of “pollution”:◮ duplicate and/or subsumed test objectives◮ harmful effect [Papadakis et al., ISSTA 2016]
detection technique:
◮ WP-based dedicated algorithms◮ enhanced with multi-core and fine tuning
achievements:
◮ detecting a large number of polluting test objectives (up to27% of the total number of objectives)
◮ scales: OpenSSL, gzip, SQLite
Nikolai Kosmatov Advanced Test Coverage Criteria 42/ 55
![Page 69: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/69.jpg)
LUncov in the LTest toolset for labels
Nikolai Kosmatov Advanced Test Coverage Criteria 43/ 55
Service cooperation
share label statuses
Covered, Infeasible, ?
Uses static analyzers from Frama-C
sound detection of uncoverablelabels
![Page 70: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/70.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 44/ 55
![Page 71: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/71.jpg)
Limitations of labels
Nikolai Kosmatov Advanced Test Coverage Criteria 45/ 55
![Page 72: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/72.jpg)
Hyperlabel Specification Language (HTOL)
Nikolai Kosmatov Advanced Test Coverage Criteria 46/ 55
![Page 73: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/73.jpg)
Hyperlabel Specification Language (HTOL) – Semantics
Nikolai Kosmatov Advanced Test Coverage Criteria 47/ 55
Formal Semantics:
![Page 74: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/74.jpg)
HTOL: Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 48/ 55
![Page 75: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/75.jpg)
HTOL: Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 49/ 55
![Page 76: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/76.jpg)
HTOL: Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 50/ 55
![Page 77: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/77.jpg)
HTOL: Taxonomy of coverage criteria
Nikolai Kosmatov Advanced Test Coverage Criteria 51/ 55
![Page 78: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/78.jpg)
HTOL: Expressiveness and support
Nikolai Kosmatov Advanced Test Coverage Criteria 52/ 55
![Page 79: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/79.jpg)
Outline
1 Labels
2 LTest: an all-in-one testing toolset
3 Efficient test generation for labelsDynamic Symbolic Execution (DSE)DSE⋆: optimized test generation for labels
4 Detection of infeasible test objectives
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 53/ 55
![Page 80: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/80.jpg)
Summary
Labels: a generic specification mechanism for coverage criteria◮ can easily encode a large class of criteria◮ a semantic view, with a formal treatment
DSE⋆: an efficient test generation technique for labels◮ an optimized version of DSE (Dynamic Symbolic Execution)◮ no exponential blowup of the search space
LUncov: an efficient technique for detection of infeasible objectives◮ based on existing static analysis techniques
LTest: an all-in-one testing toolset◮ on top of Frama-C and PathCrawler
HTOL: Hyperlabel Specification Language, extension of labels◮ capable to encode almost all common criteria including MCDC
Reminder: Goals
Specify [X] and Measure, [X], Cover [X] and Unmask [X]
Nikolai Kosmatov Advanced Test Coverage Criteria 54/ 55
![Page 81: Advanced Test Coverage Criteria · Coverage criterion = objectives to be fulfilled by the test suite Criterion guides automation Can be part of industrial normative requirements](https://reader033.fdocuments.us/reader033/viewer/2022042216/5ebef69436860d66b86d43ee/html5/thumbnails/81.jpg)
Future work
An efficient dedicated support of hyperlabels in testgeneration (DSE)
Further optimizations of LTest (e.g. detection of uncoverablehyperlabels)
Developing the emerging interest for LTest in industry
Nikolai Kosmatov Advanced Test Coverage Criteria 55/ 55