Post on 04-Jan-2016
Doron Peled,
University of Warwick
Joint work with Saddek Bensalem, Stavros Tripakis, Hongyang Qu
Unit testing: Selection of test cases (for white-box testing)
The main problem is to select a good coverage
criterion. Some standard options are: Cover all paths of the program. Execute every statement at least once. Each decision (diamond node on flow chart)
has a true or false value at least once. Each condition predicate is taking each truth
value at least once. Check all possible combinations of conditions in
each decision.
Cover all the paths of the program (skip)
Infeasible.
Consider the flow diagram on the left.
It corresponds to a loop.
The loop body has 5 paths.
If the loops executes 20
times there are 5^20 different paths!
May also be unbounded: number of iterations depend on data.
So also forget about covering all executions!
How to cover the executions?
if (A>1)&(B=0) then X=X/A; if (A=2)|(X>1) then X=X+1;
Choose values for A,B,X at the beginning that would force the right path/conditions/predicates.
Value of X may change, depending on A,B.What do we want to cover? Paths?
Statements? Conditions?
Statement coverageExecute every statement at least once
By choosingA=2,B=0,X=3each statement will
be chosen.The case where the
tests fail is not checked!
if (A>1)&(B=0) then X=X/A;
if (A=2)|(X>1) then X=X+1;
Now x=1.5
Decision coverageEach decision (diamond node in flow graph) tested with true and false outcome at least once.
Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1
Problem: Does not test individual predicates. E.g., when X>1 is erroneous in second decision.
if (A>1)&(B=0) then X=X/A;
if (A=2)|(X>1) then X=X+1;
Condition coverage (skip)Each predicate has a true and false value at least once.
For example: A=1,B=0,X=3 A=2,B=1,X=0
lets each condition be true and false once.
Problem:covers only the path where the first test fails and the second succeeds.
if (A>1)(A>1)&(B=0) then X=X/A;
if (A=2)|(X>1) then X=X+1;
Preliminary:Relativizing assertions
(B) : x1= y1 * x2 + y2 /\ y2 >= 0Relativize B) w.r.t. the assignment
becomes B) [Y\g(X,Y)]e(B) expressed w.r.t. variables at
A.) (B)A =x1=0 * x2 + x1 /\ x1>=0
Think about two sets of variables,before={x, y, z, …} after={x’,y’,z’…}.
Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after.
Here: x1’=y1’ * x2’ + y2’ /\ y2’>=0 /\x1=x1’ /\ x2=x2’ /\ y1’=0 /\ y2’=x1now eliminate x1’, x2’, y1’, y2’.
(y1,y2)=(0,x1)
A
B
A
B
(y1,y2)=(0,x1)
Y=g(X,Y)
Verification conditions: tests
C) is transformed to B)= t(X,Y) /\ C)
D) is transformed to B)=t(X,Y) /\ D)
B)= D) /\ y2x2
y2>=x2
B
C
D
B
C
Dt(X,Y)
FT
FT
How to find values for coverage?
•Put true at end of path.
•Propagate path backwards.
•On assignment, relativize expression.
•On “yes” edge of decision node, add decision as conjunction.
•On “no” edge, add negation of decision as conjunction.
•Can be more specific when calculating condition with multiple condition coverage.
A>1/\B=0
A=2\/X>1
X=X+1
X=X/Ano
no
yes
yes
true
true
How to find values for coverage?
A>1/\B=0
A=2\/X>1
X=X+1
X=X/Ano
no
yes
yes
true
true
A 2/\X>1
(A2 /\ X/A>1) /\ (A>1 & B=0)
A2 /\X/A>1Need to find a satisfying assignment:
A=3, X=6, B=0
Can also calculate path condition forwards.
How to cover a flow chart?(skip)
Cover all nodes, e.g., using search strategies: DFS, BFS.
Cover all paths (usually impractical). Cover each adjacent sequence of N nodes. Probabilistic testing. Using random number generator
simulation. Based on typical use. Chinese Postman: minimize edge traversal
Find minimal number of times time to travel each edge using linear programming or dataflow algorithms.Duplicate edges and find an Euler path.
Test cases based on data-flow analysis (skip)
Partition the program into pieces of code with a single entry/exit point.
For each piece find which variables are set/used/tested.
Various covering criteria: from each set to each
use/test From each set to
some use/test.
X:=3
z:=z+x
x>y
t>y
Some real life story
An expert programmer inspects the code of NASA MER.
He observe using his experience and intuition that some execution path is suspicious.
He decides how to force this path to execute, e.g., by figuring some inputs and initial values.
He executes the path, showing his supervisor the presence of an error.
We want to build some tools to help him with this process.
We’ll use LTL to help with formalizing the intuition on where the error is.
Learning from another technique: Model Checking
Automaton description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection
includes a counterexample. Repeat for different properties.
¬
Unit Testing Model Checking
Unit Checking
New: Test case generation based on LTL specification
CompilerModel
CheckerPath condition
calculation
First orderinstantiator
Testmonitoring
Transitions
Path
Flowchart
LTLAut
Path conditions Path in flow chart multiple executions following
path. First order formula. All executions of a path must start with initial
values satisfying the path condition. In deterministic code, there can be only one
execution starting with particular values, hence all executions starting with initial values satisfying the path condition will follow that path.
In nondeterministic code, each such initial value has an execution following a path. May need to insert synchronizing code.
Generalizations: include inputs, being more specific about decisions made in path.
Goals Verification of software. Compositional verification. Use only a unit of
code instead of the whole code. Parameterized verification. Verifies a procedure
with any value of parameters in “one shot” Generating test cases via path conditions: A
truth assignment satisfying the path condition. Helps derive the demonstration of errors.
Generating appropriate values to missing parameters.
Spec: ¬at l2U (at l2/\ xy /\
(¬at l2/\(¬at l2U at l2 /\ x2y )))
Automatic translation of LTL formula into an automaton [GPVW95]
LTL is interpreted over finite sequences.
Can use other (linear) specification.
Property specifies the path we want to find (SPIN: never claim),not the property that must hold for all paths (for this, take the negation).
¬at l2
at l2/\xy
¬at l2
at l2/\x2y
Observation:each node hasconjunctions of
predicates onprogram variables
and programcounters
Divide and Conquer Intersect property automatonproperty automaton with the
flow chartflow chart, regardless of the statements and program variables expressions.
Add assertions from the property automaton to further restrict the path condition.
Calculate path conditions for sequences found in the intersection.
Calculate path conditions on-the-fly. Backtrack when condition is false.Thus, advantage to forward calculation of path conditions (incrementally).
Spec: (only program counters here)
¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))
¬at l2
at l2
¬at l2
at l2
l2:x:=x+z
l3:x<t
l1:…
l2:x:=x+z
l3:x<t
l2:x:=x+z
XX==
at l2
at l2
¬at l2
Either all executions of a path satisfy the formula or none.
Sifts away path not satisfying formula. Then calculate path condition.
Spec: ¬at l2U (at l2/\ xy /\
(¬at l2/\(¬at l2U at l2 /\ x2y )))
¬at l2
at l2/\xy
¬at l2
at l2/\x2y
l2:x:=x+z
l3:x<t
l1:…
l2:x:=x+z
l3:x<t
l2:x:=x+z
XX==
xy
x2y
Only some executions of path may satisfy formula
Modify calculation of path condition to incorporate property
Calculating the intersection of the property automaton and flow graph (abstract variables away).
¬a
¬a a
a
as1 s2
s3 q2
q1
s1,q1
s1,q2 s3,q2
s2,q1Acceptance isdetermined by
propertyautomaton.
<>a
How to generate test cases Take the intersection of an LTL automaton (for a
never claim) with the flow graph. Some paths would be eliminated for not satisfying the assertions on the program counters.
Seeing same flow chart node does not mean a loop: program variables may value. Use iterative deepening.
For each initial path calculate the path condition. Backtrack if condition simplifies to false.
Report path condition based on flow graph path+LTL assertions.
Always simplify conditions!
How the LTL formula directs the search
Consider (x=4)U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
X=4
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
X=4
X=4
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
X=4
X=4
x<5
X=4
true
This is in acontradiction
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
X=5
X=4
How the LTL formula directs the search
Consider x=4U (x=5/\o…)x=4
x=5x<5
x:=x+1y:=7
truefalse
X=5
X=4
Example: GCD l1:x:=a
l5:y:=z
l4:x:=y
l3:z:=x rem y
l2:y:=b
l6:z=0?yesn
o
l0
l7
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0?yesn
o
Oops…with an error (l4 and l5 were switched).
l0
l7
Why use Temporal specification
Temporal specification for sequential software?
Deadlock? Liveness? – No! Captures the tester’s intuitionintuition about the
location of an error:“I think a problem may occur when the program runs through the main while loop twice, then the if condition holds, while t>17.”
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0?yesn
o
l0
l7
a>0/\b>0/\at l0 /\at l7
at l0/\a>0/\b>0
at l7
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0?yesn
o
l0
l7
a>0/\b>0/\at l0/\at l7
Path 1: l0l1l2l3l4l5l6l7a>0/\b>0/\a rem b=0
Path 2: l0l1l2l3l4l5l6l3l4l5l6l7 a>0/\b>0/\a rem b0
Potential explosion
Bad point: potential explosion
Good point: may be chopped on-the-fly
Now we add time
Detailed model, for each transition we have 4 parameters [l, u, L, U]: l Needs to be enabled at least that much. u Cannot be enabled without taken longer
than that. L Least time for transformation to occur
(after been chosen). U Transformation cannot take more than
that.
Translation to timed automata
s1
at l
s3,at lx2<u2x1<u1
s4,at lx2<u2
s2,at lx1<u1
c1c2
x2:=0
c1c2
x1:=0
c1c2
x1:=0
c1c2
x2:=0c1c2
x1:=0c1c2
x2:=0
c1c2 c1c2
c1c2
x1,x2:=0 c1c2
c1c2 c1c2
Timing out the enabledness:Zero counters,Cannot wait enabled too much.
Translation to timed automata
s1
at l
s3,at lx2<u2x1<u1
s6x2<U2
s5x1<U1
s4,at lx2<u2
s2,at lx1<u1
x1l1x1:=
0
x1l1x1:=
0
x2l2x2:=
0
x2l2x2:=
0
c1c2
x2:=0
c1c2
x1:=0
c1c2
x1:=0
c1c2
x2:=0c1c2
x1:=0c1c2
x2:=0
c1c2 c1c2
c1c2
x1,x2:=0 c1c2
c1c2 c1c2
ac
ac bc bc
Can fire only if waited enough,Zero counters again.
Translation to timed automata
s1
at l
s3,at lx2<u2x1<u1
s8s7
s6x2<U2
s5x1<U1
s4,at lx2<u2
s2,at lx1<u1
x1L1 x2L2
x1l1x1:=
0
x1l1x1:=
0
x2l2x2:=
0
x2l2x2:=
0
c1c2
x2:=0
c1c2
x1:=0
c1c2
x1:=0
c1c2
x2:=0c1c2
x1:=0c1c2
x2:=0
c1c2 c1c2
c1c2
x1,x2:=0 c1c2
c1c2 c1c2
ac
ac bc bc
af bf
Should we really look at paths? Its easy to select an
interleaved sequence.
But due to time limitations, it may execute in a different order.
Just the order on events from the same process and using same variables is to be considered.
a
b
c
d
a b
c d
Sameprocess
Samevariable
Generate an automaton for all consistent interleavings
a b
c d
a
a b
b
c
c
bd
dc
Intersect this automaton with automaton for system.Calculate “partial order” condition: start from leaves.When there is a choice, usedisjunct.
Generate an automaton for all consistent interleavings
a
a b
b
c
c
bd
dc
Generate an automaton for all consistent interleavings
a
a b
b
c
c
bd
dc
Generate an automaton for all consistent interleavings
a
a b
b
c
c
bd
dc
An example — a simple network protocol
The flow charts
Path — no timeout
Precondition
The simplified precondition: l >= 110
The diagrams
The PET tool Basic mode: interactive choice of a path,
calculating of path conditions. Model checking mode. Iterative model checking mode: apply model
checking recursively to find successive segments, control backtracking.
Unit checking mode. Calculating path condition: simplify, simplify,
simplify.Use SML and HOL for rewriting and deciding on Pressburger arithmetic. Plan using other tools!
Problem: US patent 6,408,430 belongs to Lucent!
Drivers and Stubs(skip)
Driver: represents the program or procedure that called our checked unit.
Stub: represents a procedure called by our checked unit.
In our approach: replace both of them with a formula representing the effect the missing code has on the program variables.
Integrate the driver and stub specification into the calculation of the path condition.
l1:x:=a
l5:x:=y
l4:y:=z
l3:z’=x rem y/\x’=x/\y’=x
l2:y:=b
l6:z=0?yesn
o
l0
l7
Some references
Translating LTL into automata:Gerth, Peled, Vardi, Wolper, Simple on-the fly automatic verification of temporal logic, PSTV 1995.
The PET tool:Gunter, Peled, Path Exploration Tool, Tacas 1999, LNCS 1579
Unit Checking:Gunter, Peled, Unit Checking: symbolic model checking for unit of code, LNCS 2772 (Z.M. birthday volume)
Forcing an execution under nondeterminism:Qu, Peled, Enforcing Concurrent Temporal Behavior, RV 2004
Conclusions Model checking and testing have a lot in common. Can
use ideas from model checking for generating test cases.
Unit Testing: Model checking of infinite state spaces.But: semidecidable: Don’t know when to stop search (undecideable), Don’t know when condition equivalent false
(undecideable). Tools, visual user interface. Generalization to real time systems. Wanted collaborations: integrate stronger simplifiers,
more complicated data structures, implementing abstractions.