Conformance Test Suites, Extensionally Arend Rensink University of Twente Dutch Workshop on Formal...
-
Upload
rodney-anthony -
Category
Documents
-
view
215 -
download
2
Transcript of Conformance Test Suites, Extensionally Arend Rensink University of Twente Dutch Workshop on Formal...
Conformance Test Suites, Extensionally
Arend RensinkUniversity of Twente
Dutch Workshop on Formal Testing Techniques
University of Twente13 September 2007
DWFTT, September 2007Conformance Test Suites,
Extensionally 2
Automaton construction
Approach
Specification
SuspensionAutomaton
Test Suite
Test Automaton
ImplementationUnder Test
Formal
Physical
Non-deterministicalgorithm
Closer to actual implementation
• Finite if spec is regular
• On-the-fly construction
• Completeness criteria
• Direct reasoning & manipulation
Current
New
DWFTT, September 2007Conformance Test Suites,
Extensionally 3
Background
• Specification s given by LTS– Distinguished inputs and outputs– Weakly convergent
• Stable, output-free states: quiescent– Made explicit by -labelled self-loop– Suspension language Susp(s): traces with
’s• Implementation i modelled by IOTS– Input enabled
• Conformance (ioco)i ioco s , 8 w 2 Susp(s):
Out(i after w) µ Out(s after w)
DWFTT, September 2007Conformance Test Suites,
Extensionally 4
Example
• Self-closing door
• Wrong implementation:
press
openpress press
close
input: pressoutputs: open close
press
openpress
press
closepress
Error: 2 Out(i after press¢open)
explicit quiescence
input enabled
DWFTT, September 2007Conformance Test Suites,
Extensionally 5
Example (continued)
• Alternative specs:
press
openpress press
close
press non-determinism
under-specification
for press
press
open
close
press
DWFTT, September 2007Conformance Test Suites,
Extensionally 6
IOCO analyzed
•The following are equivalent:– 8 w2Susp(s):
Out(i after w) µ Out(s after w)– 8 w2Susp(s), x2U:
w¢x 2 Susp(i) ) w¢x 2 Susp(s)– Susp(i) Å (Susp(s)¢U) µ Susp(s)
– Susp(i) Å (Susp(s)¢U n Susp(s)) = ;
•So the following are equivalent: – i conforms to s– i has no suspension trace accepted by
Susp(s)¢Un Susp(s)
DWFTT, September 2007Conformance Test Suites,
Extensionally 7
Intuition
• Susp(s): correct behaviour
• Susp(s)¢U: Just one more response…
• Susp(s)¢U n Susp(s): Forbidden paths
DWFTT, September 2007Conformance Test Suites,
Extensionally 8
Example
press
openpress
close
press
press
openpress
close
openclose
close
open
press
Spec s Construct: Susp(s)¢U n Susp(s)
press
open
press
close
openpress
close
openclose
close
open press
Example error trace: press¢open¢
final state
DWFTT, September 2007Conformance Test Suites,
Extensionally 9
reduction & -saturation
• Given a set of traces W µ L*
– W- removes all traces from W with multiple successive ’s (of the form v¢¢
¢w)– W+ adds v¢+¢w for all v¢¢w 2 W
• Facts: If W is a suspension language– W+ = W– (W-)+ = W
• Improvement on previous result:– i conforms to s iff – i has no suspension trace accepted by
Susp(s)-¢Un Susp(s)
remove -cycles
DWFTT, September 2007Conformance Test Suites,
Extensionally 10
Test automata
• Let T be an arbitrary automaton (with ’s)T accepts i , Susp(i) Å Lang(T) = ;
• Properties defined for arbitrary T– soundness: i ioco s implies T accepts i– exhaustiveness: T accepts i implies i ioco s
• Seen previously:– Susp(s)-¢U n Susp(s) sound & exhaustive for s
• General criteria (main result, part 1):– T is sound for s if and only if
Lang(T) µ (Susp(s)¢U n Susp(s)) ¢ L*
– T is exhaustive for s if Lang(T) ¶ (Susp(s)-¯ U n Susp(s))
No output after
DWFTT, September 2007Conformance Test Suites,
Extensionally 11
Main result (part 2)
• Define: T is full if– v · w 2 Lang(T) implies v 2 Lang(T)– v¢a¢w 2 Lang(T) implies v¢¢a¢w 2 Lang(T)– v¢a 2 Lang(T) implies v 2 Lang(T)– v¢U µ Lang(T) implies v 2 Lang(T)
• For all i: fill(T) accepts i iff T accepts i• Necessary criterion for exhaustiveness:– If T is sound for s, then
T is exhaustive for s only if Lang(fill(T)) ¶ (Susp(s)-¯ U n Susp(s))
fill(T) can be constructed
from T
DWFTT, September 2007Conformance Test Suites,
Extensionally 12
Example
press
openpress
close
press
Spec s
Susp(s)-¯U n Susp(s)
press
openpress
close
openclose
close
open
press
press
openpress
close
openclose
close
open
press
Susp(s)¢U n Susp(s)
Straightforward automata
constructions only
DWFTT, September 2007Conformance Test Suites,
Extensionally 13
Given:
Possible with extensional characterisation
Compositionality
Test Tsound/complete for
Spec s
Spec s’
Composition(Parallel composition,action refinement, …)
Test T’
CompositionDesired:
DWFTT, September 2007Conformance Test Suites,
Extensionally 14
Automaton construction
From test automata to test suites
Specification
SuspensionAutomaton
Test Suite
Test Automaton
ImplementationUnder Test
Formal
Physical
Non-deterministicalgorithm
DWFTT, September 2007Conformance Test Suites,
Extensionally 15
From test automata to test suites• Equivalent testing power:– Test automaton T– Set {Ti}i with i Lang(Ti) = Lang(T)
• Tretmans test cases t as test automata:– Cycle-free– One final accepting state (= fail) – One final non-accepting state (= pass) – Non-final states either stimulus or response
• Tretmans’ algorithm generates {ti}i s.t.:– Lang(ti) µ Susp(s)¢Un Susp(s) for all ti
– 9i: w2 Lang(ti) for all w2 Susp(s)¢Un Susp(s)
DWFTT, September 2007Conformance Test Suites,
Extensionally 16
Automaton construction
Direct testing with automata
Specification
SuspensionAutomaton
Test Suite
Test Automaton
ImplementationUnder Test
Formal
Physical
Non-deterministicalgorithm
DWFTT, September 2007Conformance Test Suites,
Extensionally 17
Direct testing with automata
• Given a test automaton T– Deterministic, with explicit quiescence– Corresponds to entire test suite
• Synchronisation (product) of IuT and T:– Stimulus specified by T sent to IuT– Response of IuT mimicked in T
• Test verdict– Final state of T reached: fail– Response of IuT absent in T: pass– No reachable final state in T: pass– Done testing: pass
DWFTT, September 2007Conformance Test Suites,
Extensionally 18
Conclusion
• Test automata fill gap in existing theory– Yield precise conditions for completeness– Enable constructions on test suite– Work modulo Tretmans’ “test criteria”
(not covered in this presentation)
• Re-opens discussion on– Interaction between test and IuT– Input enabledness of test cases