Jüri Vain Dept. of Computer Science Tallinn University of Technology J.Vain “...Reactive Planning...
-
Upload
nicholas-bruce -
Category
Documents
-
view
214 -
download
1
Transcript of Jüri Vain Dept. of Computer Science Tallinn University of Technology J.Vain “...Reactive Planning...
Online Testing ofNondeterministic Systems
withthe Reactive Planning
Tester
Jüri VainDept. of Computer ScienceTallinn University of Technology
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
1
Reference
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
2
Vain, Jüri; Kääramees, Marko; Markvardt, Maili Online testing of nondeterministic systems with reactive planning tester. In: Petre, L.; Sere, K.; Troubitsyna, E. (Eds.). Dependability and Computer Engineering : Concepts for Software-Intensive Systems. Hershey, PA: IGI Global (2012), pages 113-150.
Outline Preliminaries
Model-Based Testing Online testing
Reactive Planning Tester (RPT) Synthesis of the RPT Performance issues Test execution environment dTron Demo
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
3
4
What is testing for? to check the quality (functionality, reliability,
performance, …) of an (software) object
-by performing experiments-in a controlled way
In avg. 10-20 errors per 1000 LOC30-50 % of development time and cost in software
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
5
What is a Test?
Software under Test
(SUT)
Test Data Output
Test Cases
Correct result?
Oracle
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
6
Model-Based Testing
(typically)
Specifics of testing embedded systems
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
7
– non-determinism– partial observability – RT constraints– dependability test coverage issues
How to address them in testing?
Real environment in the loop!
Option 1: Offline vs Online testing
Offline testing Open “control” loop Test is not adaptive to outputs or timing of SUT Test planning – result analysis loop is long
Online testing Flexible, test control is based on SUT feedback One test covers usually many test cases
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
8
Online testing Group of test generation and execution
algorithms that compute successive stimuli at runtime directed
by the test purpose and the observed outputs of the SUT
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
9
Tester
SUT
Online testing Advantages:
The state-space explosion problem is reduced because only a limited part of the state-space needs to be kept track of at any point in time.
Drawbacks: Exhaustive planning is diffcult due to the
limitations of the available computational resources at the time of test execution.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
10
Option 2: Test scripting vs model-based generation
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
11
Scripting: does not need SUT modelling + sensitive to human errors - inflexible, needs rewriting even with small
changes of SUT specs - hard to achieve test coverage -
Model-based generation: considerable SUT modelling effort - correctness of tests is verifiable + easy to modify and regenerate + clear characteristics of coverage +
Model-Based Testing Given
Model of the SUT specification System Under Test (SUT), The test goal (in terms of spec. model elements)
Find If the SUT conforms to the specification in terms
expressed in the test goal.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
NEEDED! - Sufficiently rich modelling formalism,- Supporting tool set.
12
Model-Based Testing We assume formal specs as:
UML State Charts Extended Finite State Machines MSC OCL
etc.
UPTA -Timed Automata (timing, parallelism, test data)
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
13
”Relativized Real-Time i/o conformance” Relation
Spec = UppAal Timed Automata Network: Env || IUT
Timed Trace: i1.2½.o1.3.o2.19.i2.5.i3
Expected system reactionInput event ordering
16J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Online MBT architecture: UppAal-TRON
[www.uppaal.com]
Bottleneck of online MBT: planning
Spectrum of planning methods Random walk (RW): select test stimuli in random
inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved
RW with reinforcement learning (anti-ant) the exploration is guided by some reward function
........
Exploration with exhaustive planning MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models
17J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Bottleneck of online MBT: planning
Spectrum of planning methods Random walk (RW): select test stimuli in random
inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved
RW with reinforcement learning (anti-ant) the exploration is guided by reward function
........
Exploration with exhaustive planning MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models
???
18J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Full space
1 stepahead
Zero planning
Bottleneck of online MBT: planning
Spectrum of planning methods Random walk (RW): select test stimuli in random
inefficient - random exploration of the state space test cases that are unreasonably long may leave the test purpose unachieved
RW with reinforcement learning (anti-ant) the exploration is guided by some reward function
........
Exploration with exhaustive planning MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models
Planning with adaptive horizon!
19J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Full space
1 stepahead
Zero planning
Reactive Planning in a Nutshell (1) Instead of a complete plan, only a set of
decision rules is derived The rules direct the system towards the
test goal. Based on current situation evaluation just
one subsequent input is computed at a time.
Planning horizon is adjusable
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
20
Reactive Planning: Planning cycle Identify current state of SUT:
Observe the output (or history) of the SUT
Pick the next move: Select one from unsatisfied test (sub-)goals Gi
Compute the best strategy for Gi: Gain function guides the exploration of the model
(choose the transition with the shortest path to the (sub-)goal Gi)
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
21
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Reactive Planning Tester in a Nutshell (2)
State model of the IUT Test goal
Reactive Planning Tester (RPT) model
Reachability Analysis
Reactive Planning Tester (RPT)
implementation
SUT
Offline phase Online phase
Stimuli / responses
RPT is another
state model22
RPT autonomously generates stimuli to
reach the goal
Constructing the RPT model
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
23
Model of SUT
Test goal
Model of RPT
Test strategy parameters:- planning
horizon- timeouts
Synthesis of RPT
Extraction of the control strcture
Constructing gain guards
(GG)
Constructing gain functions
Reduction of GG
RPT: Key Assumptions
Testing is guided by the (EFSM) model of the tester and the test goal.
Decision rules of reactive planning are encoded in the guards of the transitions of the tester model.
The SUT model is presented as an output observable nondeterministic EFSM in which all paths are feasible1.
1 - A. Y. Duale and M. U. Uyar. A method enabling feasible conformance test sequence generation for EFSM models. IEEE Trans. Comput.,53(5):614–627,2004.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
24
Example: Nondeterministic FSM
s2
s3
e6:i6/o6
e7:i7/o7
s1
e0:i0/o0
e1:i0/o1
e2:i2/o2
e3:i3/o3
e4:i3/o4
e5:i5/o5
i0 and i3 are output observable nondeterministic inputs
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
25
Encoding the Test Goal in SUT Model Trap - a boolean variable assignment
attached to the transitions of the IUT model
A trap variable is initially set to false.
The trap update functions are executed (set to true) when the transition is visited.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
26
Goal directed testing: The power of traps
Several goals can be expressed all/selected transitions
transition sequences (traps with reference to other traps)
advanced goals using auxiliary variables, consequent transitions, repeated pass, …
traps with 1st order predicates data variables
Properties not expressible by traps Assertions/invariants – always/never properties
The model specifies only the allowed behaviours
Åbo Akademi University, Dec 2011
Add Test Purpose
s2
s3
e6:i6/o6, t6=true
e7:i7/o7, t7=true
s1
e0:i0/o0
t0=true
e1:i0/o1, t1=true
e2:i2/o2, t2=true
e3:i3/o3, t3=true
e4:i3/o4, t4=true
e5:i5/o5, t5=true
bool t0 = false;...bool t7 = false;
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
28
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Example: Adding the Test Goal
s2
s3
e6:i6/o6, t6=true
e7:i7/o7, t7=true
s1s1
e0:i0/o0
t0=true
e1:i0/o1, t1=true
e2:i2/o2, t2=true
e3:i3/o3, t3=true
e4:i3/o4, t4=true
e5:i5/o5, t5=true
Initial values:bool t0 = false;...bool t7 = false;
Trap variables are initially set to
false
Trap update functions are executed (set to
true) when the transition is visited
Test goal is defined by trap variables ti attached
to transitions
29
Model of the tester Generated from the SUT model decorated with test purpose Transition guards encode the rules of online planning 2 types of tester states:
active – tester controls the next move passive – SUT controls the next move
2 types of transitions: Observable – source state is a passive state (guard true), Controllable – source state is an active state (guard pS /\ pT
where pS – guard of the SUT transition; pT – gain guard)
The gain guard (defined on trap variables) must ensure that only the outgoing edges with maximum gain are enabled in the given state.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
30
Construction of the Tester Sceleton
s2
s3
e6:i6/o6, t6=true
e7:i7/o7, t7=true
s1
e0:i0/o0
t0=true
e1:i0/o1, t1=true
e2:i2/o2, t2=true
e3:i3/o3, t3=true
e4:i3/o4, t4=true
e5:i5/o5, t5=true
s1
sa
s2sb
sc
s3
sd se
sf
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
States: Transitions: - active - observable
- passive - controllable
31
Add IO and Gain Guards
s1
s4
s2s5
s6
s3
s6 s9
s7
eo0:
o0/t0=true eo
1:
o1/t1=true
ec01:
[pc01(T)]
-/i0 eo
2:o2/t2=true
eo7:
o7/t7=true
ec2:
[pc2(T)]
-/i2
ec34:
[pc34(T)]
-/i3eo
3:o3/t3=true
eo6:
o6/t6=true ec6:
[pc6(T)]
-/i6 ec7:
[pc7(T)]
-/i7
ec5:
[pc5(T)]
-/i5
eo5:
o5/t5=true
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
32
Constructing the gain guards (GG): intuition
GG must guarantee that each transition enabled by GG is a prefix of some locally
optimal (w.r.t. test purpose) path;
tester should terminate after the test goal is reached or all unvisited traps are unreachable from the current state;
to have a quantitative measure of the gain of executing any transition e we define a gain function ge that returns a distance weighted sum of unsatisfied traps that are reachable along e.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
33
Constructing Gain Guards: intuition
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Observed current state
Alternative choices
...tri trj trk
...
Planning cones to be covered for decision making
34
Constructing Gain Guards: intuition
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
tri trj trk
Alternative continuation choices
...
ge(tri,T )
...
ge(trj,T )ge(trj,T )
Gain functions characterize planning options
35
Current state
Constructing the gain guards: properties of gain functions
ge = 0, if it is useless to fire the transition e from the current state with the current variable bindings;
ge > 0, if fireing the transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap;
gei > gej for transitions ei and ej with the same source state, if taking the transition ei leads to unvisited traps with smaller distance than taking the transition ej;
Having gain function ge with given properties define GG:
pT (ge = maxk gek) ge > 0
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
36
Constructing the Gain Functions: shortest path trees
Reachability problem of trap labelled transitions can be reduced to single-source shortest path problem1.
Arguments of the gain function ge are then the shortest path tree TRe with root node e VT – vector of trap variables
To construct TRe we create a dual graph G = (VD,ED) of the tester control graph MT where the vertices VD of G correspond to the transitions of the MT, the edges ED of G represent the pairs of consequtive
transitions sharing a state in MT (2-switches)
1 - Fredman & Tarjan 1987 O(E + V log V)J.Vain “...Reactive Planning Testing...”Abo,
Feb 3, 201237
Constructing the Gain Guards: shortest path tree
(example)
The dual graph of the tester model The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition ec
01
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
38
Constructing the gain guards:
gain function (1) Represent the reduced tree TR(ei, G) as a set of elementary sub-trees
each specified by the production i j{1,..n}j
Rewrite the right-hand sides of the productions as arithmetic terms:
(3) ti - trap variable ti lifted to type N, c - constant for rescaling the numerical value of the gain function, d(0, i) the distance between vertices 0 and i, where
l - the number of hyper-edges on the path between 0 and i
w j – weight of j-th hyperedge
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
39
Constructing the gain guards:
gain function (2)
For each symbol i denoting a leaf vertex in TR(e,G) define a production rule
(4)
Apply the production rules (3) and (4) starting from the root symbol 0 of TR(e,G) until all non-terminal symbols i are substituted with the terms that include only terminal symbols ti and d(0, i)
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
40
Example: Gain Functions
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
41
Example: Gain Guards
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
42
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Example: Construction of the Tester Structure
s2
s3
e6:i6/o6, t6=truee7:
i7/o7, t7=true
s1
e0:i0/o0
t0=true
e1:i0/o1, t1=true
e2:i2/o2, t2=true
e3:i3/o3, t3=true
e4:i3/o4, t4=true
e5:i5/o5, t5=true
s1
s4
s2s5
s6
s3
s6 s9
s7
eo0:
o0/t0=true
eo1:
o1/t1=true
ec01:
[pc01(T)]
-/i0 eo2:
o2/t2=true
eo7:
o7/t7=true
ec2:
[pc2(T)]
-/i2
ec34:
[pc34(T)]
-/i3 eo3:
o3/t3=true
eo6:
o6/t6=true
ec6:
[pc6(T)]
-/i6ec
7:[pc
7(T)]-/i7
ec5:
[pc5(T)]
-/i5
eo5:
o5/t5=true
eo4:
o4/t4=true
The gain guards guarantee that only the outgoing
edges with maximum gain are enabled in the given
state
43
Complexity of constructing and running the tester
The complexity of the synthesis of the reactive planning tester is determined by the complexity of constructing the gain functions.
For each gain function the cost of finding the TRe by breadth-first-search is O(|VD| + |ED|) [Cormen], where
|VD| = |ET| - number of transitions of MT |ED| - number of transition pairs of MT (is bounded by |ES|2)
For all controllable transitions of the MT the upper bound of the complexity of the computations of the gain functions is O(|ES|3).
At runtime each choice by the tester takes O(|ES|2) arithmetic operations to evaluate the gain functions
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
44
Performance of reactive planning
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
• Test Goal: All Transitions
• Test Goal: Selected Transition
45
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Experiments: Case Study: Model of the IUT
Model of Feeder Box Controller power management (31 states, 73 transitions)
46
How to plan in large models?
Test goal: all transitions
100
1000
10000
100000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
horizon
ste
ps
Planning with horizon + anti-ant
Planning with horizon + random choice
10
100
1000
10000
0 1 2 3 4 5 6 7 8 9 10 1112 13 1415 16 1718 19 20
horizon
step
s
Planning with horizon + anti-ant
Planning with horizon + random choice
Test goal: single transition
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
48
Avera
ge leng
ths
of
the t
est
sequence
s • Adjustable planning horizon• Dependancy between horizon and test length
Horizon saturation point
Shaping RPT planning cones (i)
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
tri trj trk
Decision point
Alternative choices
In online-testing decision time must be strictly bounded!...
ge(tri,T )
...
ge(trj,T ) ge(trj,T )
Gain functions
49
Shaping RPT planning cones (ii)
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
tri trj trk
Decision point
Alternative choices
Decision time strictly bounded!
Prune the cone!...
ge(tri,T )
...
ge(trj,T ) ge(trj,T )
Gain functions
Horizon h
Different ways for defining h, and ‘visibility’ depending on depth of tree.
Fully
vis
ible
Part
iall
y
vis
ible
50
Average lengths of test sequences in the experiments
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Horizon
All transitions test coverage Single transition test coverage
Planning with horizon Planning with horizon
anti-ant random choice anti-ant random choice
0 18345 ± 5311 44595 ± 19550 2199 ± 991 4928 ± 4455
1 18417 ± 4003 19725 ± 7017 2156 ± 1154 6656 ± 5447
2 5120 ± 1678 4935 ± 1875 1276 ± 531 2516 ± 2263
3 4187 ± 978 3610 ± 2538 746 ± 503 1632 ± 1745
4 2504 ± 815 2077 ± 552 821 ± 421 1617 ± 1442
5 2261 ± 612 1276 ± 426 319 ± 233 618 ± 512
6 2288 ± 491 1172 ± 387 182 ± 116 272 ± 188
7 1374 ± 346 762 ± 177 139 ± 74 147 ± 125
8 851 ± 304 548 ± 165 112 ± 75 171 ± 114
9 701 ± 240 395 ± 86 72 ± 25 119 ± 129
10 406 ± 102 329 ± 57 73 ± 29 146 ± 194
11 337 ± 72 311 ± 58 79 ± 30 86 ± 59
12 323 ± 61 284 ± 38 41 ± 15 74 ± 51
13 326 ± 64 298 ± 44 34 ± 8 48 ± 31
14 335 ± 64 295 ± 40 34 ± 9 40 ± 23
15 324 ± 59 295 ± 42 25 ± 4 26 ± 5
16 332 ± 51 291 ± 52 23 ± 2 24 ± 3
17 324 ± 59 284 ± 32 22 ± 2 21 ± 1
18 326 ± 66 307 ± 47 21 ± 1 21 ± 1
19 319 ± 55 287 ± 29 21 ± 1 21 ± 1
20 319 ± 68 305 ± 43 21 ± 1 21 ± 1
51
Average time spent for online planning of the next step
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
13
14
15
16
17
18
19
20
21
22
23
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
horizon
mse
c
All transitions
Single transition
52
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
53
How to generate test data?
Example: RPT synthesis for INRES protocol
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
54
Model of the SUT local component
Example: RPT synthesis for INRES protocol
Local SUT component model RPT model
Timeout!
Off!
LowICONreq?
DR?
CC?
?
AK?
55J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Online planning constraints for the RPT
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
56
How to derive data constraints? For all transitions t(si,.) of state si generate reduced
reachability tree RRTi s.t. transition t(si,.) is a root and the trap labeled transitions the
terminal nodes of the RRTi .
Compute data constraint for each path j of RRTi
use wp-algorithm (starting from trap node) for pairs of neigbbour traps of j
unfold loops using gfp for termination for constructing the gain function of j record (when traversing j):
traps remaining on the path j and the lengths of inter-trap paths
construct the gain function for full path j using trap-to-trap distances on that path and the vector of trap variables.
Global data constraint for the path is a conjunction of data constraints pairwise traps of j
57J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
Summary RPT synthesis technique relies on offline static
analysis of the SUT model and test goals.
Efficiency of planning: Number of rules that have to be evaluated at each step is
relatively small (i.e., = the number of outgoing transitions of a current state)
The execution of decision rules is significantly faster than looking through all potential alternatives at runtime.
Scalability supported by RRT pruning technique Provides test sequences that are lengthwise close to
optimal.
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
58
59
DTRON: Distributed RPT execution (http://dijkstra.cs.ttu.ee/~aivo/dtron/usage.html)
SPREAD
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
60
||
LTC communicating timed automata (TA)
TA model of local SUT
component (extended with online planning
constraints)
TA modelof the local test goal (property)
MsMg
||
TA model of local SUT
component (extended with online planning
constraints)
TA modelof the local test goal (property)
MsMg
“TCI update” signals
“Test coverage item (TCI) reached” signal
TCI synchroLTC LTC
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
61
Tester components communicate over channels
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
62
DTRON: distibuted, heterogeneous components
SPREAD
Selenium TestCast
Z3
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
DTRON: dynamic test reconfiguration
63
SPREAD
Selenium TestCast Selenium
Model (Testerk ) Model (Tester1) Model (Test Plan Supervisor)
Tester component updates
64
DTRON: Visualization of test runs
Questions?
Thank You!
J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012
65