Automated Model-Based Testing of Hybrid Systems

Michiel van Osch

PROSEJanuary 25, 2007

2

Motivation• Hybrid Systems

• Testing might be expensive, dangerous, or resources might be limited

• Discrete and real-time model-based testing does not test the continuous aspects of the system

3

Content

• Part I: Theory– Model-based Testing– Input-Output Conformance & Discrete Tests– Hybrid Systems– Hybrid Input-output Conformance– Hybrid Tests– Results

• Part II:Tool– Test Architecture– Specification– Tester– The Connection with the Implementation Under Test– Adapter– Limitations and Future Work– Case Study: Vacuum Control

4

Model-Based Testing

model

SUT

IUTconforms to

model

pass fail

test

tool

testgeneration

tool

test execution

toolIUT passes tests

IUT conf model

soundexhaustiv

e

5

Input-output Conformance for Discrete Systems

Impl. ioco Spec. iff for all traces α:

out(Impl. after α) out(Spec. after α)

If there is an output action from state s thenout(s) = {o in O| s →}

elseout(s) = {δ}

Furthermore,out(S) = UsS out(s)

6

Test-case Generation and Execution

1. Terminate with verdict pass

2. Select an input from the specification and apply it to the implementation

3. Observe an output or a timeout from the implementation and check if it is allowed according to the specification

7

Example

s1s0

s2

s3

?Activate

?Button2

?Button1

!XLCoffee

!Coffee

t0

t1

?Activate

t3

?Button2

t2

! δ

FailFail

!Coffee!XLCoffee

Pass

!XLCoffee

FailFail

!Coffee! δ

8

Hybrid Systems

• In Practice:– Discrete behavior plus continuous behavior– Continuous behavior can be input observed through sensors

or output generated by actuators

• In Theory:– Discrete actions plus flow of continuous variables

(trajectories)– Variables can be input variables and output variables– Hybrid Transition Systems

9

Hybrid Systems (Output Only)

Time

Coffee

0

?Button1 ?Button2

s1s0

s2

s3

?Activate

?Button2

?Button1

Coffee’ = 4 cl/sec.Δt = 8 sec.

Coffee’ = 3 cl/sec.Δt = 5 sec.

Coffee’ = 0Δt = 1 sec.

Coffee’ = 0Δt = 1 sec..

10

Hybrid Systems (Including Input)

Water

s1s0

s2

s3

?Activate

?Button2

?Button1

Water’ = 4 cl/sec.Coffee’ = Water’Δt = 8 sec.

Water’ = 3 cl/sec.Coffee’ = Water’Δt = 5 sec.

Water’ = 0Coffee’ = 0Δt = 1 sec.

Water’ = 0Coffee’ = 0Δt = 1 sec..

11

Hybrid Conformance

• For every reachable state, the set of output actions possible by the implementation is a subset of the set of output actions possible by the specification

• For every reachable state, the set of trajectories possible by the implementation is a subset of the set of trajectories possible by the specification

• In contrast to ioco, no quiescence action because there is always continuous output.

12

Continuous Output Only

Impl. is input-output conform a Spec. iff for all traces α:

out(Impl. after α) out(Spec. after α) and

traj(Impl. after α) traj(Spec. after α)

s1s0

s2

s3

?Activate

?Button2

?Button1

Coffee’ = 4 cl/sec.Δt = 8 sec.

Coffee’ = 3 cl/sec.Δt = 5 sec.

Coffee’ = 0Δt = 1 sec.

Coffee’ = 0Δt = 1 sec..

13

With Continuous Input

• The implementation is input enabled (for both discrete behavior and continuous behavior).

• We do not require the specification to be input complete.

Impl. is input-output conform a Spec. iff for all traces α:

out(Impl. after α) out(Spec. after α) and

traj(Impl. after α) traj(Spec. after α)

Does not work!!

Solution: Look at the trajectories of the Implementation with respect to the trajectories of input variables of the Specification

14

Hybrid Conformance (Continuous Input plus Output)

infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α)

s1

s3

?Button2

Water’ = 0Coffee’ = 0Δt = 1 sec..

s4Water’ = 0Coffee’ = 0Δt = 3 sec..

s5!Out of Cups

Water’ = 0Coffee’ = 0Δt = 1 sec..

u1

u3

?Button2

Water’ = 0Coffee’ = 0Δt = 1 sec..

u4Water’ = 0Coffee’ = 0Δt = 3 sec..

u5!Out of Cups

Water’ = 0Coffee’ = 0Δt = 1 sec..

Specification Implementation

Water’ = 0Coffee’ = 0Δt = 1 sec..

Still does not work because …

15

Hybrid Conformance (continuous input plus output)

If there is a trajectory from state s thenout(s) = {o in O| s →} {ξ}

elseout(s) = {o in O| s →}

Impl. hioco Spec. iff for all traces α:

out(Impl. after α) out(Spec. after α) and

infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α)

16

Hybrid Tests

A Special kind of Hybrid Transition Systems:• Tree like structure• Two terminal states: pass and fail• Deterministic for actions• Strongly time deterministic for trajectories

17

Hybrid Tests

1. Terminate with verdict pass

2. Select an input from the specification and apply it to the implementation

s1s0?Activate

Water’ = 0Coffee’ = 0

Specification

t0

t1

?Activate

Test

18

Hybrid Test-case Generation

3. If an output action has to happen immediately according to the specification then observe an output action and check if it is allowed according to the specification or let time pass by selecting and applying and observing a trajectory

Fail Fail

s4 s5!”Out of Cups”

Coffee’ = 0Water’ = 0Δt = 1

Specification

t4

Pass

!”Out of Cups”

Test-Case

Coffee’ = 0Water’ = 0Δt = 1

19

Hybrid Test-case Generation

4. Select an input trajectory from the specification, apply it to the implementation and observe the output trajectory simultaneous, possibly interrupted by an output action.

Specification

s1Water’ = 0Coffee’ = 0Δt = 1

Test

t1

t2

Water’ = 0Coffee’ = 0Δt = 1

Fail Fail

Fail Fail

!”out of cups”Water’ = 0

Coffee’ = 3Δt = 1

20

Results

• A hybrid conformance theory• Proven Sound and exhaustive• A Natural extension of discrete and timed

conformance theories

21

Content

• Part I: Theory– Model-based Testing– Input-Output Conformance & Discrete Tests– Hybrid Systems– Hybrid Input-output Conformance– Hybrid Tests– Results

• Part II: Tool– Test Architecture– Specification– Tester– The Connection with the Implementation Under Test– Adapter– Limitations and Future Work– Case Study: Vacuum Control

22

Libraries

Tester Architecture

• Specification: The Model from which Tests are Generated• Tester: Implements The Test Algorithm and Gives the Verdict• Adapter: Translated Input/Output from Model to a format suitable

for the Implementation Under Test and vice versa• Medium: The Interface between Tester and Implementation• IUT: The Implementation Under Test

Spec Tester Adapter

Medium IUT

23

Libraries

Specification

Needs to:• Model Discrete behavior and Continuous Behavior• Make Distinction between Input Actions, Output

Actions, and Internal Actions• Make Distinction between Input Variables, Output

Variables and Internal Variables• Model in an Intuitive way

Medium

AdapterSpec Tester IUT

24

Libraries

Specification

Medium

AdapterSpec Tester

proc Control(cont V: real, chan h,out: real)=|[ *(V <= 2 -> h!!1.0; out!!1.0; V >= 10 -> h!!0.0; out!!0.0)]|

proc Env(cont V: real, chan h: real)=|[ var n: real = 0.0:: V’=3.0*n - 1.0| *(h?n)]|

model Spec()=|[ cont V: real = 10.0, chan h,out: real:: Control(V,h,out)|| Env(V, h)]|

IUT

25

Libraries

Tester

Implements:• On the Fly Test Generation

– Select Input from Specification– Apply Input– Observe Output– Compare the Observed Output with the Output allowed by the

Specification– Give a Verdict or Continue Test

Medium

AdapterSpec Tester IUT

26

On the Fly Testing

proc ControlS(cont V: real, chan h: real)=|[ var n: real = 0.0:: V’=3.0*n - 1.0| *(V <= 2 -> n:=1.0; h!!1.0; V >= 10 -> n:=0.0; h!!0.0)]|

model Spec()=|[ cont V: real = 10.0, chan h: real:: ControlS(V,h)]|

V=10

V=2

V’=3.0*0.0-1.0Δt = 8 sec.

Select Input(χ)(Manually/ Automatic)

V=2

h!!1.0

passfail

h!!0.0

CompareValues(χ , Maple)

Pass

Give Verdict(with trace)

Continue

Apply(Via adapter)

Pressure’= -1.0 mbar/secΔt = 8 sec.

IUT

Pump OFF

Observe(Via adapter)

IUT

27

Additional Libraries

• χ –stepper for computing sets of allowed transitions and current state of the specification

• E.g. Maple for comparing observed continuous output (samples) with specified trajectories and comparing observed discrete output values with specified send actions

Libraries Medium

AdapterSpec Tester IUT

28

Libraries

The Connection

Medium

Jabber χ Model

TCP/IP Labview Controller

Wires Electronics

Buttons/Sensors

Robot Arm

AdapterSpec Tester IUT

29

Libraries

Tester

The Adapter

• Implements– Mapping of Variables/Actions of Specification to a

Implementation and vice versa (e.g. channels to function calls , or variables to wires)

– Translating Input/Output of Specification to Implementation and vice versa (e.g. functions to samples, or signals)

Medium

AdapterSpec IUT

30

Limitations and Future Work

• This is just a prototype, there are shortcomings!– Real Time Testing is Not Possible Yet– The complexity of Continuous behavior is limited by the

Hybrid χ –stepper implementation. E.g. currently only standard differential equations.

– Models are not ‘ideal’ for testing. E.g. in case of identifying input and output

– For performance reasons we only deal with deterministic specifications.

– We assume that the communication medium is reliable

• Adaptation of theory for Sampling and Inaccuracy• Case Studies

31

Real Time

• Generating and applying input (e.g. samples)

• Observing output and Time at which output Occurred in the Implementation

32

Limitations and Future Work

• This is just a prototype, there are shortcomings!– Real Time Testing is Not Possible Yet– The complexity of Continuous behavior is limited by the

Hybrid χ –stepper implementation. – Models are not ‘ideal’ for testing. – For performance reasons we only deal with deterministic

specifications.– We assume that the communication medium is reliable

• Adaptation of theory for Sampling and Inaccuracy• Case Studies

33

The Vacuum Case

• Lithography Process takes place in vacuum

• Waferstepper has Five Chambers

• Chambers are kept in Vacuum by a system of Pumps and Valves

• Pumps and Valves are Controlled by Software (discrete)

• Software observes Pressure in Chambers through Sensors (continuous)

34

Activities

• Modeling Hardware in Hybrid χ and Stand Alone Simulation

• Modeling (translating) Hardware in discrete (timed) χ and Integration with Software Controller

• Modeling (translating) in Uppaal for Model Checking• Testing Models and Software Controller with the

Hybrid Tester

35

Questions?