System Design Research Laboratory

Specification-based Testing with Linear Temporal Logic

Li TanOleg Sokolsky

Insup Lee

University of Pennsylvania

System Design Research Laboratory

Outline Motivation and the framework Property-coverage criterion

Derive coverage criterion from system requirement in temporal logic.

Test generation using model checkers.

Path-truncating strategy: from the infinite traces to finite tests. Black-box testing. White-box testing.

Conclusion

System Design Research Laboratory

Motivations Formal methods play a major role in

designing and verifying safety-critical reactive systems.

Model-based design to accelerate the software design and facilitate formal verification.

Temporal logics used to specify the system’s behavior requirements.

Testing is still an important link in software development cycle. Some of its advantages,

Testing can directly check the implementation. Testing will not suffer from the state-explosion

problems faced by model checking. Question: how to produce high-quality test

suite from temporal logic-based specification?

System Design Research Laboratory

Preliminary: LTL and 9LTLLinear Temporal Logic (LTL) define the property held by

the executions of a system, Path formula: a property held by an execution.

P ::=a | : a| P Æ P | P Ç P | X P | P U P | P R P a is an atomic proposition. X is the “next time” operator. U and R are the “until” and “release” operators.

P1 U P2: P1 holds till P2 holds P1 R P2: P2 holds unless an occurrence of P1 releases it

(shorthands) G P (Always hold) for false R P, and F P ( Eventually hold) for true U P

A (for all the paths), and E (exists a path) are the path quantifiers.

LTL: AP . P holds on all the executions. LTL is an expressive and popular logic. LTL is supported by many state-of-the-art model

checkers, including SPIN and (Nu)SMV 9LTL: EP . P holds on some execution.

System Design Research Laboratory

LTL and 9LTL by Examplenon-critical section 0

critical section 0

try0:=1

try0::=1try1=1?

turn=1?turn1?

try0:=0

turn=0?

try11?grant0:=1

try0:=0grant0:=0

non-critical section 1

critical section 1

try1:=1

try1::=1try0=1?

turn=0?turn0?

try1:=0

turn=1?

try01?grant1:=1

try1:=0Grant1:=0

System requirement: • (LTL) mux=AG((try1=1)->F(grant1=1))• Every request to access the critical section 1 is finally granted.

Extended FSM model T of Dekkar’s algorithm for mutual exclusion problem

P0P1

turn:=1turn:=0

11 1’

2

3

4

5

6

8

2’

3’

4’

5’

6’

8’

7 7’

System Design Research Laboratory

Challenges and Problems LTL specifiies the property that holds on all

the paths. To prove mux on T, all the executions

need to be checked. Executions are infinite in length.

Test has to be done in finite time. Test suite has to be finite in length and in

size Challenge: which execution shall be covered

in test and how many steps we shall test?

System Design Research Laboratory

Intuition and Informal Approach

Only those “non-trivial” executions need to be checked.

A request is finally granted: If a request to access the critical section 1 has been

made at least once, then we shall check if it is finally granted.

9 LTL formula: f1=E ( F (try1=1) Æ mux)

Access is by invitation only. If P1 never access the critical section 1 after a time,

then we shall check that no request has ever been made since then.

9 LTL formula: f2=E ( FG (grant1 1) Æ mux)

Question: can we formally derive these 9LTL characterizing test coverage directly from the LTL formula?

System Design Research Laboratory

Property-coverage criterion: a formal approach

Obtain the trapping formula by syntactically manipulating (mutating) LTL formula.

Given : a LTL formula . Output: a set of 9 LTL formulae G()

G()={E( Æ : [a!(a)] | a is an atomic proposition inside }

The polarity of a is positive if a is nested in the even number of negations in .

(a)=false if and only if the polarity of a is positive.

Property-coverage criterion for a test suite TS: each formula in G() is satisfied by at least one test in TS

System Design Research Laboratory

An example: deriving trapping formulae from mux

mux=AG((try1=1) !F(grant1=1))

f’1!!!

f’2!!!

AG((try1=1) !F(grant1=1))

E(muxÆ : G((try1=1) !false))

E(muxÆ F(try1=1))

E(muxÆ:G(true !F(grant1=1)))

E(muxÆFG(grant1=1))

A(G((try1=1) !F(grant1=1))

System Design Research Laboratory

Property-coverage testing and its implications

Nonvacuity in model checking: avoid some part of formula go unchecked even when the system satisfies a LTL property.

Example: A(a->b) holds trivially on any system a doesn’t occur.

A subformula affects on the system T iff T ² but 9 f. T ² [ -> f].

T satisfies nonvacuously on T if every subformula of affects on T.

[Theorem 1] T satisfies nonvacuously iff T satisfies and T passes a property-coverage test suite.

“Tests alone can show the present of an error but hardly its absence.”

Property-coverage testing ensures that every piece of requirements is checked.

System Design Research Laboratory

Test Generation using Model Checking is a counter-example for T and Af if is

an execution of T and ² :f. Model checks like SMV and SPIN produces an

counterexample if T doesn’t satisfies Af. Goal of generating tests for M and : test

generation: find test cases as the executions of the model M that satisfy G()={Ef1, Ef2, , Efn} Model Checking : Efi on M!

No. The counterexample is an execution satisfy Efi

Yes. M doesn’t satisfies nonvacuously. Check M and .

System Design Research Laboratory

Are we done yet?

Model M

Model Checker: E fi

Yes

M doesn’t satisfy nonvacuously

Stop Check M and

ImplementationCounterexample

NoTest case t

G()={E f0, E fn}

Property-

Coverage

Criteria

What is the

missing link

Could be infinitely long Has to be finite

System Design Research Laboratory

From Infinite Trace to Finite Test A sample counterexample for M ² : E ( F (try1=1) Æ mux).

{1,1’}({2,1’} {7,1’}{7,2’}{8,2’}{8,7’}{1,7’}

{1,8’}{2,8’})

try0:=0grant0:=0

try0:=1

grant0:=1

try0:=1

try1:=1

grant1=1 turn=1

try1:=0grant1:=0

turn:=0

Lasso-shaped Trace

If M ² : , there is always a lasso-shaped counterexample for M and .

Counterexamples produced by LTL model checkers are lasso-shaped.

When shall we stop?

System Design Research Laboratory

Path-Truncating Strategy Given: a lasso-shaped test case ()

Question: Can we test the “loop” for the finite time to know if the implementation passes ().

White-box testing Yes, test till the same state is encountered

twice at the same place on Black-box testing: the only known factor is the

maxium number of states n Yes, test till we can infer that the same state

is encountered twice at the same place on [Theorem 2] For a black-box system T with at

most n states and a lasso-shaped test t=¢ , n is the least number such that T passes t if and only T passes ¢ ()(n)

System Design Research Laboratory

Conclusion

We proposed a new paradigm to test Linear Temporal Logic property on the implementation

The property-coverage criterion is proposed to select test cases by the LTL property.

Coverage criterion is derived from the LTL property automatically.

Tests are generated using model checkers.

Path-truncating strategy is proposed to obtain finite test cases from infinite traces.

G()={E f0, E fn}

Model Checker

Model M

: E fi

Yes

M doesn’t satisfy nonvacuously

Counterexample

Stop Check M and f

NoImplementationTest case t

Property-

Coverage

Criterion

Path-TruncatingStrategy

Test settings