Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.

Testing with Formal Methods

Ed Brinksma

course 2004

A Formal Framework

TT 2004, Formal Framework 2© Ed Brinksma/Jan Tretmans

Software Testing

Testing is: important much practiced 30% - 50% of project effort expensive time critical not constructive

(but sadistic?)

But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool :

“if you’re a bad programmer you might be a tester”

Attitude is changing: more awareness more professional

Improvements possible

with formal methods !?


Model Based Testingwith Formal Methods

Claims :

Formal specifications are more precise, more complete, unambiguous

Formal models can be formally verified

Formal models allow automatic generation of tests


Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box

black box

Level of detail

Accessibility

Characteristics

usability

reliability

module

security


Formal Methods

Use mathematics to model relevant parts of software

Precise, formal semantics: no room for ambiguity or misinterpretation

Allow formal validation and reasoning about systems

Amenable to tools: automation

Examples: Z Temporal Logic First order logic SDL Specification and

Description Language LOTOS Promela Labelled Transition Systems Finite state machine Process algebra ......


Formal methods: proving properties research sound theories “clean”

Testing : trial & error practice heuristics “dirty hands”

Testing & Formal MethodsA Perfect Couple ?

“Testing is not necessary after formal verification”

“Testing can only detect the presence of errors, not their absence”

“Formal methods are toys for boys”

“Formal methods have extreme potential - but not for my project”


Testing with Formal Methods

Testing with respect to a formal specification

Precise, formal definition of correctness :

good and unambiguous basis for testing

Formal validation of tests

Algorithmic derivation of tests :

tools for automatic test generation

Allows to define measures expressing coverage

and quality of testing


Testing & Formal Methods

Claims :

Combining the “mathematically clean” world of formal

methods and the “dirty hands” world of testing

Testing and formal methods are both necessary in

software development

Formal methods improve the testing process

Formal testing stimulates the use of formal methods


Goal:Testing functional behaviour

of black-box implementation

with respect to specificationin formal language

based on formal definition

of conformance

Specification BasedFunctional Testing

with Formal Methods

formal model

systemimplementatio

n

formal testing


testtool

implementationunder test

pass

fail

Test AutomationTraditional test automation

= tools to execute and manage test cases

Why not generate tests automatically ?

specification TTCNTTCNtestcase

s


test

executionpass / fail

Formal Testing

test

generation

test suite TS

specification S

implementation i

correctness criterion

implementation

relation

i passes Ts

i ioco s

soundexhaustiv

e


Formal relations

conforms-to IMPS SPECS iut conforms-to spec: implemetation under test obeys

specification hard: relates formal specification and real world system

imp MODS SPECS iIUT imp spec: model of the iut obeys specification relates formal objects

test hypothesis: iut conforms-to spec iIUT imp spec

iut passes t =def v ( EXEC ( t, iut )) = pass execute a test: EXEC: TESTS IMPS P OBS verdict function: v: P OBS { pass, fail } iut passes Ts t Ts . iut passes t


Formal Testing : Conformance

s SPECS SpecificationIUT Implementation under Test

IUT is concrete, physical object

Model the physical world

But IUT is black box ! ?

Assume that model iIUT exists

specification S

implementation IUT

correctness criterion

IUT conforms-to s


Formal Testing : Conformance

specification S

implementation iIUT

formalcorrectness

criterioniIUT imp s

s SPECS Specification

iIUT MODS model of IUT

Test hypothesis : each concrete IUT can be modelled

by some iIUT MODS

Conformance : iIUT imp s

iIUT is not known ;

testing to learn about iIUT


Conformance testing

conforms-to IMPS SPECS iut passes t =def v ( EXEC ( t, iut )) = pass

iut passes Ts t Ts . iut passes t

conformance: complete: iut conforms-to spec iut

passes Ts

sound: iut conforms-to spec iut passes Ts

exhaustiveness: iut conforms-to spec iut passes Ts

ioco: input output conformance


Formal Testing

exec : TESTS IMPS (OBS)

der : SPECS (TESTS)

Ts TESTS

s SPECS

IUT IMPS

imp

iIUT MODS

obs : TESTS MODS (OBS)

t: (OBS) {fail,pas

s}

OBS

pass

fail

Proof soundness and exhaustivess:iMODS .

( tder(s) . t(obs(t,i)) = pass ) i imp s

Test hypothesis :

IUTIMPS . iIUT MODS .

tTESTS . exec(t,IUT) = obs(t,iIUT)


Formal Testing : Test Derivation

test

generation

test suite TS

specification S

Test generation :

der : SPECS ( TESTS )

Test suite - set of test cases : T TESTSTest case : t TESTS


Formal Testing : Test Execution

test

execution OBS

test suite T

implementation IUT

Test execution leads to a set of observations : exec : TESTS IMPS ( OBS )

Model of test execution : obs : TESTS MODS ( OBS )

i IUT


Formal Testing : Test Hypothesis

OBS

test suite T

Observational framework : TESTS, OBS, exec, obs

Test hypothesis :

IUT IMPS . iIUT MODS .

t TESTS . exec ( t, IUT ) = obs ( t, iIUT )

obsi IUT

test

execution

IUT

exec

IUT

exec


Formal Testing : Verdicts

Observations are interpreted : t : (OBS) { fail, pass }

test

execution OBS t

pass

fail


Testing for Conformance

IUT passes Ts def t Ts . IUT passes t

IUT passes t def t ( exec ( t, IUT ) ) = pass

Test hypothesis :

t TESTS . exec ( t, IUT ) = obs ( t, iIUT )Proof obligation :

i MODS .

( t Ts . t ( obs ( t, i ) ) = pass ) i imp s

IUT passes Ts IUT conforms-to s?

Definition : IUT conforms-to sIUT conforms-to s

iIUT imp s

t Ts . t ( obs ( t, iIUT ) ) = pass

t Ts . t ( exec ( t, IUT ) ) = pass

t Ts . IUT passes t

IUT passes Ts


Testing for Conformance

IUT passes Ts

Proof of completeness on models

leads to completeness for tested systems :

i conforms-to s

exhaustive

sound

Proof obligation :

i MODS .

( t Ts . t ( obs ( t, i ) ) = pass ) i imp s

Test derivation: find such a Ts, der : SPECS P TESTS


Formal Testing


der : SPECS (TESTS)

Ts TESTS

s SPECS

IUT IMPS

imp

iIUT MODS

obs : TESTS MODS (OBS)

t: (OBS) {fail,pas

s}

OBS

pass

fail

Proof soundness and exhaustivess:iMODS .

( tder(s) . t(obs(t,i)) = pass ) i imp s

Test hypothesis :

IUTIMPS . iIUT MODS .

tTESTS . exec(t,IUT) = obs(t,iIUT)


Approaches to Formal Testing

Finite State Machine based approaches: language acceptance

transition tours

distinguishing sequences

Labelled Transition Systems concurrency theory

implementation relations

Abstract Data Type testing


Labelled Transition SystemsLabelled Transition System S, L, T, s0

ConReq

ConConf

Discon Data

Discon

states

actions,labels

transitionsT S (L{}) S

initial states0 S


Labelled Transition Systems

S0 S1dub

transition

dub coffeeS0 S3

transitioncomposition

kwart teaS0 executable

sequence

non-executablesequence

kwart soupS0

LTS(L) all transitionsystems over L

L = { dub,kwart, coffee,tea,soup}

dub

coffee

kwart

tea

S1 S2

S3

S0s

S4

S5


Labelled Transition Systems

dub

coffee

dub

tea

S1 S2

S3 S4

S0traces (s) = { L* | s }

s after = { s’ | s

s’ }

s Sequences of observable actions:

Reachable states:

traces(s) = { , dub, dub coffee, dub tea }

s after dub = { S1, S2 }

s after dub tea = { S4 }


Formal Testing with Transition Systems

t: (traces){fail,pass}


traces

der : LTS (TTS)

Ts TTS

s LTS

IUT IMPS

ioco

iIUT IOTS pass

fail

obs : TTS IOTS (traces)

Proof soundness and exhaustivess:

iIOTS .

( tder(s) . t(obs(t,i)) = pass )

i ioco s

Test hypothesis :

IUTIMPS . iIUT IOTS .

tTTS . exec(t,IUT) = obs(t,iIUT)

Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.

Documents

Transcript of Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.