BAiT: Adaptive Test Case Execution

in Practice

Jens R. CalaméCentrum voor Wiskunde en Informatica,

Amsterdam

13-09-2007 DWFTT 2007, Enschede 2

Agenda

1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion

13-09-2007 DWFTT 2007, Enschede 3

Introduction

• BAiT: Behavior Adaptation in Testing• Toolset to generate test cases and test oracles as well as

execute tests on nondeterministic systems with data

13-09-2007 DWFTT 2007, Enschede 4

Agenda

1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion

13-09-2007 DWFTT 2007, Enschede 5

BAiT: Test Case & Oracle Generation

• Test generation with TGV (enumerative approach)

• Chaotic data abstraction for input and output parameters of events

• Test oracle generation for Eclipse Prolog constraint solver in order to select test data

• Automatic generation of variable names for the test case

13-09-2007 DWFTT 2007, Enschede 6

BAiT: Test Proxy Generation

• Test Proxy for Java-based SUTs

• Instantiation of test object within test proxy

• Procedure-basedbidirectional communication (publisher-subscriber pattern)

• Mapping of datatypes from specification language to target platform

13-09-2007 DWFTT 2007, Enschede 7

BAiT: Test Execution

• Test execution of pre-calculated test trace (behavior and data)

• Adaptation to the IUT's reactions, rather than immediate assignment of verdict

• BAiT framework provides adapters for user-defined– trace selection and– test data selection

algorithms

13-09-2007 DWFTT 2007, Enschede 8

Agenda

1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion

13-09-2007 DWFTT 2007, Enschede 9

An Exemplary Case: The ATM

ATM specification (partially) ATM test purpose (completely)

13-09-2007 DWFTT 2007, Enschede 10

An ATM Test Case

Photos: Wikipedia and ECB

!getPin(1234)

?pinCorrect

!getAmount(100)

!initBalance(1000)

?retFifty(2)

?retFifty(1)

?retTwenty(2)

?retTen(1)

?

!initPin(1234)

13-09-2007 DWFTT 2007, Enschede 11

An ATM Test Case (2)

May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl merge

FINE: Pruning planned test trace.May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.RunTest findTrace

FINER: Examining: [init],initPin(Pin),getPin(PinUser),pinCorrect, initBalance(Bal),getAmount(Amt),[tau],retFifty(Fif),emitBankNotesMay 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl solve

FINEST: init(G1),initPin(G1,G2,lparam(nat(Pin))),[...][Pin in {[-Infinity .. Infinity]} => 1234]May 25, 2007 3:06:21 PM nl.cwi.sen2.bait.steps.TraceImpl solve

FINEST: init(G1),initPin(G1,G2,lparam(nat(1234))),[...][...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest

INFO: Trying: [init],initPin(1234),getPin(1234),pinCorrect, initBalance(1000),getAmount(100),[tau],retFifty(2),emitBankNotesMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest

FINE: initPin(1234) -> OK[...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest

FINE: retFifty(2) -> NOT OKMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest

INFO: Expected: retFifty(2), received: retFifty(1) -> Trace failed, trying alternative.

1. Precalculation of possible test trace

2. Iterative data selection

3. Stepwise execution of the trace…

…until the IUT deviates from the precalculated trace.

13-09-2007 DWFTT 2007, Enschede 12

An ATM Test Case (3)May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest

INFO: Trace deviated; executed so far: [init],initPin(1234),getPin(1234),pinCorrect,initBalance(1000), getAmount(100),[tau],retFifty(1),retTwenty(2),retTen(1), emitBankNotesMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge

FINE: Pruning planned test trace.May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge

FINE: Adding retFifty(1) to trace stub.[...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge

FINE: Adding emitBankNotes to trace stub.May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest

INFO: Test finished; executed: [init],initPin(5),getPin(5), pinCorrect,initBalance(1000),getAmount(100),[tau],retFifty(1), retTwenty(2),retTen(1),emitBankNotesMay 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest

INFO: The test case ended with verdict PASS.

5. Appending received system output to trace.

6. Solve test trace with pending system output;

assign verdict or calculate new trace.

4. Pruning not yet executed part of test trace.

13-09-2007 DWFTT 2007, Enschede 13

Agenda

1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion

13-09-2007 DWFTT 2007, Enschede 14

Structure of BAiT: The Tools

• BAiT: Behavior Adaptation in Testing• Toolset to generate test cases and test oracles as well as

execute tests on nondeterministic systems with data

13-09-2007 DWFTT 2007, Enschede 15

Structure of BAiT: Execution Part

BAiTExecution Core

Behavior MappingConstraint Solver

Behavior MappingImpl. Under Test

Data MappingConstraint Solver

Data MappingImpl. Under Test

Impl.UnderTest

ConstraintSolver

Trace Selection

Data Selection

•Trace class: trace execution management•Step class: communication with IUT•Trace selection from complete test graph (IOLTS with all possible test cases for a given test purpose)

Test Oracle

Test Case(s)

•Variable class: Codec of data to constraint solver, data instantiator and IUT•Data instantiation algorithm(s)

13-09-2007 DWFTT 2007, Enschede 16

Agenda

1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion

13-09-2007 DWFTT 2007, Enschede 17

Conclusion

• Behavior Adaptation in Testing helps finding more reliable verdicts in testing non-deterministic systems

• A prototypical framework for BAiT has been developed procedure-based setting with atomic data

• Further work:– Extension of framework

for other settings– Extension of framework

to structured datatypes– Case Study:

Mozilla FirefoxCSS subsystem

13-09-2007 DWFTT 2007, Enschede 18

Further Information

Background information• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Sidorova, Natalia; Data

Abstraction and Constraint Solving for Conformance Testing (APSEC 2005).• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Towards Automatic Generation

of Parameterized Test Cases from Abstractions. Technical Report, 2006.

The toolset/framework itself• Calamé, Jens R.; Adaptive Test Case Execution in Practice. Technical Report, 2007.• Calamé, Jens R.; The Internals of Adaptive Test Case Execution. Technical Report,

to appear.

All downloadable from www.calame.de/works,published.bib,date,.html

Contact:Jens R. Calamé

CWI, Amsterdamjens.calame@cwi.nl