CITLAB: a Laboratory for CombinatorialInteraction Testing

Angelo GargantiniUniversità di Bergamo - Italyhttp://cs.unibg.it/gargantiniJoint work with Paolo VavassoriIn collaboration with Andrea Calvagna Università di Catania

Workshop on Combinatorial Testing (CT) @ ICST 2012 April 17, 2012

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

lSome open issues in CTuser perspective

Still missing• A common abstract language • with a precise semantics for concepts like constraints, seeds …

• A standard concrete syntax/grammar • or exchange format for tools: to guarantee interoperability

• Limited assistance in writing the models• generation tools do not offer any editing capabilities (only a

grammar and a parser) and are rarely integrated in any IDE for programming or design.

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

lSome open issues in CT researcher perspective

• No standard language Comparison among tools and algorithms is difficult• Conceptual point of view: some features may be supported in an

approach but not in another or different meanings• Practical: every comparison is quite unreliable since every user must

redefine in its own language and tool the examples • No help in defining new algorithms and tools• In order to experiment a new test generation algorithm, a researcher

should define a proper grammar and a parser, develop the libraries to manipulate the model data, and translate the benchmarks

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CITLAB in brief

(Domain Specific) Language 1. A rich abstract language with a precise formal semantics2. A concrete syntax with a well defined grammar by XTEXT3. A textual editor integrated in the eclipse IDEA Framework4. based on the Eclipse Modeling Framework (EMF) which

provides tools and run-time support to manipulate combinatorial problems in Java

5. A rich collection of Java utility classes and methods6. For introducing new test generation algorithms7. For introducing code translators for importing and

exporting models and tests

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

A DSL for CIT problems

• Domain-specific languages (DSLs) are languages tailored to a particular problem domain.• They offer substantial gains in expressiveness and ease of use

compared with general-purpose programming languages • CIT can be considered a specific application domain for which

developing a domain-specific language is worthwhile.• DSL development is supported by several tools and technologies.

• XtextXtext is a framework for development of domain specific languages. It covers all aspects of a complete language infrastructure, from parsers, over linker, compiler or interpreter to fully-blown top-notch Eclipse IDE integration.

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

.ecore(metamodel)

Language project

.xtxtGrammar

Generator

Constraints

Antlr Parser EditorSyntax coloring

Manual (required)

Manual(optional)

Editor Project

Generated

Java API

Language Model

Outline

Content Assist

Template

Formatting…

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

A DOMAIN SPECIFIC LANGUAGE FOR CIT PROBLEMS

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CITLAB Language in a glance

Model ModelTypes:...endParameters:...endConstraints:# ... #endSeeds:# ... #endTestGoals:# ... #end

Parameters

Types

Constraints

Seeds

Test goals

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Parameters and their types

• To describe a combinatorial problem would be sufficient to specify the number of variables and their cardinality.

• CITLAB language forces the designer to name parameters and to specify their types by listing all the values in their domain.

Parameters: Boolean emailViewer; Range textLines [ 25 .. 30 ]; Enumerative display { 16MC 8MC BW };end

1. Enumerative for parameters that can take a value in a set of symbolic constants.

2. Boolean for parameters that can be either true or false.

3. Numerical value for parameters that take any value in an integer range.

Choice: explicit parameter names to facilitate the modeling of real systems and to ease the specification of constraints and seeds

From ACTS

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Named Types

• Types can be defined with their name in the Types section to be used in parameters declaration• Instead of an “anonymous” type.

Types: EnumerativeType cameraType { 2MP 1MP NOC };endParameters: Enumerative rearCamera : cameraType; Enumerative frontCamera : cameraType; ...end

Advantages: the use of named parameter types to make more compact and more maintainable the models in case many parameters share the same domain.

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Constraints

• In CITLAB, we adopt the language of propositional logic with equality and arithmetic to express constraints

• General Form (GF) constraints• propositional calculus and Boolean operators

a or b => c and d• equality and inequality

emailViewer = true or frontCamera ~= NOC• arithmetic over the integers • relational and arithmetic operators for numeric terms

textLines >= threshold + 10

• A test t satisfies a constraint C or it is a model of C iff C is true by substituting each parameter pi in C with the value of pi in t

Constraints Semantics

• A test t is valid if it is a model of all the constraints , i.e., or equivalently

• Challenge: finding “valid” “complete” test suites• Because of constraints, some combinations are not coverable• A test suite must cover “only” all the coverable combinations• Finding a (if any) model is an NP problem = SAT problem

• Checking if a test satisfies a constraint is however linear• Checking if a combination violates the constraints is linear

• Challenge: assuring constraints consistency• constraints are consistent if there exits at least a test that satisfies

them, i.e., • Consistency checking is again a SAT problem• FUTURE WORK: use a SAT/SMT solver for this

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

GF vs forbidden combinations

• Most CIT that support constraints, allow only the use of forbidden combinations or tuples

• A forbidden combination would be translated in our model as a not statement. • a forbidden pair x = a; y = b would be represented# not (x = a and y = b) #

Advantages:• Much more compact than forbidden tuples

Example: if b {x1 … xn }“a = x => b = xk” would need n-1 forbidden tuples

• More expressive (whilst in a finite domain …)• More similar to the natural language• No need to distinguish between implicit and explicit constraints

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Seeds

• The testers can also force the inclusion of their favorite test cases by specifying them as seed tests. • They can be tests generated by other criteria• Critical complete combinations

• The seeds must be included in the generated test set without modification

• CITLAB considers only complete seeds, i.e., seeds that assign a valid value to each parameter

# frontCamera==1MP, display==16MC, emailViewer=true, rearCamera==2MP, textLines=30 #

• Partial seeds? Critical partial combination the tester would like to be included in the test suite

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Test Goals

• Critical situation that must be tested • Predicates that must be covered by some tests• Test goals can be again in GF (as the constraints):// at least one camera is missing and the display has at least threshold lines( rearCamera==NOC or frontCamera==NOC) and textLines>=threshold• Partial combinations // both rearCamera and frontCamera are NOC and display is 16MCrearCamera==NOC and frontCamera==NOC and display==16MC• Remarks:• Checking feasibility of test goals is an NP-complete problem, since it can be

reduced to a SATisfiability problem• Checking if a test covers a test goal is however linear

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CITLAB EDITOR

DEMO

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Editor features

• Syntax Coloring• Content Assist• Template Proposals• Rich Hover• Rename Refactoring• Quick Fixes• Outline

• Folding• Hyperlinks for all

Cross References• Find References• Toggle Comment• Mark Occurrences• Formatting

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Example, model validation

• XTEXT provides several levels of validation for the defined language• the user can specify additional constraints for the model by

providing validation fragments

Example: No seed can violate any constraint

Note: • checking if a seed violates a constraints is decidable in a

linear time

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

AN EXTENSIBLE FRAMEWORK FOR CIT

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CITLAB: as a extensible framework

CitLab framework

Random generator

• For the definition and implementation of test generators and a set of exporters/importers to and from other languages

• Based on the eclipse plugin development environment

• Goals• to foster tools interoperability• to favor reuse of artifacts• to allow fair comparison

Casaexporter

generator exporter

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Extension points and extensionsCitLab accepts new

contributions as pluginsby defining extension points

Contributors add new plugins by implementing extensions• Declare the extension in .xml• Implement an Interface

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CITLAB: extension pointsExtension Point Interface/Methods

TestGenerator ICitLabTestGenerator

TestSuite generateTests(Model m, int nWise)boolean acceptConstraints(List<Constraint> c)boolean acceptSeeds(List<Seed> s)boolean acceptTestGoals(List<TestGoal> tg)

Exporter ICitLabExporter

void export(Model m)boolean acceptConstraints(List<Constraint> c)boolean acceptSeeds(List<Seed> s)boolean acceptTestGoals(List<TestGoal> tg)

Importer ICitLabImporter

Model import(Reader r)

TestSuiteExporter ICitLabTestSuiteExporter

void save(TestSuite ts, String file)

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Using Model to Text or Model to Model

• MDE approach: a possible way to define importers and exporters is to use • Model to Text (M2T) or • Model to Model (M2M) transformations

• For instance, Xtend2 for CASA exporter

def toCasaCode(Model sm,int n){ ''' «n» «sm.parameters.size» «FOR param :sm.parameters»«param.getSize» «ENDFOR» '''}

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Test Suite meta-model and Java utils

• a simple meta-model for tests and test suites

• We have developed several classes and methods capable to perform routine tasks like • generating all the test requirements for a given strength, • checking if a test or a seed violates some constraints• checking if a test goal is covered by a given test suite• …

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

CitLab site and Licence issues

• All the code is available at http://code.google.com/a/eclipselabs.org/p/citlab/

• CitLab is released under the Eclipse Public License (EPL) • an open source software license • used by the Eclipse Foundation • a business-friendly free software license

• copyleft • but weaker than other, like GNU General Public License (GPL)

• can use, modify, copy and distribute the work and modified versions, in some cases being obligated to release their own changes• CitLab Plugins can be commercial (distribute as binaries)• Any modification of the CitLab framework must be open sourced

• The EPL is approved by the Open Source Initiative (OSI) and is listed as a "free software license" by the Free Software Foundation (FSF).

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Status and Future work

• Adding new plugins is easy (as student projects)• Paolo Vavassori has a part-time contract for 1 year

• Already implemented• Generator

• Random with seeds support• IPO and AETG work in progress

• Model exporter• To CASA (constraints still in progress)

• Test suite exporter• To excel /CSV file

• New planned features• Language, e.g. output for oracles, arrays, …• Meta-algorithms (like: run 50 times and take the best)• Test suite postprocessor, e.g. remove unnecessary test cases

Ange

lo G

arga

ntini

- Ci

tLab

: a L

abor

ator

y fo

r Com

bina

toria

l

Objectives and conclusions

CitLab Language & editor• Establish a CIT “language” with a precise semantics• New concepts and constructs?

• Exchange models and benchmarks• CIT “knowledge base”

• A user friendly textual editor integrated with the eclipse IDECitLab plugin-based environment • defining new generators • to foster contributions from new parties

• making comparison easier and more reliable