ICT

Testing Product Lines of Industrial Size:Advancements in Combinatorial Interaction Testing

Martin Fagereng Johansen

PhD Thesis Defense, 2013-11-05

ICT

INDUSTRIAL MOTIVATION

2

ICT

TOMRA's Reverse Vending Machines

Finale's Financial Reporting Systems

ABB's Configurable Safety Module

Eclipse IDEs – Free and Open Source

3

ICT

About the Eclipse IDE

Initiated and funded by IBM Widely used by software engineers to develop software

Major competitor to Microsoft Visual Studio

Many third-party extensions

4

ICT 5

ICT

Eclipse IDE – v3.7.0 (Indigo) – An Example of a Software Product Line

6

The problem: Can we gain confidence that any product will work?

ICT

Which products are possible?

7

356,352 possibilities!

→ model its features and their relationships in a→ feature model:

ICT

Today: A Test Suite for Each Feature

8

http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/testResults.php

http://wiki.eclipse.org/Eclipse/Testing

ICT

OVERVIEW

9

ICT 10

ICT

SAMPLING

11

ICT

Faulty Features

12

Unit tests may find faults inside a single feature. n test suites required for a product line with n features.

What about faulty cooperation between features? What if they interact incorrectly?

ICT

Interaction Faults

13

2-wise interaction fault reproducible by including 2 specific features the others do not matter

ICT

Interaction Faults

14

3-wise interaction fault reproducible by including 3 specific features the others do not matter

ICT

Kuhn et al. 2004: Almost all bugs can be attributed to the interaction of a few features.

Empirics Show:

15

ICT

Only a few products are needed to cover all simple interactions. i.e. testing a few well-selected products might reveal almost all bugs

Examples (2-wise testing):

For the "e-shop product line" with 287 features: 21 products For the Linux kernel with almost 7,000 features: 480 products

Covering Arrays

16

ICT 17

?

ICT

Configuring Feature Models

Feature models can be solved by configuration:

…or by satisfying the corresponding Boolean formula: R (A R) (B R) (C A) (D A) (C D) ¬(C D) ∧ ⇒ ∧ ⇒ ∧ ⇒ ∧ ⇒ ∧ ∨ ∧ ∧

(E B) (F B) (E F) (D E) ∧ ⇒ ∧ ⇒ ∧ ∨ ∧ ⇒ e.g. R = 1, A = 1, B = 1, C = 0, D = 1, E = 1, F = 0 The SAT problem.

18

ICT

State of the art argument

SAT is the classic NP-complete problem. worst-case analysis (Cook 1971)

Configuring basic feature models i.e., finding a single product of a product line SPLE-SAT – Software Product Line Engineering Boolean SATisfiability

Includes only feature models that occur in SPLE.

Argument SPLE-SAT = SAT, and SAT is NP-complete i.e., SPLE-SAT is NP-complete i.e., SPLE-SAT is impractical

(unless P=NP, due to Cobham's thesis) i.e. because sampling involves SPLE-SAT, sampling is impractical.

19

ICT

Our Argument

If SPLE-SAT is impractical: Configuring a feature model is impractical. i.e., testing product lines is of no concern.

If we cannot find any products, why care about their quality?

However, if we have a product line with products: Finding them were practical. We care about their quality. i.e., SPLE-SAT is practical.

Also: If a feature model is too hard to configure then it cannot serve its

purpose as an SPLE artifact. A customer cannot use it to customize a product to their needs.

i.e., SPLE-SAT is practical.

20

ICT

Empirical Investigation: SAT time

21

SPLE-SAT is very quick. Even for the largest

models. E.g. The Linux Kernel

Routinely configured by hand.

ICT

Conclusions as Venn Diagrams

State of the Art Conclusion:

Our Conclusion:

22

SAT = SPLE-SAT

Hard SAT

ICT 23

?

ICT

?

24

ICT

Sampling: Impractical in Practice

25

ICT

A New, Efficient Algorithm: ICPL

ICPL

26

2

ICT

What makes ICPL quick?

Based on a greedy polynomial time approximation algorithm (PTAS) for the set covering problem (SCP) Chvátal's algorithm (Chvátal 1979)

We know SPLE-SAT is quick. Strategically run SPLE-SAT often and infer as much as possible.

Utilize modern hardware. large amounts of memory (128 GB) truly parallel processing (64 concurrent executions)

Separate out data-parallel sub-algorithms.

++

27

ICT

Comparison

28

State of the art:

Our new algorithm (ICPL):

ICT

Comparison

Case Features Clauses in Constraint

t SotA (s) ICPL (s) Improvement in speed

Violet 101 90 3 244 32 86.9%

Eshop 287 22 2 366 5 98.6%

Ecos 1,244 2,768 2 ? 185 ?

FreeBSD 1,396 17,352 2 ? 240 ?

Linux 6,888 187,193 2 ? 33,702 ?

29

ICT 30

?

ICT 31

ICT

MARKET-FOCUSED SAMPLING

32

ICT

Industrial Context

TOMRA's Reverse Vending Machines:

33

ICT

Feature Model of TOMRA RVM

34

435,808 possibilities!

ICT

The 12 Products in Their Test-lab

35

ICT

Full Sampling was Too Costly

The problem Too many test-products

Their Need: Optimize the selection of

12 products.

Our answer: 1. Model the market

situation.

2. Select the most relevant products according to that model.

36

1 2 30

10

20

30

40

50

60

ICT

Our Model of the Market Situation:"Weighted sub-product lines"

37

ICT

Better Coverage with 12 Products

38

1 2 30%

20%

40%

60%

80%

100%

t-setweight

t

coverage

All Inter-actionsInteractions of market

ICT 39

?

?

ICT

Interactions

40

TestCSV succeeds for both.

CSV works with and without Web Tools.

Does CSV work without GEF?

Does CSV work with CDT?

Etc…

With a 3-wise covering array, we get a few products with:

With a 2-wise covering array, we get a few products with:

ICT 41

What Eclipse Tests Today: 2-Wise Covering Array:

ICT 42

Test Results – Pair-wise Testing

ICT

Possible Causes

Two (or more) features … access the same resource have overlapping GUI

elements SWTBot tests

have dependencies that interact wrongly

wait for each other (deadlock)

+++

43

ICT

Potential Faults Found using Existing Test Cases Strategic application of

existing tests revealed potential faults. Relatively inexpensive to

apply.

Raises confidence on success.

Such a large scale, fully reproducible and documented application of a product line testing technique is not found in the existing literature.

44

ICT

Also Applied to the ABB-case

45

Two bugs identified with 5 test cases.

ICT 46

?

?

ICT 47

SPLCATool

SPLCAToolSPLCATool

SPLCATool

ICT

Future Work

Further empirical study of faults in software product lines. Complete application to the Eclipse IDE

With test cases for all features; it is possible today! A good source of further empirics. A good basis of further improvements.

Even quicker algorithms for covering array generation. Less memory usage. Higher degree of parallelism.

Improved test allocation. Based on specification, model or implementation. Based on meta-data such as versions.

48

ICT

Summary

SPLE-SAT was investigated. Realistic feature models are readily configurable. Encourages the investigation into faster algorithms.

A fast algorithm for sampling. Enables the use of sampling for product line testing.

Theory and algorithms for market-focused sampling. One approach for automatic allocation of test cases.

Enables the production of a test report from (1) an implementation, (2) a test case collection and (3) feature model.

An automatic and scalable technique for software product line testing supported by free, open source tooling. SPLCATool

49

ICT 50