Efficient and Effective Mutation Testing: Supporting the ...€¦ · cost cots covering creating...

Efficient and Effective Mutation Testing:Supporting the Implementation of Quality

Software by Purposefully Inserting Defects

Gregory M. Kapfhammer†

Department of Computer ScienceAllegheny College

http://www.cs.allegheny.edu/∼gkapfham/

University of Delhi – May 2, 2012†Joint with Rene Just and Franz Schweiggert (University of Ulm) and Jonathan Miller Kauffman (Allegheny College)

http://www.cs.allegheny.edu/~gkapfham/

Introduction Mutation Analysis Empirical Evaluation Conclusion

Important Points

Presenter Introduction: Gregory M. Kapfhammer

test

testing

softwaresuites

prioritization

components

coverageem

piric

allyregressionsuiteapplications

evaluating

algo

rithm

analysis

data

effectiveness

empirical

genetic

perf

orm

ance

understanding

using

approachcommercialofftheshelf

comparison

comprehensive

constrainedcreation

databasecentric envi

ronm

ents

execution

findi

ng

framework

identifying

interactive

javamethods

multiplots

mutation

party

prioritized

reduction

relational

study

third

timeaware

towards

105

adequacy

appr

oach

es

array

automatically

building

callchallenges

chapter

communication

compare

compressing

com

pute

r

conditional

cost

cots

covering

creating

criteria

database

databaseaware

data

base

driv

en

databases

declarativedependable

detection

devices

distributed

distributing

duringsearchbased

dynamic

efficiency

efficient

engi

neer

ing

environment

evaluate

examination

executing

experimental

family

flow

forward

frameworks

free

generation

greedy

gui

hamiltonian

handbook

heaps

implementation

impr

ove

incorporating

incr

ease

information

initial

intranode

invariant

javaspace

javaspacebased

kernel

knapsack

linux

measurement

memory

method

monitoring

operators

paths

poster

potential

preliminary

primitives

prio

ritiz

atio

ns

prio

ritiz

ers

prioritizing

problematic

receive

remote

reports

resource

resourceconstrained

results

role

runtime

science

searchbased

selectionsolutions

solvers

space

studies

stud

yingsupported

synthetic

techniques

transmission

transparently

trees

tuple

unstructured

wrappers

Kapfhammer Allegheny College

Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects


Important Points

Inspiration and Motivation

The magic of myth and legend has come true inour time. One types the correct incantation ona keyboard, and a display screen comes to life,showing things that never were nor could be.

Frederick P. Brooks, Jr.




Important Points


The magic of myth and legend has come true inour time. One types the correct incantation ona keyboard, and a display screen comes to life,showing things that never were nor could be.


In reference to software!




Important Points


I believe the hard part of building software to bethe specification, design, and testing of this con-ceptual construct, not the labor of representingit and testing the fidelity of the representation.





Important Points




What happens if the “incantation” is incorrect?




Important Points




How do we efficiently and effectively test software?




Software Testing

What is a Test Case?

MethodUnder Test




Software Testing


MethodUnder Test

Input




Software Testing


MethodUnder Test

Input Output




Software Testing


MethodUnder Test

TestSet Up




Software Testing


MethodUnder Test

TestSet Up

Input




Software Testing


MethodUnder Test

TestSet Up

Input Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output

TestVerdict

The test case passes and the code is correct!




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output




Software Testing


MethodUnder Test

TestSet Up

Input Output

TestClean Up

TestOracle

ExpectedOutput

TestVerdict

ExpectedOutput

Output

TestVerdict

The test case fails and a defect is found!




Software Testing

What is a Test Suite?

T1 T2




Software Testing


T1 T2 T3 T4




Software Testing


T1 T2 T3 T4 T5 T6




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2

Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


B2B2B2B2B2B2B2B2B2B2




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2





Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2


How Good is Test Suite T ?




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2



Coverage Analysis




Software Testing


T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

Test Suite T = 〈T1,T2, . . . ,T9,T10〉

R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2



Coverage Analysis Mutation Analysis




Fundamental Concepts

Conceptual Faults

if(a > 10)





Conceptual Faults

if(a > 10) if(a >= 10)





Conceptual Faults

if(a > 10) if(a >= 10)

Implemented





Conceptual Faults

if(a > 10) if(a >= 10)

Implemented Potential Fault





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 0





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 0

false false





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 15





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 15

true true





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10

false true





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10

false true

Can the tests differentiate between implemented and potential fault?





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10

false true

If yes, then the tests are adequate!





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10

false true

If no, then the tests must be improved!





Conceptual Faults

if(a > 10) if(a >= 10)

a

true false

a

true false

a = 10

false true

Purposefully insert faults in order to implement quality software!





Overview of Mutation Analysis

MutationOperator






MutationOperator

MutationOperator






MutationOperator

MutationOperator

MutationOperator

MutationOperator






MutationOperator

MutationOperator

MutationOperator

MutationOperator

Methodicallyinject smallsyntacticalfaults into

the programunder test






MutationOperator

MutationOperator

MutationOperator

MutationOperator






Test Case T1 Test Case T2 Test Case T3 Test Case T4

Execute thetest suite after

enabling asingle mutantin the program

under test









under test

The test suitecannot kill themutant – either

a test suiteweakness oran equivalent

mutant!









under test

Repeat thisprocess forall of the

test casesand mutants– calculate

mutation scorewhen finished





Contributions of this Presentation

EfficientMutationAnalysis







Challenges







Challenges

Solutions







Challenges

Solutions

Conditional Mutation







Challenges

Solutions


Syntax TreeTransformation







Challenges

Solutions



Expressionsand Statements







Challenges

Solutions




CompilerIntegrated







Challenges

Solutions




CompilerIntegrated

ComprehensiveEmpirical Study







Challenges

Solutions




CompilerIntegrated

ComprehensiveEmpirical Study

Efficient Technique - Fully Integrated into the Java 6 SE Compiler





Understanding Mutation Analysispublic int eval(int x){

int a=3, b=1, y;

y = a * x;

y += b;return y;

}

public int max(int a, int b){int max = a;

if(b>a){

max=b;}

return max;}

=⇒

=⇒






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

Methodicallyinject smallsyntacticalfaults into

the programunder test






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

• y = a - x;

• y = a + x;

• y = a / x;

• if(b < a)

• if(b != a)

• if(b == a)






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

Unbiasedand powerfulmethod forassessing

oracles andinput values






int a=3, b=1, y;

y = a * x;

y += b;return y;

}


if(b>a){

max=b;}

return max;}

=⇒

=⇒

Unbiasedand powerfulmethod forassessing

oracles andinput values

Useful methodfor fault seeding

during theempirical study

of testingtechniques





Mutation Analysis Challenges

MutantGeneration






MutantGeneration

MutationOperators






MutantGeneration

MutationOperators

Program






MutantGeneration

MutationOperators

Program

Mutants






MutantGeneration

MutationOperators

Program

Mutants

Often Yields aSubstantial Num-

ber of Mutants






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants

High Time Over-head for Generation






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysis






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysisTests






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants


MutationAnalysisTests Results






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants



Individually Executing theMutants is Too Expensive






MutantGeneration

MutationOperators

Program

Mutants


ber of Mutants



Individually Executing theMutants is Too Expensive

PriorSolutions?





Prior Work in Mutation Analysis

Improving Mutation Analysis






Improving Mutation Analysis Offutt andUntch







Do Fewer







Do Fewer

Sampling Selection







Do Fewer Do Smarter







Do Fewer Do Smarter

Distributed Weak Mutation







Do Fewer Do Smarter

Do Faster







Do Fewer Do Smarter

Do Faster

CompilerIntegrated

BytecodeTransformation

MutantSchemata







Do Fewer Do Smarter

Do Faster

Higher OrderMutation

Jia andHarman




Mutation Analysis with MAJOR









Encapsulates allmutants withinthe same block








Can be inte-grated withinthe compiler








Transforms theabstract syntax

tree (AST)









Transforms theabstract syntax

tree (AST)

Stmt → Conditional Stmt(if-then-else, switch)

Expr → Conditional Expr(conditional operator ?:)






Transforming the AST

public int eval(int x){int a=3, b=1, y;

y = a * x ;

y += b;return y;

}

⇓

=⇒

=⇒







y = a * x ;

y += b;return y;

}

⇓

ASSIGN

IDENT

y

BINARY

∗

a x

=⇒

=⇒







y = a * x ;

y += b;return y;

}

⇓

ASSIGN

IDENT

y

BINARY

∗

a x

=⇒

=⇒

ASSIGN

IDENT

y

COND-EXPR

THEN

BINARY

+

a x

COND

(M NO ==2)

ELSE

COND-EXPR

THEN

BINARY

-

a x

COND

(M NO ==1)

ELSE

BINARY

∗

a x





Source Code View of Inserting Mutantspublic int eval(int x){

int a=3, b=1, y;

y = a * x ;

y += b;return y;

}

1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}

2 Determine whether current expression or statement isaffected by mutation

3 Apply mutation operators






int a=3, b=1, y;

y = (M_NO==1)? a - x :a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = (M_NO==2)? a + x :(M_NO==1)? a - x :

a * x ;

y += b;return y;

}









int a=3, b=1, y;

y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :

a * x ;

y += b;return y;

}









int a=3, b=1, y;


a * x ;

y += b;return y;

}

Mutants that are not ex-ecuted cannot be killed








Collecting and Using Mutation Coveragepublic int eval(int x){

int a=3, b=1, y;


a * x ;

y += b;return y;

}







int a=3, b=1, y;


(M_NO==0 &&

COVERED(1,3))?

a * x : a * x ;

y += b;

return y;}


Determine coveredmutants with addi-

tional instrumentation






int a=3, b=1, y;


(M_NO==0 &&

COVERED(1,3))?

a * x : a * x ;

y += b;

return y;}


Determine coveredmutants with addi-

tional instrumentation

Only execute and investi-gate the covered mutants





MAJOR’s Compiler

MAJOR’sCompiler





MAJOR’s Compiler

MAJOR’sCompiler

Enhanced the StandardJava Compiler





MAJOR’s Compiler

MAJOR’sCompiler


Source Files





MAJOR’s Compiler

MAJOR’sCompiler


Source Files

CommonCompiler Options





MAJOR’s Compiler

MAJOR’sCompiler


Source Files


Domain SpecificLanguage





MAJOR’s Compiler

MAJOR’sCompiler


Source Files


Domain SpecificLanguage

Bytecode withEmbedded

Mutants





Integration into the Java Compiler





MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};

// Define replacement list

BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};

// Define own operator

myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}

// Enable built-in operator AOR

AOR<"org">;

// Enable operator myOp

myOp<"java.lang.System@println">;







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;



Specify mutationoperators in detail







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;




Define own mutationoperator groups







BIN(+)<"org"> -> {-,*};

BIN(*)<"org"> -> {/,%};


myOp{

BIN(&&) -> listCOR;

BIN(||) -> listCOR;

COR;

LVR;

}


AOR<"org">;




Define own mutationoperator groups

Enable operators fora specific package,

class, or method





Optimized Mutation Analysis Process

1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite




Compilation Efficiency

Mutant Generation and Compilation

1

2

3

4

5

6

7

8

9

10

11

12

0 20000 40000 60000 80000 100000 120000 140000

Co

mp

iler

run

tim

e in

se

co

nd

s

Number of mutants

apache antjfreechart

itextjava pathfindercommons mathcommons lang

numerics4j

Overhead for generating and compiling mutants is negligible




Compilation Efficiency

Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption

original instrumented original instrumentedwcs wcs+cov

aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303

itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149

numerics4j 5,869 1.2 1.3 1.6 73 90

• Runtime overhead is application dependent• Larger for CPU-bound applications

• Small for I/O-bound applications

• Even for large projects, applicable on commodity workstations




Mutation Analysis Efficiency

Evaluating and Improving Mutation Analysis

0

20

40

60

80

100

120

140

160

180

0 20 40 60 80 100

Nu

mb

er

of

kill

ed

mu

tan

ts

Runtime in seconds

optimized order (using coverage information)

random order (using coverage information)

original order (using coverage information)

optimized order (without coverage information)

random order (without coverage information)

original order (without coverage information)

• Mutation analysis is not feasible without coverage information• Reordering the test suite significantly speeds up the process,

especially if runtimes of tests differ by orders of magnitudeKapfhammer Allegheny College



Retrospective

Improving Test Suite Quality

MutationAnalysis




Retrospective


MutationAnalysis

Program Test Suite




Retrospective


MutationAnalysis

Program Test Suite

Mutation Score




Retrospective


MutationAnalysis

Program Test Suite

Mutation Score

Improve Tests




Retrospective


MutationAnalysis

Program Test Suite

Mutation Score

Improve Tests

Automated Manual




Retrospective


MutationAnalysis

Program Test Suite

Mutation Score

Improve Tests Use Tests




Retrospective


MutationAnalysis

Program Test Suite

Mutation Score

Improve Tests Use Tests

Test improvement is only effective if mutation analysis is efficient!




Retrospective

Reviewing MAJOR’s Contributions

MutationAnalysis




Retrospective


MutationAnalysis

Efficiency: MAJOR has ac-ceptable time and space over-heads and scales to large,real-world programs




Retrospective


MutationAnalysis


Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool




Retrospective


MutationAnalysis


Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool

We will release MAJOR as free and open source software




Conclusions and Future Work

Conclusion

Key Concepts and Features:

• Compiler-integrated solution• Conditional mutation with the abstract syntax tree• Furnishes its own domain specific language• Collects and leverages mutation coverage information

Characteristics of MAJOR:

• Fast and scalable technique• Configurable and extensible mutation tool• Enables an optimized workflow for mutation analysis




Conclusions and Future Work

Recently Published Papers

• Rene Just, Gregory M. Kapfhammer, and FranzSchweiggert. Using conditional mutation to increase theefficiency of mutation analysis. In Proceedings of the 6thInternational Workshop on the Automation of Software Test,Honolulu, Hawaii, May 2011.

• Rene Just, Franz Schweiggert, and Gregory M.Kapfhammer. MAJOR: An efficient and extensible tool formutation analysis in a Java compiler. In Proceedings of the26th IEEE/ACM International Conference on AutomatedSoftware Engineering (Tool Paper), Lawrence, Kansas,November 2011.



Efficient and Effective Mutation Testing:Supporting the Implementation of Quality

Software by Purposefully Inserting Defects

Gregory M. Kapfhammer

Department of Computer ScienceAllegheny College

http://www.cs.allegheny.edu/∼gkapfham/

Thank you for your attention!I welcome your questions and comments.

http://www.cs.allegheny.edu/~gkapfham/

Efficient and Effective Mutation Testing: Supporting the ...€¦ · cost cots covering creating...

Documents

Transcript of Efficient and Effective Mutation Testing: Supporting the ...€¦ · cost cots covering creating...