An Analysis and Survey of the Development of Mutation Testing[JH09]

About 30 minutes Tao Heelfinhe@gmail.com

Software Engineering LaboratoryDepartment of Computer Science, Sun Yat-Sen University

With thanks to Yue Jia and Mark HarmanThese slides are mainly extracted from Jia and Harman’s survey work

May 2011Sun Yat-Sen University, Guangzhou, China

1/371/37

[JH09] Yue Jia, Mark Harman (September 2009). "An Analysis and Survey of the Development of Mutation Testing" (PDF). CREST Centre, King's College London, Technical Report TR-09-06.

Outline

Objectives Scope Classification of Research Fundamental Hypotheses Related Concepts Future Trend Tools

Objectives of Mutation Testing

Provide “mutation adequacy score” Measure the effectiveness of a test set in

terms of its ability to detect faults

3/20

Scope of Mutation Testing

The unit level The integration level The specification level  As a white-box unit test technique

Theoretical work on Mutation Testing

Hypotheses supporting Mutation Testing Optimization techniques

techniques for reducing computational cost techniques for the detection of equivalent mutants

Practical work on Mutation Testing

Applications of Mutation Testing Development work on Mutation Testing tools Empirical work

Fundamental Hypotheses

Competent Programmer Hypothesis (CPH)

Programs can be corrected by a few small syntactical changes.

Coupling Effect

Test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults  A simple fault is represented by a simple mutant

which is created by making a single syntactical change

A complex fault is represented as a complex mutant which is created by making more than one change

A Mutation Operator

A transformation rule that generates a mutant from the original program

Generate ‘realistic faults’

Behaviors of Mutation Operators

Mutation Objects Variables Expressions

Mutation Operation Replacement Insertion Deletion

Problems of Mutation Analysis

Executing the enormous number of mutants Human effort

The human oracle problem The equivalent mutant problem

Cost Reduction Techniques

Classification of Cost Reduction Techniques

Reduction of the generated mutants (may be taken into consideration in our approach for Fault Localization)

Reduction of the execution cost

Mutants Reduction Techniques

Mutant Sampling Random Based on the Bayesian sequential probability ratio

test (SPRT)

Mutant Clustering Based on killable test cases

Selective Mutation By reducing the number of mutation operators applied. Ignore operators ASR and SVR - redundant generation  Only using  ABS and ROR

Evaluation of Mutants Reduction

A mean mutation score Reduction in the number of mutants Number of equivalent mutants 

Execution Cost Reduction Techniques

Strong Mutation Weak Mutation Firm Mutation

by providing a continuum of intermediate possibilities - compare state, which lies between execution (Weak Mutation) and the final output (Strong Mutation).

Equivalent Mutant Detection Techniques

10% to 40% of mutants which are equivalent

Empirical Study

Compare mutation criteria with data flow criteria such as 'all-use'

Compare mutants with real faults

Future Trend

A need for high quality higher order mutants A need to reduce the equivalent mutants A preference for semantics over syntax

mutation An interest in achieving a better balance

between cost and value A pressing need to generate test cases to kill

mutants

Life circle of Mutation Testing

Generate mutants based on specified mutation operations

Reduce mutants Run mutants against a test suite

Mutation Testing Tools

Goals of Mutation to Enhance Fault Localization

Not aim to simulate real faults Analyze the impact of mutation on the

suspiciousness of the mutated statement

Published C Mutation Testing Tools

Name Application Year Character Available Suitable

MILU C 2008Higher Order Mutation, Search-based technique, Test harness embedding

Yes No

MUFORMAT C 2008 Format String Bugs No No

ESPT C/C++ 2008 Tabular No No

CSAW C 2007Variable type optimization

Yes No

ExMAn C, Java 2006 TXL No No

SESAME C, Lustre, Pascal 2006 Assembler Injection No No

Certitude C/C++ 2006 General (Commercial) Commercially No

Plextest C/C++ 2005 General (Commercial) Commercially No

Proteum/IM 2.0 C 2001Interface Mutation,

Finite State MachinesYes Yes

Insure++ C/C++ 1998Source Code

Instrumentation (Commercial)

Commercially No

TUMS C 1995Mutant Schemata

GenerationNo No

Proteum 1.4 C 1993Interface Mutation,

Finite State MachinesNo No

Proteum

Environment Variable: PROTEUMIMHOME li -P pre-filename [-D directory] source-filename LI-filename

Call gcc to preprocess the source code file source-filename.c and produce a file pre-filename.c. Then parse the file pre-filename.c and generate various info files

opmuta [-<operator> n m] [-all n m] source-filename Li-filename Apply mutant operators to the source code and LI files. As ouput,

opmuta produces a description file in a format that muta is able to read and include in the mutant database.

Proteum – An Example: Space

space.c

li ~/smart_debugger/toolkit/proteum/li -P pre-space space li-space

li-space.nli li-space.fun li-space.cgr li-spaec.gfc

source codestrutt.h include

statement info

pre-space.c

function info call graph info def-use pair info

opmuta ~/smart_debugger/toolkit/proteum/opmuta -O Operators pre-one_statement li-one_statement > mutants.txt

mutants.txt

all mutants info

GenetrateMutants.py

Proteum.py

include

Mutants function name by directory

line number by mutant.dsc

mutation operator by mutant.dsc

Thoughts of Self-Made Mutants Generator

I once consider to implement a self-made mutants generator. Only mutate one statement Not aim to simulate real faults

IPO Input

A piece of source code Line number to mutate Mutation operators

Process Preprocess -> Scan -> Parse -> Mutate

Output Mutants

Collection of Compiler Front-end Tools

GCC-XML …failed E.g.

source code: http://www.gccxml.org/HTML/example1in.html

Parse-tree: http://www.gccxml.org/HTML/example1out.html

LLVM … not try, good for Objective-C? http://llvm.org/

Collection of Compiler Front-end Tools

GCC … failed, maybe no parse-tree for GCC -fdump-tree-fixup cfg-lineno

http://stackoverflow.com/questions/697817/how-to-make-gcc-spit-out-a-mapping-from-flow-graphs-to-source-code-line-numbers

-fdump-tree-all  -fdump-rtl-all http://stackoverflow.com/questions/1496497/how-can-i-see-parse-tree-intermediate-code-optimization-code-and-assembly-code

-fprofile-arcs -ftest-coverage gcno gcda: not readable for human beings gcov: offer the line number, separated statements

Useful links

http://en.wikipedia.org/wiki/Mutation_testing http://cs.gmu.edu/~offutt/rsrch/mut.html http://www.dcs.kcl.ac.uk/pg/jiayue/repository/

Q & A

31/37

Thank you!Contact me via elfinhe@gmail.com

32/37

Abstract

33/20

Fault localization is a technique that aims to pinpoint faults by analyzing program execution spectrum, while mutation is a testing technique used by generating faulty programs called mutants. Most of existing research in fault localization focuses on the spectrum of the given programs. Yet there are few attempts to introduce mutation’s impact to enhance fault localization techniques. In this paper, we propose a strategy that automatically introduces mutation into fault localization techniques, and present variations of a heuristic method to compute suspiciousness of each statement by considering the impact of mutation. To validate our method, experiments is conducted on benchmark programs, namely Siemens, grep, gzip, sed, space, flex, make, vim, and bash. Results indicate that the method can help programmers locate faults more effectively than methods without mutation.