Semantic Mutation Testing

AK Softwaretechnologie 1 WS 2013/14 1

Semantic Mutation TestingAndreas Voraberger

13.01.2014

John A. Clark, Haitao Dan, and Robert M. Hierons

Andreas Voraberger


Content• Introduction• Traditional Mutation Testing

• The idea• Mutation count• Possibilities• Advantages / Disadvantages

• Semantic Mutation Testing• The idea• Implementing semantic mutation• Advantages• Semantic Error Model• Common misunderstandings• Refinement• Translation• Example (Cruise-control system)• Conclusion

Andreas Voraberger


Introduction• Testing is important but expensive• 50% of development budget

• Test automation• More efficient/effective

• Mutation Testing• Distinguishing between descriptions (N)

• Semantic Mutation Testing• Distinguishing between semantics (L)

Andreas Voraberger


Introduction• Code defined by the pair: (N, L)

• N: the description (source code)• L: the semantics of the language

Andreas Voraberger


Traditional Mutation Testing

Andreas Voraberger


The idea• Mutants are classes of faults

• Produce mutants (N’)• Making changes to the program• Apply mutation operators

• {+, -}• {<=, <}• {delete parts}

• (N,L) (N‘,L)• Test Cases should distinguish N/N‘ (mutant killed)

Andreas Voraberger


Mutation count• Single mutation operator creates large number of mutations• Even in small programs

• Only use “first-order mutants“• Produced by application of one operator

• Hypothesis: all “first-order mutants“ kill most “higher-order mutants“

• Equivalent mutant: not kill able

Andreas Voraberger


Possibilities1. Judge test data against the mutants created2. Produce test data to kill all mutants

• A test set that is good in distinguishing N from N’ is good at finding faults

Andreas Voraberger


Advantages• Target particular classes of faults• Optimal for small slips or typos

• Simulate other test criteria• Mutants lead to errors provide code coverage

Andreas Voraberger


Disadvantages• High number of mutants, even with• Only “first-order mutants“• Small programs• Solution: “selective mutation“

• Equivalent mutants• Leads to manual effort• Increases costs

• Doesn’t aim on semantic mistakesAndreas Voraberger


Semantic Mutation Testing (SMT)Overview

Andreas Voraberger


The idea• Produce mutants (L’)• (N,L) (N,L‘)

• Discover misunderstandings related to semantic mistakes

• Requires description language with a semantics that allows manipulation

Andreas Voraberger


Implementing semantic mutation• Parameterisable system• Mutate through changing the parameters

• Express semantics that can be manipulated• A set of rewrite rules

• Simulate a mutation of the semantics• Changes to the syntax throughout the description

Andreas Voraberger


Advantages• Lead to far fewer mutants• Fewer equivalent mutants• Tests translation, refinement, migration, porting code• Eliminates misunderstandings

Andreas Voraberger


Semantic Mutation Testing (SMT)Scenarios

Andreas Voraberger


Semantic Error Model• Change of description in development process• Abstract Concrete

1. Requirements2. Specification3. Design4. Code5. Machine code

• Misunderstandings in every transformation

Andreas Voraberger


Semantic Error Model

Andreas Voraberger


Semantic Error Model• Refinement• Transformation to different abstraction level

• Translation• Transformation on the same abstraction level

Basis for misunderstandings Solution: SMT

Andreas Voraberger


Common misunderstandings• Find set of semantic misunderstandings • For different languages• Based on studies or experience• Implement tests, finding them

• Set of operators should reflect the environment• Different mistakes depending on programmer skills

Andreas Voraberger


Refinement• Semantic mistakes between abstraction levels• Problems on copying syntax• Maybe different semantics

• Examples• Truncation rules• Precedence rules• Datatypes• Floating point precision• Binary representation

Andreas Voraberger


Migration / Porting (Translation)• Migration / Porting to a different language

• Maybe different semantics• Use a semantic test case generation tool

• Suits of semantic mutation operators for common combinations• Point to common failures

• Example: C to Z• Division of negative numbers• Branching structures• floating-point comparison

Andreas Voraberger


Semantic Mutation Testing (SMT)Example

Andreas Voraberger


Example• Shows • cruise-control system• Not only slips have to be tested• Developers can make semantic mistakes

Andreas Voraberger


Statechart for a cruise-control system

Andreas Voraberger


Cruise-control system (Case 1)

• State: no_vehicle_in_front• Events: • brake • level=increase

• What happens?

Andreas Voraberger


Cruise-control system (Case 2)

• State: no_vehicle_in_front• Events: • Vehicle detected• level=increase

• What happens?

Andreas Voraberger


Conclusion

• Semantic Testing is necessary• SMT is a working practice• Captures a different failure type than MT• Focuses on misunderstandings• Normal MT still necessary

Andreas Voraberger


Quelle• John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation

testing. In Third International Conference on Software Testing, Verification, and Validation Workshops, pages 100-109. IEEE, 2010

Andreas Voraberger

Semantic Mutation Testing

Documents

Transcript of Semantic Mutation Testing