Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order...

48
Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing

Transcript of Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order...

Page 1: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Yue Jia, Mark HarmanKing’s College London

CREST

Constructing Subtle Faults Using Higher Order Mutation Testing

Higher Order Mutation Testing

Page 2: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Agenda

BackgroundProblemsSolutionEmpirical StudySummary

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution Empirical Study Summary

Page 3: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 4: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Which test case is better?

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 5: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 6: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 7: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Generate Mutants

------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

First Order MutantOriginal Program

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 8: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Generate Mutants

------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

Higher Order MutantFirst Order MutantOriginal Program

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 9: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Mutation Testing

Generate Mutants

------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

Higher Order MutantFirst Order MutantOriginal Program

------------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

-----------------------------------------------------------------------------------------------------

+------------------------------------------------------------------------------------------------------

>

First Order Mutant Higher Order Mutant

Mark Harman and Yue Jia King’s College London, CREST Centre

Problems Solution Empirical Study SummaryBackground

Page 10: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Problem 1970s

Computational cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 11: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Problem 1970s

Computational cost

e.g. Triangle LoC 50

FOM 500 HOM 2500

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 12: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Problem 1970s

Computational cost

e.g. Triangle LoC 50

FOM 500 HOM 2500

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 13: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Problem 1970s

Computational cost

e.g. Triangle LoC 50

FOM 500 HOM 2500

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution Empirical Study SummaryProblems

Page 14: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1980s

Coupling effect hypothesis

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 15: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1980s

Coupling effect hypothesis

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 16: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1980s

Simple Complex cost

If a test set kills simple faults, it also kills the complex faults

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 17: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1980s

Coupling effect hypothesis

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 18: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1990s

Selective mutation / Mutation sampling

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 19: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Selected First Order Mutants

+ -

> <

a b

Trivial Fault

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 20: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

90% of the real faults are complex faults (Purushothaman and Perry)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------HOMs -> Subtle faults

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 21: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

Solution Programmer

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 22: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

Solution Programmer

1

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 23: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

Solution Programmer

2

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 24: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

Solution Programmer

3

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 25: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Subtle Fault

4

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Solution Programmer

Page 26: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1990s

Selective mutation / Mutation sampling

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 27: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution 1980s

Coupling effect hypothesis

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 28: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

cost

Problem 1970s

Computational cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Solution Empirical Study SummaryProblems

Page 29: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution

Higher order mutation testing

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 30: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Solution

Higher order mutation testing

…Search Based Optimization

cost

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 31: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutation Testing

Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 32: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutant

Most common case

Test set T

Ta

Tb

Tc

Tabc

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 33: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutant

Uncommon case

TaTb

Tc

Tabc

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 34: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutant

Subsuming higher order mutant

TaTb

Tc

Tabc

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 35: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutant

Subsuming higher order mutant

TaTb

Tc

Tabc

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 36: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutant

Strongly Subsuming higher order mutant

TaTb

Tc

Tabc

If a test case kills HOM Mabc, it also kills the FOM Fa, Fb and Fc

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 37: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutation Testing

Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 38: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutation Testing

Increase the subtletyReduce the Oracle effort

Mutants Reduced number of equivalent mutants

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 39: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutation Testing

Increase the subtletyReduce the Oracle effort

Mutants Test cases Reduced number of equivalent mutants

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 40: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Higher Order Mutation Testing

Increase the subtletyReduce the Oracle effort

Mutants Test cases Reduced number of equivalent mutants

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Empirical Study SummarySolution

Page 41: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Results

Mutants LoC FOM sHOM

Triangle 50 584 47Tcas 150 679 98Schedule2 350 1014 78Totinfo 500 2570 320Printtokens 750 866 67Space 6000 7570 522

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution Empirical Study Summary

Page 42: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Algorithms

Greedy Hill ClimbingGenetic Algorithm

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 43: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Results

RND GA GR HC0

10

20

30

40

50

60

70

80

90

Subsuming HOMs

Valuable HOMs

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 44: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Triangle Program

Classify triangle by the lengths of the sides

Equilateral Isosceles Scalene

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 45: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Strongly Subsuming HOM

if(trian == 1 && a + b > c)

if(trian > 1 && a + b > c)

if(trian == 1 && a + b <= c)

if(trian == 1 && a + b <= c)

a == c && a + b > c && a + c <= b

b == c && a + b > a && b + c <= a

a == b && a + b > c

a == b && a + b <= c

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 46: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Tool

MiLuDeer

Horse

Cow

Donkey

rare valuable

Strongly Subsuming HOM

(Père David's Deer)

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 47: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

www.dcs.kcl.ac.uk/pg/jiayue/milu

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution SummaryEmpirical Study

Page 48: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing.

Summary

Higher order mutation testingQuality of mutantComputational Cost

QuestionAre real faults higher order mutants?

Thanks

Mark Harman and Yue Jia King’s College London, CREST Centre

Background Problems Solution Empirical Study Summary


Software Logic Mutation Testing Presented by

Software Logic Mutation Testing Presented by

Topics in Mutation Testing and Program Perturbationjhk39/teaching/cs576su06/L7.pdfDependable Software Systems (Mutation) Dependable Software Systems Topics in Mutation Testing and

Topics in Mutation Testing and Program Perturbationjhk39/teaching/cs576su06/L7.pdfDependable Software Systems (Mutation) Dependable Software Systems Topics in Mutation Testing and

[AnDevCon 2016] Mutation Testing for Android

[AnDevCon 2016] Mutation Testing for Android

Mutation Testing for Model based requirements€¦ · Mutation Testing Mutation Testing is a method of inserting faults into models based requirements to test whether the tests pick

Mutation Testing for Model based requirements€¦ · Mutation Testing Mutation Testing is a method of inserting faults into models based requirements to test whether the tests pick

Mutation Testing in Python - GOTO Bloggotocon.com/.../slides/AustinBingham_MutationTestingInPython.pdf · Mutation Testing in Python Austin Bingham ... exception handler deletion

Mutation Testing in Python - GOTO Bloggotocon.com/.../slides/AustinBingham_MutationTestingInPython.pdf · Mutation Testing in Python Austin Bingham ... exception handler deletion

Efficient multi-objective higher order mutation …¬ƒcient multi-objective higher order mutation testing with genetic programming ... mutation testing a highly generic and flexible

Efficient multi-objective higher order mutation …¬ƒcient multi-objective higher order mutation testing with genetic programming ... mutation testing a highly generic and flexible

Literature Survey on Mutation Testing August 2014 ... - …parsai.net/files/research/Literature Survey on Mutation Testing.pdfsoftware in question, mutation testing frameworks are

Literature Survey on Mutation Testing August 2014 ... - …parsai.net/files/research/Literature Survey on Mutation Testing.pdfsoftware in question, mutation testing frameworks are

Open Challenges in Mutation Testing

Open Challenges in Mutation Testing

Mutation Testing and MuJava

Mutation Testing and MuJava

MuClipse: Mutation Testing for Eclipsemuclipse.sourceforge.net/docs/muclipsebinder.pdfMuClipse: Mutation Testing for Eclipse A comprehensive report of accomplishments during Independent

MuClipse: Mutation Testing for Eclipsemuclipse.sourceforge.net/docs/muclipsebinder.pdfMuClipse: Mutation Testing for Eclipse A comprehensive report of accomplishments during Independent

Mutation Testing - Voxxed Days Bucharest 10.03.2017

Mutation Testing - Voxxed Days Bucharest 10.03.2017

Mutation testing in Java

Mutation testing in Java

Mutation testing in php with Humbug

Mutation testing in php with Humbug

Program-based Mutation Testing - Computer Science

Program-based Mutation Testing - Computer Science

Who Watches the Watchmen? - Mutation Testing

Who Watches the Watchmen? - Mutation Testing

Efficient Multi-Objective Higher Order Mutation Testing with Genetic Programming

Efficient Multi-Objective Higher Order Mutation Testing with Genetic Programming

A Mutation Testing Tool for Java Programs Ett verktyg for ... · foundation of mutation testing and emphasizes the role of efficient mutation operators. Mutation operators determine

A Mutation Testing Tool for Java Programs Ett verktyg for ... · foundation of mutation testing and emphasizes the role of efficient mutation operators. Mutation operators determine

Software Testing and Validation SWE 434 Mutation testing.

Software Testing and Validation SWE 434 Mutation testing.

Mutation Testing: A comparison of mutation selection methods

Mutation Testing: A comparison of mutation selection methods

2016 java-sig-mutation-testing

2016 java-sig-mutation-testing