Mutation Testing as a Safety Net for

Test Code Refactoring

Ali ParsaiAlessandro MurgiaQuinten D. SoetensSerge Demeyer

2

“Refactoring is changing a system to improve its internal structure without altering its external behavior.” – Fowler et al.

3

Does refactoring preserve the behavior?

Perform refactoring

Run the test suite

Do all tests pass?

Production Code

4

Does refactoring preserve the behavior?

Perform refactoring

Test Code

5

SolutionA metric to verify the preservation of the test

suite behavior pre- and post- refactoring

6

Mutation Coverage

Branch Coverage

Options

7

Branch Coverage

33%

8

Mutation CoverageProcedure

Example

75%

9

Branch Coverage+ Can be calculated fast+ Very good tool support – Easily integrable – Not accurate for defining behavior

10

Mutation Coverage+ More accurate + Adaptable by customizing the fault model – Computationally intensive – Poor tool support, hard to integrate

Works on Java source code

Works regardless of build system and testing framework

11

12

Main

JavaRead JavaParse JavaMutate ReportGenerator

Antlr-generatedJava Parser

CustomizedAntlr4 Runtime

13

LittleDarwin Project Report

14

LittleDarwin File Report

15

Mutation testing tool → LittleDarwinBranch coverage tool → JaCoCoToy example → SalaryReal project → AddThis Codec

Experiments

16

Salary class diagram

Toy Example

17

Toy Example

18

Toy Example

Pre- Refactoring

19

Toy Example

Post- Refactoring

20

Branch coverage and percentage of passed tests remain the sameMutation coverage changes

Toy Example

21

The mutant that makes a difference can be traced back to the problematic test

Toy Example

Pre- Version

22

Real Project

Found a version with refactorings in the test code

Identified the refactorings using RefFinder

Backported the changes in the test code to the previous version

ProductionCode

TestCode

Post- Version

ProductionCode

BackportedTest Code

23

Real Project

24

Mutation Coverage remains the same for all classes

Real Project

Limitations

25

One can catch smaller fishes with a tighter net

ConclusionMutation testing can act as a safety net for test suite refactoring by detecting behavior changes

26

Mutation testing cannot guaranty the preservation of the test suite behavior in every situationThis method can be used to verify automated refactoring tools

27

Does refactoring preserve the behavior?

Perform refactoring

Test Code

pre- and post- refactoring

Does the mutation coverage stay the

same?