Improve your tests with Mutation Testing

Nicolas Fränkel

@nicolas_frankel 2

Me, Myself and I

Developer & ArchitectAs Consultant

Teacher/trainerBloggerSpeakerBook Author

@nicolas_frankel 3

Shameless self-promotion

@nicolas_frankel 4

My job

Many kinds of testing

Unit TestingIntegration TestingEnd-to-end TestingPerformance TestingPenetration TestingExploratory Testingetc.

@nicolas_frankel 5

Their only single goal

Ensure the Quality of the production code

@nicolas_frankel 6

The problem

How to check the Quality of the testing code?

@nicolas_frankel 7

Code coverage

“Code coverage is a measure used to describe the degree to which the source code of a program is tested”

--Wikipediahttp://en.wikipedia.org/wiki/

Code_coverage

@nicolas_frankel 8

Measuring Code Coverage

Check whether a source code line is executed during a testOr Branch Coverage

@nicolas_frankel 9

@nicolas_frankel 10

Computing Code Coverage

CC: Code Coverage(in percent)

Lexecuted: Number of executed lines of code

Ltotal: Number of total lines of code

Java Tools for Code Coverage

JaCoCoCloverCoberturaetc.

@nicolas_frankel 11

100% Code Coverage?

“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”

Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/

55966620509667328

@nicolas_frankel 12

@nicolas_frankel 13

Assert-less testing

@Testpublic void add_should_add() { new Math().add(1, 1);}But, where is the assert?As long as the Code Coverage

is OK…

Code coverage as a measure of test quality

Any metric can be gamed!Code coverage is a metric…

⇒ Code coverage can be gamedOn purposeOr by accident

@nicolas_frankel 14

Code coverage as a measure of test quality

Code Coverage lulls you into a false sense of security…

@nicolas_frankel 15

The problem still stands

Code coverage cannot ensure test qualityIs there another way?

Mutation Testing to the rescue!

@nicolas_frankel 16

@nicolas_frankel 17

The Cast

William StrykerOriginal Source Code

Jason StrykerModified Source Code

a.k.a “The Mutant”

@nicolas_frankel 18

public class Math { public int add(int i1, int i2) { return i1 + i2; }}

public class Math { public int add(int i1, int i2) { return i1 - i2; }}

@nicolas_frankel 19

Standard testing

✔Execute Test

@nicolas_frankel 20

Mutation testing

?Execute SAME Test

MUTATION

@nicolas_frankel 21

Mutation testing

✗

✔Execute SAME Test

Execute SAME Test

Mutant Killed

Mutant Survived

Killed or Surviving?

Surviving means changing the source code did not change the test resultIt’s bad!

Killed means changing the source code changed the test resultIt’s good

@nicolas_frankel 22

@nicolas_frankel 23

Test the code

public class Math { public int add(int i1, int i2) { return i1 + i2; }}

@Testpublic void add_should_add() { new Math().add(1, 1);}

✔Execute Test

@nicolas_frankel 24

Surviving mutant

public class Math { public int add(int i1, int i2) { return i1 - i2; }}

@Testpublic void add_should_add() { new Math().add(1, 1);}

✔Execute SAME Test

@nicolas_frankel 25

Test the code

public class Math { public int add(int i1, int i2) { return i1 + i2; }}

@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}

✔Execute Test

@nicolas_frankel 26

Killed mutant

public class Math { public int add(int i1, int i2) { return i1 - i2; }}

@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}

✗Execute SAME Test

Mutation Testing in Java

PIT is a tool for Mutation testingAvailable as

Command-line toolAnt targetMaven plugin

@nicolas_frankel 27

Mutators

Mutators are patterns applied to source code to produce mutations

@nicolas_frankel 28

@nicolas_frankel 29

PIT mutators sample

Name Example source ResultConditionals Boundary > >=Negate Conditionals == !=Remove Conditionals foo == bar trueMath + -Increments foo++ foo--Invert Negatives -foo fooInline Constant static final FOO= 42 static final FOO =

43Return Values return true return falseVoid Method Call System.out.println("foo")Non Void Method Call long t =

System.currentTimeMillis()long t = 0

Constructor Call Date d = new Date() Date d = null;

Important mutators

Conditionals BoundaryProbably a potential serious bug smell

if (foo > bar)

@nicolas_frankel 30

Important mutators

Void Method Call

Assert.checkNotNull()connection.close()

@nicolas_frankel 31

Remember

It’s not because the IDE generates code safely that it will never change

equals()hashCode()

@nicolas_frankel 32

False positives

Mutation Testing is not 100% bulletproof

Might return false positivesBe cautious!

@nicolas_frankel 33

@nicolas_frankel 34

Enough talk…

Time for DEMO

Drawbacks

SlowSluggishCrawlingSulkyLethargicetc.

@nicolas_frankel 39

Metrics (kind of)

On joda-moneymvn clean test-compilemvn surefire:test

Total time: 2.181 s

mvn pit-test...Total time: 48.634 s

@nicolas_frankel 40

Why so slow?

Analyze test codeFor each class under test

For each mutatorCreate mutation

For each mutationRun testAnalyze resultAggregate results

@nicolas_frankel 41

Workarounds

This is not acceptable in a normal test run

But there are workarounds

@nicolas_frankel 42

@nicolas_frankel 43

Set mutators

<configuration> <mutators> <mutator> CONSTRUCTOR_CALLS </mutator> <mutator> NON_VOID_METHOD_CALLS </mutator> </mutators></configuration>

@nicolas_frankel 44

Set target classes

<configuration> <targetClasses> <param>ch.frankel.pit*</param> </targetClasses></configuration>

@nicolas_frankel 45

Set target tests

<configuration> <targetTests> <param>ch.frankel.pit*</param> </targetTests></configuration>

@nicolas_frankel 46

Dependency distance

1 2

@nicolas_frankel 47

Limit dependency distance

<configuration> <maxDependencyDistance> 4 </maxDependencyDistance></configuration>

@nicolas_frankel 48

Limit number of mutations

<configuration> <maxMutationsPerClass> 10 </maxMutationsPerClass></configuration>

Use MutationFilter

@nicolas_frankel 49

PIT extension points

Must be packaged in JARHave Implementation-Vendor and

Implementation-Title in MANIFEST.MF that match PIT’s

Set on the classpathUse Java’s Service Provider feature

@nicolas_frankel 50

Service Provider

Inversion of controlSince Java 1.3!

Text file located in META-INF/services

InterfaceName of the file

Implementation classContent of the file

@nicolas_frankel 51

@nicolas_frankel 52

Sample structure

JARch.frankel.lts.Sample

@nicolas_frankel 53

Service Provider API

@nicolas_frankel 54

Service Provider sample

ServiceLoader<ISample> loaders = ServiceLoader.load(ISample.class);

for (ISample sample: loaders) {// Use the sample

}

Output formats

Out-of-the-boxHTMLXMLCSV

@nicolas_frankel 55

@nicolas_frankel 56

Output formats

<configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats></configuration>

@nicolas_frankel 57

Mutation Filter

Remove mutations from the list of available mutations for a class

@nicolas_frankel 58

@nicolas_frankel 59

@nicolas_frankel 60

Time for DEMO

@nicolas_frankel 61

Don’t bind to test phase!

<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <executions> <execution> <goals> <goal>mutationCoverage</goal> </goals> <phase>test</phase> </execution> </executions></plugin>

@nicolas_frankel 62

Use scmMutationCoverage

mvn \org.pitest:pitest-maven:scmMutationCoverage \

-DtimestampedReports=false

maven-scm-

plugin

must be configured!

@nicolas_frankel 63

Do use on Continuous Integration servers

mvn \org.pitest:pitest-maven:mutationCoverage \-DtimestampedReports=false

Is Mutation Testing the Silver Bullet?

Sorry, no!It only

Checks the relevance of your unit testsPoints out potential bugs

@nicolas_frankel 64

What it doesn’t do

Validate the assembled applicationIntegration Testing

Check the performancePerformance Testing

Look out for display bugsEnd-to-end testing

Etc.

@nicolas_frankel 65

Testing is about ROI

Don’t test to achieve 100% coverageTest because it saves money in the

long runPrioritize:

Business-critical codeComplex code

@nicolas_frankel 66

@nicolas_frankel 67

Q&A

@nicolas_frankelhttp://blog.frankel.ch/https://leanpub.com/

integrationtest/