KILL THE MUTANTSa better way to test your tests

SHOW OF HANDSlet's do a

• Unit testing

• Continuous integration

• Code coverage

• Mutation testing

CODE COVERAGE• Measure the lines (or branches) that are executed during testing

CODE COVERAGE

QUIS CUSTODIET IPSOS CUSTODES?Who watches the watchmen?

MUTATION TESTING

• Proposed by Richard J. Lipton in 1971

• Surge of interest in the 1980s

• Time to revive this interest!

TERMINOLOGY: MUTATION

TERMINOLOGY: MUTANT

• A mutant is a mutated version of your class

MUTATION TESTING• Generate (a lot of) mutants of your codebase

• Run (some of) your unit tests

• Check the outcome!

OUTCOME #1: KILLED

• A mutant is killed if a test fails (detecting the mutated code)

• This proves the mutated code is properly tested

OUTCOME #2: LIVED

• A mutant didn’t trigger a failing test…

OTHER OUTCOMES

• TIMED OUT: Infinite loop

• NON-VIABLE: Illegal bytecode

• MEMORY ERROR: Out of memory

• RUN ERROR: Other…

TOOLING• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)

• Jester : http://jester.sourceforge.net/ (inactive)

• Jumble: http://jumble.sourceforge.net/ (inactive)

• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)

• PIT: http://pitest.org/

USING PIT

• Manipulates bytecode with ASM (at runtime)

• Mutants are never stored

MUTATORS

• Example of mutators:

> into >=< into <=>= into ><= into <

MUTATION TESTING IS SLOW?

WHICH TESTS TO RUN?

• PIT uses code coverage!

WHICH TESTS TO RUN?

• A mutation is on a line covered by 3 tests? Only run those.

EASY TO USE:

• Standalone Java process

• Build: Ant and Maven plugin

• CI: Sonarqube and Gradle plugin

• IDE: Eclipse and IntelliJ plugin

SUMMARY

• Code coverage: BAD

• Mutation testing: GOOD!

• Automatic, easy to add, and meaningful!