KILL THE MUTANTSa better way to test your tests
SHOW OF HANDSlet's do a
• Unit testing
• Continuous integration
• Code coverage
• Mutation testing
CODE COVERAGE• Measure the lines (or branches) that are executed during testing
CODE COVERAGE
QUIS CUSTODIET IPSOS CUSTODES?Who watches the watchmen?
MUTATION TESTING
• Proposed by Richard J. Lipton in 1971
• Surge of interest in the 1980s
• Time to revive this interest!
TERMINOLOGY: MUTATION
TERMINOLOGY: MUTANT
• A mutant is a mutated version of your class
MUTATION TESTING• Generate (a lot of) mutants of your codebase
• Run (some of) your unit tests
• Check the outcome!
OUTCOME #1: KILLED
• A mutant is killed if a test fails (detecting the mutated code)
• This proves the mutated code is properly tested
OUTCOME #2: LIVED
• A mutant didn’t trigger a failing test…
OTHER OUTCOMES
• TIMED OUT: Infinite loop
• NON-VIABLE: Illegal bytecode
• MEMORY ERROR: Out of memory
• RUN ERROR: Other…
TOOLING• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)
• Jester : http://jester.sourceforge.net/ (inactive)
• Jumble: http://jumble.sourceforge.net/ (inactive)
• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)
• PIT: http://pitest.org/
USING PIT
• Manipulates bytecode with ASM (at runtime)
• Mutants are never stored
MUTATORS
• Example of mutators:
> into >=< into <=>= into ><= into <
MUTATION TESTING IS SLOW?
WHICH TESTS TO RUN?
• PIT uses code coverage!
WHICH TESTS TO RUN?
• A mutation is on a line covered by 3 tests? Only run those.
EASY TO USE:
• Standalone Java process
• Build: Ant and Maven plugin
• CI: Sonarqube and Gradle plugin
• IDE: Eclipse and IntelliJ plugin
SUMMARY
• Code coverage: BAD
• Mutation testing: GOOD!
• Automatic, easy to add, and meaningful!
Top Related