How good are your tests?
-
Upload
noam-shaish -
Category
Software
-
view
156 -
download
0
Transcript of How good are your tests?
![Page 1: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/1.jpg)
How good are your tests?Noam Shaish
![Page 2: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/2.jpg)
How do we test?
• Unit testing
• Continues Integration
• Test-drive development (TDD)
• Code coverage
![Page 3: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/3.jpg)
Unit testing
• Prove code works
• Instant regression tests
• Improve code design
![Page 4: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/4.jpg)
Continous Integration
• Automated testing
• Maintain single source repository
• Collect statistics
![Page 5: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/5.jpg)
All tests passedSo f*cking what?
![Page 6: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/6.jpg)
Test that always pass
![Page 7: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/7.jpg)
Code coverage
• Measures line/branches executed during testing
• Common way to estimate testing
![Page 8: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/8.jpg)
We are all covered
• Are we?
![Page 9: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/9.jpg)
Test that gives full coverage
![Page 10: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/10.jpg)
So how good are your tests?
![Page 11: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/11.jpg)
Mutation testing
• Proposed by Richard J. Lipton in 1971
• A way to measure the quality of your tests
![Page 12: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/12.jpg)
Mutation
• A small change in your code
![Page 13: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/13.jpg)
Mutant
• A mutated version of your class
![Page 14: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/14.jpg)
Mutation testing
• Create mutants
• Run tests
• Check outcomes
![Page 15: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/15.jpg)
Outcomes: killed• Mutant killed: Mutant is killed if a test fails, which
proves the mutated code is properly tested
![Page 16: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/16.jpg)
Outcomes: lives
• Mutant lives: Mutant didn't fail any test…
![Page 17: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/17.jpg)
Outcomes: timed out• Mutant timed out: Mutant caused program to loop or
stuck
![Page 18: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/18.jpg)
Outcomes: other
• There there possible outcome but less relevant
![Page 19: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/19.jpg)
Mutation testing with PIT
• http://pitest.org
• Configurable ‘Mutators’
• ASM (byte-code manipulation) is used for manipulation
• No mutation storage
• Reports
![Page 20: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/20.jpg)
Mutators: Condition Boundary
> into >=
< into <=
>= into >
<= into <
![Page 21: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/21.jpg)
Mutators: Negate Conditions
== into !=
!= into ==
>= into <
<= into >
< into >=
> into <=
![Page 22: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/22.jpg)
Mutators: Remove Condition
if ( a == b) { attack(); }
into
if ( true ) { attack(); }
![Page 23: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/23.jpg)
Mutators: Math
+ into -
- into +
* into /
/ into *
% into *
& into |
>> into <<
<< into >>
>>> into <<<
a++ into a—
a— into a++
![Page 24: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/24.jpg)
Mutators:Others
• Replace return values (return a; into return 0;)
• Remove void calls (attack(); is removed)
• Replace value by null
• Some mutators enabled by default and some aren't
![Page 25: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/25.jpg)
Mutation testing speed
• Mutation testing is CPU intensive
• Multiple test runs
• PIT has a lot of speed optimizations
![Page 26: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/26.jpg)
Which test to run?
• PIT uses code coverage to decide which test to run
• A mutation is created only on lines covered by 3 test
![Page 27: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/27.jpg)
Summary
• Mutation testing automatically tests your tests
• Mutation testing can find bugs in your tests
• Code coverage is half the story, and gives a false sense of security
• Mutation testing with PIT is easy
![Page 28: How good are your tests?](https://reader031.fdocuments.us/reader031/viewer/2022021922/58edd2ad1a28ab521f8b4749/html5/thumbnails/28.jpg)
Show me some code