Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
-
Upload
agile-lietuva -
Category
Leadership & Management
-
view
414 -
download
1
Transcript of Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
“Program testing can be used to show the presence of bugs, but never to
show their absence!”- Edsger W. Dijkstra
Branches
boolean doFoo(boolean arg) { return arg ? "a" : "b";}
assertThat(a.doFoo(true), is("a"));
assertThat(a.doFoo(false), is("b"));
Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage is kind of like full compilation.
Wait! What exactly is a mutation?
def isFoo(a) { return a == foo;}
def isFoo(a) { return a != foo;}
def isFoo(a) { return true;}
def isFoo(a) { return null;}
>>>
Terminology
Applying a mutation to some code creates a mutant.If test passes - mutant has survived.
If test fails - mutant is killed.
//testmax([0]) == 0max([1]) == 1max([0, 2] == 2
//implementationmax(a) { m = a.first for (e in a) if (e > m) m = e return m}
Mutation// testmax([0]) == 0max([1]) == 1max([0, 2] == 2
// implementationmax(a) { m = a.first for (e in a) if (e > m) m = e return m}
Mutation// testmax([0]) == 0max([1]) == 1max([0, 2] == 2
// implementationmax(a) { m = a.first for (e in a) if (true) m = e return m}
// testmax([0]) == 0max([1]) == 1max([0, 2]) == 2
// implementationmax(a) { return a.last}
Baby steps matter
// testmax([0]) == 0max([1]) == 1max([0, 2]) == 2max([2, 1]) == 2
// implementationmax(a) { m = a.first for (e in a) if (e > m) m = e}
What if mutant survives
● Simplify your code● Add additional tests● TDD - minimal amount of code to pass the
test
Equivalent mutations
// Originalint i = 0;while (i != 10) { doSomething(); i += 1;}
// Mutantint i = 0;while (i < 10) { doSomething(); i += 1;}
Let’s say we have codebase with:● 300 classes● around 10 tests per class● 1 test runs around 1ms● total test suite runtime is about 3s
Is it really slow?
Let’s do 10 mutations per class● We get 3000 (300 * 10) mutations● runtime with all mutations is 150 minutes (3s * 3000)
Speeding it upRun only tests that cover the mutation● 300 classes● 10 tests per class● 10 mutations per class● 1ms test runtime● total mutation runtime 10 * 10 * 1 * 300 = 30s
Usage scenarios● Continuous integration● TDD with mutation testing only on new
changes● Add mutation testing to your legacy
project, but do not fail a build - produce warning report
Summary● Code coverage highlights code that is
definitely not tested● Mutation testing highlights code that
definitely is tested● Given non equivalent mutations, good test
suite should work the same as a hash function
About usVaidas Pilkauskas@liucijus● Vilnius JUG co-
founder● Vilnius Scala leader● Coderetreat facilitator● Mountain bicycle
rider● Snowboarder
Tadas Ščerbinskas@tadassce● VilniusRB co-
organizer● RubyConfLT co-
organizer● RailsGirls Vilnius &
Berlin coach● Various board sports’
enthusiast
CreditsA lot of presentation content is based on work by these guys
● Markus Schirp - author of Mutant● Henry Coles - author of PIT● Filip Van Laenen - working on a book