JDays 2015

Alexander Tarnowski

Dealing with combinatorial explosions and boring tests

Developer (2000→) Java, Perl, C, C++, Groovy, C#, PHP,

Visual Basic, Assembler

Trainer – TDD, Unit testing, Clean Code, WebDriver,

Specification by Example

Developer mentor

Writer

Scrum Master

Coach in training

Who am I?

https://www.crisp.se/konsulter/alexander-tarnowski

alexander_tar

alexander.tarnowski@crisp.se

”We want our

customers to be able

to compute their car

insurance premiums

online.

Online quotes are in

our favor, since we

outprice our

competitors!”

Who is Tim?

Image: stockimages/freedigitalphotos.netAlexander Tarnowski

Age Premium for males Premium for females

18-23 1.75 1.575

24-59 1.0 0.9

60+ 1.35 1.215

Business rules

Alexander Tarnowski

@Test

public void maleDriversAged18() {

assertEquals(1.75, new PremiumRuleEngine()

.getPremiumFactor(18, Gender.MALE), 0.0);

}

The first test

Age Premium for males Premium for females

18-23 1.75 1.575

24-59 1.0 0.9

60+ 1.35 1.215

Alexander Tarnowski

@Test

public void maleDriversAged23() {

assertEquals(1.75, new PremiumRuleEngine()

.getPremiumFactor(23, Gender.MALE), 0.0);

}

The second test

Age Premium for males Premium for females

18-23 1.75 1.575

24-59 1.0 0.9

60+ 1.35 1.215

Alexander Tarnowski

And this could go on…

Image: imagerymajestic/freedigitalphotos.netAlexander Tarnowski

Silly names

Repetitive test structure

Boredom

Smells and insights

Image: Mister GC/freedigitalphotos.netAlexander Tarnowski

How many equivalence classes?

How many boundary values?

Would we test drive all of them?

How many tests are needed?

0

0.5

1

1.5

2

18-23 24-59 60+

Male

Female

Alexander Tarnowski

@RunWith(Parameterized.class)

public class PremiumAgeIntervalsTest {

private double expectedPremiumFactor;

private int age;

private Gender gender;

public PremiumAgeIntervalsTest(double expectedPremiumFactor, int age, Gender gender) {

this.expectedPremiumFactor = expectedPremiumFactor;

this.age = age;

this.gender = gender;

}

@Parameters

public static Collection<Object[]> data() {

return Arrays.asList(new Object[][]{

{1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE},

{1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE},

{1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE},

{1.215, 60, Gender.FEMALE}}

);

}

@Test

public void verifyPremiumFactor() {

assertEquals(expectedPremiumFactor, new PremiumRuleEngine()

.getPremiumFactor(age, gender), 0.0);

}

}

The parameterized version

Alexander Tarnowski

@RunWith(Parameterized.class)

public class PremiumAgeIntervalsTest {

@Parameter(value = 0)

public double expectedPremiumFactor;

@Parameter(value = 1)

public int age;

@Parameter(value = 2)

public Gender gender;

@Parameters(name = "Case {index}: Expected {0} for {1} year old {2}s")

public static Collection<Object[]> data() {

return Arrays.asList(new Object[][]{

{1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE},

{1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE},

{1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE},

{1.215, 60, Gender.FEMALE}}

);

}

@Test

public void verifyPremiumFactor() {

assertEquals(expectedPremiumFactor, new PremiumRuleEngine()

.getPremiumFactor(age, gender), 0.0);

}

}

Alternative syntax

Alexander Tarnowski

Focus on data

Allow comparing a set of predefined inputs with

some predefined output

Make checking simple

Turn tests with silly names into data-driven tests

Parameterized tests

Alexander Tarnowski

Interface

Happy path

Learning

Error handling

TDD vs Testing

TDD Testing (checking)

Equivalence partitions

Boundary values

Decision tables

State machines

Alexander Tarnowski

@Test

public void aHamburgerIsnHealthyFood() {

assertThat(Menu.HAMBURGER.getCalories(),

greaterThan(200));

}

Speaking of food…

Alexander Tarnowski

Hamburgers contain the

least amount of calories

among common fast foods

Theory

Alexander Tarnowski

public class CalorieComparisonTest {

public static List<FastFood> foods() {

return Arrays.asList(Menu.FISH_BURGER,

Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,

Menu.HOTDOG);

}

@Test

public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods() {

for (FastFood food : foods())

assertThat(Menu.HAMBURGER.getCalories(),

is(lessThan(food.getCalories())));

}

}

}

Proving the theory

Alexander Tarnowski

@RunWith(Theories.class)

public class CalorieComparisonTest {

@DataPoints

public static List<FastFood> foods() {

return Arrays.asList(Menu.FISH_BURGER,

Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,

Menu.HOTDOG);

}

@Theory

public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods(FastFood food)

{

assertThat(Menu.HAMBURGER.getCalories(),

is(lessThan(food.getCalories())));

}

}

A theory test

Alexander Tarnowski

No fast food meal contains

less than 500 calories!

A more interesting theory

Image: marin/freedigitalphotos.netAlexander Tarnowski

@RunWith(Theories.class)

public class FastFoodMealTheoryTest {

@DataPoints

public static List<Main> mainCourses() {

return Arrays.asList(Menu.HAMBURGER, Menu.FISH_BURGER,

Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,

Menu.HOTDOG);

}

@DataPoints

public static List<SideOrder> sideOrders() {

return Arrays.asList(Menu.SMALL_FRENCH_FRIES, Menu.LARGE_FRENCH_FRIES,

Menu.APPLE_PIE, Menu.SMALL_CHOCOLATE_MILKSHAKE);

}

@DataPoints

public static List<Beverage> bevereges() {

return Arrays.asList(Menu.MEDIUM_COKE, Menu.LARGE_DIET_COKE,

Menu.MEDIUM_LATTE, Menu.LARGE_LATTE);

}

@Theory

public void noFastFoodMealContainsLessThan500calories(Main main,

SideOrder sideOrder,

Beverage beverage) {

assumeThat(beverage.isDiet(), is(false));

assertThat(main.getCalories() + sideOrder.getCalories() + beverage.getCalories(),

is(greaterThan(500)));

}

}

Alexander Tarnowski

Feed the test with all main courses and all side

orders and all beverages

Cartesian product: mains X side orders X

beverages

assumeThat prunes some combinations

Behind the scenes

(Hamburger, Small french fries, Medium coke)(Hamburger, Small french fries, Large diet coke)(Hamburger, Small french fries, Medium latte)(Hamburger, Small french fries, Large latte)(Hamburger, Large french fries, Medium coke)(Hamburger, Large french fries, Large diet coke)(Hamburger, Large french fries, Medium latte)(Hamburger, Large french fries, Large latte)…

Alexander Tarnowski

Are built around ”for all” type of reasoning

Can’t pair specific data points with specific results

Let you work with the Cartesian product of

multiple variables

Theories

Alexander Tarnowski

Let’s do the Caesar cipher…

A B C D E F G H I J K L M O P Q R S T U V X Y Z

S T U V X Y Z A B C D E F G H I J K L M O P Q R

CAESAR = USXKSJ

Alexander Tarnowski

Does it work?

… by borrowing an online implementation

Alexander Tarnowski

For a bunch of different arbitrary strings…

... and a bunch of different offsets...

... try the following:

CaesarCipher.decode(CaesarCipher.encode(string, offset), offset)

Dream scenario

Alexander Tarnowski

@RunWith(Theories.class)

public class CaesarCipherTest {

@Theory

public void caesarCipherRoundTrip(@RandomString(maxLength = 128) String plainText,

@TestedOn(ints = {0, 1, 2, 10, 26, 27, 1000}) int offset) {

assertEquals(plainText, CaesarCipher.decode(CaesarCipher.encode(plainText, offset),

offset));

}

}

We can do that!

Alexander Tarnowski

RandomString.java:@Retention(RetentionPolicy.RUNTIME)

@ParametersSuppliedBy(RandomStringSupplier.class)

public @interface RandomString {

int maxLength();

}

RandomStringSupplier.java:public class RandomStringSupplier extends ParameterSupplier {

@Override

public List<PotentialAssignment> getValueSources(ParameterSignature signature)

throws Throwable {

RandomString annotation = signature.getAnnotation(RandomString.class);

int length = (int) (Math.random() * annotation.maxLength());

final String s = RandomStringUtils.randomAlphanumeric(length);

return Arrays.asList(PotentialAssignment.forValue("random string", s));

}

}

@RandomString

Alexander Tarnowski

RandomStringsSupplier.java:public class RandomsStringSupplier extends ParameterSupplier {

@Override

public List<PotentialAssignment> getValueSources(ParameterSignature signature) throws

Throwable {

List<PotentialAssignment> values = new ArrayList<>();

RandomStrings annotation = signature.getAnnotation(RandomStrings.class);

Generator<String> stringGenerator

= strings(integers(1, 128, Distribution.INVERTED_NORMAL), characters());

for (int i = 0; i < annotation.count(); i++) {

values.add(PotentialAssignment.forValue("random string", stringGenerator.next()));

}

return values;

}

}

QuickCheck style

Alexander Tarnowski

Examples of generators booleans()

dates(Long low, Long high, TimeUnit precision)

fixedValues(T... values)

strings(Generator<Integer> length, Generator<Character> characters)

arrays(Generator<? extends T> content, Class<T> type)

excludeValues(Generator<T> generator, T... excluded)

sortedLists(Generator<T> content, int low, int high)

net.java.quickcheck

public interface Generator<T> {

/**

* Generates the next instance.

*

* @return a newly created instance

*/

public T next();

}

Alexander Tarnowski

Anything goes!

May involve huge domains

Often involves inverse functions

Generative testing

Alexander Tarnowski

Now we can execute thousands of tests!

But what if we want the opposite?

Congratulations!

Alexander TarnowskiImage: zole4/freedigitalphotos.net

Back to car insurance premiums

Gender

Male

Female

Age interval

18-24

25-59

60+

Yearly mileage

0

1-1000

1001-3000

3001-6000

6001+

Safety features

None

Airbag

ABS

HIP

Multiple

Brand

Nissan

Volvo

Ferrari

Toyota

Ford

Volkswagen

Driving record

Model Driver

Average Joe

Unlucky Uma

Bad Judgement Jed

Dangerous Dan

2 x 3 x 5 x 5 x 6 x 5

= 4500

Alexander Tarnowski

One parameter causes the error

Only 6 tests are needed!

Single mode faults

Brand Driving record Yearly mileage Safety features Age interval Gender

Nissan Model Driver 0 None 18-24 Male

Volvo Average Joe 1-1000 Airbag 25-59 Female

Ferrari Unlucky Uma 1001-3000 ABS 60+ -

Toyota Bad Judgement Jed 3001-6000 HIP - -

Ford Dangerous Dan 6001+ Multiple - -

Volkswagen - - - - -

Alexander Tarnowski

A combination of two parameters causes the error

Run through a tool that computes all pairs (or look

up in a table of orthogonal arrays)

Only ~40 tests are needed!

Double mode faults

Alexander Tarnowski

Finding all pairs by hand

Row Variable 1 Variable 2 Variable 3

1 A X Q

2 A X R

3 A Y Q

4 A Y R

5 B X Q

6 B X R

7 B Y Q

8 B Y R

Alexander Tarnowski

Theoretical foundation: orthogonal arrays

Reduce the number of tests from thousands to

just a few

Great to put into parameterized tests

Finding Single and Double mode faults

Alexander Tarnowski

Unit tests are examples

Parameterized tests make writing many similar

tests easy

Theory tests introduce general statements about

program elements

Generative tests – Anything goes!

Single mode faults & double mode faults –

Reduce the number of tests and feed

parameterized tests

Summary

Alexander Tarnowski

https://leanpub.com/developer_testing

http://web.archive.org/web/20110808084654/http://shareandenjoy.saff.net/tdd-specifications.pdf

https://github.com/junit-team/junit/wiki/Theories

https://github.com/pholser/junit-quickcheck

http://www.satisfice.com/tools.shtml

Some resources

Alexander Tarnowski