Factories, mocks, spies…

…and other tester’s little helpers

Carles Barrobés twitter: @technomilk github: @txels

Testing is a very broad topic

!

…with its own special lingo

blackbox whitebox regression unit-test

integration-test service-test pyramid

icecream-cone factory assertion spy SuT

…

Let’s start with a question… !

Why do we write tests?

We write tests to save money We tell the computer how to do [tedious] testing for us, faster and cheaper

Writing tests == automation

SuT: System-under-Test*Your “system” as a black box:

I am system !

(with a spec, if you’re lucky)

in

out

in: data, stimuli out: data, observable behaviour

SuT: System-under-TestYour “system” as a white/gray box:

I am system !

(and you can see what’s inside me)

in

out

…BTW I rely on a bunch of external stuff

SuT: System-under-TestYour “system” as a white/gray box:

zin

out

…BTW I rely on a bunch of external stuff

Explicit/Injected dependencies Implicit/Hardcoded

dependencies

Anatomy of [manual] testingTake your code up to the point you want to test Run the specific feature you are testing Verify that it worked

Anatomy of a test casedef test_something_works(): !

prepare <blank line>

exercise <blank line>

verify

aka “Arrange, Act, Assert”

Anatomy of a test casedef test_something_works(): !

prepare <blank line>

exercise <blank line>

verify

Get your code to a known state:

Generate test data Navigate

Isolate and monitor: Set up mocks Set up spies

Anatomy of a test casedef test_something_works(): !

prepare <blank line>

exercise <blank line>

verify

Call your code: result = something(data)

Anatomy of a test casedef test_something_works(): !

prepare <blank line>

exercise <blank line>

verify

Validate results: Assertions on results

Check observed behaviour:

Check reports from your mocks and spies

Time for another question… !

Which of those is the hardest?

Time for the actual talk… !

Let’s look at tools that can help us with the hard

bits

Tools for test setup(I find the preparation phase to be the hardest bit)

[Complex] test data: factories Test objects with behaviour: mocks Instrument internals: spies

FactoriesGoal: make it easy to generate complex test data structures !

Tool of choice: factory boy*

* I’ve tried others, but I prefer this one

Factories: use casesCreate test data with simple statements

Let the factory fill [irrelevant] details

Black/white box testing Explicit dependencies

Factories: simplicityExample: we need a django model instance for our test.

It has lots of mandatory fields… ..but in this test we only care about “title”

Factories: simplicityNot this: publisher = Publisher.objects.create(name=‘Old Books’)

book = Book.objects.create(title=‘Tirant Lo Blanc’,

author=‘Joanot Martorell’,

date=1490,

publisher=publisher)

But this: book = BookFactory(title=‘Tirant Lo Blanc’)

Factory Boy in actionimport factory

from books.models import Book, Publisher

!class PublisherFactory(factory.DjangoModelFactory): FACTORY_FOR = Publisher

name = ‘Test Publisher’

city = ‘Barcelona’

!class BookFactory(factory.DjangoModelFactory): FACTORY_FOR = Book

title = ‘Test Book’

author = ‘Some Random Bloke’

year = 2015

publisher = factory.SubFactory(PublisherFactory)

Maintainability FTW!When you maintain large tests suites, you want to maximise reuse [& DRYness]

Defaults and rules for building your objects live in a central place - easy to adapt

E.g. adding a mandatory field is no longer a pain

Not tied to your test framework

Factory Boy: nicetiesUse a sequence for unique values: name = factory.Sequence( lambda num: 'Name {}’.format(num) ) !

Lazy attributes to populate “late”: slug = factory.LazyAttribute(

lambda obj: slugify(obj.name)

)

Factory Boy: nicetiesFuzzy (randomised) values title = factory.fuzzy.FuzzyText() # u'phPEZzNqfkXv'

gender = factory.fuzzy.FuzzyChoice(('m', 'f')) # 'm'

age = factory.fuzzy.FuzzyChoice(18, 45) # 27 ... !

Coming soon: Faker support (realistic values) name = factory.Faker('name') # u'Isla Erdman'

email = factory.Faker('email') # u'vmitchell@stehr.org'

SpiesGoal: check if something happened inside your code !

Tool of choice: kgb** I don’t know others in Python, used Jasmine (JS)

Spies: use casesWhen it’s hard to have externally observable behaviour

It’s a bit like adding monitoring to your tests

“Blackbox” testing (with some inside knowledge) Implicit dependencies

Spies: how toYou know a little what your system does under the hood You “spy” on a method that should be called (the spy is a wrapper that “calls through”)

Your spy reports on how that method was called

KGB in actionfrom unittest import TestCase

from kgb import spy_on !def add_three(number): return number + 3 !def do_stuff(number): return add_three(number + 1) !class SpyOnTest(TestCase): def test_spy_on_add_three(self):

with spy_on(add_three) as spy: result = do_stuff(15)

self.assertEqual(spy.last_call.args, (16,)) self.assertTrue(spy.called_with(16))

KGB extrasYou can replace the spied on method and make it do nothing or something else

!

with spy_on(SomeClass.add_stuff, call_fake=add_two):

MocksGoal: make it easy to simulate behaviour of a dependency !

Tool of choice: mock*

* There are others, I haven’t tried them

Mocks: use casesYour SuT has explicit callback dependencies (objects it calls)

You want to feed valid objects and inspect what your system did to them

Simulate hard to reproduce conditions (e.g. exceptions)

Mocks vs FactoriesFactories generate “real production objects” Mocks generate fake objects (that you can throw anything at)

mock in action>>> from mock import Mock

>>> user = Mock(username='Fred')

>>> user.username

'Fred'

>>> user.save(force=True)

<Mock name='mock.save()' id='4492633872'>

>>> args, kwargs = user.save.call_args_list[0]

>>> kwargs

{'force': True}

mocking calls>>> user.save.return_value = True

>>> user.save()

True

>>> user.save.side_effect = Exception('Boom')

>>> user.save()

-------------------------------------------------

Exception Traceback (most recent call last)

...

!

Exception: Boom

mock extras: “patch”Patch existing code and replace it with a mock for the duration of a test Similar use cases to spies [but without “call through”]

Mock: “patch” use casesYour SuT has hardcoded dependencies but you want to test it in isolation You want to accelerate your tests [by bypassing expensive calls]

patch in actionfrom mock import patch

!

class MockPatchTest(TestCase):

@patch('test_sample.add_stuff')

def test_do_stuff_calls_add(self, add_stuff):

add_stuff.return_value = ‘whatever'

!

result = do_stuff(123)

!

add_stuff.assert_called_once_with(123)

self.assertEqual(result, 'whatever')

Tools for test validationBuilt-in assert_ functions from your test tool (nose, unittest)

assert statement (if you use py.test it will give useful reporting) Matchers (hamcrest)

MatchersGoal: reusable conditions for assertions !

Tool of choice: hamcrest

Matchers: use casesYou want to check complex or custom conditions in a DRY way

Matchers can be composed - no need for “combinatory” assertions or assertTrue(<complex expression>)

hamcrest highlightsA single assertion: assert_that Many matchers out of the box (plus you can write your own)

Useful reporting on mismatches (no more “False is not True” errors)

Composite matchers: all_of, any_of, not_

hamcrest in actiondef test_any_of(self):

result = random.choice(range(6))

assert_that(result, any_of(1, 2, 3, 4, 5))

!

!

!

AssertionError:

Expected: (<1> or <2> or <3> or <4> or <5>)

but: was <0>

hamcrest in actiondef test_complex_matcher(self):

user = UserFactory()

assert_that(

user.email,

all_of(

not_none(),

string_contains_in_order('@', '.'),

not_(contains_string('u'))

)

)

!AssertionError:

Expected: (not None and a string containing '@', '.' in order and not a string containing 'u')

but: not a string containing 'u' was 'clangworth@schuster.biz'

Custom matchersYou can write your own matchers The syntax is a bit verbose, so I wrote matchmaker to make it easier

Custom matchers…from hamcrest.core.base_matcher import BaseMatcher

!

class IsEven(BaseMatcher):

def _matches(self, item):

return item % 2 == 0

!

def describe_to(self, description):

description.append_text('An even number')

!

def is_even():

return IsEven()

…using matchmakerfrom matchmaker import matcher

!

@matcher

def is_even(item):

"An even number"

return item % 2 == 0

Custom matchers in usedef test_custom_matcher(self):

user = UserFactory()

assert_that(user.age, is_even())

!

AssertionError:

Expected: An even number

but: was <19>

More custom matchers@matcher

def ends_like(item, data, length):

"String whose last {1} chars match those for '{0}'"

return item.endswith(data[-length:])

!def test_custom_matcher(self):

user1, user2 = UserFactory(), UserFactory()

assert_that(

user.email,

ends_like(user2.email, 4),

)

!AssertionError:

Expected: String whose last 4 chars match those for 'vwalter@yahoo.com'

but: was 'brett.witting@bergnaum.biz'

Thanks!Any questions?

Carles Barrobés twitter: @technomilk github: @txels