Experiment Driven ProductHow do we evolve our product at Redbubble

Who is Redbubble

Artwork Marketplace

~10M visits a month

Melbourne based

San Francisco office

2 http://www.redbubble.com/people/luckyvegetable

Engineering at Redbubble

4 teams

Very-very Lean

Even more Agile

up to 12 deploys / day

We test assumptions3

http://www.redbubble.com/people/luckyvegetable

Purchase Experience

4

A/B test

5

FAILSUCCESS

Why A/B test

Isolate changes !

Establish causal relationship

6http://www.redbubble.com/people/joust/works/567864-e-mc-bananas

Funnel

1. Identify major

steps

2. Measure each

3. Profit

7http://www.redbubble.com/people/anfa/works/8412967-i-put-the-fun-in-funnel

Free Shipping Experiment

8

Free Shipping Experiment

1 week

No significant change

~$3500 loss

~$50000+ saved

9

Amazon Payment Fake Door

10

FAIL

SUCCESS

Amazon Payment Fake Door

Minimal cost to run

People who use only Amazon Payments exist!

Looks like ~8% increase in conversion

11

Amazon Payments Experiment

12

Stop assuming - start testing

13Duncan Hull - https://flic.kr/p/iVLZt