A/B Testing - What your mother didn't tell you

A/B TesAng

What your mother didn’t tell you hDp://www.allaroundtheworld.fr/

January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/

What is A/B TesAng?

•  TesAng customers, not code •  Show different versions •  Pick the best one


Who Relies On A/B TesAng?


Transform Your Company

•  Test your customers, not your code •  Stop guessing what works



Should your paycheck depend on my hunch?

Conversion Funnel


Conversion Funnel


→

A Basic Test

•  Your current behavior is the base •  A variant is a new version (just one change) •  You can have one or more variants

•  But we assume only one for this talk


Which Generates More Clicks?




10,017 Visits 9,983 Visits

125 Conversions 132 Conversions



9,950 Visits 10,550 Visits

125 Conversions 155 Conversions

The Numbers (95% Confidence)


hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/

Version Visits Conversion Rate Not Random

Red 10,017 125 1.25% —

Blue 9,983 132 1.32% 51%

The Numbers (95% Confidence)


hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/

Version Visits Conversion Rate Not Random

Red 9,950 125 1.26% —

Blue 10,550 155 1.47% 94%

Which Sells More Product?


Which Creates More Profit?


What Confidence Level?

Is 99% beDer than 90%?



The Cycle

Create Experiment

Only one difference per version!


Roll the Dice


Track Performance

Sta:s:cs


Choose the Best

Pa:ence


Create Experiment

Wash, rinse, repeat


Choose the Best

•  Ignore the first day •  Full-‐business cycle •  What if you’re wrong?

•  False posiAves

hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/


LimitaAons of A/B TesAng

•  Doesn’t tell you why •  Side effects •  Doesn’t tell you loyalty


Benefits

•  Hyundai: 62% increase in test drive requests •  Artsy Editor: 47% increase in click-‐throughs •  Visual Website OpAmizer: 50% signup increase •  AssessmentDay: 62% increase in sales •  SmileyCookie: 41% increase in sales •  BaseKit: 25% increase in conversions •  Mobal: 27% increase in sales •  You Need A Budget: 85% increase in downloads

hDp://visualwebsiteopAmizer.com/case-‐studies.php

What Are They TesAng?

•  Add-‐to-‐cart buDon placement •  Call to acAon buDons •  Using popup forms •  Removing content •  Link color •  Adding tesAmonials •  NavigaAon menus •  Removing signup forms


Gemng Started

•  Vendor or in-‐house? •  Go for quick wins •  Run an AA test •  Be pa:ent!


Problems

•  Forgemng caching •  Non-‐random selecAon

•  Forgemng selecAon

•  TesAng only the view •  Performance

•  Dead code •  Profit versus conversion


Tips

•  Most experiments fail •  Don’t roll the dice prematurely

•  A/B test bugs •  Test “stupid” ideas •  Don’t require rollouts!


SegmentaAon

•  New versus returning customers •  Different browsers •  Accept-‐Language •  Time of day

•  IP country •  Gender? Age? Something else?


See Also

•  The MulA-‐Armed Bandit hDp://stevehanov.ca/blog/index.php?id=132


Conclusion

•  Lots of money lying on the table •  Egos are bad •  Data is good •  Customer behavior trumps expert opinion

•  Customer behavior trumps code behavior


QuesAons?

hDp://www.allaroundtheworld.fr/


A/B Testing - What your mother didn't tell you

Marketing

Transcript of A/B Testing - What your mother didn't tell you