CHRIS CHERRETT

THE TITLE OF THIS TALK IS BEING A/B TESTED

Race 1 Race 2 Race 3 Race 5Race 4Win Win Win Win Win

WinnerLoser Loser Loser Loser Loser

7,776 PEOPLE DIVIDED INTO 6 GROUPS

WinnerLoser LoserLoser Loser Loser

Winner LoserLoserLoser LoserLoser

Race 1

Race 2

Race 3

Confirmationbias

the tendency to selectively search for information that confirms one's beliefs

CONFIRMATION BIASARE YOU BEING OBJECTIVE?

• Your brain is trying to find evidence to prove what

you already think.

• The trap is to accept what you know in your head,

feel in your heart and just ”know” to be true.

• Confirmation Bias shows how limiting your own

perspective can be.

• Sometimes it’s not even your perspective, but the

perspective of your seniors

• HIPPO (Highest Paid Persons Opinion)

minimalbias.com

OTHER BIASARE YOU BEING OBJECTIVE?

AMBIGUITY EFFECT

Choosing something with a known probability over something unknown.

SELECTIVE PERCEPTION

The tendency for expectations to affect perception.

STATUS QUO BIAS

The tendency to like things to stay relatively the same.

BANDWAGON EFFECT

The tendency to do or believe things because many other

people do or believe the same.

minimalbias.com

Bias blind spota bias that allows you to view yourself as less bias than other people

Disagree?

YOU’RE NOT TESTINGYOU THINK YOU ALREADY KNOW

• If you think you already know, you might not test an

issue or trend.

• So convinced by your understanding of the problem

you miss an opportunity to see the truth.

HOW TO TEST

• Question why something works the way it does.

• Create and run tests to find the answer.

REINFORCEMENTYOU'RE ONLY LOOKING AT DATA THAT WILL REINFORCE YOUR BELIEFS

• Even if you are testing, confirmation bias can still

cause problems.

• If you believe users love your Buy Now button, you

might test changing the main image instead, rather

than testing your assumption.

• Even if real data from real tests reveal something

negative, wouldn't you rather know as early as

possible?

“Every design is a hypothesis”

The end of the build,the start of the project

Your first encounter with a design makes youthink it’s the best, and you’re less likely to test it.

BUILD

LEARN MEASURE

BUILD

LEARN MEASURE

DEFINE SUCCESSDEFINE QUANTIFIABLE SUCCESS METRICS

• Are the things you’re moving and changing actually

helping your bottom line?

• Great tools to quantify your success metrics:

• Understand the user flow and identify bottlenecks

(drop out, bounce, don’t convert)

• Google Analytics• HotJar• Inspectlet• Mixpanel• Kissmetrics

Firstname

Email

Phone

Lastname

Address

SignUpPlan

Firstname

Email

Phone

Lastname

Address

SignUpPlan

Field Name Field NameHesitation Time (seconds) Average Fill Rate

How could wetest this?

A/B TESTING / MULTIVARIANT TESTING

50% of traffic 50% of traffic

You’ve been partof a test today

START TODAYGREAT TOOLS

• Always reset your metrics every time you make a

change.

• As with most things – use the tool that meets your

goals, but define your success metrics ahead of

time for optimal tool selection.

MeasureBuild Learn

Time

Value

A/B TESTING / MULTIVARIANT TESTING

Too much time given to underperforming variations

Reduce time and riskReduce the time it takes to deliver your best

performing variant to the majority of your traffic.

Reduce the risk of lost conversions because usersaren’t being served the better variant.

Fail FastUnderstand quickly whether another variant is performing better

BUILD MEASURE & LEARN

MULTI-ARMED BANDIT

SCENARIO

A gambler stands in front of a row of slot machines.

Each machine varies in probability of reward.

The objective of the gambler is to maximise the sum of

rewards earned through the minimum number of lever pulls.

QUESTIONS

• Which machines do they play?

• How many times do they play each machine?

• What order do they play them in?

N-ARMED BANDIT PROBLEM

Try the variant with thehighest average reward

THE RULE IS…

(conversion rate)

Bad Bad Good100% 100% 100%

Bad Bad Good100% 100% 100%

0%(1/0)

100%100%

Bad Bad Good

0%(0/1)

100%(1/1)

100%

Bad Bad Good

0%(0/1)

50%(2/1)

100%

Bad Bad Good

0%(0/1)

50%(2/1)

0%(0/1)

Bad Bad Good

We didn’t give ourselves the opportunity to explore

THE PROBLEM

(it was pure exploitation)

Try the variant that worked best in the past

EXPLOIT (90%)

EXPLORE (10%)

Try other variants to see if they’re better

E-GREEDY

45%(11/5)

12.5%(8/1)

50%(4/2)

Bad Bad Good

EXPLOIT (90%) / EXPLORE (10%)

45%(11/5)

12.5%(8/1)

40%(5/2)

Bad Bad Good

EXPLOIT (90%) / EXPLORE (10%)

41.6%(12/5)

12.5%(8/1)

40%(5/2)

Bad Bad Good

EXPLOIT (90%) / EXPLORE (10%)

41.6%(12/5)

12.5%(8/1)

60%(5/3)

Bad Bad Good

EXPLOIT (90%) / EXPLORE (10%)

Build Measure & Learn

Time

Value

Loss in conversions due to poorly performing versions

Time taken for statistical significance

A/B TESTING

MULTI-ARMED BANDIT

• Watch out for Confirmation Bias. Are you limited by

your own perspective?

• Are you testing enough? Every design is a

hypothesis.

• The end of the build is the start of the project.

• What are your success metrics? How do you

measure success?

• Use the tools that meet your goals.

• Learn quickly and fail fast.

SUMMARYTO TAKE AWAY

Exploremore…

EXPLORE. CAPTIVATE.CREATE.

© Adido 2016

01202 586 300

HELLO@ADI.DO

WWW.ADI.DO

DEAN PARK HOUSE | 8-10 DEAN PARK CRESCENT | BOURNEMOUTH | BH1 1HL