What is A/B Testing?

MYTH BUSTING

#ScienceOfTesting

A/B testing is not...

Validation of guesswork

Consumer psychology gimmicks “Meek Tweaking”

Images: Hubspot, Conversion Rate Experts

#ScienceOfTesting

It’s also not...

A waste of time Impossible to

get right Beyond the scope of

your job

A/B Testing: Defined

Conducting experiments to optimize your customer experience.

What is A/B testing?

OR

4 Steps of Scientific A/B Testing

#ScienceOfTesting

The 4 Steps of A/B Testing

Step 1

Analyze data

Step 2

Form a hypothesis

Step 3

Construct an experiment

Step 4

Interpret results

STEP 1 | ANALYZE DATA

Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting

#ScienceOfTesting

Use quantitative & qualitative data

Quantitative data tells you

where to test

Qualitative data gives you an idea of

what should be tested

#ScienceOfTesting

Quantitative datasets

•  Web traffic •  Email marketing

•  Order history •  CRM interactions •  Support tickets

…and more!

#ScienceOfTesting

Run high-impact tests

Don’t choose tests randomly

Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/how-to-use-data-to-choose-your-next-ab-test/

#ScienceOfTesting

Qualitative data

•  User testing •  Survey data •  Heat mapping •  Your sales & account teams

STEP 2 | FORM A HYPOTHESIS

#ScienceOfTesting

Parts of a hypothesis

“If [Variable], then [Result], because [Rationale].”

•  The element that is modified •  Isolate one variable for an A/B test •  Call to action, visual media, forms

#ScienceOfTesting

Parts of a hypothesis

“If [Variable], then [Result], because [Rationale].”

•  The predicted outcome •  Use data to determine the size of effect •  More email sign-ups, clicks on a CTA

#ScienceOfTesting

Parts of a hypothesis

“If [Variable], then [Result], because [Rationale].”

•  Demonstrate your customer knowledge •  What assumption will be proven wrong if

the experiment is a draw or loses?

#ScienceOfTesting

All hypotheses are not created equal

Weak Hypothesis

If the call-to-action is shorter, the conversion rate will increase.

Strong Hypothesis

If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.

#ScienceOfTesting

All hypotheses are not created equal

Weak Hypothesis

If the checkout funnel is shortened to fewer pages, the checkout completionrate will increase.

Strong Hypothesis

If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.

STEP 3 | CONSTRUCT AN EXPERIMENT

A/B Testing: Defined Every test

has 3 parts

DESIGN TECH

CONTENT

#ScienceOfTesting

Content: What are you saying?

VS.

#ScienceOfTesting

Design: How does it look?

VS.

#ScienceOfTesting

Tech: How does it work?

VS.

The most effective tests often combine all 3 elements: content, design, tech

#ScienceOfTesting

STEP 4 | EVALUATE RESULTS

#ScienceOfTesting

What are we looking for?

•  How confident am I that the observed difference from my experiment was not due to chance?

•  95% Statistical Significance = 5% probability that the observed difference was due to chance.

#ScienceOfTesting

Confidence intervals

High statistical confidence

Lower risk of implementing a test that won by chance

#ScienceOfTesting

Sample size calculator

http://optimize.ly/StatCalculator

#ScienceOfTesting

Once you reach significance:

•  Variation wins: Launch the variation or update your website.

•  Original wins: Learn why hypothesis was incorrect.

•  In either case: Think about what to test next.

Examples!

A/B Testing: Defined A simple test

A/B Testing: Defined

Iterative testing on a core hypothesis

A solid test

A/B Testing: Defined

Cohort analysis + website changes + biz process changes

A more complicated test

A

B

#ScienceOfTesting

Step 1: Data collection

#ScienceOfTesting

Step 2: Hypothesis

“If [Variable], then [Result], because [Rationale].”

If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.

#ScienceOfTesting

Step 3: Experiment

•  Changes to heading text •  Custom fields in Salesforce.com •  Business process changes for

sales reps •  Custom analysis in RJMetrics

based on offline conversion event

#ScienceOfTesting

Step 4: Results

TBD A

B

Arm Your Organization

Marketing

Increase the impact of your tests by bringing more team members into theprocess #ScienceOfTesting

Product

Sales

Engineering

Document your test results in a central repository. #ScienceOfTesting

Heat maps

Optimizely results

Hypothesis

What we learned

Variations

#ScienceOfTesting

Other tried and true tactics

•  Build excitement by sharing your wins with the company •  Hold a competition for the biggest winning variation •  Votes on variations to see who has the highest accuracy

of predicting winners

Thanks!