4 Steps Toward Scientific A/B Testing
45
description
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
Transcript of 4 Steps Toward Scientific A/B Testing
What is A/B Testing?
MYTH BUSTING
#ScienceOfTesting
A/B testing is not...
Validation of guesswork
Consumer psychology gimmicks “Meek Tweaking”
Images: Hubspot, Conversion Rate Experts
#ScienceOfTesting
It’s also not...
A waste of time Impossible to
get right Beyond the scope of
your job
A/B Testing: Defined
Conducting experiments to optimize your customer experience.
What is A/B testing?
OR
4 Steps of Scientific A/B Testing
#ScienceOfTesting
The 4 Steps of A/B Testing
Step 1
Analyze data
Step 2
Form a hypothesis
Step 3
Construct an experiment
Step 4
Interpret results
STEP 1 | ANALYZE DATA
Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
#ScienceOfTesting
Use quantitative & qualitative data
Quantitative data tells you
where to test
Qualitative data gives you an idea of
what should be tested
#ScienceOfTesting
Quantitative datasets
• Web traffic • Email marketing
• Order history • CRM interactions • Support tickets
…and more!
#ScienceOfTesting
Run high-impact tests
Don’t choose tests randomly
Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/how-to-use-data-to-choose-your-next-ab-test/
#ScienceOfTesting
Qualitative data
• User testing • Survey data • Heat mapping • Your sales & account teams
STEP 2 | FORM A HYPOTHESIS
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The element that is modified • Isolate one variable for an A/B test • Call to action, visual media, forms
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The predicted outcome • Use data to determine the size of effect • More email sign-ups, clicks on a CTA
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• Demonstrate your customer knowledge • What assumption will be proven wrong if
the experiment is a draw or loses?
#ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the call-to-action is shorter, the conversion rate will increase.
Strong Hypothesis
If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.
#ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the checkout funnel is shortened to fewer pages, the checkout completionrate will increase.
Strong Hypothesis
If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.
STEP 3 | CONSTRUCT AN EXPERIMENT
A/B Testing: Defined Every test
has 3 parts
DESIGN TECH
CONTENT
#ScienceOfTesting
Content: What are you saying?
VS.
#ScienceOfTesting
Design: How does it look?
VS.
#ScienceOfTesting
Tech: How does it work?
VS.
The most effective tests often combine all 3 elements: content, design, tech
#ScienceOfTesting
STEP 4 | EVALUATE RESULTS
#ScienceOfTesting
What are we looking for?
• How confident am I that the observed difference from my experiment was not due to chance?
• 95% Statistical Significance = 5% probability that the observed difference was due to chance.
#ScienceOfTesting
Confidence intervals
High statistical confidence
Lower risk of implementing a test that won by chance
#ScienceOfTesting
Sample size calculator
http://optimize.ly/StatCalculator
#ScienceOfTesting
Once you reach significance:
• Variation wins: Launch the variation or update your website.
• Original wins: Learn why hypothesis was incorrect.
• In either case: Think about what to test next.
Examples!
A/B Testing: Defined A simple test
A/B Testing: Defined
Iterative testing on a core hypothesis
A solid test
A/B Testing: Defined
Cohort analysis + website changes + biz process changes
A more complicated test
A
B
#ScienceOfTesting
Step 1: Data collection
#ScienceOfTesting
Step 2: Hypothesis
“If [Variable], then [Result], because [Rationale].”
If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
#ScienceOfTesting
Step 3: Experiment
• Changes to heading text • Custom fields in Salesforce.com • Business process changes for
sales reps • Custom analysis in RJMetrics
based on offline conversion event
#ScienceOfTesting
Step 4: Results
TBD A
B
Arm Your Organization
Marketing
Increase the impact of your tests by bringing more team members into theprocess #ScienceOfTesting
Product
Sales
Engineering
Document your test results in a central repository. #ScienceOfTesting
Heat maps
Optimizely results
Hypothesis
What we learned
Variations
#ScienceOfTesting
Other tried and true tactics
• Build excitement by sharing your wins with the company • Hold a competition for the biggest winning variation • Votes on variations to see who has the highest accuracy
of predicting winners
Thanks!
