Optimizing Your Optimization Strategy
59
Optimizing Your Optimization Strategy Benn Stancil CoFounder and Chief Analyst Mode @bennstancil Kyle Rush Head of Optimization Optimizely @kylerush
-
Upload
optimizely -
Category
Marketing
-
view
497 -
download
1
Transcript of Optimizing Your Optimization Strategy
Optimizing Your Optimization Strategy
Benn Stancil Co-‐Founder and Chief Analyst
Mode @bennstancil
Kyle Rush Head of
Optimization Optimizely @kylerush
Focus on high value flows
What to test: focus
!
HIGH VALUE FLOWS vs.
ONE OFFS
What to test: focus
Define success before you start
Track many goals
This is what 16 goals looks like
Define success before you start
But choose one primary goal.
The right primary goal.
What to test: geJng ideas
Comb through your data.
17% error rate
What to test: geJng ideas
Submit Successa
b
a
b
3.5% absolute difference.
26.3% relaNve difference.
What to test: geJng ideas
Go through the flow yourself.
100 Nmes.
What to test: geJng ideas
CompeNtor research
What to test: geJng ideas
Doesn’t have to be a compeNtor
What to test: geJng ideas
User tesNng
• Talk to friends
*She misses you.
• Call your mom*
• Talk to peers
Your strategy should revolve around a variable called MDE.
(minimum detectable effect)
Sample size
Everyone wants a low MDE.
But most of us don’t have the luxury.
Sample size
Myth: Small startups can’t a/b testYour small website gets 2,000 visitors a month
Sample size
Sample size calculator
h_p://www.evanmiller.org/ab-‐tesNng/sample-‐size.html
Significance? Power?
Sample size
Significance: 5%, Power: 80%
Determine sample size before starNng the experiment
Sample size
How big is the decision the test will help decide?
A couple tricks for a lower MDE
TargeOng
Not signed in
4.2% conversion rate
Signed in
0.06% conversion rate
average conversion rate: 2.3%*
*the “not signed in” to “signed in” visits raNo is not 50/50, but i’m saying it is for the purpose of this example.
Almost twice as many visitors would be needed.
Not signed in Signed in
So you want a lower MDE?
TargeOng
Here’s your chance.
36% lower MDE :)
Not signed in Signed in
Another Np for a lower MDE
TargeOng
conversion rate: 4.2% conversion rate: 6.7%
22% lower MDE :)
Bad targeNng Good targeNng
OpNmizing your opNmizaNon strategy
Benn Stancil Co-‐Founder and Chief Analyst at Mode
@bennstancil
Previous Now
OUTLINE
Test to Learn Testing is not just for making
individual decisions
!
OUTLINE
Test to Learn Testing is not just for making
individual decisions
!
Know What You Don’t Know Be aware of what you can’t see
!
OUTLINE
Test to Learn Testing is not just for making
individual decisions
!
Know What You Don’t Know Be aware of what you can’t see
!
Think Beyond One Test Aim, fire, adjust
Before Testing: !
TEST TO LEARN
TEST TO LEARN
“Give a man a fish…”
TEST TO LEARN
User research has drawbacks
TEST TO LEARN
Why we need controlled experiments
TEST TO LEARN
Why don’t new users post?
TEST TO LEARN
Hard questions don’t require hard tests
TEST TO LEARN
Do people respond to features that are “coming soon”?
While Testing: !
KNOW WHAT YOU DON’T KNOW
KNOW WHAT YOU DON’T KNOW
KNOW WHAT YOU DON’T KNOW
Analyzing test results isn’t just interpreting what you can from results.
!
It’s also knowing what the test doesn’t tell you.
KNOW WHAT YOU DON’T KNOW
Yadier Molina Ivan Rodriguez
KNOW WHAT YOU DON’T KNOW
Why do new customers test differently?
KNOW WHAT YOU DON’T KNOW
KNOW WHAT YOU DON’T KNOW
•Be skeptical !
•Don’t leave out details !
•Be comfortable with caveats
After Testing: !
THINK BEYOND ONE TEST
THINK BEYOND ONE TEST
Aim, fire, adjust
THINK BEYOND ONE TEST
Ask why
THINK BEYOND ONE TEST
Yammer’s new user badge
THINK BEYOND ONE TEST
Keep thorough records
THINK BEYOND ONE TEST
Getting feeds just right
YOUR OPTIMIZATION STRATEGY
Test to Learn Testing is not just for making
individual decisions
!
Know What You Don’t Know Be aware of what you can’t see
!
Think Beyond One Test Aim, fire, adjust
Optimizing Your Optimization Strategy
Benn Stancil Co-‐Founder and Chief Analyst
Mode @bennstancil
Kyle Rush Head of
Optimization Optimizely @kylerush
