Conversion World 2015 Keynote by Ton Wesseling
-
Upload
online-dialogue -
Category
Data & Analytics
-
view
681 -
download
1
Transcript of Conversion World 2015 Keynote by Ton Wesseling
C o n v e r s i o n W o r l d K e y n o t e , A p r i l 2 0 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# C R O w o r l d - @ To n W
Growth by Online Experiments
E m a i l : t o n @ t e s t i n g . a g e n c y !L i n k e d i n : t o n w e s s e l i n g !
T w i t t e r : @ t o n w !!
Ton Wessel ing !
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
More exper iments
“If you double the number of experiments you do per year
you’re going to double your inventiveness”
Jeff Bezos, CEO Amazon
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Like Amazon, Like Zalando G
erm
an E
com
mer
ce m
arke
t, th
ank
you
Andr
é M
orys
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
A big weekly change!
26 out of 44 weeks: over 5% change!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
The negat ive effect trend
Time span!
Money spend on!
Optim
ization worldw
ide!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
The negat ive effect trend
Time span!
Money spend on!
Optim
ization worldw
ide!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
The negat ive effect trend
Time span!
Aver
age
effe
ct o
f opt
imza
tion!
at c
ompa
nies
app
lyin
g op
timiz
atio
n!M
oney spend on!O
ptimization w
orldwide!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
The negat ive effect trend
Time span!
Aver
age
effe
ct o
f opt
imza
tion!
at c
ompa
nies
app
lyin
g op
timiz
atio
n!M
oney spend on!O
ptimization w
orldwide!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions per month!
1.000 conversions!per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Fake adwords test ing
Do they click
on your idea?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Fake facebook test ing
Do they like your idea?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Fake landing page test ing
Do they subscribe to your idea?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR - Opt imizat ion
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Low number test ing?
Why can it hurt you?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Warning: low significance levels
ü Big chance the winner is not a real winner
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Warning: low power levels
ü Big chance a real winner is not recognized
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Example test set-up
ü 90% significance level
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Example test set-up
ü 90% significance level
ü 100 conversions per variation
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of +- 10%
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So:
ü 100 tests will give 10 false positives (90% significance)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You need big numbers per var iat ion
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
With 1000 conversions + a month
20+ tests a year
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Tests with impact, on average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So in th is case, you’ve impact
Every 6 weeks
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
What to test? Business cases!
Delivery –test with in stock. 4 weeks
or 2 days?!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
What to test? Business cases!
Service charge, do it or not? Include in price or not?!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – Opt imizat ion – moving up
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Final ly you can test detai ls…
Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You should grow to
Customer Buying Reasons
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
How can you
help your user’s brain
to fullfil it’s needs?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Data & Psychology
Digital Data Persuasion Psychology
Analyze &
Experiment
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T – i t e n d s w i t h Te l l
&
Tell Conclude Analyze
Test Create Analyze Find
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
A n d t h e n s t a r t : FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find Find
Your optimization team needs a good analyst / researcher!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
A/B-test ing is the ul t imate resource
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find Analyze
Really a good analyst / researcher!!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
To find out: The cr i t ical locat ions
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Use that persuasion knowledge
Wheel-of-Persuasion.com
Resource & courses
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find Test
And your team needs a good front-end developer!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
But my A/B-test sof tware has drag & drop?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
They do have analyt ics – but not enough
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Calcu late resu l ts : ABTestGuide.com/calc
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So you can A/B-test wi th Tag Managers
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find
Analyze
Your analyst will analyze the results!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Not s ignificant
ü It’s not a winner, stop
ü But, it’s does not mean it’s a losing one!
ü Average: 33% of your tests will have winners If you are optimizing dialogues!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Choose what to analyze up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Don’t stop the test once i t ’s s ignificant!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Don’t stop the test once i t ’s s ignificant!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You just let i t run unt i l i t ’s s ignificant
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You just let i t run unt i l i t ’s s ignificant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Your winners are spare?
Less than 25%?
Please higher your significance level to 95% or more if it drops below this 25 number!!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: Your test takes too long
ü Cookie deletion (you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: Don’t confuse cross device users
ü Only test 1 device at a time!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: Maybe you’re test ing broken stuff
ü Always check browser compatibility!
ü Dont’ break dynamic stuff!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: You forgot to leave out s o m e v is i tors
ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: You’re not s lowing the control down
ü Add the same code to the control
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: You’re not test ing on fresh people
ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
T ip: You forgot to give them t ime to buy
ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
FA C T & A C T
&
Tell Combine Analyze
Test Create Analyze Find
Tell Conclude
Your psychologyst should conclude and your conversion lead should tell!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Conclude
ü Do nothing? or
ü Re-test? or
ü Implement?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Conclude
ü Do nothing? or
ü Re-test? or
ü Implement? or
ü Scale?
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Conclude
ü Do nothing? or
ü Re-test? or
ü Implement? or
ü Scale? and
ü Continue or change test roadmap
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Everyone should use your knowledge!
Data Insights Action
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – Opt imizat ion – moving up
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Now you’re test ing
On your way to 4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You’re already th inking of more teams
And still have test capacity
*test capacity – the number of tests you can start per week based on your audience & conversions!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You tested which locat ions are cr i t ical
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You know which dia logues are cr i t ical
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
+ Optimization! + Automation!
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – You don’t stop opt imiz ing
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – Maybe you’ l l never automate
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – Re-think
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
A/B-test ing is the ul t imate resource
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
You wi l l make big impact on Discover
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Customer knowledge center
Making money f rom convers ion improvements
Creat ing more and more cus tomer ins igh ts Which can be monet ized
on even a la rger sca le
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
So you keep on groing G
erm
an E
com
mer
ce m
arke
t, th
ank
you
Andr
é M
orys
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
Done test ing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
I f y o u u s e a m e t h o d o l o g y
&
Tell Conclude Analyze
Test Create Analyze Find
Otherwise the company will not grow enough on your efforts!
Ton Wesseling | @ TonW Growth by Online Experiments #CROworld
ROAR – when to start with what
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
C o n v e r s i o n W o r l d K e y n o t e , A p r i l 2 0 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# C R O w o r l d - @ To n W
Growth by Online Experiments