#Conversion2015 Amsterdam keynote Ton Wesseling
-
Upload
online-dialogue -
Category
Business
-
view
1.502 -
download
2
Transcript of #Conversion2015 Amsterdam keynote Ton Wesseling
O N L I N E D I A L O G U E G u i d i n g y o u t o d i g i t a l m a t u r i t y
D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5
To n W e s s e l i n g
A / B - t e s t e n
E m a i l : t o n @ w e s s e l i n g . n e t !L i n k e d i n : t o n w e s s e l i n g !
T w i t t e r : @ t o n w !!
Ton Wessel ing !
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Emerce magazine: conversion column
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Conversion Conferences wor ldwide
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Conversion speakers
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Where is Craig Sul l ivan?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
At least he quotes me
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
More exper iments
“If you double the number of experiments you do per year
you’re going to double your inventiveness”
Jeff Bezos, CEO Amazon – 2004!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Like Amazon, Like Zalando G
erm
an E
com
mer
ce m
arke
t, th
ank
you
Andr
é M
orys
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Exper iments per year
1000+
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Why not?
Compare to the past?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Why can this lead to fa i lures?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
A big weekly change!
26 out of 44 weeks: over 5% change!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Methods used to improve conversion
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Methods companies plan to use
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Importance of a structured approach
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
The negat ive effect trend
Time span!
Aver
age
effe
ct o
f opt
imza
tion!
at c
ompa
nies
app
lyin
g op
timiz
atio
n!M
oney spend on!O
ptimization w
orldwide!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
I t ’s not a solut ion
It’s a method
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
I t ’s a way of working
It’s DNA
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So i t ’s a method:
When
How What
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Growth curve – how i t was teached
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Phase 1: br idge the gap – Risk
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Fake adwords test ing
Do they click
on your idea?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Fake facebook test ing
Do they like your idea?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Fake landing page test ing
Do they subscribe to your idea?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Phase 1: br idge the gap – Risk
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Get out of the office
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR - Opt imizat ion
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: low significance levels
ü Big chance the winner is not a real winner
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: low power levels
ü Big chance a real winner is not recognized
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of 10%
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Average: 75% significance & 40% power
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You need big numbers per var iat ion
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
With 1000 conversions + a month
20+ tests a year
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tests with impact, on average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So in th is case, you’ve impact
Every 6 weeks
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Slower? Energy wi l l run out!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
What to test? Business cases!
Delivery –test with in stock. 4 weeks
or 2 days?!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
What to test? Business cases!
Service charge, do it or not? Include in price or not?!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
What to test? Big design changes
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: choose what to analyze up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Don’t stop the test once i t ’s s ignificant!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Don’t stop the test once i t ’s s ignificant!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You just let i t run unt i l i t ’s s ignificant
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You just let i t run unt i l i t ’s s ignificant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ABTestGuide.com/calc
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Opt imizat ion – moving up
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Final ly you can test detai ls…
Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You should grow to
Customer Buying Reasons
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Reasons behind the customer journey
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Why is your user here?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
How can you
help your user’s brain
to fullfil it’s needs?
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Data & Psychology
Digital Data Persuasion Psychology
Analyze &
Experiment
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Test ing takes t ime
Use it to LEARN
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Create a test schedule
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
FA C T & A C T – 7 s t e p s t o l e a r n
&
Tell Conclude Analyze
Test Create Analyze Find
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Many sources out there to find & analyze
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Test which dia logue has impact
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Which locat ions are cr i t ical
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
FA C T & A C T – 7 s t e p s t o l e a r n
&
Tell Conclude Analyze
Test Create Analyze Find
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Now you’re test ing
On your way to 4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Your winners are st i l l spare?
Less than 25%?
Please higher your significance level to 95% or more if it drops below this 25 number!!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: Your test takes too long
ü Cookie deletion (you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: Don’t confuse cross device users
ü Only test 1 device at a time!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: Maybe you’re test ing broken stuff
ü Always check browser compatibility!
ü Dont’ break dynamic stuff!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: You forgot to leave out some vis i tors
ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: You’re not s lowing the control down
ü Add the same code to the control
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: You’re not test ing on fresh people
ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
T ip: You forgot to give them t ime to buy
ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You’re already th inking of more teams
And still have test capacity
*test capacity – the number of tests you can start per week based on your audience & conversions!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You tested which locat ions are cr i t ical
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
CRO becomes user knowledge
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
More impact on discover
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Customer knowledge center
Making money f rom convers ion improvements
Creat ing more and more cus tomer ins igh ts Which can be monet ized
on even a la rger sca le
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So you keep on groing G
erm
an E
com
mer
ce m
arke
t, th
ank
you
Andr
é M
orys
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Done test ing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – when to start with what
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
O N L I N E D I A L O G U E G u i d i n g y o u t o d i g i t a l m a t u r i t y
D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5
To n W e s s e l i n g
A / B - t e s t e n