Would you bet your job on your A/B test results?

Would you bet your job on your A/B test results?

@QubitGroup #QubitABtest

Friday, 28 February 14

Agenda

@QubitGroup

• 5 digital experience technology trends that will guide your technology investment — Anjali Yakkundi, Forrester Research, Inc.

• 5 fundamental A/B testing mistakes and how to avoid them—Will Browne, Qubit

• How to run a successful A/B testing program—Melanie Kyrklund, Staples

• Q&A

#QubitABtest


5 Digital Experience Technology Trends To Guide Your Investment Anjali Yakkundi, AnalystFebruary 27, 2014


© 2013 Forrester Research, Inc. Reproduction Prohibited4

Burberry brings digital into the store



Room Number

Address & directions

Starwood’s Preferred Guest Streamlines The Check-In Process



Agenda

›Why digital experience technology matters ›Key digital customer experience

technologies ›Five trends that will drive your digital

experience technology investment›Recommendations



Agenda






Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report

Age of the customer



Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report

Your customers are always connected…



…often on multiple devices

Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012


You need to sustain your customer’s attention with engaging experiences.



Engaging experiences requires strong technology management



Agenda





Digital experience reference architecture



Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report

Many solutions make up the digital experience ecosystem

TestingOptimization Analytics

(mobile, social, Web)

PortalsSite search

Commerce platform Web content management

Online video platform

Digital asset management

Product content management

Recommendation engine

Campaign management



Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report

Testing and optimization solutions are key to engagement and measurement

TestingOptimization Analytics

(mobile, social, Web)

PortalsSite search

Commerce platform Web content management

Online video platform

Digital asset management

Product content management

Recommendation engine

Campaign management



Agenda






#1- This is the beginning of the end of the all-in-one-suite

Only 9% prefer a suite approach

Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey”



#2- Tools must create business value out of data

Solutions must effectively harness customer data, product data, social media data, and other

information to create and optimize experiences.



#3- Tools must support personalized experiences

Situation Real-time optimization Prediction

• Device / Channel • Location• Time of day• Weather

• Historical past• Immediate past • Current session

• Likely next steps



#4- Look for more than software: look for solutions ›Look for solutions that help marketers and business

users quickly adapt to changing customer needs. This can be achieved by:

•Keeping IT out of critical publishing paths •Prioritizing user interface needs



#5- Solutions must prioritize multichannel



Agenda






Recommendations

›You can’t do it all at once — prioritize key customer touchpoints ›Assess current technology gaps– most

organizations tend to have gaps around:•Optimization•Personalization•Multichannel delivery

›When sourcing new solutions, look to vendors that can help you connect the dots.


Thank youAnjali [email protected]@Ayakkundi


mailto:[email protected]


5 fundamental A/B testing mistakes

and how to avoid them

Will Browne, [email protected]








Why do I have to test?


A/B testing is an essential tool

@TheGrahamCooke



@TheGrahamCooke

• It is the best, simplest way to prove the business value of your digital technology




@TheGrahamCooke

• It is the best, simplest way to prove the business value of your digital technology

• It is the only way to get close to proving cause and effect in what you do



But there are costs associated with A/B testing



• The monthly cost of your testing platform





• The salaries and time of your developers and team




• The salaries and time of your developers and team

• The lost revenue from implementing changes that do nothing for your site (or actively harm it)




We’re here to show you how to reduce these costs, and increase ROI




• Find the ideas that are likely to have a real positive effect on user behavior (true winners)





• Detect as many true winners as possible





• Detect as many true winners as possible

• Minimize the chance of finding tests that look like winners, but have no true effect



39

Find the ideas that are likely to have a real positive effect on user behavior



Size does matter


Detect as many true winners as possible

We’ve found that tests based on data are

7.7x more effectivethan tests based on intuition alone

(Qubit, 2013)


Imagine testing for weighted coins



(Qubit, 2013)

• How many coin flips would be enough to prove that one coin was weighted?






(Qubit, 2013)


• It is impossible to be 100% certain, but the more data you have, the closer you can get






(Qubit, 2013)


• It is impossible to be 100% certain, but the more data you have, the closer you can get

• We follow scientific standards and wait until there is an 80% chance of detecting a difference





(Qubit, 2013)

Minimize the chance of finding tests that look like winners,but have no true effect




(Qubit, 2013)

• We generally assume that with good testing there is a 5% false positive rate






(Qubit, 2013)


• We can test this using A/A tests (showing both groups the exact thing)






(Qubit, 2013)


• We can test this using A/A tests (showing both groups the exact thing)

• So out of 100 A/A tests, 5 would be winners, despite doing absolutely nothing




Splitting tests too much can lead to false positives


Get a second opinion


Validate your results

We recommend a

95/5 splitto confirm your winning variation is still winning

“Most winning A/B test results are illusory”, Qubit 2014



A worked example: Imagine Mr Bean’s jelly beans shop

@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014


A worked example: Imagine Mr Bean’s jelly beans shop

@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014

• Mr Bean has a 1% conversion rate

• He runs 20 different tests in one year

• There are 2 good tests with uplifts of 5%

• There are also 2 bad tests with uplifts of -5%



TruthCorrect method

Naive method



TruthCorrect method

Naive method

No. of tests 20 20 20

Reported winners 2 3 9



TruthCorrect method

Naive method



After validation 2 2 9



TruthCorrect method

Naive method



After validation 2 2 9

Reported uplift 2 x 5% 2 x ~5%

9 x (5-40)%



Correct method Naive method

Reported uplift 2 x 5% 9 x (5-40)%

Delivered uplift 2 x 5% 2 x 5% + (? x -5%)






Wasted resource None7 unnecessary

tests






Wasted resource None 7 unnecessary tests

Customer experience

Improved Degraded

Your understanding

Improved Confused






Wasted resource None 7 unnecessary tests

Customer experience

Improved Degraded

Your understanding

Improved Confused

Your manager Happy Angry



So, to conclude, those 5 mistakes are:

• Mistake 1: Not A/B testing

• Mistake 2: Not prioritizing tests properly

• Mistake 3: Not waiting for significance and sample size

• Mistake 4: Testing too many things at once

• Mistake 5: Not validating the tests after implementation



Questions you should ask your testing provider

• How do you prioritize tests?

• How do you calculate the sample size required?

• How do you account for the multiple testing problem?

• How do you validate winning tests?



Thank you

If you have any detailed questions please contact

[email protected]

Will Browne, [email protected]










How to run a successful A/B testing program

Melanie KyrklundSenior Ecommerce Analyst


Testing programs have the potential to drive the online customer experience

66

Understand your customers and respond to their needs

Optimize the user journey

Create personalized experiences

Quickly respond to market pressures

Rapidly align marketing & onsite experience

Validate development items

Every visit represents an opportunity to establish a relationship with a customer

Customer experience

Personalization Persuasion

Targeted offers

User experience,

functionality & content


The reality of running a testing program

67

For many organizations, testing represents a cultural change with many political and operational barriers to be overcome:

Resource & operational factors

Buy-in / Multiple stakeholders

Alignment with other business priorities and development plans

Instilling a data-driven culture


A lot of testing is required to deliver results

68

1/5 tests or 1/10 tests will deliver statistically significant wins

A rapid pace of testing is required to build a program that is delivering consistent results month over month


Two factors are critical in building a successful testing program

69

Approach Speed

MethodologyIdea Generation

Operational ProcessPrioritization


The right approach to optimization

70

Be inspired – every visit is an opportunity to establish a relationship with a customer

Use data to understand how visitors are using your website, what is working and what isn’t

Customer feedback to understand why visitors are not purchasing on your website

If working over multiple websites & countries, look for insight at local level first and then identify commonalities at a later stage

Persuasion research, understand the psychology of your customers

Supplement with quick UX testing

Thoroughly research & quantify each opportunity (visitors reached, expected uplift and incremental revenue)

UX optimization Personalization Persuasion


Idea Generation

71Friday, 28 February 14

Speed – Operational Processes

72

Testing Tool selection : Map out the most efficient implementation process

Operational efficiency Pick managed services (analysis, implementation) as required

Prioritization

Build a test pipeline logging:

Idea & hypothesisExpected uplift & incremental revenue Implementation estimatesTest duration Prioritize tests weekly, to keep up momentum of testing and

focus on biggest wins


Conclusion: A rapid, data-driven approach is required

73

Testing programs have the potential to drive the online customer experience, however a rapid data-driven approach is required to deliver results

Plan the required resources upfront

Consider ease of use, test time to market and managed services in tool selection

Map out the most efficient end to end process to get tests live

Process supported by data-driven idea generation & prioritization


Thank you

Any questions?


75

bit.ly/qABresearch

More A/B testing material from Qubit

bit.ly/qABinfographic

@QubitGroup#QubitABtests


http://bit.ly/qubitforrester

http://bit.ly/qubitforrester

http://bit.ly/quresearch%0Dbit.ly/quresearch%0Dbit.ly/quresearch






Would you bet your job on your A/B test results?

Business

Transcript of Would you bet your job on your A/B test results?