Best Practice is Dead: The Need for a Test and Learn Strategy

Chris Moffatt E-commerce Consultant

Other Media, London

Save me time. Save me money. Make me feel happier. Satisfy my thirst/hunger/desire. Give me more confidence. Make me laugh. Make me feel special. Make me feel intelligent. Solve a problem for me. Educate me. Help me with my research. Answer my question. Reduce my stress. Give me something before my friends have it. Keep my purchase a secret. Surprise me. Shock me. Let me share my experience. I want to compliment you. I want to complain. Reward my loyalty. Show me why you’re better than the competition. Tell me what’s new and hot. Inspire me. Show me something to aspire to. Make me feel famous. I want to feel part of something. I want to feel unique. I want to blend in with my peers. I want to stand out from my peers. I want to help somebody else. Relieve my boredom.

Best Practice is Dead

Ask your customers what they want

Some customer effort: Focus groups Surveys Feedback forms

No customer effort: Use analytics data Test and Learn

Few online retailers do it right…

8  

6  

5  

2  

NO  TESTING   HAVE  SOFTWARE  BUT  NO  TESTING  

TESTING  LACKS  RIGOUR/VELOCITY  

RIGOROUS,  HIGH  VELOCITY  TESTING  

…but it can yield great results

“Companies with a culture of optimisation had twice the conversion rate of their peers”

Adobe 2014 Digital Marketing Optimisation Survey

Barriers

•  Cost of software •  Lack of resource/time •  Lack of skills •  Change averse •  Don’t know what to test •  Inconclusive results •  Don’t know how to measure success

HOW TO TEST

What should I test?

The whole customer experience, end-to-end

•  Acquisition (SEO, PPC, social media, email, affiliates, referrals)

•  On-site: page content and design. Key modules e.g. carousel/banner. Search/navigation. Key page templates: Home page, Landing page, Listing page, Product page, Basket, Checkout, beyond Checkout

•  Retention: Personalisation. Lifecycle emails and CRM.

Work towards a formal, rigorous testing programme

•  People •  Process •  Tool

People

•  Testing Skills and Experience •  Creativity •  Good listener •  Compelling communicator •  Negotiation skills •  Good presenter •  Can explain results to a non-technical audience

Process

•  Define KPIs •  Identify pain points •  Write and prioritise hypotheses •  Devise test and success criteria •  Manage risk (limit exposure/segment/time/product category) •  Run test •  Evaluate results in context •  Share insight with stakeholders: propose change or new test

Tool

features

cost

Adobe Target

Adobe Target premium

Optimizely

Google Content Experiments

VWO Maxymiser

Autonomy Optimost

Monetate

Unbounce

WebTrends Optimize

Accenture optimisation

Test for free

•  Pseudo tests/natural experiments (Example: GM cars) •  Test using your Content Management System •  Use a free trial of a test tool to justify its cost •  Google Content Experiments

PITFALLS TO AVOID

Pitfall 1: Starting too small

Small tests take longest to reach statistical significance Test for impact! Consider these first: •  Acquisition methods •  Home page content •  Product page content •  Basket and checkout design

Pitfall 2: Low test velocity

87% of online businesses run <5 tests per month… not enough! Some businesses run hundreds of tests per month Likely need at least 1 full time staff member for testing and analysis High ROI is only possible with high velocity testing Segment your audience, manage risk and do multiple concurrent tests

Source: Econsultancy CRO Report, 2013

Pitfall 3: Forgetting stakeholders

Brand  Guardians  

 Offline  Marke?ng  

PR  

E-­‐commerce  Team    

Content  Manager  E-­‐commerce  Manager  Online  Merchandiser  

Marke?ng/CRM  Manager  Op?misa?on  Execu?ve  

Budget  holders    

Commercial  Director  Retail/Online  Director  Marke?ng  Director  

Meet weekly to share results, agree changes and discuss next tests

Pitfall 4: Misinterpreting results

•  Ensure other variables are controlled

•  Don’t forget to segment data

•  Test for statistical significance

•  Misunderstanding of metrics e.g. conversion

Pitfall 5: Resistance to change

•  Need stakeholder buy-in •  Work closely with brand guardians •  Manage risk •  No-blame culture needed •  Set expectations

“Risk-averse companies actually risk failure when they don't embrace a vigorous campaign of experiments in the name of superior customer experience.”

Adobe/Econsultancy Digital Trends 2015

THANK YOU