Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Post on 02-Dec-2014

242 views 2 download

description

Jon Warden and Shruti Ahuja share their story how they established a Testing Culture at Haymarket during the Optimizely Experience London 2014.

Transcript of Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

HOW HAYMARKET IS ESTABLISHING !

A TESTING CULTUREJon Warden

Head of User Experience !

Shruti Ahuja Head of Analytics and Optimisation

Jon Warden Head of User Experience!Haymarket Media Group !

uk.linkedin.com/in/jonwarden/!@jonwarden

My background!10 years at News UK in various design & UX roles 5 years as a UX consultant, freelancing at various organisations Haymarket to date: Initial set up, and 2 years management of internal UX team

Shruti Ahuja Head of Analytics and Optimisation!Haymarket Media Group !

uk.linkedin.com/in/shrutiahuja/

My background!8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and Financial services Haymarket to date: Develop analytical infrastructure across 20 B2C brands Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands

– Grace Murray Hopper, Computer scientist and US Navy Admiral

“One accurate measurement is worth a thousand expert opinions”

Who are Haymarket Media Group?

AN INTERNATIONAL PUBLISHING HOUSE

• 72 market leading brands • 20 offices • Across six countries • 1500 + staff • Business Media division • Consumer Media division • For Print, Digital, Live Media and events

We create award winning specialist content for international audiences

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

Our journey with Optimizely

WITH OPTIMIZELY SINCE MAY 2013• We started off on a month-by-month Gold

package !

• We completed very basic experiments on: Whatcar.com PistonHeads.com CaravanSiteFinder.com

!

• We quickly found we were regularly hitting our traffic allocation

• In April 2014 we upgraded to Platinum

WHY WE UPGRADED• We needed dedicated support • We needed unlimited projects, user accounts,

and more features to use on all our brands • Other areas of the business were interested in

using Optimizely, including international divisions • We now have dedicated accounts for:

Haymarket Business Media!Haymarket Consumer Media !PistonHeads.com

The Optimisation process

OPTIMISATION & BUSINESS PRACTICEEach company has their own process for A/B testing

Success/fail

Assumption/hypothesis reported to

Product team

Publisher/feature request

Business criteria UX / UI / FEcreate experiment

Analytics reports

Test runs for 2 weeks minimum

Product / UX / Analytics

define test

No

Yes

Test report created

Sent to business for approval

Budget agreed to create dev work for sprintPushed to live

Test fail

Test report created

OPTIMISATION & BUSINESS PRACTICEEach test goes on the testing road map

OPTIMISATION & BUSINESS PRACTICEEach test has a reporting document and test ID

OPTIMISATION & BUSINESS PRACTICE

• The reports

OPTIMISATION & BUSINESS PRACTICE

• The reports

OPTIMISATION & BUSINESS PRACTICE

• The reports

IN AN IDEAL WORLD…• Over all our process is a good one, but… • We’ve found that we’ve had a few issues • Maintaining focus, and learning is difficult, we’re a busy group • We don’t have a dedicated Optimisation team, yet! • Everyone has their own day jobs, business, analytics, product, ux, editorial • Resourcing - we have small analytics and UX teams • Business see the validity, but it isn’t business critical in certain disciplines • Getting budget released or time for iteration is difficult due to dev roadmap

SOLUTIONS• We’re trying to change the mindset and the culture • We need to create an Optimisation process ‘all’ buy into • We’ve enlisted the help of Optimizely experts to help us • Getting onsite training for the team • Discussing testing strategies with senior management • In the past we’ve hired a third party CRO agency to get more focus • We need to create an in house dedicated team to focus efforts

Experiments - what we’ve learnt

WHAT WE’VE LEARNT

1. Do - ‘Start simple’, until you know what you’re doing 2. Do - Create as many tests as possible, and run them consecutively 3. Do - Measure the numbers, and then learn about the results 4. Do - Leave experiments to run as long as needed 5. Do - Embed a testing culture into your business 6. Do - Ask for help to learn more

WHAT WE’VE LEARNT1. Don’t - run before you can walk, but learn as you go 2. Don’t - try experiments with too many goals 3. Don’t - create A/B and MVT tests with too many variants at first,

without having the proper knowledge 4. Don’t - launch multiple experiments on the same site, at the same

time, they have a tendency to clash with each other, causing experiments to fail

5. Don’t - forget to inform stakeholders you’re running tests

WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS

• We recently created an experiment on one of our sites • Using two styles and four variants

Variant A Variant B

Variant C Variant D

WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS

• The results were astounding! • Variant D had an uplift of +197.6%

PISTONHEAD: ANALYSIS-LED OPTIMISATION

Shruti Ahuja Head of Analytics and Optimisation

ABOUT PISTONHEADS.COM

• Connects car enthusiasts to dealers and one another

• An automotive classifieds platform

• One-stop car shop

HOW DID WE CREATE OUR OPTIMISATION ROADMAP?

WHAT IS OUR OPTIMISATION PROCESS?

Insights Generation

Roadmap Creation

Test ImplementationResults

Analysis

Learning

WHAT DETERMINES OUR TESTING ROADMAP?

• Identify KPI’s that support the Business Model

• Investigate issues that hinder KPI conversion

• Generate ideas to test and improve performance on an on-going basis

!!!!!! Visitors

!!!!!! Inquiries

!!!!!! Leads

!!!!!! Sales

WHAT INSPIRES OUR TEST IDEAS?

• Monthly dashboard reviews • Ad-hoc analysis • Product related issues

OUR MOST SUCCESSFUL TESTObjective:!Determine if introducing a new layout with an inbuilt Contact Seller form and a large image increases site conversion !Hypothesis:!Currently users have a 2-step process to contacting sellers. The simplification of this and introduction of larger images will improve interaction with the ad details pages !Test Description:!A/B test. Control (Existing content: 2 step form & small images), Alternate B (In-page form, in-page images) !Success Metrics:!• Traffic • Retention • Contact Seller Conversion • Revenue

OUR MOST SUCCESSFUL TEST

WHEN TESTS FAIL…Objective:!Understand how user behaviour is affected by using a car like VRM on the parts and plates landing page. !Hypothesis:!The current parts and plates landing page design mirrors the old style SSI landing page. The new VRM form style has been seen to perform better for cars. This test will confirm that the design works for parts and plates too. !Test Description:!A/B test. Control, variation B - car-like VRM !Success Metrics:!Proceed to next step Ad creation conversion rate

WHEN TESTS FAIL…

WHAT WE LEARNT

• Optimisation is a learning process

• Never assume / preempt • You win some, you lose some • In either instance, test further!!

THANK YOU!QUESTIONS?