The Multi-variate advantageHow testing saved my life

Craig SullivanGroup CX ManagerBelron®

@OptimiseOrDie

Group CX Manager for Belron® (Safelite)I get to do it with crowds

Naked self promotion

• Contact deflection and online self service

• Site search analytics

• Site, Page and Campaign optimisation

• Test design and execution

• A long usable site portfolio

• ROI on UX improvements

• Over 18 years of boring meetings

• Conversion Rate Optimisation

• A/B and Multi-variate testing

• Cross channel optimisation

• User Centered Design

• Usability testing

• Customer Research & Insight

• Web Analytics

• Browser and email compatibility

@OptimiseOrDie

Outline• What is optimisation?• Belron® Testing Culture• What is MVT• Our testing program• Advantages of testing• The results• Cross channel optimisation• Case study : Chat ROI• Resources

What’s optimisation?

@OptimiseOrDie

A set of techniques, implemented in order to influence customer behaviour towards these outcomes:• Increased revenue or profit• Increasing NPS (Customer Sat) scores • Lower cost for business or contact centres• Increasing productivity or labour flexibility• Simply delighting customers

We may fail with 90% of what we try. The 10% of winners let us shift behaviour hugely and measurably.

Belron® Testing Culture

@OptimiseOrDie

If you go to the men's washrooms at the Schiphol airport in Amsterdam, you may notice there's a fly in the urinals. It’s screen printed .

So what do you think most men do? That's right, they aim at the fly when they urinate.

They don't even think about it, and they don't need to read a user's manual; it's just an instinctive reaction that means 85% less spillage!

The interesting feature of these urinals is that they're deliberately designed to take advantage of this inherent human male tendency.

This is my job.

Small team, big challenge

• Team of 3 - 2 Optimisers, 1 x part time designer• Higher ROI or lower costs, without spending more• Stuff that doesn’t hinder SEO, PPC, Performance• Measurable ROI - not ego or opinion• Fast return and low cost techniques• Over 26M page tests, 17 international tests live now• ROI is > 50 x optimisation costs• Total cost = ~800k for two years. ROI > $40M USD

Primer on testing

=

Control Version A Version B

A/B test

+9% +2%

Multi variate Pizza

=

ExperimentIngredients Recipes

+2% +3% +1% +2%

+9% +4% -2% +1%

+5% +13% -7% -9%

Month 1 - results

+20% -2% +5%

+3% +2% +6%

+6% +10% -18%

So what about ingredients?

+2% +3%

+9% +4%

+5% +13%

Weed out some losers…

+10% +13%

The final…

Our testing program

@OptimiseOrDie

Insight - Inputs

Insight

Usability testing

SurveysOps and CS

teams

Clicktale

Social analytics

Customer contact

Eye tracking

Remote testing

Forms analytics Search

analytics Depth interviews

Market research

A/B and MVT testing

Unstructured data

Web analytics

Competitor evalsCustomer

services

Killer idea generation sources

• Analytics data• Customer insight – contact, customer feedback, research• Usability testing• Diary studies• Clicktale recordings• Business owners – market data, customer mix, local knowledge• Customer contact centre immersion (listening)• Books, Websites, Tips• Other optimisers• Test data – variables plotted on axis• Making lots of mistakes – Fail faster

Typical test lifecycle

Instrument Gather and review inputs

Other test results – good

variables & learnings

Brainstorm & Hypotheses

Design variables & creative elements

Test Plan Document

Signoff (Legal, Marketing,

Brand)

Instrument analytics

Wave design Results of wave Washup Next Wave (repeat)

Declare winnerDo verification wave + 90/10%

split

Business led reporting (with = £x, without = £y)

Repeat from start

Hub and Spoke model – homepage trafficWhat’s our testing mix – The traffic picture

What’s our testing mix?

• A/B testing – Homepage, Landing pages, Funnels• MVT testing – high traffic landing or home pages

– Mini (less than 8 variables)– Midi (8-32)– Maxi (>32 variables)

And 5 types of specialised testing:• Funnel testing – bedding in new funnels• Landing page optimisation• Cross channel testing • Iterative feature and product testing• Micro testing• Telephony MVT

Ad Hoc

Local HeroesChaotic Good

Level 1Starter Level

GuessingA/B testingBasic tools

AnalyticsSurveys

Contact CentreLow budget

usability

Outline process

Small teamLow hanging

fruit

+ Multi variateSession replayNo segments

+Regular usability

testing/research

PrototypingSession replay

Onsite feedback

_____________________________________________________________________________________________ _

Dedicated team

Volume opportunities

Cross silo teamSystematic

tests

Ninja TeamTesting in the

DNA

Well developed Streamlined Company wide

+Funnel optimisationCall tracking

Some segments Micro testing

Bounce ratesBig volume

landing pages

+ Funnel analysis

Low converting & High loss

pages

+ offline integration

Single channel picture

+ Funnel fixesForms analytics

Channel switches

+Cross channel testing

Integrated CRO and analyticsSegmentation

+Spread tool use

Dynamic adaptive targetingMachine learningRealtime

Multichannel funnels

Cross channel synergy

_______________________________________________________________________________________________

________________________________________________________________________________________________

Testing

focus

Culture

Process

Analytics

focus

Insightmethod

s

+User Centered DesignLayered feedback

Mini product tests

Get buyin

________________________________________________________________________________________________

Mission

Prove ROI Scale the testing Mine value

Continual improvement

+ Customer sat scores tied to

UXRapid iterative

testing and design

+ All channel view of

customerDriving offline using online

All promotion driven by testing

Level 2Early maturity

Level 3Serious testing

Level 4Core business value

Level 5You rock, awesomely

________________________________________________________________________________________________

International testing examples

@OptimiseOrDie

UK Wave 2 – 21/9 to 26/10

+12.5%

+8.3%

+13.9%

+5.9%

What is the ROI then?

+11.2%

Another example

+1-2.5%

Testing advantages

@OptimiseOrDie

Your ‘design’ team?

Who designs this

stuff anyway?

Who designs this

stuff anyway?

Web/Online

Marketing

Legal

IT

THE CUSTOMER

Pesky Meddling Kids?

CEO

Brand

BENEFITS OF TESTING:

• Reduced ego and opinion (execs + others)• Measurable ROI• Faster time to market• Increased conversion in the pocket early• Iterative and rapid product lifecycles• Faster, smaller, focused, measured releases• Continual testing = continual improvement• SLED vs. Agile = faster return, better focus• Less reliance on anecdotal evidence• Business owners part of testing culture• Introduces a testing culture• Large cross silo team now gets involved• Reduces timidity – be brave and curious!• Helps us hit the sweet spot...

Cross Channel Optimisation

@OptimiseOrDie

Insight - Inputs

ExplodingChannels

PPC

SEO

Direct

Affiliate

Display

TV

Retargeting

Phone RingbackEmail

Chat

Mobile handsets

Apps

Tablets

Desktops

Social & Micro

PartnersDM

TV advertising and Web

+2.4%

TV - Off

+6.4%

TV - On

-1.3% +14.8%

People images that suck…

Photo Guidelines• Groups, vans, stores, corporate guff =

NO• Single person – not groups or even

person + customer• Open, Friendly, Natural, Smiling,

Engaged = YES• Uniform – has huge effect, even

wearing branded ball cap improves conversion

• Look straight at viewer or slightly towards CTA

• Plain backgrounds only• No folded arms, arms on hips, pointing,

scowling, cheesy smiles, hands behind back, hanging useless by the side, clasped by groin, fiddling with hands

• Arms with prop instead – hold a clipboard, rag, tool – I’m working for *you*

• Female images work best in almost every country – they slay the guys

• Natural, authentic and not ‘model’ or ‘stock’ image types work best for people

• We get better images now, but we keep working on the guidelines!

• We still got a genital framing picture the other week

• Photoshoots and tests are now being used to drive advertising strategy

• Pre TV test of potential ‘actors’ on website. X factor meets split testing.

Cross channel mini studies• Mini studies:

– Abandon testing– Emergency phone switchoff & Overspill systems– Contact deflection systems and self servicing– Scripting Experiments– APIs and other systems integration– Chat testing– Phone tracking

Abandon testingReduce unwanted phone hangups.

– Remove corporate guffspeak– Don’t mention the website, esp.

if this is the web number (D’oh!)– Don’t give a generic or corporate

drone apology message– Quote real KPIs and timings (ING

direct, average queue time)– Make routing emotionally driven

– not confusing categories that map poorly onto customer needs

– Experiment with queue timing – find sweet spot, don’t overquote

Test plan

– Two countries, big phone systems,– Test variants of messages, music,

timings, amounts quoted– Messages make a big impact –

experiment freely with volumes– Opportunity is large due to high call

volumes we have– What people hate personally vs.

what those same people have on their company lines, lol

– On a mission here!

Switchoff OverspillIf you have spikes, have a throttle:

– Busiest TV & Radio campaign so far, Q2 2011 for Autoglass®

– Designed a phone ‘switchoff’ system, permanently on tap

– Routed majority of contact to a web form

– Prevented a backlog situation from developing

– Bought time and still exceeded customer expectations (<4h excellent)

– Used twice to dampen inbound call volumes

Overspill systems:– Get a UX designer to make you a fast

entry system– Make them simple, easy to learn and

leverage your web platform– Give API access to overspill

companies– Sometimes heavy IT integration isn’t

needed – “Nobody needs a freaking training

course to use facebook” - Make sure your internal apps and call centre overspill systems are built for instant learnability

– This could save millions in process efficiency. Same as web, really.

Contact deflection & self service• People like immediate gratification!• The iceberg of contact opportunity• Make people like you for not talking

to them on the phone –double win• If you stick it in help, will it go away?• No elephant graveyard, please (FAQ)• Instrument for analytics and ROI • Set it up to measure and improve• Gamification of content? Adoption?• LOVEFiLM Help Centre:

– Ratings, Deflection tracking, self service tracking

– V1 :Please view in your account secion, V2 : Please visit your account, V3 : Please check your next billing date, V4 - Final version – to show the real info : “Your next billing date is 1 Feb”

• What about your measurement?• Basics – Ratings, Did you find this

answer helpful?• Intermediate – Tracking behaviour

and contact channels• Add tracking of self service events or

behaviour• Set up a game or reward structure for

staff• Continually improve and get proof• Add phone tracking - make avoidable

contact visible• Remove all barriers to self servicing

for high cost/volume contacts• Try Amazon help – where is phone?• Help experts? Sticky Content are

brilliant (www.stickycontent.co.uk)

Script Optimisation• How can we optimise phone conversion?

– Dedicated test teams– Script testing often limited range/shift in companies now– Now – we can use A/B or MV testing, with API integration– Try out 3,4,5 different versions of key areas or entire script– ‘Wrapping’ sends XML message event, triggers analytics tag– Analyse in the same way as an A/B or multi-variate web test– Fast optimisation of handling scripts, using a mini lab– Should yield 3-5% higher performing scripts, minimum– Shows what variables drive script performance globally– These emotional and psychological triggers can be mined– Aim for Q2 2012 for a proof of concept

Platform APIs• Use of web platform APIs now for:

– Contact centre messages– Facebook app– Mobile site and apps– Scheduling – Analytics tag firing

• Just making APIs, automagically makes:– People use them– Improvements happen– New ideas happen, that you didn’t expect– Incompatible stuff get glued together– Complex IT integration projects go away

Chat Testing

A - Chat

Split

A B

B - Chat C - Chat

C

Contact Us

Contact FormRingback

You must measure behaviour across contact channels during tests like these

A - Chat Free B – With Chat

Damage

Appt

Details 1

Details 2

Damage

Appt

Details 1

Details 2

0844 371 0047

0844 371 0048

0844 371 0049

0844 371 0050

0844 371 0051

0844 371 0052

0844 371 0053

0844 371 0054

Chat testing

• 1 month trial – properly A/B tested• Measured all the contact channels for

movements – what happened?• Over a week, chat saves 17 calls –

yay!• But swapped that for 329 chats @

8.25 minutes. Ouch.• Increased ringback requests? Email

form about the same.• No statistically significant impact on

conversion• Assumption was big impact, huge

impact!• Anecdotally, a wonderful tool.

Agreed.

• Customers happy, yes, but not in a good business way

• Like putting your customer service desk into the street

• You must measure influences in contact volumes when trialling Chat

• Be very careful – prove your ROI on any pilot.

• Only 7-10% use and out of those, less than 50% are influencable etc. etc.

• The productivity Myth of chat• Your situation may differ – please

TEST!

Call tracking

How it works:• Rent a bank of numbers• Add a dynamic ‘tag’ to your page• Phone number appears automatically• Customer calls this number• Captures the keywords, landing page, converting page, traffic source• Geo mapping captures customer location• When the call is hung up, optionally record the outcome code (sale, loss)• Have a special number for Facebook, SEO, Adwords traffic etc.• Lets you analyse from web -> phone call -> wait time -> wrapping• Builds a link between web and call, finally

Call tracking

Why you should use it!• Most people under-report on web driven phone contact or sales• You can’t work out your advertising effectiveness, if you don’t include

phone• If you don’t measure phone, your A/B tests may be wrong• You can capture all your web AND phone, in one easy interface • Allows you a multi-channel view of all your contact sources• Lets you record offline campaigns as well with specific numbers (print, TV

etc.)• You can experiment with routing, messages, on hold music – much more

easily• If you can’t afford lots of phone numbers, this is an easy way to get

tracking

Who?Company Website Coverage

Mongoose Metrics* www.mongoosemetrics.com UK, USA, Canada

Ifbyphone* www.ifbyphone.com USA

TheCallR* www.thecallr.com USA, Canada, UK, IT, FR, BE, ES, NL

Call tracking metrics www.calltrackingmetrics.com USA

Hosted Numbers www.hostednumbers.com USA

Callcap www.callcap.com USA

Freespee* www.freespee.com UK, SE, FI, NO, DK, LT, PL, IE, CZ, SI, AT, NL, DE

Adinsight* www.adinsight.co.uk UK

Infinity tracking* www.infinity-tracking.com UK

Optilead* www.optilead.co.uk UK

Switchboard free www.switchboardfree.co.uk UK

Freshegg www.freshegg.co.uk UK

Avanser www.avanser.com.au AUS

Jet Interactive* www.jetinteractive.com.au AUS

* I read up on these or talked to them. These are my picks.

Call tracking

What data can you get?• Data to keyword, group or traffic source level – for Phone

contact / conversions• Splits by location, traffic source, time, date• Data on the call – how long to answer, to abandon, call

duration• Data post call – was there a sale? (call wrapping)• Linkage between marketing activity and phone ROI on web

channels (very important)• Which parts of your site drive contact• Tagging supports GA, Omniture, Others• Complete call analytics (including recording, if wanted)

Tracking tips• Make contact points trackable (phone, email, form, chat etc.)

For example:• Collapsible phone section on Contact Us form – tracks intent• Phone ‘reveal’ icon, button or widget – tracks intent• Tap to call wrapper on Phone• Use a smart email form instead of email address• Use ‘ringme’ ringback functionality• If email being generated, tweak subject line• Integrate call centre events for maximum coverage• If you can’t buy phone tracking, make your own (mail me!)• Use phone tracking for major traffic groups, say:• PPC groups, Organic, Print, PPC, email, TV, Brand keywords• See your phone traffic in GA, Omniture and most packages

Follow the money – ROI example• 12.4% from Web increase• 12.5% increase from Multi Variate Testing• 3.5% increase in downstream conversion rate• 4% increase in online conversion from mobile optimised site• Reduced call time • Faster booking time• 5.5% increase in NPS (Customer sat) score • Customer delight – feedback. Less questions and worries• Total increase in conversion (measured) = 32%• Actual increase in conversion = 41%• NO additional marketing spend needed• Web Channel : ROI positive within 2 weeks

Email

Twitter

Slideshare

: sullivac@gmail.com

: @OptimiseOrDie

: linkd.in/pvrg14

: slidesha.re/nlCDm6

More reading. Slides and resources on slideshare.net

CRO and Testing resources

• 101 Landing page tips : slidesha.re/8OnBRh • 544 Optimisation tips : bit.ly/8mkWOB• 108 Optimisation tips : bit.ly/3Z6GrP• 32 CRO tips : bit.ly/4BZjcW• 57 CRO books : bit.ly/dDjDRJ• CRO article list : bit.ly/nEUgui• Smashing Mag article : bit.ly/8X2fLk