UX & Split Testing

UX, MVT, A | B testing, UCD, persuasion architecture, web analytics and any other buzz words you can think of...

Death by PowerPoint courtesy of Lucy Spence

What are we discussing?

• Different types of split testing and how those fits into our UX world– What it is?– Why you should do it?– Who should do it?– When to use it?– How to use it?– What to avoid?

• How does all that fit with UX

What is it? The different types of split tests (terminology)

• A | B– Two versions of the same thing – traffic split 50 | 50– Useful for deciding between radically different solutions

• A | B | n– Same as A | B test but with more than 2

• MVT (multivariate tests)– Many different variations of many different section of a page– Useful for optimising established funnels

What is it?

What sort of things can you test?

A | B | n• Radically different designs

– Designs don’t need common elements

– Templates can be structured in completely distinct ways

– Content doesn’t need to be shared

• Focus on:

– Key messages

– Content structure

MVT• Primarily used to test different

treatments of elements:

– Layout tweaks & positioning of elements

– Text amends

– Buttons - size, colour, style, labels & icons

– Removal of particular elements

• Focus on:

– Optimizing what is already working for you

A diversion...

My first thoughts when I heard about MVT

1. Wow. Cool.

2. OMG! I’m going to be out of a job

3. Hang on – no - wait (thinking, thinking, thinking...) ...maybe there’s a way I can be part of it...

4. Maybe designers no longer design solutions but design experiments to test possible solutions

5. Yay – we can do whatever we want! It’s play time!

6. It doesn’t really matter anyway it’s all just random anyway

A diversion...

What I’ve discovered (mostly through mistakes)• MVT can be fun, informative and deliver strong commercial results

• MVT can also be painful, confusing, time consuming, divisive and deliver no commercial benefits

• Its not quite as simple as everyone makes it out to be

• You need to be familiar with the tools that you’re using and what is / isn’t possible– Most only work on a page at a time – yet mostly we’re testing multi step funnels

• You need to understand statistical significance in order to get results you trust

• You need to be crystal clear about your goals and what you’re trying to achieve

Why should you do it?

What are the benefits?

• Because on the whole we’re a strange and peculiar species– We’re unpredictable - people are strongly influenced by subconscious

thoughts and subtle cues in most environments– We can’t simulate someone’s actual purchasing decisions

• We don’t have a better way of measuring the persuasiveness of different solutions

• Businesses can make quite a bit of money out of it without doing very much

Who should do it?

Is it suitable for your organisation?• Organisations with lots of traffic - you’ve got to get a reasonable

amount of visitors through a test to get a statistically significant result– Testing 2 versions of the same page with a 30% conversion rate and 1000

visitors a day will take 77 days to statistically prove a 2% improvement with 95% confidence.

– If you need to work it out, use a calculator: https://www.google.com/analytics/siteopt/siteopt/help/calculator.html

• Organisation that are making money through their site– It helps clarify your goals– Do you have targeted landing pages for PPC activity? If not, it’s probably not for

you.

• Bloggers (this is an exception) - it’s a easy / cheap way to do engagement experiments such as bounce rate or commenting

When to use it?

Can I use it for everything I do?

• It’s not a magic wand - it’s just a tool – use it appropriately

• Use A | B | n testing early on to help determine direction– Use static HTML for tests, don’t bother developing complex solutions– Don’t test complex solutions that you’ve already built

• Use MVT on your final funnel solutions after all other user testing is done.

• Repeat MVT until you’re struggling to come up with any other options, ie. never stop

How to use it?

Where do you start and how do you progress

• Start off simply – Google’s Website Optimizer is free and easy

• Build up your level of complexity as you get more experience– Different vendors provide different levels of sophistication– Factors you might want to consider:

• Level of support / analysis• Level of self service• Segmentation • Integration with other data centres & web analytics• Email, PPC campaigns, mobile, other platforms

• Move toward the ultimate – full functional split testing / experience testing (but only a very small number will ever make it that far)

What to avoid?

What are the pain points?

• Don’t test something if you’re just going to put it live anyway

• Multiple sources of lies can build distrust if not reconciled / understood

• Lift isn’t guaranteed even when testing positively

• Nothing remains static

• Seeing trends against natural variation is almost impossible

• Control limit analysis often doesn’t work – timescales are too long

• i18n / localization may produce different results

UCD / UX / UED

How does it fit with UX?

• It’s not a magic wand - it’s just a tool – use it appropriately

• It’s about the only real way to measure persuasiveness accurately

• Use A | B | n testing early on to help determine direction– Use static HTML for tests, don’t bother developing dynamic solutions

• Use MVT on your final funnel solutions after all other user testing is done.– Make what works, work harder

• The next step – design with MVT in mind– Know what elements you want to move around in the funnel

Contact details

If you found this interesting and want to stay in touch, or have some questions:

• Linked in: http://uk.linkedin.com/in/lucyspence

• Twitter: http://twitter.com/lucyjspence

• Email: lucyjspence@googlemail.com

And if you want a 2 month free trial for LOVEFiLM use the promo code: HBAQRRMDD