A/B Testing…The Right Way

The What’s, Why’s and How’s to do A/B Testing Successfully

Hello!

• Holly Wright• Marketing Manager• holly.wright@spectate.com• @hollygowrightly

“I’m still experimenting.” – Stevie Wonder

“While you are experimenting, do not remain content with the surface of things.” – Ivan Pavlov

A/B Testing

• A/B Testing is a method for experiment with your web design and/or content

• The purpose of A/B Testing is to improve some aspect of your website, ads, landing pages or other digital content

The Concept

• A/B Testing has been used for a long time in marketing and advertising

• Traditionally, it has included testing ads in different publications, testing product packing in different markets, testing price points in different geographies

The Basics

• The most basic A/B Test for web involves one original piece of content and one variation

• You present the two options to visitors, see which one gets the best response and then make that your standard

The Goal

• A/B Testing is not about creating the perfect ad, landing page, website, etc.

• The goal is just to create a better ad, landing page, website, etc.

The Process• You don’t have to stop after one test. • A/B Testing is all about the iterative process of

revising, optimizing and refining

The Experimental Method

• Independent Variable• Dependent Variable• Control Variables• Hypothesis

Independent Variables

– Images– Form Fields– Colors– Fonts– Capitalization– Button Size– Word Selection– Pricing Offers– Social Buttons

– Delivery Method– Layout– Length of Form– Format of Text– Header Text– Required fields– Emails

Dependent Variable

• This is the thing you’re testing• It can be just about any behavior:– Clicks– Conversions– Time on Page– Purchase Behavior– Social Shares

The Dependent Variable

• You must decide on a behavior to measure in order to “declare a winner”

• Measured behavior should match test– In other words, if you’re testing an ad, then you

want to measure clicks– If you’re testing a landing page or promotion, you

may want to measure conversions or purchases– If you’re testing a content format, you may want to

test the time visitors spend on your page

The Experimental Method for a Callout

• Independent Variable(s): the callout• Dependent Variable: clicks• Control Variables: logo, font size, images used

on page, content on page, layout, colors and menu items (in other words, everything else!)

• Hypothesis: A blue callout will get more clicks than a red callout

The Experimental Method for a Design Element

• Independent Variable(s): the font size• Dependent Variable: time on page• Control Variables: logo, images used on page,

content on page, layout, colors and menu items (in other words, everything else!)

• Hypothesis: A larger font size will increase the average time people spend on my homepage

Humans are Irrational

“Engineers think everyone else should think like them. Marketers know that everyone else doesn’t think like them”

The Ever-Important Hypothesis

• You do not have a psychic gift• It’s important to make hypotheses, so you

think about what you’re testing and why, but don’t expect to be right much of the time

A/B Testing vs. Multivariate Testing

A/B Testing• A/B Testing involves two or

more versions of a single page or page element

• This is a simpler way to test visitors’ responsiveness to different messages

• It can be relatively quick to test this way

• AKA “split testing”

Multivariate Testing• Multivariate Testing involves

multiple versions of multiple page elements that are presented in random combinations

• This method requires either more traffic volume or more time in order to “declare a winner”

Which one?

• If you don’t have a lot of traffic or a lot of time, the best approach is going to be testing just two versions of a single page or element

• If you have more time and/or traffic, you can test three or more versions at a time

• If you also have money to invest in a multivariate testing tool, go nuts!

The First Step

• If you’re just starting out with this, start broad– Test very noticeable things– Change up the entire design of a page– Use a completely different message

Declaring a Winner

• This is all about traffic (which is all about the rate of traffic and time)

• You need to have a sufficient number of visitors and the desired behavior before you have enough data to make a decision

Declaring a Winner

• If you make a decision too early or with too small a sample size, you could be making your site WORSE!

A Word About Statistical Significance

• If your results between the two tests are significantly different, you can make a decision with less data

• If your results are very close, you need a lot more data to be confident that one variation is significantly better than the other

• The good news is, we don’t have to do the math ourselves…

A Word About Statistical Significance

http://getdatadriven.com/ab-significance-test

The Next Steps

• If you’ve found a winner from your first test, try refining that version– Test smaller elements to see what converts best– Run a longer test with a few different versions– Get creative--Try tests you’d never expect

someone to test, like line spacing, italics, bullet point length

– The web is your laboratory!

What If There IS No Winner?

• If your test doesn’t yield a clear winner, then keep testing

• You can – Extend the time of the test,– Change up one of your samples, or– Just pick one and go with it!

SEO & A/B Testing

• Google wants us to do A/B Testing• It’s better for them if our web content is the

best it can be!• There are some guidelines for A/B Testing so it

won’t affect your SERP placement

SEO & A/B Testing - Guidelines

• Don’t do anything sneaky with your JavaScript, like showing different content to bots vs. humans

• This is called cloaking and is not a typical A/B Testing practice, but can get you in trouble when you are using JS

SEO & A/B Testing - Guidelines

• After a test is over, use 301 (permanent) or 302 (temporary) redirects to your original URL

• This ensures that any visitors who have bookmarked your page or linked to it from their sites will always find the right page

SEO & A/B Testing - Guidelines

• Use the rel=“conanical” link element if you can’t use redirects

• This tells Google which version of a page is your preferred version, which influences the content is serves up in search results

Google’s explanation of this:http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394

Call-to-Action Example

Call-to-Action Example

Call-to-Action Example

A

B

Call-to-Action Example

Call-to-Action Example

Popular Tools

• Google Content Experiments• Optimizely or Visual Website Optimizer• Spectate!

Google Content Experiments

• Test up to five different versions of a page and select specific metrics and goals

• Essentially A/B Testing for entire pages (or A/B/C/D/E Testing)

• Integrates with Google Analytics• FREE!

Optimizely / Visual Website Optimizer

• Advanced visual page builders• Allow visitor targeting to customize test• Only need a single line of code to implement• Good reporting features• VWO has more features but a higher base

price than Optimizely; pricing is based on monthly visitors

Spectate

• Allows A/B Testing of callouts and landing pages

• Includes a visual landing page editor• Is not just a testing tool; also includes a host of

other features, such as social posting, form builders, email auto-responders, SEO scoring, notification messages, nurture programs, etc.

Putting It All Together

• A/B testing can help you optimize your content and improve your conversions

• The process is iterative because you can always improve• Search engines have no problem with this as long as you follow

some general rules• There are tons of tools out there to assist with both A/B Testing

and Multivariate Testing• Statistical significance must be taken into account before

deciding on a winner• Dependent variables need to match up with independent

variables being tested

Q&A

Resources

• http://spectate.com/?s=a%2Fb+testing&submit.x=-1129&submit.y=-136

• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394

• http://getdatadriven.com/ab-significance-test

• http://rich-page.com/website-optimization/which-low-cost-testing-tool-google-content-experiments-vs-optimizely-vs-visual-website-optimizer/

Contact Info

• Holly Wright• Marketing Manager• holly.wright@spectate.com• @hollygowrightly

Want to try out Spectate for free? Contact me for a FREE 15-Day Trial!