Conversion Jam

How to run 200 tests a year & what do you learn?

Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch

Emerce Conversion - @MartijnSch

Title

Know why, you test and how you scale what matters:

@MartijnSch

WHAT YOU GET OUT OF THIS?

We’ll be sharing what tools + process we have at The Next Web to get started!

How it allStarted

@MartijnSch

THE START

Why we feelThe need for testing?

@MartijnSch #ConversionJam

“So what would we have

to do to grow x10?”

Boris (CEO)

“I have no idea what I’m doing”

@MartijnSch / #ConversionJam

MEDIA News

E-commerce

EVENTS Events

Conferences

WORK SPACES TQ

INTELLIGENCE Index

Why am I showing you this slide?


But how did we get to200 A/B Tests a Year?


Testing Figures Data from 2015

2,5 CRO Team in 2015

38.000.000 Sessions with Tests

0 CRO Team in 2014

5-7 Tests per week

Priceless Amounts of fun

35% Traffic v.s. Tests

@MartijnSch

@MartijnSch

Our roadmap for getting to 200 tests a yearJan ‘15

Let’s do more testing!After running 20-30 tests in 2014 we knew we had to scale up.Resources were low, but with the addition of a web analist we could grow.

Apr ‘15

Hitting some limitsGoogle Analytics Content Experiments (JS) was our first tool to use. But soon we hit the limits of the tool.

May ‘15

Setting the visionAfter running 40 tests by May we needed a beter process that could scale and that aligned with our testing vision.

Dec ‘15

Reaching XXX testsWith the testing vision and updated testing program we were able to run XXX tests in total in 2015.

3 Months

1 Month

7 Months

Quantity v.s. Quality?

@MartijnSch

But why would you like to run at high velocity? Why not go for quality?

#ConversionJam

Testing Figures Data from 2015

26% Winning Tests in 2015

@MartijnSch

:( :)

“If you double the number of experiments you’re going to double your inventiveness”

- Jeff Bezos

@MartijnSch

What makes it different: Publishers v.s. E-commerce?


The opportunity is 350-400 tests a

year. As ROI per winning test is

lower we need to run a lot to have

a positive return.

Efficiency <> Costs


How is our (marketing) teamStructured?


SEOCRO/ANALYTICSTNW DEALS

INDEX.COCONTENT

GROWTH

@MartijnSch

Front-end Developers

F*CK it, we’ll do it liveIt’s not rocket science

“Innovate through in-house solutions”

@MartijnSch

What’s ourProcess

@MartijnSch

PROCESS

We failed, not once but also not twice…


We needed to learn,Knowledge building?

@MartijnSch

You got to love working with people who are like minded.

#ConversionJam

Thanks for all your support!

We learn from you!


The more we test the more

we innovate and the higher

inventiveness.

Find theperfect process





a positive return.

Know your metrics?


@MartijnSch

1

2

Unique UsersGrowing our unique user base willincrease the potential to reach Generation T.

2

3 3

Our metrics/KPIs for success;

User Engagement: PV/SessionA higher engagement with our user base will increase the number of pageviews per session.

User Retention: Returning %Increasing the percentage or returning users will increase our brand awareness.

Newsletter SignupsTo get a relationship with our users we’d like to have their email address for the newsletter.

Social SharesMore shares will influence our reach amongst new and returning users.




a positive return.

Do you have a swimming pool?





a positive return.

Segment Users


Users are the same

@MartijnSch

KNOW YOUR SEGMENTS

X

X

X

Google Tag Manager (GTM) is making our lives easier + scalable.

What tools we use


So we need to improve most of our communication around testing.

Communicate more


How did we overcome our fails?

@MartijnSch

You fail once, you fail twice!

#ConversionJam

We didn’t move to any ‘real’

CRO tool but decided to

iterate on our own: GTM <> JS.

GTM

Download:

Github.com/TheNextWeb/CRO


http://github.com/MartijnSch/CRO

Building our own solutions for

saving testing documentation

+ analysis of tests.

Innovate


We moved most of our

communication towards

Slack. Making the teams more

involved in the whole process.

Always communicate


We hired two front-end

developers, to make sure in

the long-run we can test

bigger things.

Front-end Dev


5-7 Tests per week

2-4 Tests per week

30-40 Tests per year

20% More efficient

@MartijnSch

Improvement What does progress look like?

What’s in our

Future?@MartijnSch #ConversionJam

Who’s able to deploy winning

variants within minutes from their

codebase?

Continous Improvement


This will make segmentation +

automation even easier. In addition

it will decrease the flickering effect.

Back-end Testing


Testing small doesn’t work, so we

need to grow even bigger. On

average we grow traffic with around

10-20% yearly. At the same rate we

can increase our testing program.

Big > Biggest


TL;DR Process

1. Have the right company vision 2. Have the right team structure 3. Have access + support from C level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation


What tests did we learn from

@MartijnSch

LEARNINGS

MEDIA News

E-commerce

EVENTS Events

Conferences

WORK SPACES TQ

INTELLIGENCE Index

Sharing


More content on social media drivesbrand awareness + reach

Sharing brings in more loyalty,

more engagement and

hopefully more traffic.

Share buttons

• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users

Winner: Variant

Original:

Variant

Engagement


More engagement increase loyalty which provides advertising revenue

Engagement is important,

focus on getting more clicks

on the related stories in the

sidebar.

Rankings!

• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users

Winner: Variant 1

Original: Variant:

Engagement is important,

focus on getting more clicks

on the related stories below

the article.

Related Stories

• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users

Winner: Variant

Original:Variant

So what if we remove the

whole sidebar, wouldn’t people

be more focused on the stuff

that is out there?

Big, bigger

• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users

Winner: Original

Now you know how to run200 A/B Tests a Year?

@MartijnSch

But did we get to200 tests?

@MartijnSch

“Why didn’t we run 200 tests?”

“What went wrong?”

“Do I need to start crying?”

Boris (CEO)


“Well, if this guy wouldn’t have stuck

his fingers into a lawn mower, YES!”

Missing out on ±20 tests in 2015.

The Lawn Mower


Take Aways:

1. Focus on running X tests a year

2. Improve the process

3. Have the right (free) tools 4. Have the right people + support 5. Document/Communicate everything

6. Know what the future looks like

7. Have fun!


Emerce Conversion - @MartijnSch

Title

Tack!Ha så kul!


Do you have any

Questions?@MartijnSch #ConversionJam

Conversion Jam

Marketing

Transcript of Conversion Jam