Learn From Your List: How to A/B Test Your SMS Campaigns

Copyright © 2015. Mobile Commons, Inc. All rights reserved. | www.mobilecommons.com | [email protected]

Learn from Your List: How to A/B Test Your SMS Campaigns

July 14th, 2015

Ti Chesley Lead Mobile Strategist

Mobile Commons [email protected]

Jessica Fraser Sotelo Deputy Director, Online Marketing

World Wildlife Fund [email protected]


What is A/B Testing?Revisit Your MissionDefine Your Ladder of EngagementDefine Your GoalsIdentify Your Metrics for SuccessTest, Test, Test!Identify Your TargetsStatistical SignificanceCase Study: World Wildlife FundMobile Commons Demo

Agenda

Mobile Commons, Inc. | www.mobilecommons.com | [email protected] 3

What is A/B Testing?

Comparing the results of two versions of a message to see which one performs better.

Revisit Your Mission4


The first step in deciding how to incorporate A/B testing into your text message campaigns is to revisit your campaign goals and objectives.

1.  What do you want to accomplish with your text message campaign?

2.  What actions do you need your subscribers to take in order to achieve your mission?

Define Your Ladder of Engagement5


Make a list of actions and segment your subscribers based on level of engagement.Start with small, easy actions and work your way up to higher value asks.

For example:

1.  Sign a petition

2.  Make an advocacy phone call

3.  Refer someone else using Tell-A-Friend

4.  Volunteer at an event

5.  Make a donation during a key season or an emergency

Don’t scare off new subscribers with premature donation asks!

Define Your Goals6


Once you have created a ladder of engagement, determine the goal of each text message in your campaign.•  What “results” do you want to see?

•  For example:

•  Drive more visitors to your website

•  Drive more advocacy calls to Congress

•  Recruit more volunteers

•  Generate more fundraising leads

Identify Your Metrics for Success7


Once you’ve identified the goal of each message, you should have a very clear sense of how to measure your success:

Click-through rate

Number of phone calls

Number of responses

Information collected

Donations made


Testing, 1, 2, 3!

Test, Test, Test!9


Choose which aspect(s) of your text message you’re going to test:•  Start with elements that you think have the

biggest impact on your metrics. For example:

•  Calls-to-action

•  Content

•  Format

•  Tone

•  Almost every component of your texting has the potential to make a big difference, no matter which one you are testing.

WWF: Happy World Wildlife Day! Check out this exciting news that is cause for celebration.

Hi, it’s Jess at WWF. I want to wish you a Happy World Wildlife Day! Check out this exciting new that is cause for celebration.

Identify Your Targets10


Identify your target test population(s):•  Once you have selected a variable to test, you will need to choose a target

population. This can be your entire mobile list, or a segmented group of subscribers.

•  If you already segment your list into different groups based on subscriber behavior or past actions, it can be interesting to run tests on action-based messages and see how they respond.

•  For example, you can compare how volunteers vs. non-volunteers respond to two different versions of an ask to sign a petition.

Statistical Significance11


In order to provide meaningful results, your A/B tests will need to be conducted with groups large enough to provide statistical significance.•  Statistical significance tells us the likelihood that

something is caused by something other than mere random chance.

•  The larger your test groups, the more likely your test results will be statistically significant.

Use a statistical significance calculator•  You can find free statistical significance calculators

online that will tell you how large your test groups need to be for a specific test.


Case Study: World Wildlife Fund

About World Wildlife Fund13


WWF's mission is to conserve nature and reduce the most pressing threats to the diversity of life on Earth.

Our vision is to build a future in which people live in harmony with nature.

Why WWF Uses SMS14


Why SMS?

•  Contact your supporters where they are.

•  Their phones are within an arm’s reach!

Who are our subscribers?

•  Casual supporters

•  Activists

•  Donors

Take them on a journey from least to most committed!

Why Does WWF A/B Test?15


SCIENCE!

Think:

•  What’s the goal of the SMS?

•  What variable(s) will be test?

•  What is our hypothesis?

•  What metrics will we use to evaluate the results?

•  How will we apply the learnings to future broadcasts?

Example A/B Test: The Plan16


Goal: To get subscribers to take WWF’s tree quiz.

•  Variable: Including an image vs. no image

•  Hypothesis: An image will result in a higher click-through rate.

•  Metrics: (1) click-through rate, (2) response rate, (3) unsubscribe rate

•  Application of learnings: Determine whether or not to include an image in future broadcasts.

•  Targets/Segments:

1. Non-activists

2. Activists

Non-activists: Results17


Message with the image had a higher click-through rate (statistically significant, with 98% confidence).

0.7% 0.7%

12.4%14.0%

0.3% 0.4%0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

No image Image

Non-activists

Response RateClick-Through RateUnsub Rate

Activists: Results18


Message with the image did better in terms of click-through rate, but there were more responses to the version with no image. Neither result is statistically significant, but the trend favors the SMS with the image.

1.2% 0.8%

18.5% 20.3%

0.4% 0.3%0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

No image Image

Action Takers

Response RateClick-through RateUnsub Rate

Lessons Learned19


•  Including an image increased engagement… but not always.

•  An image may not be appropriate for all messages.

•  Optimize the image – what should it feature? should it include copy?

•  Action takers clicked through at a higher rate.

•  Should we send them the same ask as the rest of the list?

•  And, as always, an unexpected lesson: It’s important to make sure you have the full picture – beware of “failed” SMS!


Mobile Commons Demo

We’re Here to Help!21


Finally, don’t be afraid to loop in your Mobile Strategist!

Talk to us about your Mission, Goals, and Metrics! We can help you think through the nuances of effective text messaging.


Additional Resources

Online Resources23


1.  Blog post about A/B testing:

https://mobilecommons.zendesk.com/hc/en-us/articles/205086550-How-do-I-A-B-Test-

2. Tutorial Video:

https://youtu.be/RZ7YcYZC_yo

3. Statistical Significance Calculator:

http://getdatadriven.com/ab-significance-test

4. WWF’s “What Tree Are You” Quiz:

http://www.worldwildlife.org/pages/what-tree-are-you


Thank you!

[email protected]

[email protected]

Learn From Your List: How to A/B Test Your SMS Campaigns

Mobile

Transcript of Learn From Your List: How to A/B Test Your SMS Campaigns