The Online Laboratory:

Perfecting Your Outreach

Through Email Testing

#NNOnlineLab

WHO ARE YOU?

• Ben Betz

– Online Strategy Manager, People For the

American Way

• Damien Shirley

– Account Executive, Chapman Cubine

Adams + Hussey

WHAT IS AN EMAIL TEST?

• Send multiple versions of an email

• Identical except for a single test variable

• To a random, representative sample of your

email list

• Measuring which version performs better

• Goal: Try a new approach and see if it works

VS.

WHY SHOULD I TEST?

• Every email list is different

• There are best practices, but they don’t always apply to every email list.

• You may know how your supporters respond to your emails, but not why

• If you’re not testing, you’re guessing

• You might be leaving something on the table

• Bottom line: Test to optimize your email engagement.

WHAT DO I NEED TO GET

STARTED?

• Email list

• Email platform (CRM)– Many have built-in testing modules

• Email messaging program

• Confidence calculator

WHAT SHOULD I TEST?

• Anything

• Test approaches that could improve

your email performance

– Let your data guide you

• Make your tests scalable

• Think outside the inbox

WHAT SHOULD I TEST: TESTING

IDEAS

• Sender

• Send time

• Send date

• Subject line

• Personalization

• Callout box position

• Callout box image

• Salutation

• Signer

• Length

• Tone

• Landing page

• Landing page

layout

HOW DO I TEST?

• Components:

– Control and test email versions

• Only difference: your test variable

• Only one test variable at a time

– Control and test audiences

• Random, representative samples of your list

– Bottom line: the only difference between

email versions should be the variable

you’re testing

HOW DO I TEST: TEST TYPES• A/B split test

– One control email variant & one test email

variant

– Most common

– Easiest to set up

Control Test

• Multivariate Test

– One control email variant & multiple test

email variants

– Still only one test variable

– Better for larger lists

Control Test 1 Test 2

HOW DO I TEST: TEST TYPES

• Pre-Launch Test

– Send your control and test variant(s) to

small segments of your email list

– Launch the winning version to the rest of

your list

– Takes more time to set up

Control TestTo Full List

HOW DO I TEST: TEST TYPES

HOW OFTEN SHOULD I TEST?

• Always

• Or, as often as you can

• Continue until you get significant

results

HOW DO I KNOW WHAT WON?

• Check your data

• Use a confidence calculator

–Determine statistical significance

–Calculate your confidence interval

HOW DO I KNOW WHAT WON?

• What if neither variant wins?

–No significant difference: go with

your preference

–Mixed results: prioritize your metrics

WHAT NOW?

• Apply your results

• Plan your next test

• Build a testing library

• Embrace failure

• You’re not done yet: retesting

– No set schedule

– Let your data guide you

WHAT TO TEST: MEMBER DRIVE

• Test idea: Test including a member

drive versus standard renewal

messaging.

• Test type: A/B split test

• Control: Standard renewal messages.

• Test: Messages included the member

drive goal for new or renewed

members.

Standard Renewal Member Drive

WHAT TO TEST: MEMBER DRIVE

• Results: – The member drive test had a higher

clickthrough rate.

– The member drive also test generated more revenue per donor.

– Using a confidence calculator, we determined that both of these results were statistically significant.

• Recommendation: – Include member drives in future renewal

campaigns.

WHAT TO TEST: MEMBER DRIVE

WHAT TO TEST: EMAIL LAYOUT

Personalized Member Invoice

vs. Standard Email Format

Invoice was the winner

WHAT TO TEST: CALLOUT BOX

IMAGE

Member Card had a higher

clickthrough rate

Thermometerhad a higher response

rate (winner)

WHAT TO TEST: SENDER

• Testing campaign-y names as sender

versus names of staff

– Stop GOP Witch Hunts vs. NAME, PFAW

– Overturning Citizens United vs. NAME,

PFAW

– PFAW Membership Director vs. NAME,

PFAW

WHAT TO TEST: MESSAGE

• Message A vs. Message B

• Roll out a message to smaller segment

of the email list to test its strength prior

to full-list launch

QUESTIONS?