Metrics and Testing
-
Upload
neworganizingist -
Category
Technology
-
view
950 -
download
2
description
Transcript of Metrics and Testing
![Page 1: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/1.jpg)
1
Stats are Sexy!Metrics and Testing
That’s HOT!
![Page 2: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/2.jpg)
2
2
“The only truth is what actually happens.”
The number of people who did the thing you wanted them to do. Give.
Act. Volunteer. Come to Get Together.
![Page 3: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/3.jpg)
3
3
The data
Oh NO! Let’s take a closer look...
![Page 4: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/4.jpg)
4
4
What are all these metrics?
• It’s easy to get lost in the result metrics, some common calculations are:• Delivery rate• Open rate• Click-through rate• Conversion rate• Tell-a-Friend rates
![Page 5: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/5.jpg)
5
5
Delivery
Delivered Emails, not yet opened
Delivery Rate =
Number of emails Received/
Total emails Sent
![Page 6: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/6.jpg)
6
6
Opening the email
Opened email
Number of emailed members who opened at least 1 email/
Total Emails Sent
Open Rate =
![Page 7: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/7.jpg)
7
7
Beware, Preview Pane Affects Open Rates
![Page 8: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/8.jpg)
8
8
Opened email
![Page 9: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/9.jpg)
9
9
Click-ThroughClick-through Rate =
Number of emailed members who clicked on at least one link/
Total Emails Sent
![Page 10: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/10.jpg)
10
10
The ResponseResponse Rate =
Number of emailed members who participated in the campaign /
Total emails Sent
![Page 11: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/11.jpg)
11
11
Tell-a-Friend Usage
Number of emailed members who told at least 1 friend/
Total emails Sent
Tell-a-Friend Usage Rate =
![Page 12: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/12.jpg)
12
12
Tell-a-friend Recruitment
Number of Friends who joined a center/
Total Emails Sent
Tell-a-Friend Recruitment Rate =
![Page 13: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/13.jpg)
13
13
Which Rates are the most important?
• Conversion Rates are the best measure of a campaign’s immediate success
• Click-through rates can be good indicators of how members are interacting with your messages
• Open rates are deceiving for short term data, but if measured over several years can become interesting. They also are good for normalizing other data.
![Page 14: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/14.jpg)
14
14
So, what can you do with all these statistics?
![Page 15: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/15.jpg)
15
15
This is a numbers game …… do the math
• If you want 10 people to TAKE ACTION, then:• (at least) 20 people need to click to the
advocacy page• (at least) 30 people need to open your e-mail
message• (at least) 100 people need to receive your
message
![Page 16: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/16.jpg)
16
16
This is a numbers game …… do the math
• If you want 10 people to DONATE, then:• (at least) 50 people need to click to the
donation page• (at least) 250 people need to open your e-
mail message• (at least) 1,000 people need to receive your
message• Yes, your results may vary…dramatically!
![Page 17: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/17.jpg)
17
17
It all starts with the list• Good direct marketing is all about the
list• It starts with the list• It ends with the list• It is all about the list
• But building a good list requires you to be good at everything else: your work, copy, layout, timing, etc.
![Page 18: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/18.jpg)
18
18
Typical response rates• Nothing is typical, really
• Big lists usually = lower response…
• …and smaller lists usually ROCK
• But if you make us…
![Page 19: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/19.jpg)
20
20
Making the most of the numbers to get the most out of your online program…
Segmenting and Testing
![Page 20: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/20.jpg)
21
21
Segmenting is…
…dividing your e-mail list when running campaigns to learn more, personalize, and generate higher results.
![Page 21: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/21.jpg)
22
22
Segmentation example• Donors
• People on direct mail list with e-mail
• E-mail subscribers
Recipients
Amt of Gifts
# of Gifts
Avg gift
Resp. rate
3,500 $2,290 40 $57.75 1.13%
Recipients
Amt of Gifts
# of Gifts
Avg gift
Resp. rate
7,250 $3,385 92 $36.79 1.26%
Recipients
Amt of Gifts
# of Gifts
Avg gift
Resp. rate
7,000 $775 26 $29.81 0.37%
![Page 22: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/22.jpg)
23
23
Before segmenting…
![Page 23: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/23.jpg)
24
24
After segmenting…
![Page 24: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/24.jpg)
25
25
Other segmentation strategies
• Giving level • Campaign-based
• Donors who have taken action• Donors who have not taken action• Non-donors who have taken action• Non-donor, non-action-takers
• Recency• Source• One-time and lapsed online donors• Active and inactive non-donors
![Page 25: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/25.jpg)
26
26
Testing …
…Once you’ve determined how best to segment your particular list, you can begin to test what works within each of these segments.
![Page 26: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/26.jpg)
27
27
Testing• Open rates
• Time of day, day of week• Subject lines
• Straightforward subject lines vs suggestive ones. • Your action will help a child eat today
vs.• How can you help the orphans?
• To Mention The Ask Or Not? • Looking ahead, more animals need your help in 2007
vs.• Last chance for your tax-deductible gift to animals
• Subject lines that in different ways ask for a renewal of commitment.
• Renew your commitment for 2007vs.
• Your membership is expiringvs.
• Renew today—and save even more puppies
![Page 27: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/27.jpg)
28
28
Testing, continued• Click through rates
• Premiums• Offer • Signer• Short or long copy• Link placement
• Conversion rates • Short vs. long page• Images and insets• First things first! • Giving levels
![Page 28: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/28.jpg)
29
29
How to create a good test...
• Pick one variable you want to test• Keep everything else the same• Randomly 50/50 split the population of
the campaign• Compare the results from each test
group using a “significance calculator”• “Significant” results mean the results
can be repeated
![Page 29: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/29.jpg)
30
30
Testing Do’s and Don’t’s
• DO• Ask a question and design a test to answer it• Select a RANDOM sample of your list• Test ONE thing at a time• Keep everything the same except for the thing
you are testing• Check for statistical significance• Make sure your list segments are large
enough to produce meaningful results• Understand that some test results are only
applicable to the message you were testing
![Page 30: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/30.jpg)
31
31
Testing Do’s and Don’t’s
• DON’T• Assume that test results apply across lists
or list segments• Bite off more than you can chew• Read too much into the numbers• Assume that every test will yield
statistically significant results
![Page 31: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/31.jpg)
32
32
Testing Gone WrongVersion Description
Messages Sent Open Rate
Click Through Rate
Unique Signatures
Response Rate (unique sigs)
Better Than Control?
Statistical Significance?
CONTROL Regular Email 21,059 20.90% 8.40% 1,395 6.62%
Test 1
Masked URLs – used hyperlinked text instead of spelled out URL 20,953 22.70% 10.60% 1,645 7.85% Y Y
Test 2
I n Line URLs –URLs are in the line of text, not separate paragraphs 20,377 21.30% 8.30% 1,318 6.47% N N
Test 3
Longer subject line – Everything is same as control, except the subject line is a longer version 20,419 20.30% 7.90% 1,262 6.18% N N
Test 4
Header Images- Everything is same as control, except header image does not have a picture of candidate 20,784 21.50% 8.30% 1,391 6.69% Y N
Test 5No Header Image – header image is removed 21,734 22.50% 8.80% 1,524 7.01% Y N
Test 6
Large Button - take action button in the upper right hand corner is larger 21,014 21.10% 8.50% 1,431 6.81% Y N
Test 7
Move On Style- email is stripped of header image and all graphics 21,591 22.60% 10.70% 1,804 8.36% Y Y
Test 8Short Email – middle paragraphs are removed 21,146 23.90% 8.90% 1,483 7.01% Y N
Test 9
Greeting - lead in to the email says “Dear Friend” instead of Dear First Name 20,702 21.30% 7.90% 1,328 6.41% N N
Total 209,779 21.82% 8.84% 14,581 6.95% Y N
![Page 32: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/32.jpg)
33
33
What Happened?
• Too many variables
• Tried to answer every question at once
• The result:• Hours of work to set up 10 segments• Data was messy and difficult to decipher• We didn’t learn much about our list
![Page 33: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/33.jpg)
34
34
What We Should Have Done
• Picked one variable
• Had a clear question to answer
• Split the list in half or thirds instead of in tenths
![Page 34: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/34.jpg)
35
35
And the winner is….
• Subject Line Test• What metric should be used to evaluate a subject
line?• Which is the winner?
![Page 35: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/35.jpg)
36
36
And the winner is…
• Open Rate• Two-way tie
Message Emails SentOpen Rate
Click Through Rate Actions
Response Rate
Bush's Budget: Guns Not Butter 1,000 28.50% 13.00% 53 5.30%
What's At Stake 999 31.90% 15.50% 52 5.20%
Action Alert: Defend 100 Hours Agenda 999 27.70% 14.80% 49 4.90%
Action Alert: Organize agaist Bush's budget 1,000 31.90% 18.20% 51 5.10%
Organize Against Bush's Budget 1,000 28.80% 16.70% 54 5.40%
Total 4,998 30.60% 16.90% 259 5.18%
![Page 36: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/36.jpg)
37
37
And the winner is…
• Link placement test• Which is the most important metric?• Which segment is the winner?
TestMessages Sent Open Rate
Click Through Rate
Unique Signatures
Response Rate
Below the first paragraph 20,419 22.70% 10.60% 1,645 8.06%
Below the third paragraph 20,567 21.30% 8.30% 1,318 6.41%Below the fifth paragraph 20,489 20.30% 7.90% 1,262 6.18%
![Page 37: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/37.jpg)
38
38
And the winner is…
• Click Through Rate
• Below the first paragraph
![Page 38: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/38.jpg)
39
39
And the winner is….
• Short Email vs. Long Email• What metric(s) should you look at?• Which message is the winner?
Messages Sent Open Rate
Click Through Rate
Response Rate
Long Email 95,917 20.90% 7.50% 5.62%Short Email 96,953 21.10% 7.30% 5.48%
![Page 39: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/39.jpg)
40
40
And the winner is…
• Click through rates and response rates• Trick question – the results are statistically
insignificant.• You can’t always eyeball test results – be sure to
test for statistical significance
Messages Sent Open Rate
Click Through Rate
Response Rate
Long Email 95,917 20.90% 7.50% 5.62%Short Email 96,953 21.10% 7.30% 5.48%
![Page 40: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/40.jpg)
41
And the winner is…
• Which campaign is more successful? Why?
41
CampaignMessages Received
Open Rate
Click Through Rate Donations
Amount Donated
Average Gift
Response Rate
Issue #1 63,101 42.06% 0.49% 102 $6,578 $64.49 0.16%Issue #2 62,653 41.48% 0.62% 125 $4,960 $39.68 0.20%
![Page 41: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/41.jpg)
42
And the winner is…
• Arctic campaign most likely to raise money (fewer donors, but higher average gift)
• Public lands campaign most likely to get engage donors (more donors, lower average gift).• Depending on the goals of the campaign
(raise a lot of $, or get more people to give), both are a success.
42
![Page 42: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/42.jpg)
43
And the winner is….
• From line testing – by list segment
• What should the organization do moving forward?
43
From "ORGANIZATION"
From "Kathy, ORGANIZATION Difference
Super Activists (10+ actions) 41.20% 42.10% 0.90%Activists (1-10 actions) 35.40% 38.10% 2.70%Inactive (0 actions) 10.10% 9.40% -0.70%Newbies (less than 2 months on the list) 25.40% 26.40% 1.00%Donors 40.70% 43.50% 2.80%
![Page 43: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/43.jpg)
44
And the winner is…
• Even thought the inactive segment had a lower open rate with the from line “Kathy, Organization” all of the other segments saw improved open rates.
• You could segment out the inactive folks and continue to send them messages with the old from line, but it’s probably not worth the effort.
44
![Page 44: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/44.jpg)
45
45
Digging Deeper:5 Questions you could ask
![Page 45: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/45.jpg)
46
46
Question 1
Which group of people on my email list is the most likely to take action, tell a friend, or donate?
![Page 46: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/46.jpg)
47
47
Question 1, Sample answerSource 1 Source 2
Members in Group 2,403 4,014
Activists* 59.84% 29.45%
Donors 5.45% 2.72%
TAF 35.41% 9.12%
Super activists 4.99% 2.99%
Members who opened an email in 2005
57.26% 55.23%
Members who clicked 70.41% 54.53%
![Page 47: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/47.jpg)
48
48
Answering Question 1
• Two words: Source Codes• Suffix on imports or on any link someone may
click to come on your list, starts with ‘?source=’• Advocacy campaign (?source=advo1)• Viral Flash game or movie (?source=funny_movie)• Banner ads (?source=yahoo_banner)• Paid source (?source=care2)• Donation campaign (?source=moneybags)
• Source codes are attached to a member so you always know where they originally came from
![Page 48: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/48.jpg)
49
49
Answering Question 1
• Each cell in the table is a member select
• Query members who have the source code you want AND have donated, taken action, etc.
• The results can help you compare how valuable names from various sources are to your organization
![Page 49: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/49.jpg)
50
50
Question 2
How do I know how fast my list is growing?
![Page 50: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/50.jpg)
51
51
Answering Question 2
• List Growth Rate: # new sign-ups/# of deliverable addresses• Shows % list growth over a period of time (month,
3-months, year)
• List Hurdle Rate: # removals/# of deliverable addresses• Shows % of list that leaves over a period of time• # removals tells you how many new names you
need to get before you achieve net growth
![Page 51: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/51.jpg)
52
52
Question 3
How come my advocacy response rates are so low?
![Page 52: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/52.jpg)
53
53
Question 4
How do I find out if I’m sending too much email?
![Page 53: Metrics and Testing](https://reader033.fdocuments.us/reader033/viewer/2022051312/547b4bfbb37959582b8b4cfe/html5/thumbnails/53.jpg)
54
54
Question 5
How valuable is an email address?