Learning from Obama: Redesigning Analytics
description
Transcript of Learning from Obama: Redesigning Analytics
![Page 1: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/1.jpg)
Learning from Obama: Redesigning Analytics
![Page 2: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/2.jpg)
In 2008, Obama campaign raised $750 million Would not be enough in 2012
The fundraising challenge
Not impressed.
$750 million?
![Page 3: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/3.jpg)
The fundraising challenge But fundraising was proving more
difficult in 2012 than in 2008 President less available for fundraising
events In early campaign, we saw average online
donation was half of what it had been in 2008
We had to be smarter, and more innovative
![Page 4: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/4.jpg)
Overview A/B testing in Obama’s digital
department
Lessons learned Don’t trust your gut Foster a culture of testing Make it personal
![Page 5: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/5.jpg)
Winning with A/B Testing
![Page 6: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/6.jpg)
Example: Draft language
![Page 7: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/7.jpg)
What impact can testing have?
version Subject line donors moneyv1s1 Hey 263 $17,646v1s2 Two things: 268 $18,830v1s3 Your turn 276 $22,380v2s1 Hey 300 $17,644v2s2 My opponent 246 $13,795v2s3 You decide 222 $27,185v3s1 Hey 370 $29,976v3s2 Last night 307 $16,945v3s3 Stand with me today 381 $25,881v4s1 Hey 444 $25,643v4s2 This is my last campaign 369 $24,759v4s3 [NAME] 514 $34,308v5s1 Hey 353 $22,190
v5s2There won't be many more of these deadlines 273 $22,405
v5s3 What you saw this week 263 $21,014v6s1 Hey 363 $25,689v6s2 Let's win. 237 $17,154v6s3 Midnight deadline 352 $23,244
ACTU
AL ($
3.7m)
IF SEN
DING AVG
IF SEN
DING WORST
$0$2$4
Full send (in millions)
$2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average
Test sends
![Page 8: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/8.jpg)
Test every element After testing drafts and subject lines, we
would split the remaining list and run additional tests Example: Unsubscribe language
Variation Recips UnsubsUnsubs
per recipient
Significant differences in unsubs
per recipient
578,994 105 0.018% None
578,814 79 0.014% Smaller than D4
578,620 86 0.015% Smaller than D4
580,507 115 0.020% Larger than D3 and D4
![Page 9: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/9.jpg)
No, really. Test every element.
We also were always running tests in the background via personalized content
![Page 10: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/10.jpg)
Then, keep testing Example: how much email should we
send?+6 emails per week
![Page 11: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/11.jpg)
The results Campaign raised over one billion dollars
Raised over half a billion dollars online Over 4 million Americans donated
Recruited tens of thousands of volunteers, publicized thousands of events and rallies
Did I mention raising >$500 million online? Conservatively, testing probably resulted in
~$200 million in additional revenue
![Page 12: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/12.jpg)
Lessons
Click icon to add picture
![Page 13: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/13.jpg)
Don’t Trust Your Gut
Lesson #1
![Page 14: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/14.jpg)
Don’t trust your gut We don’t have all the answers
Conventional wisdom is often wrong Long-held best practices are often wrong You are not your audience
There was this thing called the Email Derby… If even the experts are bad at predicting a
winning message, it shows just how important testing is.
![Page 15: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/15.jpg)
Experiments: Ugly vs. Pretty We tried making our emails prettier
That failed So we asked: what about ugly?
Ugly yellow highlighting got us better results
![Page 16: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/16.jpg)
Foster a culture of testing
Lesson #2
![Page 17: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/17.jpg)
The culture of testing Check your ego at the door
Use every opportunity to test something
Compare against yourself, not against your competitors or “the industry” Are you doing better this month than last
month? Are you doing better than you would have
otherwise?
![Page 18: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/18.jpg)
When in doubt, test In a culture of testing, all questions are
answered empirically
Example: With the ugly yellow highlighting, we worried about the novelty factor Maybe highlighting would only work for a
short time before people started ignoring it (or being irritated by it).
We decided to do a multi-stage test across three consecutive emails
![Page 19: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/19.jpg)
The ugly highlighting experiment
Experimental design:
Determined through this test that novelty was indeed a factor
Group 1Group 2Group 3Group 4Group 5Group 6Group 7Group 8
Group 1Group 2Group 3Group 4Group 5Group 6Group 7Group 8
Group 1Group 2Group 3Group 4Group 5Group 6Group 7Group 8
First Email Second Email Third Email
![Page 20: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/20.jpg)
Use data to make the user experience more personal
Lesson #3
![Page 21: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/21.jpg)
Big data ≠ big brother
Testing allows you to listen to your user base Let them tell you what they like Whether through A/B testing or behavioral
segmentation, optimization gives them a better experience
Usually, the interactions that are the most human are the ones that win
![Page 22: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/22.jpg)
Be human! In general, we founds shorter, less
formal emails and subject lines did best. Classic example: “Hey”
When we dropped a mild curse word into a subject line, it usually won “Hell yes, I like Obamacare” “Let’s win the damn election” “Pretty damn cool”
![Page 23: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/23.jpg)
Good segmentation: behavioral Behavioral segmentation was much more
effective than demographic segmentation Donor vs. non-donor High-dollar vs. low-dollar Volunteer status What issues do people say they care about?
After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results
![Page 24: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/24.jpg)
Experiments: Personalization Adding “drop-in sentences” that reference
people’s past behavior can increase conversion rates
Example: asking recent donors for more money
Added sentence significantly raised donation rate Confirmed in several similar experiments
…it's going to take a lot more of us to match them.
You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today?
…it's going to take a lot more of us to match them.
Will you donate $25 or more today?
![Page 25: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/25.jpg)
Conclusions
![Page 26: Learning from Obama: Redesigning Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062813/56816696550346895dda7e87/html5/thumbnails/26.jpg)
Conclusions Test everything, especially your gut
instinct
Foster a culture of testing
Use data to make it personal