Targeting Compiled Formatted v8

1

TARGETING

TARGETING ...................................................................................................... 1

INTRODUCTION TO TARGETING ...................................................................... 3

OVERVIEW .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 SUCCESSES AND CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 RECOMMENDATIONS FOR THE FUTURE .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

BUILDING TURNOUT PROJECTIONS ................................................................ 8

SYNOPSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 SUCCESSES/CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

BUILDING YOUR ORGANIZATION ................................................................... 14

BUILDING GOALS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 BUILDING YOUR ORGANIZATION – TURF BASED SEGMENTATION .. . . . . . . . . . . . . . . . . . . . . 20 VOLUNTEER RECRUITMENT – MICHIGAN CASE STUDY .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 VOLUNTEER FLAKING – VIRGINIA CASE STUDY .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

PREDICTIVE MODELING I: MICROTARGETING ............................................. 27

TYPES OF MODELS: DEFINITIONS AND USES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 SUPPORT MODEL .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 TURNOUT MODEL .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 UNDECIDED MODEL .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 PERSUASION MODELS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 CONTACT RATE MODEL.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 MODELING TECHNIQUES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 SUCCESSES AND CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 RECOMMENDATIONS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

PREDICTIVE MODELING II: THE MODELING PROCESS .................................. 40

SYNOPSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 GATHER, CHECK, AND CLEAN THE DATA .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 ASSEMBLING THE DATASET .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 ESTABLISH ESTIMATION PROCEDURE .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 CONSTRUCT THE MODEL .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

PREDICTIVE MODELING III: BUILDING THE DATASET AND CHECKING

RESULTS ........................................................................................................ 49

SYNOPSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 GATHER PAID ID DATA .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 BUILDING PAID ID DATASET .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 CHECKING RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 SUCCESSES AND CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 RECOMMENDATIONS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

TARGETING PERSUADABLES .......................................................................... 61

TARGETING: INTRODUCTION

2

SYNOPSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 SUCCESSES & CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 RECOMMENDATIONS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 INDEX .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

TARGETING DEMS: BUILDING A GOTV UNIVERSE ........................................ 68

SYNOPSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 GOTV INDEX .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 SUCCESSES AND CHALLENGES .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

TARGETING NEW REGISTRANTS ................................................................... 75

GEOGRAPHIC TARGETING .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 TARGETING INDIVIDUALS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 MEASURING PROGRESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

UNIVERSE IMPLEMENTATION ........................................................................ 78

PERSUASION .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 SPECIFICS TO IMPLEMENTING A SPORADIC UNIVERSE .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

POLLING ......................................................................................................... 81

OVERVIEW .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 POLLING LIMITATIONS .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 SAMPLE BIAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 SURVEY STRUCTURE .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

PAID MEDIA TARGETING ............................................................................... 89

PURPOSE .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 WHY IT’S IMPORTANT, WHAT IT DOES.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 SPECIFIC APPLICATIONS OF PAID MEDIA TARGETING .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 TYPES OF DATA USED .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 ALLOCATING SPEND ACROSS MARKETS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 THINGS TO KEEP IN MIND .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 GIS AND PAID MEDIA TARGETING .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

INTRODUCTION

WHAT IS TARGETING?

Targeting is the study and practice of ccampaign has limited time, staff and monetary resources, which tanational leadership) direct at different uses

Specifically, targeting staff do several things:

• Work with pollsters, paid media, technical consultants, national leadership and mail vendors to help

synthesize electoral information into useful information for voter contact

• Define voter contact universes for registration, persuasion, early vote motivation and get out the vote

(GOTV)

• Establish statewide and local goals based on estimated turnout projections and voter contact requirements

• Build predictive models to help understand how likely v

turnout, how likely voters are to be undecided and how likely voters are to be persuadable

rates

• Provide analytical consulting generally for important campaign resource and investment decisions

Microtargeting - the statistical practice of good targeting, but it is only one component

WHY DO WE NEED TARGETING?

There is an immense amount of information available databases. Targeting staff provide an analytical joint between contact decisions derived from that information

3

INTRODUCTION TO TARGETING

OVERVIEW

is the study and practice of campaign resource optimization for the purpose of earningcampaign has limited time, staff and monetary resources, which targeting staff (in consultation with state and

different uses based on quantitative analysis of campaign information

, targeting staff do several things:

Work with pollsters, paid media, technical consultants, national leadership and mail vendors to help

synthesize electoral information into useful information for voter contact

ct universes for registration, persuasion, early vote motivation and get out the vote

statewide and local goals based on estimated turnout projections and voter contact requirements

Build predictive models to help understand how likely voters are to support, how likely voters are to

turnout, how likely voters are to be undecided and how likely voters are to be persuadable

Provide analytical consulting generally for important campaign resource and investment decisions

the statistical practice of predicting voter behavior based on sample evidence but it is only one component.

ING?

There is an immense amount of information available on a campaign, typically from different sources and an analytical joint between these various information sources and critical

derived from that information.

TO TARGETING

for the purpose of earning votes. The rgeting staff (in consultation with state and

formation.

Work with pollsters, paid media, technical consultants, national leadership and mail vendors to help

ct universes for registration, persuasion, early vote motivation and get out the vote

statewide and local goals based on estimated turnout projections and voter contact requirements

oters are to support, how likely voters are to

turnout, how likely voters are to be undecided and how likely voters are to be persuadable and contact

Provide analytical consulting generally for important campaign resource and investment decisions

based on sample evidence - is critical to

lly from different sources and these various information sources and critical voter

During 2008, the campaign used a regional targeting dcampaign regions. Campaign consultants provided Microtargeting models and the technology team provided data infrastructure and specific targeting products. provided support for national paid media analysis

FINDING THE BEST TARGETS

A target is a voter or group of voters market for a commercial, a city for an eventthe amount of average work it takes to get an additional vote. On this,

• How effective will our contact be? If we’re trying to persuade, what % of vote

after we talk to them?

• How efficient will our contact be? Of voters that we contact for persuasion, what % will

(so we don’t waste resources)?

In the case of a persuasion phone call done by a volunteerlikely to participate (efficient):

Voter is…

Minimally Persuadable

Highly Persuadable

EFFECTIVENESS


4

During 2008, the campaign used a regional targeting desk system – with one targeting desk for each of the six Campaign consultants provided Microtargeting models and the technology team provided data

infrastructure and specific targeting products. In addition to this, a “Paid Media Targeting Desk”or national paid media analysis.

is a voter or group of voters that the campaign can direct its resources at – a voter for event – to try and earn more votes. A target is good or bad

takes to get an additional vote. On this, there are two considerations

contact be? If we’re trying to persuade, what % of voters will change their minds

contact be? Of voters that we contact for persuasion, what % will

In the case of a persuasion phone call done by a volunteer, a good target is both persuadable (

Voter is… Unlikely to Vote

Minimally Persuadable Bad Target

Highly Persuadable Decent Target

EFFICIENCY

with one targeting desk for each of the six Campaign consultants provided Microtargeting models and the technology team provided data

ing Desk” (not shown)

a voter for a phone call, a A target is good or bad depending on

there are two considerations:

rs will change their minds

contact be? Of voters that we contact for persuasion, what % will actually turn out

is both persuadable (effective) and also

Likely to Vote

Decent Target

Good Target

EFFICIENCY


5

This framework is useful for all forms of voter contact – the campaign is looking for targets that we can most affect and that we can do so most cheaply (contact rate would provide another efficiency screen). These targets will provide the campaign the most additional votes.

PRODUCTS – WHAT WE DID

The list below shows a series of products the Targeting Department provided in 2008, in the order provided:

Targeting Need Targeting Analyses / Products

Campaign Strategy Statistical Summaries of each state – including breakdown of the electorate and estimated requirements to win (through registration, turnout, persuasion)

Turf Cutting / Staffing State regional breakdowns and staffing needs by region

Turnout Projections Statewide turnout projections, projections by sub-demographic groups (age, race, gender)

Persuasion Persuasion universes for mail and field, state by state reporting by universe over time, polling and messaging analysis

Registration Statewide registration goals, geographic and individual registration targeting

Early Vote Early vote target universes for mail and field, state by state reporting by county and sub-demographic, early vote effectiveness analyses

GOTV (Get Out the Vote) GOTV universes for mail and field, Election Day targeting program, paid & volunteer phone allocation

Paid Media Market Analysis, Cost Efficiency Analysis, Media Buy Allocation

Other Vote Goals, Scheduling and Trips analysis

SUCCESSES AND CHALLENGES

The OFA 2008 targeting program successfully developed a framework through which future Democratic campaigns will better use their information for more effective and efficient voter contact. While there were several challenges, the information gathered through the process will serve as valuable information for future campaigns.

SUCCESSES

The targeting team designed, built and implemented dozens of national and state products, but the most important success was organizational. There are several key examples:

• Targeting program provided a new framework for data-driven decision making at the national and state

level

• Extended robust targeting information for the first time to non-Field departments - including Mail, Paid

Media (TV and Radio), Scheduling and Voter Protection. These departments should rely on

comprehensive targeting to drive their planning going forward

• Coordinated three departments – Regional Targeting Desks, Technology Team Support, and

Microtargeting Consultants – to provide unified targeting products for 23 battleground states


6

Specifically, there were several strategic successes as well:

• Designed new methods and practices for predictive modeling, universe construction, database

management, reporting and general campaign analytics

• Designed effective voter contact universes across all battleground states for 4 distinct programs -

registration, persuasion, early vote and GOTV - based on extremely accurate predictive models

• Provided guidance and targeting analysis for national and state leadership that helped important strategy,

field and investment decisions

CHALLENGES

There were several challenges in 2008 as well, both organizational and strategic:

Organizational:

• Because targeting was so new to campaigning, targeting literacy was lower across the organization than

anticipated, which compromised communication at several occasions

• Some tension over the ownership of targeting decisions – vis-a-vis pollsters, targeting desks, mail

vendors, state leadership. Future efforts should clearly delineate exact roles in the information and

decision-making process

• Did not initially anticipate the importance of the relationship between Regional Targeting Desks and state

Voter File Managers, who are ultimately responsible for targeting implementation

Strategic:

• Data management (table creation, code management, etc.) was much harder than we predicted, and led to

serious delays in Microtargeting model roll-outs.

• Initially neglected to put in place adequate quality control processes for model creation.

• Department never developed a successful message testing program, which would have greatly improved

the persuasion program.

• Persuasion model rolled out slightly late, which prohibited sufficient testing and examination.

RECOMMENDATIONS FOR THE FUTURE

The OFA 2008 targeting program was the first time that a Democratic campaign had tried to build an internal targeting program on such a large scale. Overall, the effort was very successful, and should be incorporated into future campaigns at an equivalent (or larger) scale.

There are many lessons from 2008, but several highlights:

• Targeting department should make a very early effort to build foundational knowledge for state and

national leadership – both to enforce buy in and also to make targeting a more inclusive and state-

informed process

• Also, targeting desks should build early, solid relationships with state voter file managers, and they

should be included in the targeting process. Several reasons:

o Voter File Managers will ultimately be the most familiar with a state’s voter file and numbers

o Voter File Managers will be responsible for implementing targeting information


7

o Voter File Managers will be able to provide valuable context and consultation for targeting

• Quality control should be formally built into processes – especially during model preparation and creation

• Department needs to agree on a data architecture and database management system early – defined by

clear roles, management and outputs

• Targeting information can – and needs to – be extended for application beyond field. This would include:

o Paid Media

o Travel

o Staffing

o Messaging (both national and state)

• Targeting program needs to start as early as possible. When field hits the ground, they will need to have a

general understanding of the electorate, plans for staffing, and initial goals. The sooner the targeting

program can roll-out the better.

• Doing targeting work most requires three important skills: deep analytical experience, communication,

and stress tolerance. Future hires should demonstrate all three.

8

BUILDING TURNOUT PROJECTIONS

SYNOPSIS

WHAT A TURNOUT PROJECTION IS

A turnout projection is an estimate of the number of people who will vote in a particular election. To build turnout estimates, we use previous turnout results, population change information, registration estimates and other factors.

WHY TURNOUT IS IMPORTANT

The turnout estimate is important for two reasons: firstly, estimated turnout will be used to derive the statewide vote goal:

�� 51%

Throughout the campaign, resource decisions will be derived based on the statewide vote goal (and current estimates against it). More importantly, however, turnout is important for calculating local vote goals: the state-wide vote goal will be broken down across turfs down to the most local areas – from which voter file managers will build voter contact and volunteer recruitment goals, all contingent on the original state-wide turnout goal.

SOME CONSIDERATIONS

Turnout goals are notoriously difficult to predict – it’s nearly impossible to know how the electorate will respond to unforeseen changes in the coming months and how excited voters will be on Election Day for each state. For this reason, turnout estimates should serve a guideline for goals and resource allocation, but state and national leadership should keep in mind that turnout estimates are best – but not precise – calculations.

WHAT VARIABLES GO INTO TURNOUT PROJECTIONS

In 2008, the following variables were used to project turnout state-wide turnout:

Variable Description

Prior Turnout • Raw Turnout

• Turnout as % of the Voting Eligible Population

Population / Demographic

Changes

• % Change in total population

• % Change by Demographic Group

Competitiveness / Interest in

Downballot Races • Interest in downballots – e.g., Congressional, Senate, State

House races

Voter ID Laws • Recently introduced laws that may repress some votes (true

effect is unknown)

Size of Field Operation • Registration and GOTV effects

TARGETING

PROCESS FOR ESTIMATING TURNOUT

Turnout will vary based on changes in two parameters

• Change in the number of eligible voters (Voting Eligible Population)

• Changes in participation (% of Voting Eligible Population that participates)

Below is an example for Nevada:

RAW TURNOUT AND TURNOUT

2004

93,360

0

200,000

400,000

600,000

800,000

1,000,000

2004 Turnout Population

Growth

Change in Population

TARGETING: BUILDING TURNOUT PRJOECTIONS

9

ESTIMATING TURNOUT

in two parameters:

Change in the number of eligible voters (Voting Eligible Population)

Changes in participation (% of Voting Eligible Population that participates)

TURNOUT AND TURNOUT % PROJECTION: NEVADA 2008

2008

56.4% 0.5%2.3% 1.3% 0.4%

2008 Adj. for

Growth

Caucus Reg &

Turnout

Early Vote

Programs

Registration

Programs

National/

State Trends

Change in Participation Change in Population

A 2008

2008 Proj

60.9%0.4%

National/

State Trends

2008 Projected

Turnout


10

Population growth for the voting eligible population was constructed using census data: below is an example for

Nevada:

2004 VEP 2008 VEP Growth in VEP

Change in % of Electorate

Total 1,458,208 1,635,648 12%

Af-Am 104,591 132,636 27% 0.9%

Hispanic 173,078 220,899 28% 1.2%

First Americans 35,323 41,204 17% 0.1%

Other 55,003 69,455 26% 0.6%

White 1,090,213 1,171,454 8% -2.7%

White, Male 542,328 583,863 8% -1.2%

White, Male, 18-34 145,377 157,198 8% -0.3%

White, Male, 35-59 263,220 279,975 6% -0.8%

White, Male, 60+ 133,732 146,689 10% -0.2%

White, Female 547,885 587,592 7% -1.5%

White, Female, 18-34 144,708 155,318 7% -0.4%

White, Female, 35-59 254,713 269,842 6% -0.9%

White, Female, 60+ 148,464 162,431 9% -0.2%

Population and demographic changes can give a baseline turnout estimates (assuming no changes in participation rates from 2004):

NEVADA: BASELINE TURNOUT ESTIMATES

Demographic

2008 Voting

Eligible

Population

(Thousands)

% of Total

2008 VEP

% Point

change from

2004

Total

projected that

will Vote in

2008

% of VEP

that votes

(from 2004)

% of the

Electorate in

2008

Total 1,677,150 100.0% 993,273 59.2% 100.0%

White / Other 1,343,532 80.1% -2.7 786,926 58.6% 79.2%

Af/Am 124,592 7.4% 0.4 66,218 53.1% 6.7%

Hispanic 209,026 12.5% 2.3 107,143 51.3% 10.8%

18-24 182,570 10.9% -1.8 83,060 45.5% 8.4%

25-44 614,186 36.6% 0.4 342,423 55.8% 34.5%

45-64 605,240 36.1% 2.2 414,819 68.5% 41.8%

65+ 284,450 17.0% -0.2 185,957 65.4% 18.7%

* Estimates from Current Population Survey (Census)

** Totals may vary slightly due to rounding

Estimating changes in participation is somewhat more subjective. In the case of Nevada, targeting staff identified four factors likely to drive higher turnout:

• Caucus turnout: voters who participated or registered for the caucus would cause marginally higher

participation rates amongst this group

• Early vote: Field driven GOTV and early vote interest would both enable and enforce higher turnout


11

• Registration: Field driven registration programs would directly increase the number of available

participants

• National trends: Generally higher identified interest in the election (e.g., higher African American

turnout)

For Nevada, the targeting team estimated strongly increased participation amongst African Americans and Hispanics – both due to increased competitiveness, and the historic nature of the race. These participation estimates were corroborated by paid IDs indicating very high enthusiasm amongst these two groups. On Election Day, both these groups turned out at very high rates.

NEVADA: ESTIMATED CHANGE IN PARTICIPATION RATES

(% of VEP by Demographic group – 2004 to 2008)

Total % of VEP: 2004 % of VEP: 2008 ∆ 2008 - 2004

Total 56.9% 61.1% 4.2%

Afam 51.0% 62.4% 11.4%

Hispanic 39.6% 50.2% 10.6%

First Americans 30.9% 39.5% 8.6%

Other 61.5% 66.0% 4.5%

White 60.8% 63.5% 2.7%

White, Male 59.1% 61.7% 2.6%

White, Male, 18-34 57.8% 60.2% 2.4%

White, Male, 35-59 55.4% 58.1% 2.7%

White, Male, 60+ 67.6% 70.3% 2.7%

White, Female 62.5% 65.3% 2.7%

White, Female, 18-34 62.1% 64.7% 2.6%

White, Female, 35-59 61.2% 63.8% 2.6%

White, Female, 60+ 65.1% 68.2% 3.1%

SUCCESSES/CHALLENGES

COMPARING PROJECTIONS TO ACTUAL RESULTS

In ten of the twenty-two states we had made official projections for by 9/18/08, our projections were within 5% of

the actual results. In twenty of the states, our projections were within 10% of the actual result. In most states, we

over-projected the turnout number. There are two main reasons projections were not exact:

• We assumed an over-optimistic level of electoral excitement vs. 2004, especially in states that previously

had strong turnout

• In many states, the data was incredibly imperfect. State Voter Files, for example, carried large amounts

of voters on the rolls that may have moved or were deceased.


12

The chart below shows how accurate each state’s turnout projections as of 9/18/2008 were relative to the actual turnout numbers.

We under-estimated turnout in three states: GA, NC and VA. There are a few possible reasons:

• We underestimated our registration efforts (and total registration)

• We underestimated the turnout percentage of African Americans in these states, which turned out at

especially large margins for early vote in NC and GA

We significantly over-estimated (> 7.5%) in five states: AK, ND, OR, WA and WI. All these states began as competitive states, but closed the race as very non-competitive states (for the presidential election). Programmatically, however, it made more sense to plan based on more conservative estimates that assumed a competitive race through Election Day.

All these examples serve to highlight the difficulty of providing good turnout estimates. In the future, states should be mindful of current information to re-update turnout estimates as needed.

The following page contains a breakdown of the different turnout estimates used compared to 2008 actuals (current December 2008):

8.0%

6.0%

7.3%

-2.0%

5.6%

6.9%6.1%

2.9%

4.2%3.5%

5.7%

3.5%

2.2%

4.9%

-5.1%

9.7%

1.7%

15.8%

5.0%

-3.1%

13.5%

8.3%

-7.5%

0.0%

7.5%

15.0%

AK CO FL GA IN IA ME MI MN MO MT NV NH NM NC ND OH OR PA VA WA WI

2008 Estimated Turnout - 2008 Actual Turnout (% of VEP)

13

State 2008

Unofficial

VEP

Turnout %

Actual

Turnout

2004

VEP

Turnout

Rate

2008

Minus

2004

Turnout

Atlas Obama

State

Snapshots

Obama

8/26

Estimates

Obama

10/30

Estimates

Atlas Obama

State

Snapshots

Obama

8/26

Estimates

Obama

10/30

Estimates

United States 61.6% 131,313,171 60.1% 1.50%

Alaska 67.9% 326,197 69.1% -1.20% - 313,223 351,270 350,034 - 4.0% -7.7% -7.3%

Colorado 69.4% 2,401,349 66.7% 2.70% 2,338,313 2,293,266 2,419,342 2,419,427 2.6% 4.5% -0.7% -0.8%

Florida 67.1% 8,390,744 64.4% 2.70% 8,298,257 8,490,558 8,788,180 8,792,218 1.1% -1.2% -4.7% -4.8%

Georgia 61.1% 3,924,303 56.2% 4.90% - 3,513,095 3,845,485 3,974,911 - 10.5% 2.0% -1.3%

Indiana 59.3% 2,751,054 54.8% 4.50% - 2,795,934 2,901,115 2,901,052 - -1.6% -5.5% -5.5%

Iowa 69.7% 1,536,820 69.9% -0.20% 1,560,170 1,581,286 1,627,723 1,649,484 -1.5% -2.9% -5.9% -7.3%

Maine 71.1% 731,163 73.8% -2.70% - 768,659 - 778,945 - -5.1% - -6.5%

Michigan 68.6% 5,001,766 66.6% 2.00% 5,143,737 5,001,008 5,143,654 5,143,422 -2.8% 0.0% -2.8% -2.8%

Minnesota 77.9% 2,910,369 78.4% -0.50% 2,894,093 2,982,702 - 3049881 0.6% -2.5% - -4.8%

Missouri 68.0% 2,925,205 65.3% 2.70% 2,863,312 2,946,437 3,030,470 3,030,254 2.1% -0.7% -3.6% -3.6%

Montana 65.9% 490,109 64.4% 1.50% - 523,159 516,313 516,313 - -6.7% -5.3% -5.3%

Nevada 57.5% 967,848 55.3% 2.20% 1,001,221 993,273 999,622 1,055,687 -3.4% -2.6% -3.3% -9.1%

New Hampshire 71.0% 710,970 70.9% 0.10% 711,194 720,829 723,932 746,167 0.0% -1.4% -1.8% -5.0%

New Mexico 59.6% 830,158 59.0% 0.60% 818,283 881,702 851,974 880,470 1.4% -6.2% -2.6% -6.1%

North Carolina 65.8% 4,310,789 57.8% 8.00% - 3,993,040 4,049,999 4,311,419 - 7.4% 6.0% 0.0%

North Dakota 64.8% 316,621 64.8% 0.00% - 356,265 350,050 350,008 - -12.5% -10.6% -10.5%

Ohio 66.3% 5,650,000 66.8% -0.50% 5,969,520 5,759,734 5,967,149 5,967,189 -5.7% -1.9% -5.6% -5.6%

Oregon 67.4% 1,827,864 72.0% -4.60% - 2,070,175 2,036,017 1,906,878 - -13.3% -11.4% -4.3%

Pennsylvania 63.9% 5,992,384 62.6% 1.30% 5,893,350 6,143,426 6,144,883 6,185,964 1.7% -2.5% -2.5% -3.2%

Virginia 67.4% 3,723,260 60.6% 6.80% 3,568,654 3,472,085 3,604,305 3,772,458 4.2% 6.7% 3.2% -1.3%

Washington 66.8% 3,036,878 66.9% -0.10% - 2,968,539 3,114,384 3,114,371 - 2.3% -2.6% -2.6%

Wisconsin 72.5% 2,983,417 74.8% -2.30% 3,300,000 3,138,378 3,208,804 3,233,774 -10.6% -5.2% -7.6% -8.4%

2008 Turnout Estimates vs. Actual Turnout

BUILDING YOUR ORGANIZATION

There are two distinct ways to construct your organization’s goals

• Win-based goals: the movement needed within the e

• Capacity Goals: the work that th

Both of these are used to develop a program. Areas of discrepancy exist where there is a difference between what we estimate the program needs to do to win and what we estimate the program can feasibly do with its resources. In cases where a program doesn’t have enough capacity to hit its voter contact goals, we say there is a projected voter contact deficit.

The ideal field plan will be a marriage of these two processes. The field program should and work needed to achieve your electorate vote goalshas to do so.

UNDERSTANDING THE ELECTORATE

Within the electorate, there are three ways through which a voter contact program can gain more votes: register voters, turnout voters, and persuade voters

In a competitive state, the voter contact program will be responsible for getting additional votes through these three channels.

14

YOUR ORGANIZATION

BUILDING GOALS

s to construct your organization’s goals – win-based goals vs. capacity

movement needed within the electorate to win the election

he work that the field program expects to do with its available resource

e are used to develop a program. Areas of discrepancy exist where there is a difference between what we estimate the program needs to do to win and what we estimate the program can feasibly do with its

program doesn’t have enough capacity to hit its voter contact goals, we say there is a

ge of these two processes. The field program should provide the organization achieve your electorate vote goals while being cognizant of the amount of resources that it

ECTORATE

Within the electorate, there are three ways through which a voter contact program can gain more votes: register turnout voters, and persuade voters:

In a competitive state, the voter contact program will be responsible for getting additional votes through these

YOUR ORGANIZATION

capacity-based goals:

with its available resources

e are used to develop a program. Areas of discrepancy exist where there is a difference between what we estimate the program needs to do to win and what we estimate the program can feasibly do with its available

program doesn’t have enough capacity to hit its voter contact goals, we say there is a

provide the organization while being cognizant of the amount of resources that it

Within the electorate, there are three ways through which a voter contact program can gain more votes: register

In a competitive state, the voter contact program will be responsible for getting additional votes through these

TARGETING: BUILDING YOUR ORGANIZATION

15

ESTABLISHING VOTE GOALS

Your overall vote goal is simply 51% of your expected turnout

�� 51%

Once you have that number, the objective is to determine the breakdown of those votes. Vote goals cannot be established until you have your electorate clearly broken down and a good understanding of where the different groups of voters exist.

Initially, basic assumptions need to be applied to each of these voter segments. As the cycle goes on, the targeting team will use modeling from paid IDs to estimate support, but some level of assumptions will be needed to start. For each segment of the electorate, two assumptions are needed: support and turnout.

Support assumptions can come from three sources: polling, ID samples, and previous cycle exit polling Turnout assumptions are best gathered from historical data: how did that similar universe turnout in the last election?

Once these assumptions are agreed upon, each segment of the electorate should be multiplied by those assumptions to gain an understanding of the current vote levels of the electorate. Below is an example:

INDIANA 2008 ESTIMATES

INITIAL SUPPORT AND TURNOUT ASSUMPTIONS

Universe

Total Unregistered

People Expected

Registration Total Voters

On Nov 4 Expected Turnout

Expected BO

Support Obama Votes

BO Base currently on voter file 1,016,099

Frequent Voting 744,307 90% 83% 555,997

Sporadic Voting (no new reg) 271,792 50% 85% 115,512

JM Base currently on voter file 993,159

JM Base 993,159 81% 10% 80,347

Total Unregistered People 1,259,508 19% 236,552

Unreg'd Af. Am. 139,607 20% 27,921 69% 95% 18,303

Unreg'd Youth 406,455 25% 101,614 65% 75% 49,537

Unreg'd Nontargets 713,445 15% 107,017 75% 50% 40,131

Non Base Voters currently on voter file 1,542,769

Persuasion Voters 370,000 75% 50% 138,750

Non-Persuasion Voters 1,172,769 69% 50% 404,605

Total 3,788,579 77% 48% 1,403,181

1,436,052

Difference -32,870

These initial assumptions leave a 32,870 person vote deficit.

Here, you adjust the turnout and support to account for movement: what is going to be the effect of your campaign and field program? What turnout do you need? What support do you need? Here, you must decide the focus of your field program, and you have three main options, all highlighted on the above chart:

• Increase turnout among the sporadic voters, through voter education efforts and GOTV programs.

• Increase support among the persuadable voters, through voter contact and conversations.

• Increase the total number of new registrants through extensive voter registration efforts.


16

Other programmatic options – increasing turnout among likely voters, or increasing support among your sporadic base – show a diminishing return for the effort required. The above three options are the “low hanging fruit” for your field program.

To return to the previous example, here is the same electorate, but with the highlighted areas changed to reflect programmatic goals:

INDIANA 2008:

TURNOUT AND SUPPORT GOALS

Universe

Total Unregistered

People

Expected

Registration

Total Voters

On Nov 4

Expected

Turnout

Expected

BO

Support

Obama

Votes

BO Base currently on voter file 1,016,099

Frequent Voting 744,307 90% 83% 555,997

Sporadic Voting (no new reg) 271,792 60% 85% 138,614

JM Base currently on voter file 993,159

JM Base 993,159 81% 10% 80,347

Total Unregistered People 1,259,508 22% 270,836

Unreg'd Af. Am. 139,607 30% 41,882 69% 95% 27,454

Unreg'd Youth 406,455 30% 121,937 65% 75% 59,444

Unreg'd Nontargets 713,445 15% 107,017 75% 50% 40,131

Non Base Voters currently on voter file 1,542,769

Persuasion Voters 370,000 75% 58% 160,950

Non-Persuasion Voters 1,172,769 69% 50% 404,605

Total 3,822,863 76% 51% 1,467,542

1,436,052

31,491

Now, Indiana’s total estimated votes are 1,467,542 - higher than the vote goal. We have adjusted values in three areas:

• Sporadic turnout from 50% to 60%: Our field goal is to increase sporadic base turnout by 10%.

• Persuadable support from 50% to 58%: Our field goal is to increase support among the persuadable voters

by 8%.

• New registrants – Af-Am and Youth - from 20% to 30% and 25% to 30%, respectively

The balance between these three options will come from qualitative assessment and reflection on the program and its capacity, and the realities of the electorate. For example, OFA ran an almost exclusive voter registration program in Georgia, a heavy turnout program in North Carolina, and a long-lasting and ambitious persuasion program in Ohio. Of course, each of these states also had goals (sometimes aggressive ones) in all other areas of program; but they were tailored to the state.

INITIAL VOTER CONTACT ASSUMPTIONS

With programmatic goals to increase turnout, support, and registration and achieve your statewide “win number,” the objective should be to analyze your resources and program and determine when and where efforts should be focused. Field leadership decisions should determine:


17

• The size of each universe your program will need to focus on (taken from vote goals).

• The program needed to achieve each goal; i.e. how many conversations with persuadable voters, how

many voter registrations to collect, how many contacts with sporadic voters.

• The basic assumptions of volunteer recruitment and production – how many doors a volunteer can knock,

how many people will answer their door, etc.

In larger states or districts, these assumptions can be broken up by “turf type” – for example, a volunteer can knock more doors in a shift in a city than they can in a rural county; but more people in rural areas will answer their door or phone.

Below is a list of assumptions provided by the Ohio field program:

Step Name Explanation

Step 1: Set Turf Type Assign each turf to a turf category which will decide its assumptions and priorities

Step 2: Universe Summary Assumptions Needed: (by Turf Category) 1. How many times do we want to touch our persuasion universe? 2. How many times do we want to touch our sporadic base universe?

Step 3: Field Assumptions Assumptions Needed: (by Turf Category) 1. Hours per volunteer shift 2. Dials per hour 3. Knocks per hour 4. Contact rate - Calls & Doors 5. Split of volunteer time on doors vs. phones 6. Voter registrations per shift

Step 4: Volunteer Capacity Assumptions Needed: (by Turf Category, by Week) 1. Number of organizers 2. Volunteer Shifts per week per organizer 3. Split of volunteer resources between persuasion, turnout and registration

Step 5: Persuasion Contacts Expectations

Assumptions Needed: (by Turf Category) 1. Undecided rate on each persuasion pass 2. Supporter rate on hard IDs on each persuasion pass 3. Expected breaking rate of undecideds at the end 4. Expected support rate of untouched voters

Step 6: Sporadic Contacts No Assumptions Necessary - Calculates the passes on our sporadic universe based on universe & volunteer capacity assumptions (steps 2 & 4)

PERSUASION PASSES:

Calculating the total number of contacts needed for the persuasion universe is more complicated than calculating the total number of contacts needed for the base universe. There are two reasons:

• Want to touch persuadable voters several times (usually a minimum of three)

• The universe is very dynamic: undecided voters from paid IDs and the base universe will be dumped into

the persuasion universe as other decided voters are pulled out.

In this regard, if you are calculating out the contacts needed for each pass, accommodate for this movement: project the decided rate in the universe after each pass, and remove those people from future passes; and anticipate the growth of the universe and the rate of identifying undecided voters outside the persuasion universe.

Below is a simplified example for Pennsylvania that at least highlights the concepts behind estimating the number of persuasion passes needed:


18

PENNSYLVANIA PERSUASION PASS CALCULATOR

ASSUMPTIONS

Persuasion Contacts Program Can Do 724,360

Touches per Persuasion Target 3

PROJECTED SUPPORT RATES ON FIELD IDS

Use Suppporter Rates from Current Field IDs

* Currently set to recontact 3Ls, 3s and 4Ls only

Pass 2 3L 3 4L 5 Other Total Recontact*

1 15% 10% 40% 10% 15% 10% 100% 60%

2 15% 15% 35% 15% 15% 5% 100% 65%

3 20% 15% 30% 15% 20% 0% 100% 60%

Best Universe

Size

364,000

CONTACTS PER PASS

Pass Contacts Per Pass % to Recontact Total Contacts

1 364,000 60% 364,000

2 218,400 65% 582,400

3 141,960 60% 724,360

Assuming the program estimates they can do 724,360 contacts, and they want to touch each persuasion target 3 times, the best universe size is going to be 364,000. This will allow them 3 passes – 364,000 on their first pass, 218,400 on the second (knocking out committed voters from the first pass) and 141,960 on the third (again eliminating committed voters):

Additional assumptions and parameters are needed to make this calculation more accurate – it needs to incorporate people that have moved, vacant houses, contact rates, etc.

After initial goals and expectations are established, it is important to continuously check and revisit progress toward those goals, and adjust to change in program. Capacity assessment is extremely similar to calculating initial capacity, but with the benefit of being informed by real data and less assumptions.


19

The chart below illustrates a projection of voter contact, using real data from the Ohio program, June 2nd through September 5th. From those values, projections of continued growth were made: this is the blue line.

CUMULATIVE AND PROJECTED CUMULATIVE CONTACTS:

OHIO FIELD PROGRAM

SUCCESSES

We were able to produce vote goals and capacity estimates which informed good programmatic decisions for a few reasons:

• Calculations acknowledged the importance of capacity and programmatic possibility; and focused less on

non-actionable vote goals.

• Capacity was consistently reassessed and adjusted, to ensure that program was on the right course.

• Assumptions and goals were flexible, and could often be adjusted by geography or timeline.

CHALLENGES

• Constant tension between creating win-based goals vs. capacity based goals – i.e., what program needs to

win vs. a realistic assessment of what it can do organizationally (regardless of goals)

• Difficulty creating voter contact assumptions. For example - what’s the effect of a touch on persuasion?

What’s our GOTV effect? Future campaigns should research 2008 data to verify these values

RECOMMENDATIONS

The 2008 field program was forced to create assumptions in many areas where data was lacking – e.g., contact rates, volunteer capacity, etc. Future campaigns should try as hard as possible to use the 2008 database as a base of information from which to build their goals.

0

200

400

600

800

1,000

1,200

7/2/2008 8/2/2008 9/2/2008 10/2/2008

Th

ou

san

ds

of

Co

nta

cts

Projected Contacts

(Cumulative)

Actual Contacts Projected Contacts


20

BUILDING YOUR ORGANIZATION – TURF BASED SEGMENTATION

QUANTIFYING DIVISION OF TURF

There are many qualitative factors to consider in making turf-cutting decisions: political factors, personnel management, local knowledge, geography and culture. This section focuses on the data-informed quantitative analysis of turf division.

The most important takeaway is that turf was in no way divided by a raw vote goal. Instead, it was divided by the quantification of field work goals. For the 2008 campaign, field work was defined in three broad buckets:

• Persuasion: the percent of the non-base which needed to be persuaded to vote for Barack Obama, in

addition to the percent of the non-base which would break for Obama despite field effort and voter

contact.

• Sporadic Base Voter Turnout: the percent of the supportive base which needed to be actively turned

out, who wouldn’t have voted without an extensive GOTV program.

• Voter Registration: the voters the campaign hoped to register in addition to organic self-registration

rates.

The vote goals portion of this document defines these goals more exactly. For turf cutting purposes, the universe definitions and distribution must be broken up by geography – if your goal is to increase sporadic turnout, how many sporadics are in each county? How many are in each precinct? The geographic criteria used to cut the turf depend on the level of turf cutting, regional or organizer. Regional turf cuts, which should always precede any subsequent layers, may be divided by county or district. Organizer turf was most often cut at the precinct level. It is essential that turf cutting never breaks up a precinct – precincts must remain the smallest geographical distinction, as Voter Files do not distinguish voters in to political geography smaller.

MAPPING AND TURF CUTTING

GIS should be used to illustrate geographic regions of demographics and natural boundaries within the area, to provide a spatial context to the quantitative analysis. Below are two examples of maps from Missouri: The first shows demographics by census block, which illustrates where geographic regions of people exist, and the second demonstrates the team-based turf that was ultimately cut in an area. Maps of turf are essential to organizers, so that they have a spatial understanding of their turf and its boundaries.


21

MISSOURI NEIGHBORHOOD TURF

CUTTING BY TARGET DISTRIBUTION

A B C D E F G

Persuasion Targets (1)

Sporadic Targets (1)

Registration Targets (1.5)

Total Field Work Persuasion +

Sporadic + 1.5*Reg % of Total Field Work

Organizers Needed Total Organizers x

% of Total Region

Total 276 335 87 742 100% 30 --

Americana 35 78 13 133 18% 5 1

Freedom 43 45 7 99 13% 4 1

Victory 65 10 2 78 11% 3 3

Change 11 80 25 129 17% 5 2

Red 76 45 11 138 19% 6 2

White 34 54 21 120 16% 5 3

Blue 12 23 8 47 6% 2 3

The chart above illustrates the thought process for cutting turf based on target distribution. Columns A through C are determined by vote goal analysis – what percentage of the non-base must be targeted for persuasion? How does that universe fall across the state? Often, early in program building, this required layers of assumptions. For example, it is often assumed that the targeted “persuasion universe” was going to be spaced geographically


22

relatively equal to the non base. For example, if there are 100 non base voters in one county, and our statewide goal is to persuade 5% of the non base, we assume that five persuasion targets come out of that county. However, in reality, if that county contained a particular demographic, such as older people, who tend to be more susceptible to persuasion efforts, it may be relatively over targeted for persuasion.

In this particular example, the field work involved in registering a new voter is weighted at 1.5 of the field work involved in targeting a persuadable voter. This is a somewhat arbitrary value for this example – assessment of the resources needed for each voter contact track varied by state and program.

TURF-TYPE SEGMENTATION

Splitting organizing turfs in to different types, which indicates the demographics and electorates of that turf, is covered thoroughly in the field portion of this document. From a targeting perspective, it is extremely beneficial for two primary reasons:

• The structure within which to adjust program and universes more precisely and provide a mechanism by

which to define that program

• The ability to neutralize factors and isolate variables: for example the ability to adjust contact rates in

urban turfs vs. contact rates in rural turfs.

Turf type assignments can be decided qualitatively or quantitatively (or a combination of both). In qualitative assignments, staff members with a strong presence on the ground assess the area and assign it to a category. Quantitative “checks” can be applied to confirm the assignment makes sense (i.e., an “urban/base” turf should have a high ratio of supporters to non-supporters).

Turf types, however, are not solely a quantitative definition of a turf. They should be designed to accommodate for and correct for organizing challenges and opportunities unique to that area. For example, rural turf and suburban turf are both relatively red, heavy-persuasion programs. However, contact rates on rural canvasses vary greatly from contact rates on suburban counties – organizing is different, despite similar demographics or targeting. These nuances allow for greater flexibility and precision.

SAMPLE TURF TYPES

Turf Type Program Focus Unique Challenges

Urban Turnout and voter registration Lower contact rates per attempt

Rural Persuasion Sparser canvasses; high contact rates

Mix Turnout, voter registration, persuasion Balance of programs and timelines

Youth Turnout and voter registration Out of date data and voter files

Suburban Persuasion Lower registration opportunity


23

VOLUNTEER RECRUITMENT – MICHIGAN CASE STUDY

Michigan, unlike the majority of the states in 2008, did not have a DNC-sanctioned primary. President-Elect Obama did not contest Michigan, and, as such, the grassroots network of Obama volunteers that was available in other states was not available to the Campaign for Change in Michigan. In order to create a volunteer network, the campaign initially built out “Tiers” of potential volunteers for the field to look to for volunteer recruitment.

BUILDING TIERS

Hard Codes and Past History

The first set of prospects came from hard data within the voter file and our donor databases. The Michigan Party had been on the VAN systems for over 2 cycles and had many people in the Voter File tagged as volunteers from previous general elections. These prospects were immediately moved into the My Campaign system. Additionally, any potential precinct captain (Michigan has had an active precinct captain program in past cycles) were immediately transferred. Finally, donors to the Obama campaign were added to the pool for additional recruitment options.

Website Leads

Additionally we had two sets of people from the web: Anyone who signed up on our website in the state of Michigan was dumped into our volunteer database. A smaller subset of those who had indicated that they wanted to volunteer were identified and prioritized.

“Soft” Prospects

The campaign also created a “blind” recruitment universe of those people who were likely supporters. These were people who were identified as a Strong or Leaning Democrat (the likely party model meant that anyone tagged in such a way was given party points by a past ID for Democrats or through some sort of activity) or were in a base area. This set was narrowed to those with good vote history, with the thought that those people would be more likely to be civically active. As the campaign continued, we used the support score to find the “cream of the crop” and narrowed to a universe of likely supporters with the highest scores.

RESULTS

In the end, the Michigan Campaign for Change had nine tiers for volunteer recruitment. These exclusive tiers were:

• Confirmed 2008 Precinct Captains from the Michigan Democratic Party

• Potential 2008 Precinct Captains from the MDP (were a precinct captain in the past)

• 2006 Volunteers

• 2004 Volunteers

• Online Volunteer Signups

• Donors

• The rest of My Campaign

• Likely Supporters with strong vote history in My Campaign

• ID-ed supporters from volunteer or staff calls or knocks

RECRUITMENT RESULTS

The way My Campaign was treated up until the end of the campaign made it difficult to accurately tease out results of calls within My Campaign. Volunteers were tagged if they were active and the tagging of refusals was something that tended to happen sporadically. Despite these flaws, broad trends emerged within this framework.


24

Coding Description % That Said "Yes" to a

Volunteer ask

Total

Sample (n)

MDPPC

Confirmed Active volunteers with the Michigan party 68% 967

MDPPC

Prospect Former volunteers from the Michigan party 49% 5,656

Obama-2006

Volunteer Active volunteers from the 2006 election 52% 4,861

Obama-2004

Volunteer Active volunteers from the 2004 election 42% 11,236

Volunteer

Prospect

Online signups who specifically expressed interest in volunteering

72% 18,315

Donors People who donated to OFA 61% 7,234

Reg Form

Collected

New registrants who registered with the Michigan Campaign for Change

50% 4,311

VFC New

Registrant

People who downloaded a registration form off of VoteForChange.com

29% 1,824

The highest volunteer recruitment gains came from online signups who specifically expressed interest in volunteering – 72% “yes” rate. Less effective were older lists and prospects – the Michigan Democratic Party’s lists of 2004, 2006, and “prospective” volunteers. In Michigan, the VoteForChange.com online signups were the least fruitful, with only a 29% recruitment rate.

In “blind” recruitment, where uncontacted voters were asked to volunteer, the model proved to be a strong indicator of volunteerism. Those scored under 65 were recruited to volunteer at 11%, while those scored above 65 were recruited at 19% - a 73% increase in recruitment.

CONCLUSIONS AND RECOMMENDATIONS

For volunteer recruitment, self-selecting prospects that approach the organization themselves are always the best recruitment choices. After that, actual volunteer activism is the best indicator of recruitment success; and recent activism is more valuable than historical. “Blind” recruitment, even well targeted to supporters, does not yield anything close to the results from online signups and previous campaign volunteer data.


25

VOLUNTEER FLAKING – VIRGINIA CASE STUDY

In Virginia, the calendar in My Campaign was used aggressively and actively to track volunteer events during the general election. Using it, we are able to see who did and did not show up. Looking at Virginia, we can see that there are relationships between certain demographics and behaviors, and volunteer commitment/flake rates.

Variable Correlation with

Flake Rate Description

Income - Wealthier volunteers were less likely to flake than non-wealthy volunteers

Team Membership - Team Leaders were less likely to flake

Previous Flake + The more someone flakes, the less likely they will show up the next time

Time 0 Flake rates remained relatively constant throughout the campaign

Age - Older volunteers were less likely to flake

Gender Female – Male +

Female volunteers were less likely to flake than their age-adjusted male counterparts

Region 0 Variance between regions did not show strong significance, save for other demographic factors already accounted for

Race/Ethnicity / African American and Latino volunteers completed shifts at lower rates.

TEAM LEADERS

Team leaders are much more likely to complete volunteer shifts, as illustrated on the chart below.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Team Member Not a Team Member, but

has Teams in the Area

No Team in Area All volunteers where there

are Teams

Total Percentages: No Shows and Completed Shifts

No Show

Complete


26

An interesting take away is that not only were team leaders less likely to flake, their flake rates decreased as the campaign went on – evidence that the organization were getting more disciplined and more committed over time. The chart below illustrates this change over time among team leaders in Virginia: their completion rate jumped from under 70% in June to over 90% in November.

RESCHEDULING VOLUNTEERS

Not surprisingly, consistent flakers continued to flake. Volunteers that failed to show up to volunteer for events were unlikely to show up for future events, even as the end of the election approached. After 5 or more flakes, without consistent showing, rescheduling volunteers was generally unproductive. For the future, data managers should drop continual flakers from their recruitment lists.

It was much easier to reschedule volunteers who showed up, and the field program took advantage of those who came in to volunteer by scheduling them for more and more shifts as Election Day approached.

AGE AND GENDER BREAKDOWN

Our volunteer base consisted of a large number of individuals across the nation. However, our largest volunteer base in Virginia was women over 35. Middle age and Senior Women provided a large base of volunteerism that we were able to work off of.

Although young volunteers appeared in the summer, middle aged and senior women increasingly made up a large portion of our volunteer base. Despite the fact that the gender was such a factor in our volunteer makeup, it is surprising that showing up to volunteer once scheduled was more determined by age than gender – older volunteers being a bit more likely to show up than their younger counterparts.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Vo

lun

tee

r C

om

mit

me

nt

Ra

te

Change in Shift Completion Rates Over Time Among Team Leaders

% Complete % no show

27

PREDICTIVE MODELING I: MICROTARGETING

TYPES OF MODELS: DEFINITIONS AND USES

There were several different types of models used in the general election. In general, with a sufficient number of IDs, a model can be built that predicts the likelihood of any voter giving a certain response to any given question. Useful models, however, are those that predict responses to questions that the campaign needs answered.

The Obama campaign ended up producing models that predicted 5 things:

• Obama support

• Turnout probability

• Probability of being undecided

• Various measures of persuadability

• Contact Rate

SUPPORT MODEL

Choosing a Dependent Variable

The Obama support score was the most widely used microtargeting score. This score gives the probability that a voter who has a candidate preference will pick Obama. Although the concept of Obama support seems straightforward enough, considerable thought went into selecting the exact dependent variable to be used. The dependent variable is the voter characteristic that the model is trying to predict.

There are a number of options for the dependent variable for support:

A. Percent that choose Obama of all voters that answer the phone (include refusals)

B. Percent that choose Obama of all voters that indicate candidate preference

C. Percent that choose Obama of all voters that choose Obama or McCain

Ultimately, option C was chosen because it provided the clearest universe for GOTV, and also because the campaign is programmatically indifferent to voters that are undecided or refuse.

The question of whether to model all Obama support (including leaners), or only strong support is less straightforward. During paid IDs, voters with candidate preference were asked strength of support – strong or lean. Voters who said that they were undecided were also asked if they were leaning toward one candidate or the other. Combined, this gave us 4 subgroups of voters:

• Strong Obama

• Lean Obama

• Lean McCain

• Strong McCain

TARGETING: PREDICTIVE MODELING I

28

Put together, this data presents two possible dependent variables for Obama support:

1. ��

�� !�� "#$�%�

2. �� !&'��

�� !&'�� !&'�� "#$�%�!�� "#$�%�

In the end, the Obama campaign chose option 1 for two reasons. First, it was statistically more predictive than option 2. Second, there was a programmatic incentive to keep lean Obama supporters in the persuasion universe, which was usually restricted to the middle support score range, approximately 35 to 65 (see Targeting

Persuadables section for more information).

Uses

The Obama support models were used in different ways during different phases of the campaign:

Phase Use

Phase 1: Capacity Building

Because the support models ranked voters from most to least likely to support Obama, they were used to prioritize field calls for volunteer recruitment, event invitations and other capacity-building activities. By calling voters with the highest Obama support scores first, the campaign maximized the number of successful contacts per attempt.

Phase 2: Persuasion Universe Exclusions

The support models were used to exclude solid Obama or McCain supporters from the campaign’s persuasion universes. In most states, voters with an Obama support score of less than 35% or greater than 65% were excluded from the persuasion universe.

Phase 3: GOTV Universe Definitions

The support score was the foundation of GOTV universes. Other targets, including hard IDs, contributors, and online signups were included in the GOTV universe as well. In general, voters with a support score of 65%+ were included in the GOTV universe. For specific sub-universes, the support score cutoff could be lowered on a state by state basis. For example, in some states it was found that Hispanics performed at 65%+ for Obama even with a support score cutoff below 65%. These sub-universes were defined and tested on a state-by-state basis in the final weeks of the campaign.

TURNOUT MODEL

Turnout models predict the likelihood that a voter will turn out in a given election. During 2008, the turnout model was based on two criteria:

• Vote history & demographics (Catalist Propensity Score)

• Self-reported turnout (From Paid IDs)

Vote history is the strongest predictor of future turnout: a voter who has voted in three of the last three general elections is much more likely to vote in the next general election than is a voter who voted in only one of the last three. The predictive power of this select is limited for states with voter files that do not include vote history, and for younger voters who were ineligible to vote in the elections in question.

The Catalist Propensity Score estimates turnout in a generic presidential election based on vote history and demographic criteria. The advantage of this model is that it incorporates out of state vote history into turnout predictions for individual voters.


29

The other component was self-reported turnout likelihood. In 2008, two questions were used:

• How closely are you following the presidential election?

• How likely are you to vote?

Previous research – including analysis of Obama primary IDs – indicates that voters overstate their likelihood of voting. However, self-reported turnout is a good relative measure of how likely it is that someone will vote. Not everyone who says that they are very likely to vote will actually vote, but those saying they are “very likely” are more likely to vote than those saying they are only “somewhat likely”, “fifty-fifty”, or “not very likely”. These responses can be modeled, producing a score that gives the likelihood that a voter would say they are “very likely” to vote.

The final Obama turnout model was a combination of the Catalist Turnout Propensity Score and self-reported turnout probability. This model uses past vote history where available, fills in the blanks where past vote history was unavailable, and reflected the increased enthusiasm for Obama among certain groups of voters.

The final score was weighted three-fourths from the Catalist turnout propensity score, and one-fourth from the self-reported turnout model. The score was intuitive and easily to understand. Voters with strong past vote history had high scores, so selects based on the turnout score were not that much different from selects done using traditional vote history screens. This weighting deserves further research.

UNDECIDED MODEL

The Obama undecided models predict the likelihood that voters will respond “undecided” to the horserace identification question. There are a number of reasons why voters say they are undecided. They may not be following the election closely enough to have developed an opinion, they may not want to reveal their true candidate preference, or they may be truly undecided. Because of this, being undecided is not synonymous with being persuadable. Undecided models were the basis for persuasion during the primary, but were not the foundation for most states’ persuasion universes during the general election. (See the persuasion model section for a discussion of the different measures of persuadability that were used).

PERSUASION MODELS

Background

Persuasion is not a directly measurable attribute, so it can be difficult to model. In order to estimate voter persuadability, the targeting team developed preliminary models by combining different attributes likely to be associated with persuadability.

In the end, most states chose not to use these persuasion score models for the definition of their field universe. However, the creation of these models provided valuable insight into the electorate and merits much future research.

The Models

In total, we examined several variables including: engagement, likelihood of shifting candidate preference toward Obama, and responses to issue questions and other candidate IDs. These responses were merged into three different models and appended to the voter file.


30

The three main persuasion models that were appended to the voter files in all battleground states were:

Model Description

Move Obama

This model predicts the likelihood a voter will shift their candidate preference in the pro-Obama direction when re-ID'ed. These shifts include moving from strong McCain to lean McCain, from lean McCain to undecided, from undecided to lean Obama, and from lean Obama to strong Obama.

This model does not differentiate different sizes of pro-Obama shifts; a small shift from undecided to lean Obama has the same weight as a large shift from strong McCain to strong Obama.

Long Form Points

A subset of Obama ID calls included a series of issue questions. In some states, ID questions were also asked for Gubernatorial and U.S. Senate races.

The issue questions included:

• Change vs. experience

• Iraq

• Pro-choice vs. pro-life

• Whether neighbors were ready to vote for an African American for President.

In general, we expect Obama voters to prefer change over experience, favor withdraw from Iraq, be pro-choice, and believe their neighbors are ready for an African American President.

The model calculates a score for each voter who received the long-form ID script. They were given one point for each pro-Obama response and one point was subtracted for each pro-McCain response. By looking at the net pro-Obama vs. pro-McCain responses, we corrected for not all voters being asked the whole set of questions. In states where a gubernatorial or U.S. Senate ID question was asked, support for the Democratic candidate was a pro-Obama response, and support for the Republican was a pro-McCain response.

The long form points for voters who received the long form issue questions were used to build a model predicting the net pro-Obama responses expected from other voters.

Persuasion Variable

B

This variable used the two variables above, as well as three others:

• Shift in candidate support over time (Move Obama)

• Response to issue questions (Long From Points)

• Level of engagement

• Candidate certainty

• Possibility of supporting another candidate This variable was calculated for the individuals for whom there was re-ID information. A model was also built to predict the Persuasion Variable B value for voters who had not been re-IDed.

TESTING THE PERFORMANCE OF PERSUASION MODELS

The goal of our persuasion program was to move voters in the pro-Obama direction. With that in mind, a “shift points” variable was defined to measure the direction and size of a voter’s shift in candidate preference.

The largest shift would be moving from strongly supporting one candidate to strongly supporting the other. A shift from strong McCain to strong Obama would be a +4 shift. A shift from strong Obama to strong McCain


31

would be considered -4. A shift from undecided to lean Obama, or lean Obama to strong Obama would be considered +1 shifts.

Because shift in candidate preference was one of the components of two of the persuasion models, we limited this test to voters whose first re-ID occurred after the persuasion models were built on August 22nd, 2008.

For each of the models we took the pool of voters re-IDed after 8/22 and split it into deciles based on each of the three persuasion scores. The large majority of voters (70%) did not shift their candidate preference when re-IDed. Because of this the average shift in points is close to zero even at the high and low ends of each score. Nevertheless, if the scores were successfully rank-ordering voters based on their likelihood of shifting in the pro-Obama direction, we would expect to see the average pro-Obama shift increase in each decile.

As can be seen from the charts below, the pro-Obama shift points do increase as each of the scores increases. The one exception is a slight drop between the 9th and 10th deciles of the MO (Move Obama) model. This is likely because the MO model predicts only whether a voter will shift in the pro-Obama direction, not the size of the shift.

-0.15

-0.1

-0.05

0

0.05

0.1

1 2 3 4 5 6 7 8 9 10

Shift Points by PVB Score

Decile

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

1 2 3 4 5 6 7 8 9 10

Shift Points by MO Score

Decile


32

UNKNOWNS

There are two main unknowns:

It is clear that higher-scoring individuals were more likely to shift in the pro-Obama direction. However, what is unknown is the degree to which the models accurately found voters most likely to be persuadable, and the degree to which we targeted the higher scored individuals and successfully persuaded them.

The second unknown is the degree to which we succeeded in keeping voters from moving in the pro-McCain direction. It is possible that many of the voters in our persuasion universe who showed no shift in candidate preference actually would have shifted in the pro-McCain direction if we had not targeted them.

Also, the persuasion score often correlates strongly with support score. Higher supporting voters are more likely to move to Obama, and some of the positive relationship between the persuasion score and movement may have been driven by higher support amongst voters with higher persuasion scores.

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

1 2 3 4 5 6 7 8 9 10

Shift Points by LFP Score

Decile


33

CONTACT RATE MODEL

OVERVIEW

Phone calls and walk attempts do not always lead to a canvass. Indeed, a substantial portion of the volunteers’ time is spent simply trying to make contact with a potential voter. Aggregated across millions of volunteers, this is a huge source of inefficiency. The contact rate model was developed to help ameliorate this inefficiency by predicting the probability of contact for every voter in the voter file. With these probabilities of contact, voter file managers could purge their lists of people with very low probabilities of answering their phone or door. Also, call and walk lists could be tiered according to the contact rate (e.g. one list would be the high probabilities of contact; the other would have the lower probabilities). Since lists were often called/walked multiple times to attempt to contact those not home in the first attempt, the second round could be prioritized on the high contact rate group.

To construct these probabilities, we used simple linear regression (a linear probability model) of canvass on a set of demographic variables and (when applicable) previous contact history. For those whom we had not previously attempted contact, we estimated the probability simply based on demographics. For those whom we had previously attempted contact, we included previous contact outcomes in the estimation.

The contact history file from VoteBuilder was a perfect dataset for this estimation. Each observation was an action (e.g. phone call, walk, paid id call, etc.). The dataset recorded the outcome of each action as “Canvass”, “Not Home”, “Busy”, “Wrong Number”, “Wrong Address”, etc...We then linked up the subject of each action (e.g. who was called) to the voter database. This provided the demographic information we used in the analysis. With this, we ran two regressions. First, we regressed a dummy variable (equal to 1 if the action resulted in a canvass) on a set of demographics (age categories, race, sex, etc.). This allowed us to extrapolate the contact rate for people whom we had never called or walked. Second, we regressed the same contact dummy variable on both demographics and previous contact results (e.g. number of total attempts, number of successful attempts, etc.). These formed the contact rates for people we had previously attempted to canvass. These estimates were repeated for calls and for walks, although similar results were found with each type of contact.

RESULTS

The following graph shows a histogram of the predicted contact rates by phone for Virginia. In this example, the average probability of contact is 18.1%. In other words, randomly calling the database should lead to a canvass 18.1% of the time. But, if one were to instead call only the upper ¾ of the contact distribution (e.g. remove people with contact rates below 10%), the average probability of contact would be 22.8%, which would lead to 26% more canvasses per phone call.


34

VIRGINIA CONTACT RATE HISTOGRAM

The most powerful indicators were: confirmed vs. unconfirmed phone, contact history (whether we’d talked to them before), count of not homes, vote history, registration date, turnout score, age, gender and race.

In order to ensure accuracy, the model was tested out of sample by comparing the predicted contact rates to the actual contact rates seen in the days after running the initial regressions. We compared the deciles of the predicted contact rate distribution to the actual contact rate distribution as seen in the following graph:

VIRGINIA CONTACT RATE BREAKDOWN: PROJECTED VS. ACTUAL (BY DECILE)

0%

2%

4%

6%

8%

10%

12%

14%

Contact Rate

Projected Contact Rate Distribution

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

1 2 3 4 5 6 7 8 9 10

Co

nta

ct

Rate

Decile

Contact Rates 10/24/08

Actual

Predicted


35

MODELING TECHNIQUES

There are a number of different modeling techniques, each with their own strengths and weaknesses. The final Obama microtargeting models were a combination of several modeling techniques. As described below, this hybrid process yields models that are more accurate than models used by any single methodology.

Regression

Regression is the most widely used and most easily understood modeling technique. In regression analysis, the dependent variable (the attribute being modeled) is defined as a function of one or more independent variables (the indicator data). When the dependent variable has only two possible values, as in Obama support yes or no, logistic regression is the preferred technique.

Strengths Weaknesses

Ease of Interpretation:

Regression analysis generates a set of coefficients for each indicator used in the model, and it is easy to tell which indicators have a positive or negative impact on the model score.

Missing Interaction:

Regression analysis generates coefficients for each indicator used. This works well for single indicators like age that have strong predictive value. Sometimes, however, the true power of an indicator is only evident in combination with other indicators. For example, income and commute time might not be strong predictors in isolation, but having both a high-income and a long commute may have strong predictive value. Simple regression analysis does not capture these interactions. Synthetic indicators can be built that combine two or more indicators. This depends on the combined indicators being intuitive enough for a human operator to have thought to build the combined indicator. It is possible to use statistical software to automatically analyze each combination of two or more indicators. The downside of this approach is that by examining each combination, analysis may run extremely slowly. In a model with 1,000 indicators, for example, it could be one million combinations.

Speed:

Regression analysis is faster than many machine-learning algorithms.


36

Segmentation

Segmentation refers to a number of algorithms that divide a list into separate segments. Generally, a list is divided based on the value of an indicator, and the resulting segments are further segmented based on other indicators. The indicator and the split point used are picked in such a way as to maximize the difference between the resulting lists.

Segmentation techniques include algorithms such as CHAID, as well as proprietary algorithms such as SPSS’s C5.0.


Speed:

Segmentation algorithms are fast compared to most machine learning algorithms.

Large number of IDs required:

Because the list is sub-divided in each step, a large set of IDs is needed in order to maintain a meaningful sample size while segmenting the list multiple times.

Interactions:

Because segments can be further segmented based on other indicators, powerful interactions between multiple indicators are automatically found.

Machine Learning

Machine learning refers to a wide variety of artificial intelligence techniques in which computer software “learns” over time. Neural networks, which simulate the operation of the human mind, and genetic algorithms that simulate the process of evolution are two examples of machine learning algorithms.


Accuracy:

Because machine learning algorithms automatically adjust themselves, they are sometimes able to generate more accurate models than can be built using more traditional techniques.

Relatively Slow Speed:

Machine learning algorithms are generally iterative processes. That is, they run the same process repeatedly, making small changes in order to continue to improve the predictions. While this is the key to the algorithms’ ability to “learn”, it also results in machine learning algorithms being slower. In many cases, a machine learning algorithm does not ever finish. Instead, it is allowed to run until it is interrupted and the current best prediction is used.

Interactions:

Most machine learning algorithms automatically find patterns in a large body of data, including complex interactions between multiple indicators.

Difficulty in Explaining Results:

The output of machine learning algorithms is generally a score giving the likelihood of the behavior being modeled. The algorithms do not generate a set of coefficients or formulas like regression or segmentation. The results of machine learning algorithms are opaque and often referred to as “black box”. That is, they produce a result, but the process of achieving that result is not easily described.


37

COMBINING METHODOLOGIES

Each of the modeling and data mining methodologies discussed above has its own strengths and weaknesses. No one methodology is best for all situations. By combining methodologies it is possible to build a model that is more accurate than what could be produced by any single methodology.

All modeling methodologies produce predictions that include some level of error. If the error is random, then combining different methodologies can have the effect of canceling out the error, yielding a final prediction that is more accurate than the prediction produced by any one of the algorithms by itself.


SUCCESSES

1. Models were accurate: The support models were exhaustively tested, and almost invariably succeeded in correctly rank-ordering the probability of voters supporting Obama. Close to the election, narrow testing was done of voters predicted to be 65%+ in Obama support score. The support scores were also tested in combination with other demographic data and past political IDs to determine if certain sub-groups could be included even if the support scores were below 65%. To a remarkable degree, the support scores held up even among fairly narrow subgroups.

2. Models were actionable: The purpose of microtargeting models is to be used by field staff. The Obama campaign field team understood and used the models in making targeting decisions. State field staffs were included in the construction and testing of models throughout the campaign, and thus were bought into them.

3. Improved understanding of persuasion: As discussed earlier, being undecided is not identical to being persuadable. The Obama campaign broke new ground in finding and modeling other measures of persuadability.

4. Voter File Updates: The campaign made an unprecedented effort to collect information for new registrants. To apply microtargeting scores to these voters, a modified set of models was built using only the enhancement data that would be available for last-minute updates. This made it possible to score every battleground state voter who was on the file as late as two weeks before the election.

CHALLENGES

1. Bad ID Data: As part of weekly testing of the ID models, some states did not perform as the models predicted. Although some change in the support levels was to be expected as voters responded to external campaign events such as conventions and debates, the models should still correctly rank order the probability of different groups of voters supporting Obama. When the models did not seem to be correctly rank ordering voters, both the model and the week’s ID calls were examined further. In all of these cases, it was determined that there were problems with the ID calls.

A further test of the quality of the ID calls was to verify the phone vendors’ data on voters’ ages. The accuracy of the age responses (when compared to age on the voter files) was used to determine how well the phone vendors were doing collecting and recording information, and how well they were doing matching the responses back to the correct individual in a household. Interestingly, just the act of asking age seems to have improved the quality of the ID responses. ID calls where age was asked were consistently closer to the modeled predictions, and showed the expected breaks based on party registration, race and age. One possibility is that vendors exercised greater supervision of their call centers when they knew that age was being asked and verified.


38

2. Models conflicting with past data: In some states, the Obama campaign further analyzed certain groups

of voters who had past ID information or likely party identifiers from the state party’s voter files. While further analysis showed that different models more accurately predicted the support level of these voters with past ID information, this improvement came at the expense of the model’s accuracy among voters with no past ID information. Because of this, the models that most accurately rank ordered all voters were used to build a GOTV universe, but it was also supplemented with voters who did not have 65%+ support scores, but who did have past ID information or likely party ratings indicating that they would be safe GOTV targets.

3. Contamination of Indicators: Where past ID information and party ratings exist, they can be helpful indicators in support models. State party’s likely party classifications are generally built from a combination of past ID questions. ID information from the 2008 cycle was excluded from these indicators because ID information from questions about Obama support could produce a model that looks surprisingly strong, but that fails to predict Obama support among voters for whom that question was not asked.

In some states, Minnesota for example, the party rating system was updated continuously based on 2008 ID results. As a result, anyone IDed for Obama was automatically classified as a likely Democrat. To guard against this contamination of indicators, the Obama campaign acquired and used an archival copy of the Minnesota party grades as they existed before 2008 Obama IDs were added.

4. Late Start: The drawn-out nominating process, followed by negotiations about which voter file provider

to use resulted in the general election modeling process starting later than originally planned. While this did not hamper our ability to produce the basic support turnout and persuasion models described above, it did mean less time for additional state-specific modeling projects

RECOMMENDATIONS

1. Aggressively monitor and test paid phone IDs. We saw wide variation in the quality of IDs from different vendors, and even from the same vendors in different states and weeks.

2. Archive copies of past IDs and party rating systems before IDs for the 2012 re-elect begin.

3. Begin early. As discussed above, the late start was a challenge for the general election modeling program. Barring unforeseen circumstances, we know that Obama will be the Democratic nominee in 2012. Work on microtargeting can begin years ahead of time. Models predicting Obama support in a direct head-to-head matchup with the Republican nominee must wait until the Republicans select their candidate. Other modeling, however, can begin much earlier.

4. Develop and automate a standard set of reports describing new models. As discussed above, the “black-box” nature of some of the more powerful modeling methodologies made it difficult to describe exactly what types of voters are being selected by a given model. The original plan in 2008 was for targeting desks to develop and run cross-tab reports describing the demographics of voters in different score ranges. The exercise of developing these reports would make the targeting desks more familiar with the models. In the end, desks did not have the capacity to develop these reports as comprehensively as hoped.

5. Develop and use a message testing framework throughout President Obama’s first term. There are four years in which to conduct controlled experiments on the best ways to reach and persuade different


39

types of voters. The Republicans did this in President Bush’s first term. We have the opportunity for research that will eclipse anything the Republicans were able to achieve.

40

PREDICTIVE MODELING II: THE MODELING PROCESS

SYNOPSIS

In this section we lay out the steps a campaign must take in order to use predictive modeling. Predictive modeling is a statistical technique that helps use data about some voters to make informed estimates about voters for which there is less information. It is easy to do basic predictive modeling, but it takes rigor and attention to detail to do accurate predictive modeling well.

This is technical topic, and each section begins with a summary for non-technical staff. The majority of each section’s content is designed for campaign database managers and targeting analysts. Points covered are:

1. Gather, Check and Clean the Data:

a. What kinds of data you will need and things to think about when preparing a data acquisition

program.

b. Technical instructions for initial data processing. This is an important step that is often

overlooked.

2. Assembling the Dataset: Ultimately the statistical work will require a single dataset which will be

assembled within a database

3. Establish Estimation Procedure. A few analyses will be used frequently, but it will not be immediately

clear which those will be

4. Construct the Model. Advice for the campaign analyst

GATHER, CHECK, AND CLEAN THE DATA

This section covers some basic points about campaign data. The most important point is that past behavior is the best predictor of future behavior. Your best donor prospects are your previous donors. Your most likely supporters are people who have demonstrated support for Democrats in past elections.

This section focuses on data quality, but campaigns should not overlook the importance of data quantity. 1000 data points that are 70% accurate will provide the same direction as 50 data points with 90% accuracy, but with more data, there is more certainty (about a factor of 4 in the margin of error). The ratio of 1000 to 50 represents the cost differential between our “paid id” phone surveys (about $1 per complete) and our polling data (about $20 per complete).

CONSIDER THE SOURCE

As they say, “Garbage in, garbage out.” Know the source of data is imperative. In 2008, this was frequently a challenge with Catalist, which had a number of variables whose source was difficult to determine. There were also challenges with census data on a county-level, versus census data on a census-block level.

TARGETING: PREDICTIVE MODELING I I

41

You should construct and continue to update a data dictionary. The archive contains various versions of the Obama data dictionary. Each dictionary contains at least a variable name and description. It is helpful also to have a field for data type.

OFA 2008 DATA DICTIONARY – SELECTED EXAMPLE

VARNAME Data Type Revised Description

AA_INC_F Integer Median earnings for the neighborhood age 16+ who are employed full-time, year-round -- African-American, female

AA_INC_M Integer Median earnings for the neighborhood age 16+ who are employed full-time, year-round -- African-American, male

AA_INC_TOT Integer Median earnings for the neighborhood age 16+ who are employed full-time, year-round -- African-American

AA2544MINC Integer Median household income for households with an African-American householder who is 25-44 years old

THE MOST USEFUL DATA IS BEHAVIORAL

Start with historical behavioral data. In 2008 we inadvertently built our first round of models without data gathered by the state parties including likely party variables. No demographic inference will ever be as predictive observations like the donations to Democratic campaigns, email sign-ups, or previous involvement with Democratic campaigns. The large archive of Obama campaign IDs will be a helpful resource for campaigns in 2008 swing states.

TRAINING AND VERIFICATION

In addition, you will have to collect new data. As with all other data, you must understand and verify the data you collect from end-to-end. For example, as above, the Obama campaign went to great lengths to verify the validity of our “paid id” phone vendor data.

We did not realize the role of training until late in the campaign. In order to match our pollsters, we asked the same questions about certainty and chance of switching to the other candidate. For example, we asked “Even

though you’re not supporting him now, what are the chances you might end up supporting [name of candidate the

voter did not pick in Q2 or Q2B]: (1) Fair chance, (2) Small Chance, (3) Just a Very Slight Chance (4) No Chance at All”. In reflecting on this question, it becomes clear how difficult it would be for a typical call center employee to code a common response, such as, “I don’t know?”, “Maybe,” or “Are we almost done here?”

In re-ID tests this question in particular acted differently when asked by pollsters versus paid id vendors. The pollsters can justify their expense because they screen the phone respondents before beginning the poll and because they reputedly train their phone survey workers more carefully. In the polling data, the stated chance of switching to the other candidate correlated with measured rates of switching. In the paid id data, the only distinction was in the most definitive answer (”no chance at all”) versus the others categories.

CHECK AND CLEANSE THE DATA

Once data is available, many are tempted to immediately ask the critical questions. However, it is a waste to make inferences from corrupted data. The first urgent questions should be about the quality of the data.

• Does it look right? Request raw counts of the main indicators such as candidate support questions.


42

• Is it consistent? Your data sources should and will contain some redundant information. Those pieces

of data should line up. For example, party registration, past candidate preferences, and current candidate

preference should correlate to a reasonable degree.

• Is it clean? Do your analysts have a test for statistical outliers and a plan for scrubbing them? How

many variables required scrubbing?

ASSEMBLING THE DATASET

Although dataset assembly is conceptually simple – all the various data sources must be pulled together in one place for use in some statistical packages – dataset assembly is easier said than done. A campaign or technical manager who wishes to marshal this process can try asking for intermediate results (various crosstabs, for example) or ask for a step-by-step plan for the dataset construction.

Before anything else, you will have to import your data into a table or set of tables. Flatter is usually better. If you’ve had academic database training, you’ve been taught to normalize everything. Resist that. Go ahead and have 10 million rows with contact type “phone” instead of contact type ID of “1”. Disk space is cheap; time spent puzzling out your variables is lost forever.

If you have any choice in the matter, tab-delimited text files are better than comma delimited. Voter data often has commas and rarely has tabs.

Complete match tables will be needed for the different ID systems. This is very important. In 2008 voters were identified by DWIDs from Catalist, Van IDs from VAN, and Cons IDs from BSD. There was a similar profusion of precinct ID schemes. Even counties are variously labeled by name, truncated name, or FIPS code.

It is helpful to standardize variable names and table names. Database variable names are introduced into code at all levels, so it is particularly important to standardize early in the process before you know what the data will be.

Example:

• To capitalize or not to capitalize?

• Underscores between words?

• Standardized marker for keys and foreign keys (such as “ID”).

• Standardized key name (such as “<table_name>_ID”).

• Name marker for data type (such as “address1_s” to indicate that address1 is a string)?

• Tables named with singular or plural word for the things in it (such as “donor” or “donors”)?

Remember to use productivity tools to the extent possible. If an appropriate development environment is not available, Excel is a good substitute in many code-construction problems. If you used rational naming conventions, Excel makes it easier to construct your data dictionary. Excel can convert your data dictionary into a SQL CREATE statement very quickly.

QUICK CHECKS

Examine the data. Make sure every variable has the range of values it is supposed to have. Make sure the strings aren’t truncated.

Not all problems are obvious. For example, in 2008 an Obama analyst recognized that it was a problem that all of our binary variables had values of 0 or 1 for every record in the voter file. Many of these variables (such as the marker for Veterans status) could not possibly have known values everywhere. On examination we discovered that MySQL converts blanks into 0s when it is importing text to integers. We needed a script to convert the


43

blanks into explicit NULL symbols in order to make the import routine set those missing variable values to NULL instead of 0.

There is no way to know in advance all of the potential sources of error in your computing environment, but you will have to find them and fix them.

CHECKING DISTRIBUTIONS

It is important to clean up the outliers in variables that are poorly behaved for modeling purposes. For example, consider the census variables that document the percentage of the population that is Native American. In most census blocks, there are zero Native Americans. In many census blocks, the number is near 0.1%. But there are a small number of blocks in the United States where the number hits 30%, 50%, or even 100%. The odds of hitting one of these heavily Native American census blocks are low. If you happen to get one of these while testing, you will get over-fitting in the model. More likely, a large test set will have a few samples with a somewhat higher Native American population (say 5%) which can lead to excessive model extrapolation in census blocks on or near reservations.

The Obama campaign used a slight variant of the standard Grubb’s test for statistical outliers. The Grubb’s test is a blind threshold on the maximum number of standard deviations from the mean tolerated in a data sample. In addition to the Grubbs test, we examined the count of cases which were deemed to be outliers and the ratio of the largest value to the mean (this task was simplified because few data had negative values). We set a rough threshold of 6 Grubb’s units, a max-to-mean ratio of 10, and 100 cases for outliers. We then made a handful of adjustments by examining the variables that were or were not labeled as having outliers, based on knowledge of the meaning of the variable. The resulting cleaning script, loeb_analy_cleaning_script.sql.tmpl, limited the range of 154 variables, and that script is available in the Obama archive.

Although it is less rigorous, a common-sense review of the data was an important part of the cleaning process. Before you pursue a rigorous outlier test, you may wish to build some basic summary stats for all your variables; average, max, min, and standard deviation. Whether you keep or discard outliers, it is important to know that they are there. For example, secretaries of state often insert incorrect birthdates, which leads to improper age information such as 200 year old voters.

USING INDEXES

You must index every variable on which you will be joining or searching. A database index works like the index of a book: if you want to know what page a certain piece of information is on, you look in the index to find all the pages where it appears; then you look through those pages to find just what you want. In your database you will often need to look up information on a voter when you have the voter ID. If the database is not indexed, then in order to find voter number “8549093” the database computer must start reading through each voter record until it either finds that one or gets to the end of them all. This is time-consuming. Once you have indexed the voter ID variable, the computer can use the index to find all the memory pages with voter IDs starting with “85490” and then look through just those to find the right record. Once indexed, it can find 1000s of records per second.

A query that is taking more than 10 minutes to run probably needs an index.

DATABASE MANAGEMENT PITFALLS

Most data planning and processing steps take longer than anticipated. As the number of fields grows, the complexity of the database and its requisite management grows exponentially. Here are several possible issues:

• Data will be bad or too old, and will require revision

• Different ID schemes will require several translation tables

• Improper indexing (or lack of indexing) will slow down query processing


44

So, proper planning means:

• Leave plenty of time to do the job

• Save the steps in case they need to be repeated

• Plan or write tests into the sequence of steps that check errors

• Review query processing to make sure the query is not taking too long

• Make each step small rather than writing one big query – it’s more ordered and computationally efficient

USING DERIVED VALUES

Be prepared to add new derived variables to your modeling dataset. Our Obama modeling work almost entirely relied on census and Catalist variables “as-is”. For example, the Catalist variable for “Number of Children in Household” ranged from 0 to 9. For analytical purposes, it became clear that a binary variable indicating whether a voter had children or not provided the vast majority of the predictive power. Similarly, binary variables for “Married with Kids” and “Has missed mortgage payments and lives near foreclosures” had explanatory value in our analytics. For their analytical success, we can expect that these derived variables would have been good explanatory factors in our modeling work, possibly helping to eliminate spurious factors and thus strengthen model reliability.

Note that to add a derived variable you will have to assemble the dataset again.

ESTABLISH ESTIMATION PROCEDURE

This is a highly technical section. The take-away for managers is that wise use of simple statistics is better than some use of the greatest statistical procedure you can’t understand. A campaign needs relatively straight-forward statistical procedures that are used and re-used often, as opposed to many new and different procedures run once. The campaign environment more-or-less demands this approach, as it can take substantial time to become comfortable with a new statistical technique. Advanced techniques are also, often, difficult to describe. An obscure list of weighting coefficients is often not useful. Our most widely used models (such as the Polling Place Capacity Tracker) were structural – wherein we put intuitive understanding into equations that were accessible to a wide audience. When that was not possible, in the cases of the support and persuasion models, we used the models to generate scores for every voter so that Voter File Managers in the field could use these models.

USING MULTIPLE MODELS

Our modeling consultant used averages of multiple models. This was widely successful for the Obama campaign.

• Observed result = Model_1 + error_1

• Observed result = Model_2 + error_2 …

If we use an average of these two models, Model_final = (Model_1 + Model_2) / 2, then the error terms will tend to cancel out. Statistical procedures operate to remove information so that the errors are noise with a mean value of zero. Consequently the error terms of the two models should not be correlated. If the random error_1 term is high for some voter, then we expect the error_2 term will be lower. In other words if there is a 10% chance of getting such a high error_1 term, there is only a 1% chance that the error_2 term will be equally high or higher.

Note that the drawback of averaging multiple models is that the final model is not transparent. The Obama campaign saw reducing errors as more important than an easy understanding of exactly how the model works.


45

CHAID MODELS

We used CHAID models to predict binary or multinomial dependent variables from a combination of continuous or discrete variables. The output of a CHAID model is a decision tree which shows which variables (and in which priority order) are most predictive of the output/dependent variable. The decision tree information is particularly helpful in exploratory statistics, but a CHAID model can also be used as a model in its own right.

REGRESSION MODELS

We used multiple regression models almost exclusively for modeling work. They can sometimes be deceptive in analytical work, as illustrated below.

Here we see a chart constructed from 10s of thousands of observations. Each data point is the mean certainty value for all people of a given age who were surveyed as leaning toward McCain. The higher the numerical certainty value the more likely they are to switch to the other candidate. In other words, the McCain leaners who were near 70 years old were more certain they would vote for McCain than were the younger and older McCain leaners.

FIVE THINGS TO BE AWARE OF ABOUT REGRESSIONS

Step Comments

1. Check the residuals The difference between the data and the model should not be correlated with the independent variables. This check passed.

2. Use the residuals as a

dependent variable

In order to control for some effect that isn’t interesting, you can first do a regression using that variable. We ran a regression of mean McCain certainty against Income in order to remove income effects. We then plotted the residuals of that regression against age in order to see this U-shape curve once the correlation between age and income correlation is removed. As it turns out, the graph above is entirely an income effect: young and very old McCain leaners had lower incomes and more uncertainty than older McCain supporters.


46

Step Comments

3. Square terms and

higher

If you want to look for quadratic or higher order effects and your statistics package doesn’t do that for you automatically, you can construct variables like age squared and use them as variables in a multiple regression.

4. Interaction terms Just like you can construct a polynomial term, you can construct interaction terms, like age * income.

5. Logistic regression

You should not really do a multiple regression with a binary or ordinal dependent variable. For these you should use a form of non-linear regression like logistic (binary) or multinomial logit (ordinal). You can convince yourself that straight linear regression is wrong for these cases by going back to step 1: try a linear regression on a binary variable and look at the resulting error distribution. Does it look like a zero-mean normal distribution?

DISCRIMINANT ANALYSIS

The Obama campaign made heavy use of a modified version of discriminant analysis (DA). Standard DA finds the best high-dimensional plane to separate two populations. In practice, DA would allow us to map some huge number of variables (cat owner, retired, low income, high education, rural area, etc.) into a best guess of the plane in those dimensions that best separates McCain and Obama supporters.

Our modified DA algorithm is in the archive. Here is how it works:

Step Description

1. Pick groups

Decide which pair of groups to compare. For example, we first compared voters under 30 to voters over 35 and found reasonable differences, including the biggest difference being the likelihood of being retired.

2. Run program The analysis is a library of python code run within the SPSS environment.

3. Add in normalizing

factors

The output of the program is a spreadsheet with the mean values for the two groups for each variable. The spreadsheet also gives the significance value (F statistic) and whether it is significant at the p < 0.01 level. The normalizing factors were the mean values for each variable computed once from a random set of 200,000 voters drawn from the whole US. We would then add 4 columns by hand:

• Normalized Mean for group 1. Group1 mean/US mean

• Normalized Mean for group 2. Group2 mean/US mean

• Absolute difference of normalized means. Abs(norm mean 1 – norm mean 2)

• Variable description. The description of the variable

4. Sort and read tea-leaves

Sort by significance first and the abs_norm_diff (descending) second. It is also reasonable to sort in descending order by the F statistic. The first sort puts large significant differences on top; the second sort puts the most distinct significant differences on top. Once the spreadsheet is sorted we would review the variables to put together a story of what makes the two groups different.

5. Do follow-up analysis

Do follow-up analysis. Use the story of the differences to drive analysis. If the story is correct then it will play out in standard statistical tests. It usually was, but follow-up analysis occasionally invalidated hypotheses.


47

We used this analysis to great effect. For example, we found that white suburban undecided women were more likely to have a variety of retail credit cards than white suburban McCain-leaning women. Once validated, this result led to a pilot project to put Obama kiosks in suburban malls. That program proved ineffective due to staffing (only supporters approached and spoke to a lone volunteer sitting at a kiosk).

Another analysis showed that Obama supporters were far more likely to live near public transit and/or commute to work via public transit. That analysis led to inexpensive GOTV advertising in buses. In retrospect it is obvious that our supporters were more likely to be riding buses (even when those buses were driving through swing areas), but statistics alerted us to this possibility, and confirmed its validity.

CONSTRUCT THE MODEL

It is important for campaign managers to ask a lot of questions about analyses. Obama managers were involved in monitoring the accuracy of analytical models and results. They asked a lot of questions and the analytical side of the campaign was much stronger as a result.

BUILDING MODEL SCORES

The distribution should fall into the 0 – 100 range to make implementation easy. If necessary, rescale the scores - a straight re-scaling of the scores saves time and changes nothing about distribution. If a straight re-scaling squeezes too much of the model into a tight range, then re-work the distribution (set values below some threshold to zero and above some threshold to 100) in order to stretch out the functional range.

Do not give deciles only. Voter file managers frequently have to use model scores to find the best set of people (for example the 50,000 most persuadable). They also often have to balance multiple pairs of models (for example, find the 50,000 most persuadable in the 35 to 65 support score range with the highest turnout model scores). These tasks are impossible with deciles only. In addition, nodes in a distribution can result in misleading breaks in the deciles.

Provide a histogram of the model scores. Voter File Managers should not have to hack at the break points in the model scores to find out where to set the limits. Give them a picture of the distribution up front. Sophisticated Voter File Managers will want to look at model scores within universes (for example, how is persuasion score distributed among suburban women). Be prepared to answer those questions.

ALWAYS QUALITY CONTROL FIRST

This is particularly difficult with analytical results. In the Obama campaign the targeting group would generally review results internally in a peer review process. In a campaign without those resources, remain skeptical of results and remember that social science rarely provides complete clarity. Think of every alternative explanation you can before and get help from your peers before you report the result.

TRACKING AND COMMUNICATION:

Track what you do:

It is everybody’s job to be skeptical and ask every question they can before acting on your results. You must be completely transparent. Analytical results in particular often depend on sub-selects and computed values. You will waste time if you try to go back and figure out what you did after the fact. Keep a complete record as you go.

Include a summary at the top:

Get straight to the bottom line. What are you going to say and why is it useful?

Explain your graphs more completely than you think you have to:


48

What is the X-axis; what is the Y-axis; what does each data point or bar or line represent? What would you say if you were showing the graph to somebody in person?

Include complete information for replication:

As an appendix to your write-up it’s helpful to include the selects and calculations you used.

49

PREDICTIVE MODELING III: BUILDING THE DATASET AND

CHECKING RESULTS

SYNOPSIS

In this section we cover the paid ID data collection and processing used in building our support models. We used multiple phone vendors to issue short, 5 to 10 question surveys to lists of voters selected from the voter file. Points covered are:

• Gather Paid ID Data. What data you will need, and how we selected voters to contact.

• Building Paid ID Dataset. What data we pulled together in order to build Obama support models. All data available in Obama archive.

• Checking Results: Performing post-estimation quality checks

GATHER PAID ID DATA

Databases are perishable products. This section accordingly covers our weekly process for gathering new data: choosing a sample of voters for the following week’s calls and managing data from the previous week’s calls. However, we start with a reminder that not all old data have gone bad.

USE WHAT YOU HAVE

You already have the most valuable and reliable data you can get. Your state party has IDs from previous elections, and your donors and volunteers are all supporters. In a small race in a 2008 battleground state, the database of millions of Obama and McCain field and paid IDs will put targeting at an initial advantage.

CHOOSING A SAMPLE FOR PAID IDS

When choosing a sample for paid IDs, there were four primary roles:

• Choosing the list of voters to sample (Targeting)

• Cutting the lists of voters from the voter file (Data)

• Providing quality control on paid ID results (Targeting)

• Managing scripts and relationships with the paid vendors (Targeting)

We ultimately used three sample types plus slight modifications: broad spectrum, middle support, and balanced.

TARGETING: PREDICTIVE MODELING I II

50

Sample Description

Broad Spectrum

Broad spectrum samples were essentially a straight random sample with the following adjustments. Attempted to invert the contact rate drop-off function by increasing the number of young voters in the sample. Eliminated all but a few African Americans (about 500 complete per state). Adjusted the samples with re-IDs and special requests from the states as described below. We used broad spectrum samples three times: initial set; the week after the Republican convention; and the week after the last debate.

Middle Support

Most weeks sample was drawn from the middle of the modeled support range. These samples had a small number of completes (~500) in each state for African Americans, modeled Obama supporters (65+ support score) and modeled McCain supporters (under 35 support score). After these and the re-ID and special request samples described below, all remaining samples needed for the state were drawn from the middle support score range (35 to 65) with informal efforts to correct for under-representation of young voters in the completed calls.

Balanced

The last paid ID sample inverted the contact rate function for a 3 x 3 x 2 demographic grid: age (18 to 35, 36 to 59, 60+) X race (African American, Hispanic, other) X gender. This was a process of counting the number of completes and attempts in a state; and looking at the proportion of each demographic bucket within the voting population in a state. The code for these operations can be found in the archive files.

Re-ID samples were drawn from the set of completed calls that occurred more than a week prior. We generally used about 10,000 samples in order to achieve 1000 completes. In large states, we generally attempted 2000 completes. In practice it was not helpful to re-ID more than twice (ie. 3 ID calls). The small segment of our re-ID dataset who we re-ID’d 4+ times may be interesting for additional analysis.

Note, one can also re-ID a specific population. We did this once for Jewish voters in Pennsylvania and Florida after McCain sent anti-Obama mailers on the issue of Israel. These specific re-IDs showed a sharp drop in Obama support among younger Jewish sporadic voters.

Finally, we added special samples in states as requested. Many states requested larger Hispanic samples. We also sampled Native American voters in Nevada.

Data was collected weekly until 3 weeks before the election, when we moved all vendors and data processing to nightly data dumps. Daily data processing required more automation with fewer errors, so this shift was necessarily an iterative one requiring cooperation from our phone vendors.

QUALITY CHECKING PAID ID RESULTS

The Obama campaign used several methods to check the quality of data from paid ID vendors. These included monitoring calls, comparing collected data to known data, comparing across vendors within a state, and observing cross-vendor re-ID movement. We also computed cost-per-completed calls for the various pricing scheme and learned that hourly rates were best.

When the paid ID call program began in June, our targeting team members signed up for shifts to randomly monitor calls. We also seeded phone lists with phone numbers of targeting team members. Each team member gave a fake name for seed lists, to alert members that they were receiving a paid ID call. This demonstrated major differences in vendor quality, as some vendors deviated noticeably from scripts, or failed to ask for the specific person they were supposed to speak to.

When we received phone data from vendors, we compared each respondent’s age on the voter file to the age they reported in the phone interview. When we plotted the histogram of the difference in those values, the better vendors had tight distributions with mean values of 1 year (leading us to learn that the age variable in our voter


51

file had not been updated for 8 months). The vendors who had not pushed to speak to the right person had wider age distributions with 3 modes: one peak near zero and two peaks near plus and minus 20 years. Further examination revealed that these 20-year peaks were due to multiple generations in the same household.

We canceled our contracts with the poorer vendors or demanded improved performance.

Over the next several weeks we rotated vendor assignments so that each state was sampled by several vendors. These tests were less revealing than the age comparisons and the direct observations.

After vendor quality assessment, we calculated cost-benefits resulting from the various vendors’ pricing schemes. We found that hourly pricing was clearly more cost-effective than completed-call pricing. Hourly vendors had no incentive to keep bad data and their callers had no incentive to hurry a call. Simple hourly pricing gives the right incentive structure to ensure that the caller will ask to speak to the right person.

Finally, we looked at cross-vendor re-IDs (individual voters who had been contacted by different vendors) in mid-August. We looked at the probability of the second ID being different from the first as a function of who the first vendor was. Two vendors stood out as having high movement rates, but once the worse of these two was removed, we found no problem with the second vendor.

BUILDING PAID ID DATASET

Our paid ID dataset consisted of several sources of data for each respondent. Included were the information from the call itself, their commercial data (from Catalist), census data (several sources), political data (from the voter’s state Democratic Party) when available, and recent model scores. This dataset enabled the targeting desks to answer a broad range of questions for their states.

PROCESSING THE RAW PAID ID DATA FROM THE VENDORS

Each vendor’s results arrived in a tab-delimited file on Sunday night (or, at the end of the campaign, every night). Vendors uploaded weekly or daily survey files to the campaign’s ftp server. For ease of processing, the files had pre-arranged codes (vendor name, state, date) built into the file names in a pre-arranged order and format. Files that did not comply were returned, and the vendors quickly adopted naming standards. When we moved to daily processing, the vendors continued to send the weekly counts (so that we could perform consistency checks). The daily files were named in the identical manner with an extra “_d” to mark them.

Before processing began the survey questions and allowed response codes were stored in the database (wells) on which the processing code would run. After the initial month, changes in the survey questions became less frequent but continued. The processing script ensured that every question was answered by its allowable response codes only. This check helped stop data alignment errors in which answers to one question were contained in the column for another. To build further on this error check, one could insert question ID and/or expected response frequency into the response ID, depending on the patience and malleability of the phone vendors. This system does not work if any one vendor cannot conform to the required format.

The processing script checked the FTP site for new files every 2 hours. When a new file was found and that file passed basic validation, a database record was created for the file. In this way we could back out and re-process any bad files that we discovered post-automation. If the file did not pass validation, an error alert was emailed to the vendor and the responsible technology staff.

Each processed call resulted in one set of records in a heavily normalized data model. That schema is in wells MySQL paidids_import database.


52

KEY ELEMENTS

Element Description

Survey The highest level table of the Survey node. It simply identifies a unique survey by name

SurveyQuestion This is the master list of all survey questions that were asked. Each survey question is a separate record with the text of the question.

SurveyQuestionResponse

Each SurveyQuestion has a different set of responses, represented in this table. The SurveyQuestionResponseCode is the numeric value that was assigned to that answer choice on the survey that was called. FlatColumn and FlatValue are used when generating a normalized view of the data. The FlatColumn describes which data the question is trying to determine. The FlatValue is the normalized value for the given response code. The multiplier column is not used.

SurveyQuestionOrder Joined to the Survey table, this table specifies the order in which specific questions were asked in a given Survey.

SourceFile

Any file that was processed successfully gets a record in this table. Each file must be identified by the SurveyID of the script used in the call. Week is identifier of the weeks in the program, growing larger towards election day. DateCalled is a more specific measure of time. BadData is a flag for sets of IDs marked by the campaign as suspicious and not used in further analysis. Daily is a flag for whether the file was a daily or weekly file. Daily files should be a subset of the weekly file; weekly files should compose the complete set of IDs and daily files would only add duplicates.

PhoneCalled

Every phone number called is given a unique record in the PhoneCalled table. Phone and SourceFileID should form a unique identifier on this table. Universe describes the population this person belongs to. Call center, operator, duration, attempts are specified by the vendor and not standardized. Every record should have a DispositionCode that joins to the DispositionCode table.

DispositionCode All of the Dispositions you can get from a call, including things like “Not Home” or “Disconnected”.

PersonResponse

This table has the person’s identifying information, in the form of VANID and/or DWID. In early weeks (1 to 3), voters were identified to the phone vendors with DWIDs and had VanIDs joined later. Weeks 4 on were called with just VANids and had the DWIDs joined in later. After a certain point, records should have come in with both VANIDs and DWIDS (from exports for the VAN). Every record in this table represents a person’s response to an individual question.

SurveyQuestionResponseCode Joins to the SurveyQuestionResponse table. Any person’s answer that did not match to the SurveyQuestionResponse table is put in the SurveyQuestionResponseMisc field in PersonResponse.


53

Finally, we used two views to flatten this normalized structure into one record per PhoneCalledID, for any person who answered any survey questions:

• Paidids_flat_a flattens the table by taking the min of a lot of columns and grouping on PhoneCalledID . This view does not include responses from files that were marked as bad (SourceFile.BadData = 1) and includes a piece of SQL to only include daily files where its weekly version doesn’t exist. TODO (FlatColumn, FlatValue)

• Paidids_Flat takes the Paidids_flat_a view and simply adds a column for rolled up Obama/McCain. It is a complicated case statement that pulls different combinations of question answers into “SO”, “LO”, “UN”, etc. TODO (multiplier)

STRUCTURE OF DATASET

The most difficult component of the paid ID dataset was the commercial information on each voter. This data was difficult to manage because it was extremely large – several trillion bytes for the file in the large states. We initially tried not to house the whole state Catalist file at the campaign, but the time costs of arranging for correct data to be sent and then processing that data each week were prohibitive. Once the campaign agreed to house the entire Catalist dataset at the campaign, the overhead was reduced to dealing with entire state files when Catalist performed updates. We punted on the first of these operations. The second, in October, took nearly a week to complete. The construction code we used to import these tables is on wells in /var/opt/ofa/catalist-imports.

RECOMMENDATIONS:

1. All voter data should be centralized and provided with significant database management staff and oversight. Syncing all data from all sources is necessary, and time-consuming. To provide the same success with targeting that the Obama Campaign had, future campaigns will need to either provide internal organizations to managing all the data, or establish better contracts with outside vendors that are capable of managing and centralizing the campaign’s voter information. This cannot be overstated.

2. State party data, when it was available, was some of the best data– but getting it from state parties was sometimes difficult. Democratic organizations need to coordinate and formalize data exchanges and usage licensing early.

3. The model scores existed in a thin table, one per state, with a model score and a Catalist ID (“DWID”). The scripting for these scores was simplified once we created MySQL views with the most recent score for each state. Analysts wishing to examine different versions of a state’s model needed to perform joins themselves. The code for constructing the paid ID dataset nightly is in the archive files.

CHECKING RESULTS

OVERVIEW

After building a model, the next step is to check the model for errors and present results. There are two main goals:

• Verify whether the model is accurate

• Verify what the model says about the electorate

These two steps are often overlooked during campaigns’ compressed timelines, but they are as important as the estimation procedure itself. Clear reporting on what the model says will communicate to state and national leadership explicitly how the electorate is behaving, and will give guidance on how a program should respond. A rigorous quality control process will guarantee that those results are indeed accurate, ensuring program quality.


54

QUALITY CONTROL: MOST COMMON PROBLEMS

• Bad data: Model inputs (paid ID data, VAN data) are incorrect because of inaccurate data collection or

database construction

• Missing data: Two common varieties

o Missing variables – e.g., vote history

o Missing records – e.g., limited age or gender records for entire voter file (North Dakota)

• Model misspecification: predicted modeled relationship between two variables is incorrect – e.g., a

positive linear relationship (positive and not decreasing) is specified instead of a positive logarithmic

relationship (positive and decreasing)

The most important preventative measure is to develop methods that avoid these errors from happening as much as possible. The second most important measure is to develop processes to check the results to see if one of these problems may have occurred unnoticed during pre-modeling quality control.

QUALITY CONTROL 1: DETECTING BAD DATA

Bad data is the most common, and probably most malignant, problem when building predictive models. There are several preemptive checks one can do to check for bad data:

• Summary Stats: Maximum, Minimum, Average, Standard Deviation, Distribution Graphs

• Cross-checking results by source when data came from several sources but provides the same variable

Here is a simple example where summary stats can be very valuable:

AGE VALUES IN THE INDIANA VOTER FILE: DECEMBER 2008

Statistic Age

Minimum Age -5,974

Maximum Age 157

Average Age 46

This is clearly an issue of bad data in the voter file. If this were in the modeled paid set, it could alter the estimates considerably.

When a specific set of data comes from one source, you can check it against other sources to see if it provides similar results. In the example below, we checked to see how the model predicted a week of paid IDs supplied by a specific phone vendor.


55

Notice that there is little difference in observed support (in blue) from quintile 2 to 5, while the model (in red) predicts some difference. The results indicated that the model had poor explanatory power, though it had performed well the previous week. Further examination revealed the paid ID data had serious problems.

QUALITY CONTROL 2: MISSING DATA

There are two types of missing data – missing records and missing variables – each which present their own problems.

The front-end checks for missing data are straightforward: counts and crosstabs. The North Dakota voter file, for example, is missing age information for roughly 50% of all voters in the state.

AGE BREAKDOWN – NORTH DAKOTA VOTER FILE (2008)

Age Voters

18 to 24 1,893

25 to 34 9,681

35 to 49 42,045

50 to 64 59,728

65+ 46,479

Unknown 147,310

Total 307,136

0%

20%

40%

60%

80%

100%

1 2 3 4 5

Ob

am

a S

up

po

rt %

Quintile

Nevada Week 16 Paid IDs

Strong Obama % Compared to Modeled %

Strong Obama % Average Score


56

Because age tends to have strong explanatory power on support, the distribution for voters without age information is far narrower than for voters with age information:

Missing variables – when those variables are important – can easily distort what would otherwise be a good model. The case below shows the average predicted support score for primary voters and non-primary voters when primary data is excluded. The differences between predicted and actual results are far greater for primary participants than non-primary participants.

DIFFERENCE BETWEEN ACTUAL AND PREDICTED SUPPORT: OHIO 2008

2008 Primary

Participation

% Supporting Obama

in Paid IDs (Actual)

Average Support

Score (Predicted)

Actual – Predicted

Support

Voted in Democratic

Primary 68.3 60.6 7.7

Voted in Republican

Primary 13.4 25.0 -11.7

Did not Vote in

Primary 39.2 41.5 -2.3

QUALITY CONTROL 3: MODEL MISSPECIFICATION

Models are sensitive to the way data is entered. In the example below the blue dots represent the average support for Obama at every age from 0 to 85. The correlation between age and support for Obama is positive. That says the older people are, the more they support Obama. High rates of supports among seniors drive this correlation; the positive correlation is represented with the red line below. If the model included only a linear term for age, this would be the final conclusion.

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100

% o

f V

ote

rs

Support Score Range

Support Score Distribution: North Dakota Voters With and Without Age Information

Age Data Known Age Data not Known


57

However, this correlation masks an important characteristic of the electorate: young people support Obama at a higher rate than even seniors. Ideally the model would reflect that fact in addition to reflecting higher support among seniors. With an age-squared term in the model, the result is the green line below. The green line also indicates that support is higher among seniors, but now it also reflects the fact that Obama has higher support among youth. The quadratic version of the model fits the data better than the linear version. By including the age-squared term, we improved how well the model matches reality.

When constructing a model, it is important to enter the data in a variety of different ways, in order to allow the model to capture these subtleties. In addition to including age-squared terms, you can also include log(age), age-cubed, etc. The more information provided to the model, the better job it will do. Looking at data in many different ways and testing different specifications to see if they affect results should be done with every model.

QUALITY CONTROL 4: POST-MODEL CHECKS

There are two immediate checks that should be done after a model is completed. The first – stair-stepping – checks the overall fit of a model. The second – crosstabs – looks at how different variables are performing in the model. Both the larger picture and the more specific details are important to ensure model quality.

Stair-stepping

When the model is estimated, only two-thirds of the available observations are used in the estimation. The remaining 1/3, which are called the “test set”, are withheld from the estimation so that they can provide a post-estimation check on model quality. If the model correctly predicts support in the test set, then we can be confident that the model has been estimated correctly.

Below is an example of a comparison between the test set and the model predictions.

0%

10%

20%

30%

40%

50%

60%

70%

0 20 40 60 80 100

%

Supporting

Obama

Age

Obama Support by Age

Linear Model

Quadratic Model


58

There is a very close correlation between the model’s predicted support (in green) and the observed support in the test set (in blue). This is a model that validates well. In some cases correspondence will not be so accurate. A common problem is that the distribution is “flat”, which means that there is little difference between the test set support in the lower deciles and the upper deciles. In this case, it is important to return to the above steps to check data quality.

Crosstabs

The second test is to look at how the model performs across specific variables. A good model should behave like a comprehensive set of cross-tabs. If males have higher support than females, then males should have higher scores than females. This should be true across all variables in the model. The way to check this is to look at the cross-tabs for support and then compare to the model scores for the various groups.

The example for missing primary data in Ohio above shows a case where the cross-tabs do not match the model scores. Below is an example where the model is performing well and the cross-tabs match the predicted support from the model. In the observed data, there is an 11 point difference in observed support between people living in precincts that voted for Bush in 2004 and those that living in precincts that voted for Kerry in 2004. There is a corresponding 12 point difference in the predicted support across these two groups. The close correspondence of the observed support and the predicted support is evidence of an accurate model.

OHIO SCORE CROSSTABS

Kerry 2004 Performance

(NCEC Precinct Data)

% Supporting Obama

in Paid IDs (Actual)

Average Support

Score (Predicted)

Actual – Predicted

Support

0%-35% 49.1% 52.8% 3.7%

35%-65% 59.7 62.6 2.9

65% + 60.9 64.4 3.6

These post-estimation cross-tabs have the significant advantage that they can be easily explained to the field leadership (cross-tabs are accessible to most field leadership).

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10

Su

pp

ort

%

Decile

Ohio Obama Support Score

Test Set Average Score


59


SUCCESSES

The success of the models can be seen in the following two tables.

Model Score Observed Support in Paid IDs

50-55 57%

55-60 57%

60-65 60%

65-70 67%

70-75 83%

75-80 81%

80-85 71%

85+ 98%

This first table shows that the voters identified as supports (model scores >= 65) did in fact support Obama at greater than 65% in the final set of paid IDs. This is real data from North Carolina paid IDs in the final weekend of the campaign.

Variable Category Model Score Observed Support in Paid IDs

Media Market Cincinnati 38.8 38.3%

Media Market Cleveland Metro 57.5 59.1%

Media Market Cleveland non-Metro 49.8 51.5%

Media Market Columbus 46.0 46.6%

Media Market Dayton 42.2 41.3%

Voter is in MyCampaign No 52.7 55.5%

Voter is in MyCampaign Yes 68.2 82.8%

Married Yes 54.7 55.9%

Married No 44.1 44.9%

% Owner Occupies House 0%-66% 53.6 54.5%

% Owner Occupies House 66%-82% 45.3 46.4%

% Owner Occupies House 82%+ 42.3 43.2%

This second tables shows that across many important variables – regional variables, campaign volunteer variables, consumer variables (marriage) and census variables (owner occupied housing) – the observed cross-tabs are accurate in predicting support in the model score. Together this defines a successful model. The model must predict overall support, especially among supporters. And the model must fit the cross-tabs across the important variables. The final models were successful in meeting both criteria.


60

CHALLENGES

It took us a long time to finalize these models. The first few months of models were far from comprehensive and were much less accurate in fitting paid ID results. The biggest challenge was compiling all relevant data and including it in the model. The first sets of models were deficient in the following ways:

1. They did not include political data on the VAN

2. They did not include primary participation data

3. They were estimated with paid IDs that were later determined to be of poor quality

4. They did not include NCEC precinct level data

5. They did not include regional controls

The bottom line is that most of these problems were addressed through quality-control. The simple questions, “What is going into these models?” and “What is not going into these models that should be?” were important in improving models. Once we started framing the problem in those data quality and data completeness terms, the models improved dramatically.

RECOMMENDATIONS

The most important thing is to put in place a process with multiple checks on the input of data into the models and on the post-estimation validation of the models. This should include, at a minimum, the following checks on the inputs:

1. Check that the paid ID data is high quality

2. Check that the data includes: past political IDs (from VAN or state party), NCEC data, and vote histories

3. Check that the number of voters being scored matches the number of voters on the voter file.

4. Check that Polimetrix or race data is included.

5. Check that the voter file being used for the model is the latest available voter file.

The post-estimation check system to validate the models should include the following steps:

1. Stair-steps charts that compared the test-set to the model predictions.

2. Cross-tabs that compared the observed support levels with model scores

3. Comparison of the next week’s IDs with the new model scores

4. Additional examination of the most important variables to ensure they are included and behaving as expected.

Building a system of explicit checks for each of these things is critical for constructing accurate models. This process, while admittedly time consuming, is the most important thing future targeting groups can do.

61

TARGETING PERSUADABLES

SYNOPSIS

DEFINITIONS: PERSUADABLE VS. PERSUADABILITY

A persuadable voter is a registered voter who does not currently support our candidate, but through sufficient messaging can be convinced to support our candidate. Persuadability indicates how persuadable an individual or a group is – i.e., the likelihood that their preference for a candidate can be changed with sufficient messaging.

Within a group of voters, only a fraction will be persuadable (to any degree). If one group of voters – e.g., “Senior Women in the Suburbs” - appears to have more persuadable voters than another group, we’d say that the group is, on average, is more persuadable than the other (even though there are many non-persuadable voters in that group).

PROBLEMS WITH IDENTIFYING PERSUADABLE VOTERS

For many reasons, identifying persuadable voters is difficult:

• We can directly ID candidate commitment (having candidate preference vs. being undecided), but we

can’t directly ID how willing a voter is to change their mind

• Being undecided is often as much a degree of apathy as open-mindedness

• As such, there’s no identifying variable for being persuadable. Even if we asked every voter “are you

persuadable?”, that wouldn’t communicate whether or how they would respond to messaging

Persuadability, as we understand it, is “unobservable” – we can’t see it or measure it. As a result, building a universe of persuadable voters – and communicating what parameters are reasonable and why - is a challenging exercise.

MEASURING PERSUADABILITY

As there is no way to directly identify persuadable voters (or how persuadable they are), we have to rely on measurable variables most likely to be associated with persuadability.

In 2008, we used three principle variables:

• Open-mindedness: % of undecideds and non-supporters that indicate there is a “Fair Chance” they will

support Obama

• Receptiveness to message: % of undecideds and non-supporters that switch candidate preference after

positive or contrastive messaging

• Non-partisanship: Initial likelihood of being undecided or uncommitted

These variables can be sourced and combined in different ways – to different effects – when building a persuasion universe for voter contact.

TARGETING: BUILDING

BUILDING A PERSUASION UNIVERSE

The persuasion universe should be the set of voters that the campaign can move with its message, given the amount of resources that it has.

Building a persuasion universe is not a straight or obviouothers, but there are no bright line criteria that distinguish persuadable from nonnumber of voters that we can contact available monetary and organizational (e.g., volunteers) assets.

During the general election, the OFA targeting team used a twothe first step from polling and the second from micro

For the first step, the team identified subthe greatest % non-supporters (undecided + McCain supporters) indicating a “fair chance” they would support Obama and also the greatest % non-supporters that moved to Obama after messaging.

The example below, taken from an OH poll, supporters indicating a ‘fair chance’ they would support Obama and also the highest % that suafter messaging.

SAMPLE POLLING RESUL

Polling Results Horse Race in Poll

Demographic Obama

White, Female,

<35 years 54%

White, Female,

35-59 years 45%

White, Female,

>60 years 42%

* % that support Obama + Undecided / McCain supporters that indicate t

** % That support Obama (of all polled voters) after receiving positive on Obama and contrastive messaging on McCain

TARGETING: BUILDING A PERSUASION UNIVERSE

62

N UNIVERSE: INITIAL FRAMEWORK

should be the set of voters that the campaign can move with its message, given the

Building a persuasion universe is not a straight or obvious process. Some voters are more persuadable than others, but there are no bright line criteria that distinguish persuadable from non-persuadable voters. Also, the number of voters that we can contact – and the number of times we can contact each one available monetary and organizational (e.g., volunteers) assets.

During the general election, the OFA targeting team used a two-step framework to build the persuasion universe, the first step from polling and the second from micro-targeting1:

For the first step, the team identified sub-demographics – with sample size greater than 100 voters supporters (undecided + McCain supporters) indicating a “fair chance” they would support

supporters that moved to Obama after messaging.

The example below, taken from an OH poll, shows that White Females > 60 showed the highest % of nonsupporters indicating a ‘fair chance’ they would support Obama and also the highest % that su

SAMPLE POLLING RESULTS FOR OHIO – AUGUST 20082

Horse Race in Poll Movement in Poll

McCain Undecided Initial BO Support + Fair

Chance* / Difference

BO Support (Post

Message)** / Differ

42% 4% 62% / 8% 54% / 0%

41% 14% 52% / 7% 50% / 5%

39% 19% 52% / 10% 51% / 9%

% that support Obama + Undecided / McCain supporters that indicate there is a “Fair Chance” they will support Obama


Non-Partisanship Open-Mindedness

Receptiveness to Message

should be the set of voters that the campaign can move with its message, given the

s process. Some voters are more persuadable than persuadable voters. Also, the

and the number of times we can contact each one – is bounded by

step framework to build the persuasion universe,

with sample size greater than 100 voters - that showed supporters (undecided + McCain supporters) indicating a “fair chance” they would support

that White Females > 60 showed the highest % of non-supporters indicating a ‘fair chance’ they would support Obama and also the highest % that supported Obama

2

BO Support (Post

Message)** / Difference

54% / 0%

50% / 5%

51% / 9%

here is a “Fair Chance” they will support Obama


TARGETING: BUILDING

While the sample size was small, the information seemed to indicate that a high % of Whisupporters were both open to an Obama candidacy and also changed their minds (to support Obama) at high rates after receiving (positive) information on Obama and (negative) information on McCain. Repeated polling which tested the robustness of this claim became a critical part of the Ohio persuasion program in 2008.

In the next step, the targeting team used modeled support scores to eliminate statistically likely partwithin target demographics. Most frequently, states chose to place a lower bound support score of 35 and an upper bound of 65 for the persuasion universe. A voter with a support score of 35 is likely to support McCain at a 2:1 margin on average; a voter with a support score of 65 is likely to support Obama at a 2:1 margin on average and those in between are likely to be statistically less partisan.

In Ohio’s case, the voters within score range 35partisan than voters that don’t participate in primaries.

Voters in this score range also tended to be more considerably more undecided:

0%

20%

40%

60%

80%

100%

0-9 10-19 20-29

% o

f T

ota

l V

ote

rs

% of Voters by Support Score Range by Party Tag


63

While the sample size was small, the information seemed to indicate that a high % of Whisupporters were both open to an Obama candidacy and also changed their minds (to support Obama) at high rates after receiving (positive) information on Obama and (negative) information on McCain. Repeated polling

ustness of this claim – provided consistently similar results and, this subbecame a critical part of the Ohio persuasion program in 2008.

In the next step, the targeting team used modeled support scores to eliminate statistically likely partwithin target demographics. Most frequently, states chose to place a lower bound support score of 35 and an upper bound of 65 for the persuasion universe. A voter with a support score of 35 is likely to support McCain at a

ge; a voter with a support score of 65 is likely to support Obama at a 2:1 margin on average and those in between are likely to be statistically less partisan.

In Ohio’s case, the voters within score range 35-65 tended to be non-primary voters – who arpartisan than voters that don’t participate in primaries.

FIGURE 1 :

Voters in this score range also tended to be more considerably more undecided:

30-39 40-49 50-59 60-69 70-79 80-89

Support Score Range

% of Voters by Support Score Range by Party Tag

Ohio: August 2008

Democratic Primary Voters

Non Primary Voters

Republican Primary VotersLess Partisan Voters

While the sample size was small, the information seemed to indicate that a high % of White Female, >60, non-supporters were both open to an Obama candidacy and also changed their minds (to support Obama) at high rates after receiving (positive) information on Obama and (negative) information on McCain. Repeated polling –

provided consistently similar results and, this sub-demographic

In the next step, the targeting team used modeled support scores to eliminate statistically likely partisan voters within target demographics. Most frequently, states chose to place a lower bound support score of 35 and an upper bound of 65 for the persuasion universe. A voter with a support score of 35 is likely to support McCain at a

ge; a voter with a support score of 65 is likely to support Obama at a 2:1 margin on average –

who are intuitively less

Democratic Primary Voters

Non Primary Voters

Republican Primary Voters


64

CANDIDATE SUPPORT BY SUPPORT SCORE RANGE: WHITE WOMEN >603

Score Support

McCain

Support

Obama Undecided

10-19 71.7% 14.8% 13.5%

20-29 65.6% 20.0% 14.4%

30-39 52.4% 28.2% 19.4%

40-49 41.2% 36.6% 22.1%

50-59 38.0% 39.7% 22.3%

60-69 31.8% 46.9% 21.2%

70-79 23.0% 57.5% 19.4%

80-89 18.0% 65.7% 16.4%

Total 41.9% 39.2% 18.9%

The Ohio final universe construction was the following:

OHIO PERSUASION UNIVERSE: 2008

Demographic Modeling Constraint Final Count – Voters Final Count - Mailboxes

Senior Women, Non Rural

• >60 • Urban + Suburban

Support Score : 30-60 104,053 103,802

Middle-Aged Voters • 35-59 • Urban + Rural + Suburban

Support Score: 45-65 531,773 480,392

Senior men, Non Rural

• 60+ • Urban + Suburban

Support Score: 35-65 54,243 54,116

Rural East Spill Markets • Southern Spill Markets • Not covered by previous universes

Support Score: 35-65 17,457 15,854

Field IDs outside target

demographics

• Undecided / BO Leaners • JM Leaners with Scores > 40

Support Score: All 48,792 45,101

BUILDING A PERSUASION UNIVERSE: ADDITIONAL SCREENS

There are several other screens / modifications that can be built into the persuasion universe:

• Narrow to a more narrow support score range (farther towards 50/50)

• Narrow to support score range that has the highest undecided % from paid IDs

• Narrow to likely (or most likely) voters within a specific support score range

A vote screen to eliminate non-voters and/or strongly infrequent voters is almost always necessary – a persuasion contact on someone that doesn’t plan to vote is poorly utilized. In 2008, most states chose to restrict to voters that had either voted in the 2004 election (who had the opportunity) or those that registered after the 2004 election. A more careful methodology would be to limit voters above a certain turnout score (where a turnout score is available) or to combine a turnout score with a standard vote screen.


65

SIZING THE PERSUASION UNIVERSE

There are two main ways to think how big a persuasion universe should be – based on vote goal or

capacity:

• Vote Goal: what we need to win (contacts needed to win): (��') *''+'+

% �, (��'�) -'�).�+'+ /'� $��#� �� 0 12��3� �14�4

• Capacity: how many contacts organization can provide with its resources: *�.�, $��#�) ��%6��%�� #�� -��7%+'

$��#�) -'� -'�)�� 0 12��3� �14�5

Alternatively, one can select an arbitrary modeling cut-off (for use across states, for example). Some possibilities:

• Voters with support scores 35-65 and turnout score > 85

• Voters with support scores 40-60 and undecided score > 70

• Voters with support scores 40-65 and persuasion score > 65

BUILDING A PERSUASION UNIVERSE WITH AN UNDECIDED MODEL

Throughout the primary, we frequently built GOTV and persuasion universes based on their respective support score and undecided score combinations. For visual purposes, score matrices served to highlight the number of voters within different buckets, which made universe creation easier.

This is a specific example from the Indiana 2008 primary with potential universe selection – each cell representing thousands of voters (see below):

UNDECIDED SUPPORT SCORE MATRIX

Score

Range0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100

0-10 8 12 16 15 18 18 24 29 46 79

10-20 15 17 20 21 24 24 29 33 34 36

20-30 27 27 28 27 30 30 31 33 28 23

30-40 30 30 30 29 31 30 32 30 28 19

40-50 34 36 33 30 29 30 29 28 28 21

50-60 31 29 29 26 26 27 27 26 24 20

60-70 31 28 30 27 27 28 27 27 25 22

70-80 43 35 34 30 28 31 27 24 23 19

80-90 43 29 37 32 31 31 23 30 22 14

90-100 23 18 21 19 23 23 22 20 24 19

Support

Score

Supporter (GOTV) Universe Persuasion Universe

Low Undecided Score High

Low

High


66

Previously, we had used a “persuasion score”, which was the support score * undecided score, and chose voters with the highest score that had support scores below 65.

SUCCESSES & CHALLENGES

EVALUATING THE INTIAL PERSUASION FRAMEWORK

The initial framework for persuasion universe was generally successful for a few reasons –

• The methodology made sense intuitively – it used polling information and micro-targeting for their

relative strengths in a logical combination

• It provided an answer that worked – it gave each state a good number of persuasion targets that yielded a

heavy number of truly persuadable targets (per reactions from field)

• It was easy to communicate

On the other hand, there may have been several drawbacks

• Often relied on extremely small polling data sets for movement numbers (especially for small sub-groups)

• Voters with support scores around 50 often tended to be voters with disproportionately less vote history

and, in general, less information on the file – i.e., the less information we had on a voter, the more their

predicted support converged to the mean.

• Finally, there may have been better ways that are yet unexplored or underexplored. The persuasion model

provides an example of that.

UNDECIDED & PERSUASION MODELS

Building the persuasion universe based on an undecided model presents a number of problems:

• Being undecided is much harder to predict than support – empirically, we haven’t been able to provide the

same level of predictive power (or accuracy) as with the support model

• A good undecided model is contingent upon good paid ID data to distinguish between leaners, true

undecideds and apathetic voters

• At times, the universe suggested by undecided scores suggested a far different universe than suggested by

polling or message testing. These differences were frequently irreconcilable.

On the other hand, there are some clear benefits from building persuasion universe using undecided

model:

• It’s extremely transparent

• It’s testable after the fact. Unlike universes built from polling results, it’s possible to evaluate a universe

derived from an undecided model on specific score ranges and sub-demographics (e.g., Male voters with

undecided scores 60-70).


67

RECOMMENDATIONS

There is a great more study needed on persuasion universe construction and persuasion tracking and reporting.

There are several recommendations:

• Need to do a more thorough examination of undecided models earlier in the process. Need to put in an

early investment with paid IDs and modeling testing (and re-testing) to see if we can identify at least the

most “undecided” voter blocs that are also likely to vote

• Better reporting on persuasion effectiveness is clearly needed. For example - after several field touches

does one universe show greater movement than another universe (vs. a control group?). This type of

information would have provided invaluable insight for any changes and or specific sub-group targeting

• Message testing needs to draw from larger samples and it needs to be done more frequently. Small

sample “horserace polling” is can probably be done more effectively with paid IDs – but polling can find

great amounts of voter information beyond candidate preference

• The persuasion model – which was done for the first time in the 2008 presidential – needs to further

investigated

Most importantly, though, building a good persuasion universe is contingent upon a clear organizational understanding of what persuasion is and is supposed to be. Throughout 2008, the definitions – and goals – of persuasion were often prohibitive to building a universe that everyone could agree on early enough from which to build a cross-department persuasion program. In the end, though, incredible progress was made towards building a final understanding that will go farther in future cycles.

INDEX

1. Targeting Team Presentation Document – August 2008

2. Poll Question was “Even though you are not supporting Barack Obama now, what are the chances that

you might support him in the presidential election?” – With the following answers: Fair Chance, Small

Chance, Slight Chance, No Chance.

3. Targeting files oh_final_mail_universe

4. This is more of an explanatory definition than an actual description – see the capacity section for a more

complete definition.

5. This is somewhat simplified, and does not account for the dynamism of the universe. See the capacity

section for more explanation of this process.

TARGETING: NEW REGISTRANTS

68

TARGETING DEMS: BUILDING A GOTV UNIVERSE

SYNOPSIS

CORE UNIVERSE

A GOTV Target is a supportive voter that our campaign can motivate to turn out that wouldn’t turn out under their own initiative. In practice, the Get Out the Vote (GOTV) universes are built using a combination of support variables and turnout criteria. Better GOTV targets are voters with high confidence support and low turnout likelihood.

During 2008, each state built a core / priority GOTV universe consisting of “high-confidence” supporters that were also sporadic (but still active) voters. In most states, a high confidence supporter was defined as follows:

• Base Universe:

o Support model 65 or above

o African Americans (who should already fall in to the initial category).

o All identified supporters and leaners (field IDs and paid IDs)

• Add: Event signups or online signups (e.g., everyone in My Campaign); often this was combined with a

support score screen when there was no ID, such as anyone in My Campaign scored above 35, to clear out

very unlikely supporters.

• Remove: identified non-supporters (including McCain leaners). Identified undecided voters were treated

neutrally – if they were modeled in to support, they were included, if not they were not.

A “sporadic voter” was defined as someone with infrequent voting history. Below is the example from Ohio:

Voting History

Registration Description 00 01 02 03 04p 04 05 06 07 08p

SP

OR

AD

IC

One-off General

Election Voter ('04-

'08) Exactly 1 election Before 2004

Non-Recent Voter Any or no elections Before 2002

New Registrant (w/

No Voting History) No elections After 2000

New Registrant ('08

Primary Voter) Voted After 2004

LIK

EL

Y

Recently Active Voter Voted At least 1 election Any

Recently Active Voter Voted At least 1 election Any

Recently Active Voter At least 1 election Voted Any

Recent Registrant At least 1 election

2005 – 2006

Two-Cycle Voter* At least 1 election At least 1 election Before 2002

NON-VOTER No elections Before 2000

* In most states, two-cycle voters were categorized as sporadic voters. Definition can be adjusted based on size constraints

In some states, the sporadic universe also included likely voters with low turnout scores – which would incorporate demographics with relatively rich history but low modeled engagement.


69

One consideration to take when defining a “sporadic” voter is previous GOTV history in the state. In Ohio, for example, extensive GOTV efforts were made in both 2000 and 2004 Presidential Elections – so voting during those years did not necessarily indicate that a voter would be self-motivated to vote again in a Presidential Election. While the existing definition of a sporadic voter remained, any additions/supplemental, which will be discussed further in this document, drew directly from that pool of General Election only voters.

The Core GOTV universe is dependent on model accuracy (the support score), so the national targeting team performed consistent and thorough model testing constantly going in to Election Day – not only statewide but among demographics and geographic regions. They compared the observed paid ID support for demographic/geographic groups within each state to the modeled support for that group. If the model over-predicted support (model score for group greater than measured Obama support), the model threshold got the GOTV universe was moved up from 65 appropriately to compensate. Where the model under-predicted observed support, “blind pulls” were added below the 65 model threshold (see “Blind Pull Universes” below)

GAUGING CAPACITY

We compared the Core GOTV universe to our expected capacity and voter contact goals in each region. We had the choice of changing our capacity to match the GOTV needs or changing the GOTV universe to match capacity. In practice, we used a combination of both options

To adjust capacity to match the Core GOTV universe, tactical decisions have to be made to decide where capacity can be shifted. For example, in Ohio, there was a lot of excess volunteer capacity in rural and suburban areas, but capacity fell short in the urban city centers, where the GOTV universe was very large. Attempting to move around volunteers resulted in a loss of total capacity (if 10 volunteers are willing to volunteer in their own neighborhood, perhaps only 5 of them will drive to a nearby city and volunteer there). Ultimately, the decision in Ohio was to move volunteers in suburban and rural areas where there was a large city nearby; but add supplemental universes to match capacity in rural areas that were nowhere near cities.

If the Core GOTV universe is simply too large for any adjustment, then we reduce the GOTV universe by removing the lowest priority groups first. Our main priority indicator was the GOTV index, which incorporated support score, turnout score, and contact rates in to one model which prioritized voters with high support, low turnout, and high contact rates. New registrants, also, are targeted as primary GOTV targets, because the benefit of the voter education contact is increased among brand new voters.

Removing voters is especially effective in call universes; and has to be done carefully in canvass universes. A less-dense canvass universe does not necessarily mean more doors will be hit, due to the expanded space between doors. In those cases, geographical turfs could be prioritized using the GOTV index or measures of density, so that the canvassers and FOs could make adjustments on the fly if they wouldn’t be able to get through everything.

A precinct prioritization model may look like:

[(# of GOTV targets)/precinct area]*average GOTV score of all targets

Where the total number of targets is divided by the area of the precinct, to prioritize dense precincts; and that figure is multiplied by the average GOTV score of the targets in order to prioritize the best demographics. A model like this is more useful for tiering GOTV canvass universes, where taking out voters does not necessarily help.

The following chart illustrates Ohio volunteer capacity for Election Day as of October 20, 2008 – two weeks out. For each region, the light blue represents the number of volunteers needed to hit the GOTV universe, and the dark blue represents the volunteer deficit. The negative deficits in region 1 and elsewhere indicate a volunteer surplus. You can immediately see that the greatest deficits existed in regions with big cities (with the notable exception of


70

Cincinnati) – Region 6 (Dayton), Region 9 (Columbus), Region 19 (Akron), and Regions 22 and 23 (Cleveland). Some of the regions had surplus volunteers already. The decision on whether or not to give those regions supplemental universes was wholly dependent on their proximity to the regions with deficits. Of course, when looking at these volunteer numbers two weeks out, one had to account for additional recruitment and flake rates.

-200

0

200

400

600

800

1000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Region

Election Day Volunteer Needs & Deficits:

Ohio, 10/20/2008

Volunteers Recruited Volunteer Deficit


71

BUILDING SUPPLEMENTAL UNIVERSES

In areas where there was excess capacity which isn’t being directed to areas of deficit, we created supplemental GOTV universes. The chart below illustrates where supplemental universes come from on a grid of the electorate. Groups 2, 3 and 4 are all supplemental universes to the Core GOTV universe, group 1.

Diagram of GOTV & Supplemental Universes: Turnout X Support

Su

pp

ort

Ob

ama

Bas

e

100

90 3

1

2

80

70

No

n-B

ase

4

60

50

40

McC

ain

Bas

e

30

20

10

0 Non Voters Sporadic Voters Likely Voters

Turnout

Core GOTV Universe

Likely Voters Supplemental

Non Voters Supplemental

Blind Pull Supplemental

Option One: Adjusting Turnout Screen

Groups 2 and 3 on the grid adjust the turnout screen in either direction to include more supporters.

• Group 2 includes likely voters: one option is to tighten the definition of a likely voter, which moves some “less likely” likely voters in to the sporadic voting category. To do this, we applied our turnout model score to the voters we had labeled as “likely voting” due to their vote history. We created the desired universe size (to fit capacity) by starting with a threshold at the bottom of the available turnout model scores and increasing the threshold until the count of the selected voters was approximately right. This should only be done up to a certain point, after which the voters are too likely to vote to yield a valuable GOTV contact. Instead a different supplemental should be pulled once the modeled turnout likelihood reaches. That “point” is wholly dependent upon the turnout model’s normalization and the natural breaks within the scores.

• Group 3 includes non voters: the other turnout adjustment is the opposite – tightening the definition of a non voter, which moves some of the “more likely” non voters in to the sporadic voting category. Again, we use the turnout score, but in the opposite way. We start with the “non-voters” (such as people who voted in 2000 – 2003 but not since) and we decrease the turnout score threshold until we have a group of the desired size. As before, this should only be done to a point.


72

• This supplemental universe requires extensive contact rate analysis to determine its yield: contact rates among non voters as a whole were too low to justify attempts, but pieces of the universe could be cut out where contact rates were equal or close to active voter universes. A non voter supplemental should not compromise contact rates, especially for GOTV when cleaning the list is not a priority or even beneficial in the short term.

Option Two: Blind Pull Universes

Group 4 illustrates a supplemental which includes pieces of the electorate who do not fall in to the core universe’s definition of support: most likely, they are modeled in the 55-65 range, and share a characteristic that the model may not be picking up.

The important thing to note about this option is that it must not be done indiscriminately. Find the groups that are most likely to be supporters and add lower modeled support members of that group to the GOTV universe. Dropping the support score threshold globally, without substantial justification, is unwise and compromises the GOTV program’s effectiveness.

The targeting team in headquarters provided analysis of dozens of demographics, broken out by support score, and the rates they were IDing at in paid IDs and field IDs. These demographics included, in addition to age/sex:

• Democratic Performance Index

• IDs within household

• Native American tribes

• Renters vs. Owners

• Married vs. Single

• Internet usage

An example, illustrating the ID rates of registered, white, under 50, unmarried, non-Republican women in Indiana, with all supportive universes highlighted in yellow, demonstrates a potential blind pull universe:

Paid IDs Calls/Walks

Score Total Reg BO JM Sample BO JM Sample

-25 48 0% 0% 0 0% 100% 12

25-35 3,723 16% 84% 44 28% 72% 19

35-45 60,181 40% 60% 867 53% 47% 922

45-55 207,00 55% 45% 3,149 65% 35% 1,179

55-65 254,92 66% 34% 2,708 76% 24% 2,680

65-75 69,863 75% 25% 599 85% 15% 6,630

75+ 18,608 89% 11% 133 93% 7% 6,235

Women who were modeled 55-65 were actually measured supporting Obama at 66% - a rate high enough to include in the GOTV universe.

Note also that field IDs broke for Obama higher than paid IDs did (on average more than 10 points higher). For this reason, only use paid IDs for these sensitive and critical GOTV universe adjustments.

We used analyses of this sort across many demographics and geographies. The table above was statewide. Before any of these universes could be included as a supplemental for a limited geographic region, the ID rate of that limited universe had to be tested to ensure it was still a supportive universe. Often times, the supplemental wasn’t included in major urban centers where the universe was large enough; once these urban centers were taken out of the sample the support rate dropped significantly.


73

GOTV INDEX

Deciding who to put into a GOTV list and in what priority to call/walk them can be one of the most important last-minute decisions of a campaign. Unquestionably, this list should be constructed to maximize the probability of earning an additional vote. The GOTV Index eases this process by ranking each potential voter in the voter file on a scale of 0 to 100, where 100 indicates that placing this person on the GOTV list is likely to have the largest impact on Obama’s share of the votes. In 2008, the GOTV Index was used to help construct GOTV universes, and also to tier those universes into high, medium, and low priority, etc. Furthermore, the score offered an easy way to tier lists at the last minute (especially on Election Day).

The GOTV Index attempts to answer the question: What is the expected change in Obama’s share of the votes if this person is put on the GOTV list? Intuitively, the effect on Obama’s vote share from putting someone on a GOTV list can be viewed as the combination of three pieces:

• Will the person being called support Obama? (i.e., Is this someone we want to turn out?)

• Will the volunteer reach the person if they try to contact? (i.e., Is this attempt a good use of volunteer

resources?)

• Will the phone call motivate the person to turnout? (i.e., Would they not have otherwise turned out and

are they receptive to the call?)

The first two pieces are relatively easily measured from the support model and the contact rate model (described in the modeling section). The third piece, the casual impact of a contact, is the most difficult to estimate. For this, we rested on the assumption that the effect of a contact was inversely related to someone’s current likelihood to turnout – modeled through the turnout score. The lower the turnout score, the higher the estimated GOTV impact of a contact. This was probably the most controversial and important assumption of the model.

The final equation can structured as following:

89:; �� <� ��=� >�� ?@@�=�

Where:

�� A2 � �C�D� �� =�� E 1F

<� ��=� >�� <� ��=� >�� =��

�� ?@@�=� 10%H .6 � A1 E �� =��F

The TurnoutEffect variable was dampened slightly (by a factor of 0.6) to account for noise/error in the Turnout Score. This computation yields an estimate of the expected (probabilistic) increase in vote share for Obama by putting an individual on a GOTV list. The GOTV Index is then just an index from 0 to 100 of the above effectiveness measure.


74

FLORIDA GOTV INDEX SUMMARY CHARTS

Going forward, recommendations include better understanding of the relation between turnout and the effectiveness of a phone call. Indeed, it is possible that people who have a very low chance of turnout are also not moved to go vote by a phone call or a walk. Also, those readers familiar with statistics will recognize that the validity of this analysis rests on numerous assumptions of independence of turnout probabilities with support scores and contact rates. These assumptions are almost certainly overly restrictive and could be improved upon. A richer model would attempt to estimate these 3 pieces of the GOTV Index jointly, as opposed to taking separate pieces and multiplying them together.


GOTV universe creation was successful for a few reasons:

• It was a thorough and deliberate process, where no hasty decisions were made without testing to prove the

effectiveness of the universe.

• It was creative and didn’t “shut the door” on any potential GOTV universe.

• It accommodated both necessity and capacity.

However, there were some areas that could be improved:

• The decisions of whether to re-allocate resources or supplement the universe was often made

qualitatively, without a strong quantitative way of analyzing the returns from resources/volunteers.

• Less information in some states led to varying consistency within the support model; meaning extensive

work was done to gauge support among blind pull universes in the last weeks.

GOTV & Support Score

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

GOTV Score

Su

pp

ort

Sco

re

GOTV & Turnout Score (Support Score 80-85)

0

20

40

60

80

100

120

50 60 70 80 90 100

GOTV Score

Tu

rno

ut

Sco

re

GOTV Score and Contact Rate (Support Score 80-85)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

50 60 70 80 90 100

GOTV Score

Conta

ct R

ate

75

TARGETING NEW REGISTRANTS

Targeting for registration efforts is a very different problem than targeting for persuasion or GOTV efforts. While in the latter you have a known and stable set of names and addresses, that information is inherently imprecise for unregistered people. For the general election, targeting for new registrants was done at both geographic and individual levels – each with its own set of challenges.

GEOGRAPHIC TARGETING

WHAT IT IS:

Unlike persuasion or GOTV, we don’t always know exactly which individuals we’re trying to register. While broad demographics are defined – African Americans, young people, etc. – without the benefit of a voter file, their exact contact information cannot easily be defined. Often, clipboarding or tabling in dense areas of supporters is used as a tactic to register new voters. The challenge for targeting is to help send volunteers to the areas where they will find the most unregistered supporters per square mile. Additionally, at the macro level, targeting at the state level is useful for determining if registration is a tactic worthwhile pursuing in the state at all.

HOW WE DO IT:

• Define a geographic target level (precinct, ward, county).

• Accumulate demographic data from census and voter file data

• Use these two sources to locate areas that are likely to have both a high number of unregistered

voters and also have high average support

Geographic targeting is based on census data, registered voter numbers and aggregates of individual targeting. This targeting is particularly useful for setting goals (statewide, regional, and turf-specific). Projections to 2008 from 2000 census data were acquired from ESRI. The projections are adjusted to find Voting Eligible Population by accounting for minors, non-citizens, and (in some states) felons. By subtracting the number of registered voters from VEP, an approximate number of unregistered voters can be found.

This method is fairly straightforward at the state level. As the geography becomes narrower, however, the accuracy of the estimate decreases. Since census numbers are only updated in some cities and counties in off-years, accuracy of projections at the county level is dubious. Obtaining precinct-level estimates is also challenging. One method used was to use counts off the Voter File of precincts and census blocks. The factor of what percent of a census block is in a precinct can be used to map population numbers to precincts.

Factors such as DPI, average support score, or demographics (in conjunction with polling) can be used to set these goals. The resulting numbers can be used to relatively weight precincts. Note that the relationship between these numbers and goals should not necessarily be linear: In places with higher percentages of unregistered people, it can be exponentially easier to find people to register.

To order precincts for canvassing, we used precinct attributed to score and prioritize precincts. Based on current polling data and some hand adjustments, census blocks were shaded according to the following formula:

TARGETING: NEW REGIS

0.50 (Constant)

0.45 Percent African American

0.10 Percent White, aged 18-44

-0.05 Percent White, aged 45-59

-0.10 Percent White, aged 60+

0.15 Percent Hispanic * (NCEC Dem

Performance - .5) 0.25 Percent household income <$39,000

0.10 Percent household income >$100,000+*

(NCEC Dem Performance

The results were put on maps and colorscore:

TRACKING PROGRESS

In Missouri, maps of individuals who had been recently registered, both by the campaign and otherwise, were created. Towards the registration deadline, these maps allowed organizers to spotential voters still unregistered.


76

Percent African American Registration Potential

44 0.50 (Constant)

59 x 60% + 0.10 Displacement

Percent White, aged 60+ 0.20 (1 – Percent registered)

(NCEC Dem 0.30 NCEC Democratic

Performance Percent household income <$39,000

Percent household income >$100,000+*

(NCEC Dem Performance - .5)

The results were put on maps and color-coded for field use. The darker the blue, the higher the average precinct

In Missouri, maps of individuals who had been recently registered, both by the campaign and otherwise, were created. Towards the registration deadline, these maps allowed organizers to see where there may be pockets of

x 40%

Percent registered)

NCEC Democratic

the blue, the higher the average precinct

In Missouri, maps of individuals who had been recently registered, both by the campaign and otherwise, were ee where there may be pockets of


77

TARGETING INDIVIDUALS

The campaign acquired a VAP list, which was added to the VAN. These lists, compiled by aggregating consumer data and removing registered voters, were used with widely varying success. Anecdotally, they were more successful in places that had lower rates of housing turnover (i.e. people had been in their houses longer). Unfortunately, this was often negatively correlated with the support level of an area. This made them most useful for adding to persuasion walks, whereas strong base areas had more success collecting forms with less targeted methods (tabling, etc).

MEASURING PROGRESS

Success of the voter registration program is measured using the same metrics used in geographic targeting. By assigning a precinct or census block to voters on collected registration forms (GeoCoding is one way to do this), a score is applied to each. The average score of these new voters can be compared to the average score of the state, or to the average of all new registrants obtained from a Secretary of State file update.

78

UNIVERSE IMPLEMENTATION

PERSUASION

CREATING A FIELD PERSUASION UNIVERSE

After the persuasion universe is built, the most important factors in adjusting it for field voter contact programs were contact rates and contactability – i.e. having a phone number and/or a walkable up to date address. Contact rate analysis went go across calls versus walks and day versus evening.

After the initial definition of the field persuasion universe (which varied usually within 10% of the overall persuasion universe and mail universes), in many states it became a dynamic, living list. For example, self-identified undecided voter could automatically become a prime persuasion target, for two reasons:

• The goal of the model is to predict what people will say when we talk to them – a self identification is

infinitely more valuable than the model.

• An identified voter is, almost by definition, contactable.

Identified leaners, either toward Barack Obama or John McCain, were treated differently in different places. Some used score screens – some used screens on undecideds as well. For example, an identified leaner toward Barack Obama who was modeled in to support was considered a supporter, but one modeled in to the non-base was a persuasion target; or a John McCain leaner in the non-base was still a target, but one modeled low was not. Decisions to make this used re-ID analysis, looking at what percentage of those leaners was still likely to change their mind or become undecided, and within what universes.

IMPLEMENTING THE UNIVERSE

As covered in the persuasion universe section, the universe has been cut to a size which matches capacity within that state. This capacity, however, is determined around a program of passes and attempts. Identified voters who support a candidate already are removed, but undecided voters continue to be contacted within a methodical timeline – after a certain number of attempts, within a pass schedule

In many states, the passes were clearly delineated and separated for the field to use, especially for calls. For example, “pass one” included the initial universe, “pass two” added back in undecided voters, who were again re-added in “pass three”.

In Ohio, each pass was further split in to three attempts. Any contacted voter was removed within attempts; so that attempting the universe 3 times within one pass meant that 53% of the universe was contacted assuming a declining contact rate (declining because with each attempt, the voters you’re calling have already been called and not home, so the odds that they were uncontactable went up.

Attempt Universe Size as

% of Original

Contact

Rate

% of Original

Universe Contacted

% Remaining for

Next Attempt

One 100% 25% 25% 75%

Two 75% 22% 17% 59%

Three 59% 19% 11% 47%

Final sum of three attempts: 53%

TARGETING: UNIVERSE IMPLEMENTATION

79

In Missouri, each pass was one attempt: undecided voters were added back in immediately for the next pass, in the “second pass” universe. Field staff was also provided with a system to remove recently contacted voters, to adjust their attempts-program to their turf.

Both options, however, shared the commonalities of a dynamic universe, segmented out into varying phases of persuasion contact.

Where the universe was dynamic, it required monitoring and adjustments. For example, in both Ohio and Pennsylvania, in early September, the rate of identifying undecided voters outside the initial persuasion universe was greater than the rate of identifying decided voters within the persuasion universe. As such, the targeted persuasion universe was growing, not shrinking, over time. This fed back in to capacity projections, which relied on a universe shrinking at an expected rate as decided voters were removed. Capacity estimates had to be revisited, and program adjusted appropriately to ensure all targets were contacted effectively.

Call and walk universes needed different considerations – the key difference that targeted persuasion universes are rarely sufficiently dense for canvasses, and must be combined with other targets. This decision varied by state. In some cases, other non-base targets were included; often prioritized based on vote history: likely voting non base voters were a higher priority than a sporadic voting non base voter, because the contact only has one goal – to increase support – without also including a turnout push. Persuasion targets could also be combined with Democratic targets; especially once the VAN developed the technology to put targets on list reports for each individual voter. The target that a voter fell into could be designated on a walk sheet, and the volunteer could alternate scripts.

Example of target display on walk sheets:

The target of the voter is circled in red, displayed next to the voter. “Motivate” signifies a sporadic voter, who must be motivated to vote; while “persuade” signifies a persuadable voter. Volunteers carried both scripts with them.

Another consideration for both call and walk universes is tiering and building supplemental universes.

Tiering of the persuasion universe took in a myriad of factors, examples including:

• Prioritizing likely voters

• Prioritizing demographics with high contact rates

• Prioritizing identified undecided voters

• Prioritizing demographics with high undecided rates

These are all similar factors to the initial persuasion universe building, covered more thoroughly elsewhere in this section.

Supplemental universes also follow the same “rules” as the initial universe building, but adjusting the opposite way. Rather than tightening the criteria to find priority targets, criteria was loosened to find targets outside the original universe, but using the same factors - the “next best” targets.

Finally, especially in programs of ongoing persuasion, such as states which started persuasion and June and ran as late as November, the universe must be constantly revisited, assessed, and “purged” of uncontactable voters and bad targets. If contacted, decided voters are consistently removed, and contacted, undecided voters are recycled in and out through passes, what can happen is this skeleton of the universe left of people that just can’t be contacted;

TARGETING: UNIVERSE IMPLEMENTATION

80

who are consistently called over and over again without garnering an ID. Many states established systems of removing voters who hadn’t been contacted after a certain number of attempts from all persuasion contact attempts, to maintain contact rates and save time making unsuccessful calls.

SPECIFICS TO IMPLEMENTING A SPORADIC UNIVERSE

The field universe for sporadic voters in most states took on a “non-negotiable” quality. Unlike the persuasion universe, which was sized to capacity, once a sporadic-voting supporter was defined, it was prioritized equally with other sporadic-voting supporters.

However, the dynamism, supplementing, and purging of the universe followed the same prescription of the persuasion universe, with a few nuances.

The most important thing to consider with a field sporadic universe is contact rate: because these are voters who aren’t active, they are much less likely to have accurate information on the file. Increased rates of moved voters depress contact rates and can affect the program.

However, this list-cleaning is crucial, especially when it’s GOTV time and volunteer resources no longer make sense cleaning lists. As such, many programs made the decision to embark on heavy list cleaning efforts early on. In Ohio, volunteers doing voter registration “blind knocks” in supportive precincts knocked with a list, so they could gather and clean data as they went – even though voter contact and identification was not the goal of the canvass. By Election Day, the Ohio field program had cleaned over 600,000 addresses out of its canvass universe – the equivalent of 24,000 volunteer hours. Well over half of that list cleaning was concentrated in the three or four weeks from late September to late October, illustrated in the chart below.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Relationship Between Contact and Moved Rates

Ohio: Sept. 28 - Oct. 23rd

Contact Rate Moved Rate

81

POLLING

OVERVIEW

Political polling uses survey research to assess what percent of the electorate holds a particular opinion. Done well, political polling is the most accurate way of knowing how the electorate feels about a candidate or issue. A poll interviews a sample of likely voters in order to provide a picture of the opinions and beliefs that the likely electorate feels.

HORSE RACE

The most discussed question on a political poll is the horse race, or head-to-head match-up between candidates. This question is used to indicate which candidate is ahead at a particular point in time. The results of a horse race question can provide helpful insight into the current state of a race. Additional insights can be gained by examining the following

• Dynamics – A poll provides a statistically representative snapshot of how likely voters feel at that exact

moment. It should not be mistaken as a guide for how voters will always feel. The more unknown a

candidate or issue, the more voter opinion is likely to shift as they learn more.

• Intensity – While horserace numbers are often reported as two numbers (e.g. 42% to 35%), survey results

generally provide more detail on intensity. Polls often ask some variation of, “Are you certain to vote for

X, or might you still change your mind.” The results of this question can indicate both the overall fluidity

of the electorate, as well as which candidate has more committed supporters. This does not always

correlate with which candidate holds the lead in the horse race.

• Undecideds – The proportion of respondents that are undecided helps to indicate how much of the

electorate remains persuadable. If, for example, candidate X leads 42 to 35, but 21% of respondents

remain undecided, the race is considerably close than if candidate X leads 52 to 45, with 1% of

respondents undecided.

• Would not vote – The proportion of respondents that say they will not vote in a particular race gives an

indication of the enthusiasm about that particular race. More respondents who say they will not vote in a

race may indicate a low turnout, or a high drop-off rate for a down-ballot race.

• Refusals – Some respondents are unwilling to indicate their vote preference. While not generally a high

percent, in close races, refusals can be more significant.

• Margin of error – Any race that is within the margin of error should be considered a statistical tie. Only

if one candidate holds a lead greater than the margin of error can it be said that he or she holds a lead.

TARGETING: POLLING

82

ISSUES

Polling can provide helpful insight into what issues are on the top of voters’ minds, as well as into how they feel about a particular issue.

• Important Issues – One of the most common open-ended questions is some variation of, “what issue is

most important to you?” This question can provide insight into what issues voters are thinking about, and

what issues it is important for candidates to address.

• Specific Issues – Polling can provide insight into how voters feel about a specific issue. In particular, it is

often helpful in determining which issues voters have solid opinions of, and which issues voters are

undecided or neutral about. It can also indicate which issues voters generally agree on, and which issues

voters are more divided on. This can help candidates focus on issues on which the majority of voters and

the candidate hold the same position.

MESSAGING

Polling can be used to test the effectiveness of different messages. This can provide helpful insight into which is the most powerful argument on a particular issue, or for a particular candidate. There are many ways that polling can be used to provide this information. A few are outlined below

• Overall Persuasiveness – polling can be used to ask voters to rate a series of statements on a scale of

how persuasive or convincing they are. This can help determine which statements are the most persuasive

to the most people.

• Subgroup Persuasiveness – polling can be used to determine how persuasive different arguments are

with specific subgroups. This can be used in conjunction with paid ID targeting to determine what is the

most persuasive argument for a subgroup, which is determined to be more likely to be undecided, or to

respond to a particular message.

• Movement – another technique that can be used to measure the persuasiveness of an argument is

analyzing movement in response to new information. For example, the horse race is asked initially, the

persuasiveness of a battery of arguments is measured, and then the horse race is asked again. By doing

this, respondents who change their vote after hearing additional information can be isolated and

examined. An argument that is more effective with voters who move is more powerful.

• Movement and Regressions – another similar technique is to use regression analysis to determine which

arguments are the most salient predictors of movement. That is, the persuasiveness of which arguments is

the best indicator of whether a particular respondent is likely to change their mind about a candidate or

issue.

ATTRIBUTES

In candidate races, measuring attributes can provide insight into how voters’ perceive a candidate. There are two comparisons

that can provide helpful insight.

TARGETING: POLLING

83

• Comparisons Over Time – measuring a standard battery of attributes, such as “strong”, “intelligent”, or

“likely to bring change to Washington,” in a series of polling can provide helpful insight into how the

campaign is affecting voters opinions of a particular candidate. This provides more context and nuance

into how opinions are changing than simply examining the horse race question.

• Comparisons Between Candidates – measuring how two candidates compare on the same set of

attributes can be helpful in providing insight into where a candidate has advantages and disadvantages.

Further, if voters are unsure of which candidate is stronger on a particular attribute, it may indicate an

opportunity for a campaign.

LIKELY VOTERS

In order to accurately predict voter opinion, pollsters must first define who they believe is likely to vote. There are two levels of screening before a person is considered a likely voter.

• First, people voters have to be included on a list. Pollsters can either use a voter list or a Random Digit

Dial (RDD) list. A voter list is a list of people. The most narrow voter list is a list of registered voters who

have a specific vote history. The most inclusive voter list is a list of registered voters. An RDD list is even

more inclusive than a list of registered voters, including every phone number, prescreened for bad

numbers and businesses.

• Second, voters have to pass a likely voter screen. This is a question or a series of questions in a poll

designed to determine who is likely to vote. The simplest voter screen is asking respondents whether they

plan to vote. More complicated voter screens can measure engagement and knowledge about elections

and politics as a proxy for voter likelihood.

Important considerations in examining likely voters is what proportion of the electorate is expected to be new voters, what percent turnout a likely voter model assumes, and what the party distribution of a particular likely voter model is.

POLLING LIMITATIONS

Polls tend to be small – generally between 400 and 1,000 voters - for two main reasons: survey cost and survey depth. Polls cost a lot of money, and are designed to capture depth about the voters that they survey. As a result, they contain valuable voter information but can also present a series of statistical shortfalls that are important to keep in mind when interpreting polls for campaign management and decision-making.

There are three areas in which polls can provide misleading results:

• Small Sample Size – How the quantity of interviews affects accuracy

• Improper Sample Bias – How of the composition of the sample affects accuracy

• Improper Survey Structure – How the structure of a survey affects accuracy

TARGETING: POLLING

84

SAMPLE SIZE

The problem that sample size presents is variance. Generally, as sample size decreases, variance increases. Variance and standard deviation measure how much the level of support moves from the average. This occurs in two ways: variance over time and internal variance among subgroups. To test this assumption we plot the standard deviation of Obama support against the average sample size of each demographic subgroup.

Ohio and Colorado: Average Sample Size and Standard Deviation of Support

This shows that as sample size decreases standard deviation increases, at an increasing rate.

This graph shows that the relationship is similar in multiple states. However, Ohio shows greater deviation of support as sample size approaches zero than Colorado does. However, it appears that for both of these states, a subgroup sample size of less than 100 appears to have a standard deviation of 10% or more, meaning that it is not statistically reliable.

VARIANCE OF SUBGROUP SUPPORT OVER TIME

The relationship between sample size and standard deviation determines the accuracy of a poll. This is illustrated below, showing the variance of three subgroups over time. Subgroups with a smaller sample size experience greater variance.

Ohio and Colorado: Movement of Sub-Demographics from the Average over Time

R² = 0.7024

0%

10%

20%

30%

40%

0 200 400 600 800

Sta

nd

ard

De

via

tio

n

Average Sample Size

R² = 0.6213

0%

10%

20%

30%

40%

0 200 400 600 800S

tan

da

rd D

ev

iati

on

Average Sample Size

-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

6/1 8/1 9/16 9/25 10/2 10/9 10/16

Va

ria

nce

fro

m A

ve

rag

e B

O S

up

po

rt

Polling Date

White

(n=655)

White,

Male

(n=310)

White,

Male,

<35

years

(n=83)

-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

6/1 8/1 9/16 9/25 10/2 10/9 10/16

Va

ria

nce

fro

m A

ve

rag

e B

O S

up

po

rt

Polling Date

White

(n=655)

White,

Male

(n=310)

White,

Male,

<35

years

(n=83)

TARGETING: POLLING

85

As sample size, n, decreases, variation in Obama support is greater over time. For small subgroups, this can give the impression of movement in support where no movement actually exists. However, there the magnitude of the effect varies across states. Whites males under 35 show a dramatic spike in Ohio, while this effect does not appear in Colorado or with White men overall. This may have been a real spike, but is likely a random fluctuation caused by an insufficient sample size.

SAMPLE BIAS

Sample bias relates to the distribution of the polling sample and its effect on the data. Sample bias is mostly outside of the control of the pollster, because of the challenge of reaching certain demographic groups with varying level of success. The ideal pool should be a representative sample of the demographics of the given geographical area. To counteract this problem our campaign used the Polling Vote Model to weight the sample based on the demographics of the likely electorate.

WHO DOES IT AFFECT?

Sample bias leads to an under representation of an Obama supporter in a poll. This is mainly driven by the demographic distribution of who a pollster is able to interview.

The following table describes the distribution of the polling sample in relation to the likely voting population as a whole for Ohio. The positive numbers are areas in which the poll overrepresented a group and conversely a negative number shows where the poll sample underrepresented a group.

Support

Score

Poll Sample

Distribution

Registration

Date

Poll Sample

Distribution Marital Status

Poll Sample

Distribution

0-9 4% < 2004 7% Married 13%

10-20 5% 2004 -3% Single -12%

20-29 0% 2005 -1%

30-39 -1% 2006 1% Wealth Poll Sample

40-49 -5% 2007 0% 0-30,000 -12%

50-59 -5% 2008 -6% 30,001-60,000 0%

60-69 0% 60,001-100,000 1%

70-79 4% Gender Poll Sample 100,001-300,000 9%

80-89 -1% Female -5% 300,001-1,000,000 5%

90-100 0% Male 6% >1,000,000 1%

This table shows that the distribution of the sample is not representative of the distribution of the voting

electorate. The polling sample is skewed in the following directions:

• Likely McCain Supporters

• Male

• Wealthy

• Married

• Long-time voters

Pollsters often attempt to remedy this problem by weighting their samples so they more accurately represent the voting electorate. The assumption in weighting is that there isn’t anything inherently different about people within a subgroup who answer a telephone survey and people who don’t.

Pew Research has conducted research indicating that respondents that require the most effort in for a survey are more likely to be youth, African Americans, and people who live in urban settings.

CELL PHONE BIAS

Over the last few election cycles, there has been a significant rise in the number of cellhas caused concerns about a potential bias in polling sample, because of the difficulty of many pollsters to reach cell phone only users. Pew Research estimates that 17% of adults are cell phone only reachable with an annual increase of 2% a year. During this election cycle there was a limited group of polls that incorporated cell phone only users into their samples, and their findings have important implications for understanding how this bias affects polling data.

The cell phone only problem seems to affect effect on youth.

The effect of a cell phone bias is unsubstantial in comparing the landline and combined numbers of all voters. There is only a slight increase in Obama support of 2 prespondents Obama’s support rises to 55%. This can be attributed to the celldisproportionately being under the age of 30. Pew found that there was no difference between landline andcombined data for adults over 30, but that the effect on voters under 30 is substantial. As illustrated above, support for Obama was 10 points higher among under 30 cell

The effect of this bias on a poll is linked to the turnout estimaincrease in youth turnout could substantially undermine the predictive quality of polls who only sample land line only voters.

45% 46%

45% 44%

10% 10%

0%

20%

40%

60%

80%

100%

120%

Landline

(n=1960)

Combined

(n=2509)

Cell Phone Only

Su

pp

ort

%

All Voters: Cell Phone Effect

TARGETING: POLLING

86

Pollsters often attempt to remedy this problem by weighting their samples so they more accurately represent the he assumption in weighting is that there isn’t anything inherently different about people within

a subgroup who answer a telephone survey and people who don’t.

Pew Research has conducted research indicating that respondents that require the most effort in for a survey are more likely to be youth, African Americans, and people who live in urban settings.

Over the last few election cycles, there has been a significant rise in the number of cell-phone only users. This aused concerns about a potential bias in polling sample, because of the difficulty of many pollsters to reach

cell phone only users. Pew Research estimates that 17% of adults are cell phone only reachable with an annual election cycle there was a limited group of polls that incorporated cell phone

only users into their samples, and their findings have important implications for understanding how this bias

The cell phone only problem seems to affect different demographic groups to a varying degree, with the largest

The effect of a cell phone bias is unsubstantial in comparing the landline and combined numbers of all voters. There is only a slight increase in Obama support of 2 points. However, when looking at cellrespondents Obama’s support rises to 55%. This can be attributed to the cell-phone only respondents disproportionately being under the age of 30. Pew found that there was no difference between landline andcombined data for adults over 30, but that the effect on voters under 30 is substantial. As illustrated above, support for Obama was 10 points higher among under 30 cell-phone only voters.

The effect of this bias on a poll is linked to the turnout estimates for 18-29 year olds. A greater than expected increase in youth turnout could substantially undermine the predictive quality of polls who only sample land line

55%

36%

9%

Cell Phone Only

(n=176)

All Voters: Cell Phone Effect

52%

39%

9%

0%

20%

40%

60%

80%

100%

120%

Landline (n=250) Cell Phone Only (n=146)

Su

pp

ort

%

Youth 18 - 29: Cell Phone Effect

Pollsters often attempt to remedy this problem by weighting their samples so they more accurately represent the he assumption in weighting is that there isn’t anything inherently different about people within

Pew Research has conducted research indicating that respondents that require the most effort in order to interview for a survey are more likely to be youth, African Americans, and people who live in urban settings.5

phone only users. This aused concerns about a potential bias in polling sample, because of the difficulty of many pollsters to reach

cell phone only users. Pew Research estimates that 17% of adults are cell phone only reachable with an annual election cycle there was a limited group of polls that incorporated cell phone

only users into their samples, and their findings have important implications for understanding how this bias

different demographic groups to a varying degree, with the largest

The effect of a cell phone bias is unsubstantial in comparing the landline and combined numbers of all voters. oints. However, when looking at cell-phone only

phone only respondents disproportionately being under the age of 30. Pew found that there was no difference between landline and combined data for adults over 30, but that the effect on voters under 30 is substantial. As illustrated above,

29 year olds. A greater than expected increase in youth turnout could substantially undermine the predictive quality of polls who only sample land line

62%

27%

11%

Cell Phone Only (n=146)

29: Cell Phone Effect

Other

McCain

Obama

TARGETING: POLLING

87

SURVEY STRUCTURE

The structure of a survey can have an effect on the polling data in 4 important ways.

• Question Order Effect –This occurs when the ordering of the issues being discussed affects the

responses. The magnitude of the effect is greater when the respondent is less informed about the subject.

Research has found that this bias is most pronounced on less prominent issues and among less educated

voters.

• Response Order Effect – This occurs when one response category is asked first. For example, “if the

election were held today, would you vote for Barack Obama or John McCain?” This effect is easily

remedied by rotating which response is offered first.

• Acquiescence Bias – This occurs when respondents are more likely to reply affirmatively, than

negatively. For example, respondents may be more likely to answer agree to, “Do you agree or disagree

that Barack Obama isn’t ready to be President?” Questions should be alternated between being asked

positively and negatively to avoid bias.

• Forced Choice Bias – This occurs when respondents are forced to make a false choice, For example,

respondents are more likely to respond “not enough” in response to, “Does Barack Obama hate too much

or not enough experience?”, than if the question was, “Does Barack Obama have too much, note enough,

or just the right amount of experience to be President?” To remedy this question should have exhaustive

options including “don’t know.”

INDEX

1. Ohio Polling data source: Harstad Strategic Research, June - October 2008.

2. Colorado Polling data source: Harstad Strategic Research, June – October 2008

3. Distribution data: Harstad Strategic Research and Catalist voter registration data

4. ”How Different are People Who Don’t Respond to Pollsters?" http://pewresearch.org/pubs/807/how-

different-are-people-who-dont-respond-to-pollsters

5. “Cell Phones and the 2008 Vote”, http://pewresearch.org/pubs/964/

APPENDIX – SAMPLE BIAS GRAPHS

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

Po

ll -

Po

p D

istr

ibu

tio

n

Age

Age Distribution

-5%

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

5%

Rural Suburban

Po

ll -

Po

p D

istr

ibu

tio

n

Population Density

Geographical Distribution

TARGETING: POLLING

88

SAMPLE BIAS GRAPHS

-15%

-10%

-5%

0%

5%

10%

15%

Po

ll

-P

op

Dis

trib

uti

on

Wealth

Wealth Distribution

Urban

Population Density

Geographical Distribution

-15%

-10%

-5%

0%

5%

10%

15%

Married

Po

ll -

Po

p D

istr

ub

tio

n

Marital Status

Marital Status Distribution

Wealth Distribution

Single

Marital Status

Marital Status Distribution

89

PAID MEDIA TARGETING

PURPOSE

Improved targeting can make paid media – including TV, radio, print, etc. - more effective and less expensive in both time and money spent. Due to the varying sizes, demographics and costs to advertise in each media market, certain markets are more efficient and cost-effective than others. When trying to decide between putting a TV ad on the air in one media market versus another, having a numeric and metric-driven understanding of the efficiency and cost-effectiveness of each market can be very useful in resource allocation, both within and across states.

WHY IT’S IMPORTANT, WHAT IT DOES

• The most important aspect of paid media targeting is its potential for resource allocation. When resources

are scarce, decisions have to be made about how to divide those resources both within states and across

states. By providing a set of data to help visualize and analyze each situation, it allows for context to be

added to informed decision-making.

O Inter-state resource allocation: It is very difficult to assess the allocation of resources across

states in a national campaign. Each state has its own needs and is essentially running its own

campaign. However, when dealing with budgets such as a paid media budget – which is

distributed across states – it becomes necessary to determine the appropriate metrics for the

apportionment of those resources.

O Intra-state resource allocation: In addition to determining budgets across states, including the

state leadership in the discussions of which metrics are the most useful in each individual state,

the budget can be best utilized across the media markets of a particular state. This is especially

useful in states like Ohio and Florida which have a number of media market with significantly

different demographics, and quite a variance in cost-effectiveness.

• In addition to providing an overall backdrop and context with regard to the allocation of resources, paid

media targeting can be tremendously useful for both constituency-specific and campaign timeline-specific

paid media needs. It is about cost optimization and having information to back-up decision-making.

O Being able to quantify the value of each media market with regard to distinct demographics

allows for the most efficient use of resources with regard to each specific need. For example, by

combining the field information indicating who is part of the GOTV universe, it becomes

possible to analyze the most efficient markets in each state and across the states for targeted

GOTV buys. The same applies for persuasion ads early on in the campaign.

O Due to the vastly different demographic make-ups of each market, there are certain areas where a

much different type of messaging will be needed. Measuring those metrics and quantifying them

provides very useful context and replaces intuition and gut-instinct with data-driven decision-

making.

TARGETING: PAID MEDIA

90

• The final aspect of paid media targeting that is useful, and is perhaps the most tangible, is the ability to

make the paid media environment transparent. Visualizing information is often the best way to illustrate

a point and by providing the paid media information in sensible ways, it becomes easier to understand the

current situation and make informed decisions in response to, or in anticipation of, the paid media

environment throughout the campaign.

SPECIFIC APPLICATIONS OF PAID MEDIA TARGETING

Determining the efficiency and effectiveness of markets can prove very helpful in the allocation of limited resources. This can apply to any of the following examples of paid and earned media opportunities:

• TV and Radio Ad buys

• These can include general persuasion or GOTV ads.

• Constituency-Specific Ad buys

• Ads that are aimed specifically at Latinos, African Americans, rural voters, youth, suburban voters, etc.,

all have different markets that are the most effective and efficient. Knowing which markets are the best

for each constituency can be very helpful.

• Satellite Interview Scheduling

• When determining which media markets to schedule an interview between the candidate or surrogates and

a local news or radio station, being able to see which markets we are behind our goals or where the

opposing candidate had recently visited – or is planning to visit – can make the limited candidate’s time

much more effective.

• Travel Scheduling

• The same principles could apply when determining principle and surrogate travel schedules.

TYPES OF DATA USED

We created a database of every media market in our 22 (23?) battleground states that contained all of the below pieces of data:

Data sets Source (s) Specific Examples

Demographic

data VoteBuilder, Catalist

% of each media market that is African American; # of Latino voters in each market; large concentrations of young and unregistered populations

Field data VoteBuilder # of persuasion voters in each market; % of each market that is in the GOTV universe

Buy data GMMB, cmag (TNS-MI) Point levels and dollars spent in each market by each campaign; the types of ads and messages being delivered

Polling data Pollsters Current horse-race numbers by media market; distance to goals; favorability measures

Travel data Scheduling and Research Dept’s

Principle and surrogate travel schedules for our campaign and theirs from our scheduling and research departments, respectively


91

• Demographic data by media market

o Using counts and crosstabs from both VAN and Catalist, data such as the number of the African

Americans in each market and the % of the market’s population that were African American can

be included in analyses. The same is true for Latinos, rural voters, youth, elderly and

unregistered populations. For example, the Atlanta, Detroit and Philadelphia media market have

the greatest African American populations, but the Columbus (GA), Albany (GA) and Augusta

(GA) markets all have the highest African American %.

• Field data by media market

o Counts for each market of the number of voters in the persuasion and GOTV universes, as well as

the % of each market that fell into each category should also be included.

• Buy data

o The most important data set for determining the paid media environment is the buy data. It

indicates the point levels that both campaigns were buying advertising in each market, how much

they were spending and what types of ads they were airing.

o GRPs – Gross Rating Points are the measure of exposure for a particular advertisement. 1 GRP is

generally considered to be the equivalent of reaching 1% of the media market’s target audience.

A fairly typical amount of exposure during the persuasion period of the campaign was about 800

– 1000 GRPs per week per market. Rather than thinking about it as 800% - 1000% of the target

audience, another way to describe it would be to say that the average viewer in that market will

see the ad 8 – 10 times over the course of the ad’s airing (usually one week). Those numbers

increased to upwards of 2500 – 3000 GRPs during GOTV and the last few weeks of the

campaign.

o In addition the weekly GRP amounts for each campaign, it is also important to receive estimates

of the Cost per Point (CPP) for each market. The CPP is largely based on the population of the

media market with markets such as Washington, DC, Philadelphia and Atlanta representing some

of the most expensive (~$400 – 500) and markets such as Glendive (MT) and Alpena (MI)

representing some of the least expensive (~$10 – 25). So, an ad that is to be aired at 800 GRPs in

the Philadelphia media market would cost roughly $400 * 800 = $320,000. Covering that same

amount of the Glendive media market would cost roughly $10 * 800 = $8,000.

o The CPP of a media market is not necessarily, however, a good indicator of how cost-effective

the market is. The best way to calculate the efficiency of a market is by looking at the cost per

point per voter (or per non-base voter for persuasion purposes). The chart below of two media

markets in Michigan illustrates this example:

Media Market % of Non-base Goal

# of Non-base Voters

Cost per Point (CPP)

CPP per 1K Non-base Votes

ALPENA 1% 8,418 $26 $3.09 DETROIT 43% 562,408 $326 $0.58

o Even though it is far less expensive to advertise in the Alpena media market, when the number of

non-base voters that can potentially be reached in a market is taken into account, Alpena is

actually a very inefficient market. It also shows that even though the Detroit market is very

expensive in raw cost, it is a very efficient market since it reaches so many non-base voters.


92

o In addition, keeping track of the ad messaging (Economy, Foreign Policy, etc) and whether the

ads are positive or negative is very important. It provides for the analysis of which types of ads

were effective in each market and also gives insight into the type of strategy each campaign is

deploying.

• Polling data by media market

o Polls should be broken down by media market (where sample sizes provide for statistically

significant results) so that it is possible to see how the candidate is doing in each market and

where there is potentially room to grow among non-base voters.

• Travel data

o Daily data on where each of the eight principles (BO, MO, JRB, JBB, JM, CM, SP, TP) and

select surrogates (HRC, WJC, President Bush) travelled to do public events and when is also very

important. Including this data into the database creates the ability to target markets where the

McCain campaign had recently visited and where we had not.

ALLOCATING SPEND ACROSS MARKETS

After gathering all of the above data into the database, it is important to create a model that scores each market according to the different variables. Some of the more important variables should include CPP per voter, % of the market that is in the persuasion universe, state tier, ad buy spending differentials, distance to goal, travel differentials and African American population. Each of these variables should be given a different weight according to the type of paid media pertinent to the situation. Different weighting schemes can range from “Persuasion Ads” to “Af-Am GOTV Ads” to “Principle Satellite Interviews”, with each scheme weighting the variables differently

For example, in the “Persuasion Ads” scheme, the “Persuasion Universe Market Share” variable should be weighted fairly high, as should the “Distance to Goal” from the polling. Constituency-specific variables such as “% Latino” and “% rural” should be given weights of zero. The weightings for each scheme will vary and should constantly be updated and changed to become more accurate. The results will be every media market ranked according to these variables:

Persuasion Ads Rank

(national) Market, State Targets Score

1 Cleveland, OH Af-Am, Suburban Women, Youth 34.4190

2 Orlando_Daytona, FL Hispanic/Latino, Independents, Af-Am 32.9031

3 Columbus_OH, OH Suburban Women, Af-Am, Independents 32.8375

4 Denver, CO Suburban Women, Independents, Youth 32.7904

5 Springfield_MO, MO Independents, Suburban Women, Youth 30.5941


93

THINGS TO KEEP IN MIND

The media market rankings are not meant to be considered a final or complete list. They are meant to aid in decision-making and allow for specific data to provide the backdrop for resource allocation. Some of the more useful times that the ranking model was used were for targeting a rural-specific radio ad and Af-Am GOTV ads. The reasons behind this were that without the data displaying all of the information, it is often easy to overlook certain markets. The other benefit to resource allocation was the ability to view the most efficient markets within states and also across states.

GIS AND PAID MEDIA TARGETING

As mentioned before, one of the key aspects of the paid media targeting analysis is the ability to visualize the paid media environment. A key component of this was mapping. This is very useful in providing a “big-picture” style glimpse at the situations both within and across states. Below is an example of the difference in point level buys for the campaigns from one week in September. Redder markets indicate places where the McCain campaign outspent us.

94

POINT BUY DIFFERENTIAL (OFA + SURROGATES VS. MCCAIN + SURROGATES)

September 8 – September 14, 2008

•

Targeting Compiled Formatted v8

Documents

Transcript of Targeting Compiled Formatted v8