Planning Campaigns with Evaluation in mind

By Jorge Restrepo M.G.AEureka Facts LLC

Design• Clearly define the intended outcomes of a program and

ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal.

• Educational, language and cultural barriers are necessary considerations in good evaluation design.

• Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.

Design

• The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment.

• Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal.

• Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.

Methods - data collection

Tips for in-person data collection• Protocols and interviewer training are key• Supervision and redundant verifications are

important • Dress-codes and interviewer profile• Compensation/ Investment• Controls• Timing• Response rate

Methods - data collection

Mail surveys• Lists – sample frame• Who is your respondent?• Sponsor, package, timing, incentive• Respondent burden• Education/language• Coding visibly, invisible coding• Flexibility• Timing• Response rate - examine non-response closely

data collectionOnline

• Who is your respondent? - Your sample frame is everything

• Spam filters are a bigger concern than non-response

• Web links create disproportionate sampling• Make web links and pop-ups as unobtrusive

as possible• Respondent fatigue• CANSPAM • COPA

data collectionTelephone

• Who is your respondent? - Your sample frame is everything

• RDD sampling – often no longer representative• Combine methods • No land line households• Reach• Incentives• Cooperation• Do not call registry

Challenges in Analysis• Most distributions – especially in public education

efforts are not normal so common centrality statistics (mean, median) are not very meaningful.

• People often articulate categories (i.e. well, not well) better than continuum scales.

Analysis• Think outside the box – examine

findings spatially

Stars represent respondents to a campaign

Analysis• Think outside the box –

examine findings spatially

Flat findings examined by other means revealed strong differences by county

Simplify constructs• Build your assessment on existing frameworks

and models, but create high level aggregates as well

Mean Levels of Knowledge and Action

Source: Random survey

State

SCNCGAFL

Me

an

8

6

4

2

0

Know ledge

Action

This evaluation measures knowledge and action

High and Low Impact Groups• A powerful tool is the use of classification trees

to identify pockets of a population with high or low impact.

Compare - Always

Mirrored instruments are especially valuable in examining outcomes

Parents can move their child to a different school when their school has a failing grade.

0

20

40

60

80

100

Jan

Feb.

Mar

.Apr

ilM

ayJu

ne July

Aug.

Sept.

Oct. Nov.Dec.

Month

Random survey

Callback survey

Response cards

Segment for Success

Outcomes are rarely evenly spread out throughout a population

Common cross-tabbing analysis

• Ethnicity

• Hispanic Origin

• Gender

• Age

• Education

• Urban locale

• Generational cohort

• Socioeconomic status

• Life stage, career stage

• Primary language

• Geography

Who we are

Eureka Facts brings you the smart marketing Information you need to

discover opportunities

maximize results

identify issues

make decisions

reduce market risks

Smart Evaluation

www.EurekaFacts.com

EurekaFacts LLC451 Hungerford Drive

Suite 515 Rockville MD 20850

(301) 610-0590