1

PM 536Program Evaluation & Research

• Course Outline, Objectives & Expectations

• “Health Promotion Program Evaluation”

• Program Evaluation is Fun

• Program Evaluation is Difficult

2

Course Materials

1) Text (Everything you always wanted to know about program evaluation, but were afraid to ask)

2) Supplemental readings (Some case studies and other methodology points)

3) Lab materials

Labs

SAS/SPSS/STATA

4) Overhead copies (6 slides/page)

3

What is Evaluation

• Systematic application of research procedures that assess the conceptualization, design, implementation, and effectiveness intervention programs

Did it work

How effective was it

What does it mean

4

The Evaluation Field

Evaluation has had an explosive growth since WWII as social programs have expanded

Evaluation has emerged as a distinct field of inquiry boasting its own professional association and own journals.

Three stages of evaluation: Shadish, Cook and Leviton (1991)

1. Defining scientific rigor (Campbell & Stanley; Scriven, etc.)

2. Practicalities of evaluation in the field (Weiss, Wholey, Stake)

3. Integrate lessons to construct evaluation theory that is appropriate and rigorous (Cronbach, Rossi)

5

Evaluation can be Threatening

Acknowledge that evaluation can be “value” laden.

In reality evaluation is rarely used to terminate someone’s job

However, evaluation is used to decide on where to focus resources and this causes concern

Evaluation can be empowering

6

Why Do We Evaluate Programs

• Determine whether the program met its objectives• Force programmers to set objectives• Determine how the program worked (if it did)• Determine who was influenced by the program• Often required (Fed. Gov’t Acct. Act, GRPA)• Provides research opportunities• Plan for future programs• Interesting

7

Evaluation is Difficult

Control groups hard to createa. Hard to restrict programs (mass media)

b. People move

c. People talk

Programs are often culturally-specific

Control variables are hard to specify

Cost is often perceived to unjustifiable

8

Evaluation Framework

• Formative - pre-intervention planning

• Process - intervention implementation

• Summative - post-intervention impact analysis

9

Old Figure 1-1 Communication Campaign Evaluation FrameworkN

eeds

Ass

essm

ent/

Ide

ntif

icat

ion Formative

ResearchObservations

Depth Interviews

Focus Group Discusions

Key Informants

Community Mapping

SecondaryAnalysis

Dis

sem

inat

e F

indi

ngsDesign

Stage

Design & TestMessages

Design & TestInstruments

Design & TestMonitoringMethods

Sha

re F

indi

ngs

wit

h S

take

hold

ers

AnalysisStage

DetermineImpacts

DetermineOutcomes

Conduct Program & Effects Monitoring

Col

lect

Bas

elin

e D

ata

Col

lect

Fol

low

-up

Dat

a

Impl

emen

t Pro

gram

/Cam

paig

n

10

Figure 1-1. Program Evaluation FrameworkNeeds Identification and Assessment

Formative: Observations, Interviews, FGDs, and Community Mapping

Design: Design and Test Messages,Instruments, and Monitoring Methods

Data Collection:Collect Baseline DataMonitor Implementation of ProgramCollect Follow-up Data

Analysis: Determine Impact and Outcome

Interpretation: Share Findings with Stakeholders and Interpret Results

Disseminate Findings

11

Figure 1-2: Campaign Evaluation Framework

Formative Research Process Research Summative Research

ActivitiesFocus GroupsDepth InterviewsSecondary AnalysisParticipant Observation

Design Program Launch Program Program Ends

ActivitiesImplementation Monitoring (viewer logs, broadcast schedule)Effects Monitoring (sales data, visitation data or interviews)

ActivitiesAnalyze Survey DataKey Informant Interviews

Objectives, Understand:Barriers to ActionAppropriate LanguageConstellation of Factors

Objectives, Understand:Frequency of Broadcasts Potential Audience ReachPreliminary Data on Effects

Objectives, Understand:Level of EffectDegree of Efficiency

12

Formative - pre-intervention planning

• To develop messages

• Determine barriers to behavioral adoption

• Determine baseline benchmarks

13

Process - intervention implementation

• Determine the “reach & freq”

• Make adjustments to implementation

• Determine Impacts of program (esp. for long programs)

14

Summative - post-intervention impact analysis

1) determine impact

2) determine efficiency

3) future planning

15

Impact vs. Outcomes

• Immediate impact• Long-term outcomes• Many programs designed to change

outcomes in an overall model of behavior, but waiting until those outcomes are realized is unrealistic.

• Hence indicators are developed to assess programs for immediate impact.

16

Evaluators must be versatile

• Program changes

• Evaluation changes (usually for the worst)

• Programs and evaluations change– stakeholders change

– the priorities and resources change

• Program Stages - Evaluation should be tailored to the program stage such that more resources devoted to evaluating a pilot project or innovative program than an on-going one.

17

Chapter 2: Health Promotion Interventions (What Do We Evaluate)

• Provider Training - educational materials such as manuals, flyers

• CBD/CBO

• Community Mobilization

• Social Marketing

• Mass Media Advertising - Promotional campaigns

18

Interventions (cont.)

• Entertainment-education such as dramas, Soap operas, theater, film, music (songs, music videos, concerts)

• Interactive Health Communication

• Multi-media or community-wide Programs

19

Figure 2-1: Intervention Impact SpaceTradeoff Between Impact and Reach

Impact: Effect

on Audience

Reach: Percent of Audience Exposed to Message

One-OneCounseling

Outreach

Street Theater

Soap Opera

Mass Media PromotionIneffectiveProgramsIneffectivePrograms

EffectiveProgramsEffectivePrograms

community-wide programs

CommunityMobilization

20

Tailoring

• Objectives

• Program Stage

• Impact Models

21

Conditions Needed to Show Impact (4)

• Situational: Population can adopt behaviorPromoting behaviors that cannot be adopted is

frustratingBe sure creating a health promotion program, be sure

it is appropriate

22

Conditions for impact (cont.)

• Programmatic: Program delivers appropriate message, appropriately

1) reach

2) appeal

3) tailored

4) appropriate channels

5) messages match needs

6) coordinate supply

23

Programmatic conditions (cont)

7) logos & slogans

8) source credibility

9) source trustworthy

10) replication

11) call to action

12) continuous involvement

24

Programmatic conditions (cont.)

13) Maximize IPC

14) Feedback

15) Promise a reward

16) Use fear cautiously

25

Impact Conditions (cont.)

• Theoretical: Researchers and programmers understand the population and have an articulated theory of behavior change

• Methodological: Researchers appropriately measured population characteristics, mediators and outcomes

26

Chapter 3: Behavior Change Theory

Theories• Diffusion of Innovations• Hierarchy of Effects• Attitudes Intentions and Beliefs• Social Learning Theory (Self-Efficacy)• Health Belief Model• PRECEDE/PROCEEDPerspectives• Networks• Ecological Perspectives• Social Marketing• Framework vs. Theories

27

Theories

Diffusion of Innovations• "Diffusion is the process by which an innovation is

communicated through certain channels over time among the members of a social system” (Rogers, 1995).

• Innovativeness• Stages: Knowledge, Persuasion, Decision• Implementation, Confirmation• Characteristics of the Innovation (Radical vs.

Continuous; Perceived attributes)• Mathematical Curves and Properties

28

Figure 3-1: Typical Diffusion of Innovations Curves

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19

Time

Pe

rce

nt

Cumulative Adoption

(S-shaped)

New Adopters(Bell-shaped)

29

Figure 3-2: Knowledge Attitude and Practices Curves

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19

Time

Cum

ulat

ive

Perc

ent

Know ledge

Attitude

Practice

30

Figure 3-3: Campaigns/Programs Can Accelerate Behavior Change

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19

Time

Perc

ent Campaign

No Campaign

31

More TheoriesHierarchy of Effects

• Expanded diffusion stages to 12/16 steps to behavior change that are more measurable

Posits rate of progress through steps

Provides expected rates for each step (for calculating expected effects)

32

Hierarchy Model

VeryHigh High ModerateStep 80.0 60.0 40

1 80.0 60.0 40.02 64.0 48.0 32.03 51.2 38.4 25.64 41.0 30.7 20.55 32.8 24.6 16.46 26.2 19.7 13.17 21.0 15.7 10.58 16.8 12.6 8.49 13.4 10.1 6.7

10 10.7 8.1 5.411 8.6 6.4 4.312 6.9 5.2 3.4

33

More Theories

Prochaska’s Stage Model • Specifies cognitive behavioral change stages

Attitudes Intentions Motivations and Beliefs

Social Learning Theory and Self-efficacy• Bandura (1985) individuals imitate models who

are similar to the individual and likeable, and the behavior must be demonstrated and then rewarded

34

More Theories

Health Belief Model

• HBM states that individuals change health-related behaviors if their perceptions of: Risk, Severity, Susceptibility, Cost-effectiveness, Cues)

PRECEDE

• presents categories of factors that inhibit and facilitate behavior change: PREdisposing, Enabling, Reinforcing

35

Theories, Frameworks, Theories

• Frameworks guide implementation of programs and describes steps in the process.

• Theories explain behavior change and predict human behavior- they explain how and why change occurs

• Models are particular embodiments of theories • Hypotheses are testable statements derived from

theories or models.

36

Frameworks/Theories/Models (2)

• Thus, the framework guides program steps, the logic model of what is going to happen

• The theory explains why or how you expect the behavior to occur

• The model depicts the relationships expected between key variables.

37

Perspectives

• Perspectives are points of view that direct programmers and researchers to variables or activities that affect the program, but

• They do not provide a proscribed prediction on behavior change, rather they

• Specify variables or activities that affect the outcome and should be included in the models

38

Perspectives

Ecological• Ecological Perspective (emphasizes analysis and

intervention at multiple levels: individual, interpersonal, institutional, community and public policy)

• Individual behavior is conditioned on environmental factors

• Ecological levels include interpersonal, family, community, organizational, societal, policy

• Ecological levels may vary depending on behavior

Figure 3-4: Ecological Models Specify Multilevel Influences on Behavior

Individual

Community

Society

Interpersonal

Policy

40

Perspectives (cont.)

Social Network Analysis

• Relations rather than attributes; and Structural positions are important

• People have many different networks and they can both constrain or promote health behavior

• A community of scholars (INSNA) has been formed to study social networks

• Methods and techniques can be simple or quite complex

41

A Typical Social NetworkData come from one class asking students to name others whom they think would make good leaders

42

Social Marketing

“Why can’t we sell health like we sell soap?”

•Product

•Price

•Place

•Promotion

Perspectives (cont.)

43

Supply vs. Demand Perspective

• Behavior change and diffusion have both supply and demand characteristics:

• Supply refers the components of a product or service that involve making the service product available for consumption.

• For example, the supply-side of family planning services is the provision of contraceptive counseling and devices used to regulate fertility.

44

Supply vs. Demand

• Demand refers to the characteristics of the consumer population in terms of their desires and preferences for the service or product.

• For example, the degree that a given population chooses to use a contraceptive such as pill is their demand for the pill.

• Supply and demand factors interact in a dynamic way

Figure 3-5: Supply and Demand Variables and Interventions

Provision, accessibility, and quality ofservices and products

Examples

Number of locationsto get vaccines

Politeness of trained service providers

Knowledge, attitudes,and preferencesof the public

Examples

Target audience knowsof the importance of vaccines

Audience members believethat vaccines areimportant

Supply Demand

Integrated/comprehensive programsaddress both supply and demand

46

Use & Selection of Theory

• Theory helps define behavior and specify the mechanism thought to influence it

• Thus it provides the target concepts for the programs

1. Program design2. Variable measurement3. Setting Goals/objectives4. Theory vs. program failures

47

Theory for Program Design

• Theory can be used to design programs.• For example, if studies show that

adolescents are most influenced by their peers, programs should use peers to implement the program

• Theory says that people smoke because “it’s cool” programs should be designed to show that it’s not cool

48

Theory forDefining Setting Goals & Objectives

• Goals are general statements about what is to be achieved (Decrease unwanted fertility, Reduce smoking, Increase seatbelt use)

• Objectives are operational, specifications of goals and should be SMART: (Specific, Measurable, Achievable, Realistic, Timely)

49

Hints:

1. Use strong active verbs

2. State only one purpose or aim

3. Specify a single end-product or result

4. Specify timeframe

Examples:– Reduce homelessness by 10 percent– Decrease fertility to from 3.6 to 3.1– Reduce smoking by 35 percent

50

Absolute vs. Relative Objectives

• Absolute objectives specify that everyone (or no one) will have the condition - Percentage Point Change

• Relative objective establish some proportionate improvement - Percentage Change

• Modifications to objectives are often necessary

51

Figure 3-6: Weiss’ View of Program vs. Theory Failures.

ProgramSuccess

Theory Success

Desired Effect

sets in motion which leads to

ProgramSuccess

which does not lead to

ProgramFailure

does not set in motion

which may or may not lead to

Theory Failure

TheoryNot Tested

sets in motion Desired Effect

Desired Effect

52

Figure 3-7: Conceptual Model of Program Impact.

SES

Gender

Income

Education

PsychographicCharacteristics

FamilyCharacteristics

InterpersonalContacts

ProgramExposure

Knowledge

Attitude

Practice

53

Table 3-2 Behavior Change Theories

Theory Citation

Diffus ion of Innovations Rogers, 1995

Hierarchy of Effects McGuire, 1989

Stages of Change Prochaska et al., 1992

Attitude/ Behavior Fishbein & Ajzen, 1980

PRECEDE Greene,

Health Belief Model Becker, 1974

Networks Valente, 1995

Social Marketing Kotler & Roberto, 1989

Ecological Factors McElroy, 1989

54

Developing an Impact Model

• An intervention or impact model is an attempt to translate conceptual ideas regarding the behavior under study into hypotheses on which action can be based.

• There is usually a causal hypothesis which must consist of concepts that can be measured.

• Causal Hypothesis

• Intervention Hypothesis

• Action Hypothesis

• Manipulability & Feasibility

55

Theory Selection

• One may have the impression that specification of an impact model is a scholarly monastic activity involving one or two persons who cogitate about the impact of the intervention on the outcome measures and systematically sets down the relevant hypotheses.

1. Prior experience

2. Peer Information

3. Literature Reviews

4. Meta-analysis

56

Internal vs. External Evaluators:Internal Evaluators

Advantages

1. Learn from each other

2. Can work w/ programmers (to modify objectives, gather information, get cost estimates

3. Results are cumulatively compiled and fed into future programs

Disadvantages

1. Not perceived as credible

2. Not able to maintain objectivity

57

Chapter 4:Formative & Qualitative Research

• Programs should conduct considerable formative research to design strategy.

• Formative research significantly improves the likelihood that programs will be successful

• Include audience in problem definition, message design, pre-testing of materials

• Makes impact evaluation much easier

58

Good qualitative research

Consist of four characteristics (SIFT)

1) Systematic

2) Iterative

3) Flexible and

4) Triangulate

59

Depth Interviews

• Establish rapport• Maximize homophily (usually)• Use a semi-structured guide• Conducted in 3 phases: Introduction, body of

interview, closure (tie up loose ends, solicit information)

• Generally don’t try to extract quantitative data• No magic number on how many to conduct• Segment respondents (men/women, age groups, etc.)

60

Intercept Interviews

Solicit interviews at public places

1) Approval from all relevant persons

2) pre-test the instruments and be flexible to changing them based on early results

3) vary the days, times and persons collecting the data

Randomize to minimize selectivity bias

61

Focus Group Discussions

• ... a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, non-threatening environment” (Krueger 1994, p. 18)

• Recruit participants (4-12) in advance

• Have them be similar

• Moderator by trained expert

• Use an agreed upon guide

62

FGDs (cont.)

• Introduction phase

• Have participants introduce one another

• Usually last about 2 hours

• Provide refreshments

• Tape record (notify participants)

63

FGDs (cont.)

• Advantages: socially oriented, flexible, cheap, quick and have face validity.

• Disadvantages can be difficult to organize and analyze; require trained moderators; and their results can be unpredictable

• Used to pilot test materials and for impact evaluation.

64

Limitations to FGDs (cont.)

Can be misleading in 3 ways:

1) the group is influenced by a dominant personality,

2) the topic of study is heavily influenced by normative pressures of conformity, or

3) the topic is not of sufficient interest for participants to generate opinions.

65

Observational Studies

• Conducted to observe behavior in its natural setting.

• Declared and undeclared

• Mystery or simulated client studies, done by

1) researcher

2) hired community persons

3) service providers themselves

66

Pilot Testing of Materials

• Crucial to pilot test materials

• Can pilot test at various stages

• Can conduct experiments to pilot test

• Use qualitative techniques

67

Professional Review of Materials

It is often advantageous to have noted experts in the field review program materials:

1) To ensure accuracy

2) To act as experts in cases of controversy

68

Simulations

• Hypothetical scenarios, usually computer generated, to determine “what if” outcomes

• Important to ask: How relevant and meaningful are assumptions and ruls in the model: Isomorphic to reality?

• Used to estimate how long for behavior change to occur or project potential benefits of program expansion

69

Ratings & Coverage Data

• Ratings: % of HH saw program

• Share: % of viewing HH saw program

• Reach and Freq. measures important to have in any program evaluation

• Gather from independent sources and compare to Rs self-reports

• Hire monitors

70

Content Analysis

• Used to code and describe programmatic materials (TV shows, ad campaigns)

• Must decide on universe, sampling scheme, coding scheme, and formulate hypotheses

• Ex: Daytime soap operas portray alcohol consumption as stress reduction technique

• Code all drinking all stress behaviors

71

Case Study ResearchUsed under 4 conditions:

1) the program is unique and unrelated to other activities,

2) the program’s implementation is complicated and data collection would be unwieldily,

3) the population being addressed by the program is small and/or unique, and

4) the program does not have a clear goal or objective.

Methods:

1) Observations

2) Key informant interviews

3) Archival

72

Qualitative Techniques for Impact Evaluation

Reasons include:

1) costs constraints,

2) program is small and not expected to bring about much change,

3) quantitative data will be impossible to acquire, and

4) as a complement to quantitative impact evaluation.

Chapter 5:Process Evaluation

• Information about how a program is implemented

• Documentation for that implementation

• Program corrections mid-implementation

• Information useful for determining impact

• Information needed for replication

Literature is thin

• Literature on process research is thin because it is a relatively new preoccupation.

• Doesn’t often provide much data as outcome is still what is interesting.

• Process research is most interesting when programs fail (implement poorly).

• Contradiction in mutation

Terms

• Fidelity

• Dose

• Compatibility

• Reach & Freq

Longer the Program the Greater the Need for Process Evaluation

1) the resources devoted to the program can change,

2) the implementing agencies can change,

3) unanticipated problems with intervention delivery may (and do) occur, and

4) unanticipated problems may occur in the implementation of the evaluation.

Process Evaluation is Overlooked Because:

1. Assume uniform implementation (particularly for mass media stuff)

2. Scientists ignore issues of program & message creation

3. Assume program reception is uniform

4. Programs are often multi faceted so hard to track

78

5-1: Process Evaluation can be Conducted at 5 Points in the HP process

NeedIdentified

AudienceReception

ProgramCreated

ProgramImplemented

BehaviorChange

ServiceDelivery

Supplier

4

5

32

1

Six Points of Program Monitoring

1. Program Created

2. Program Disseminated

3. Audience Reception (different from outcome research in 3 ways)

4. Audience Effects

5. Product Supplied

6. Interorganizational relations

Uses of Process Research

1) Alert Programmers to Problems w/ Dissem.

Make mid-term corrections

2) Interference from other campaigns

3) Measure contamination

4) Replication

Challenge: Collecting and reporting data in a timely manner

81

Four Sources of Data for Monitoring (use checklists)

1. Direct observation (careful of reactivity a.k.a. Hawthorne effect)

2. Service record data (A few items of data gathered consistently and reliably are generally better for monitoring purposes than a more comprehensive set of information of doubtful reliability.)

3. Service provider data

4. Program participant data

82

Monitoring Delivery of Services

Is the actual service provided similar to that which is expected to occur?

1. No treatment or not enough treatment2. Treatment is unstandardized or uncontrolled

a. mode of delivery impairs treatmentb. treatment requires delivery system that is too

sophisticated

3. Varies across target populations - Unstandardized (e.g., head start, DARE)

How to Conduct Process Evaluation

• Can require trained researchers or audience members trained for such purposes

• Inspection of primary documents (memos, contracts, etc.)

• Interviews

A Note on the Distinction Between Research Types