IDEV 624 – Monitoring and Evaluation

Evaluating Program Impact

Elke de Buhr, PhDPayson Center for International Development

Tulane University

Process vs. Outcome/Impact Monitoring

Process Monitoring Outcome Impact

Monitoring Evaluation

LFM

USAID Results Framework

04/20/23

What is the problem? Situation Analysis & Surveillance

What are the contributing factors?

Determinants Research

What interventions and resources are needed? Needs, Resource, Response Analysis & Input Monitoring

What interventions can work (efficacy & effectiveness)? Efficacy & Effectiveness Studies, Formative & Summative Evaluation, Research Synthesis

Are we implementing the program as planned? Outputs Monitoring

What are we doing? Are we doing it right?Process Monitoring & Evaluation, Quality Assessments

Are interventions working/making a difference? Outcome Evaluation Studies

Are collective efforts being implemented on a large enough scale to impact the epidemic? (coverage; impact)? Surveys & Surveillance

Understanding Potential Responses

Monitoring & Evaluating National Programs

Determining Collective Effectiveness

ACTIVITIES

OUTPUTS

INPUTS

OUTCOMES

OUTCOMES & IMPACTS

A Public Health Questions Approach to HIV/AIDS M&E

Are we doing the right things?

Are we doing them right?

Are we doing them on a large enough scale?

Problem Identification

(UNAIDS 2008)

04/20/23

Most Some Few*All

Input/ Output Monitoring

Input/ Output Monitoring

Process EvaluationProcess

EvaluationOutcome

Monitoring / Evaluation

Outcome Monitoring / Evaluation

Levels of Monitoring & Evaluation EffortLevels of Monitoring & Evaluation Effort

Number of

Projects

Number of

Projects

*Disease impact monitoring is synonymous with disease surveillance and should be part of all national-level efforts, but cannot be easily linked to specific projects

Strategic Planning for M&E: Setting Realistic Expectations

4

Impact Monitoring / Evaluation

Impact Monitoring / Evaluation

Monitoring Strategy

• Process Activities

• Outcome/Impact Goals and Objectives

Impact Evaluation

Impact Evaluation• Impact evaluations are undertaken to find out whether

a program has accomplished its intended effects • Directed at the net effects of an intervention,

impact evaluations produce "an estimate of the impact of the intervention uncontaminated by the influence of other processes and events that also may affect the behavior or conditions at which the social program being evaluated is directed” (Rossi/Freeman 1989: 229)

• Ideally, impact assessments establish causality by means of a randomized experiment

Outcome vs. Impact

• Outcome level: Status of an outcome at some point of time

• Outcome change: Difference between outcome levels at different points in time

• Impact/program effect: Proportion of an outcome change that can be attributed uniquely to a program as opposed to the influence of some other factor

(Rossi/Lipsey/Freeman 2004)

Outcome vs. Impact (cont.)• Impact/program effect: the value added or

net gain that would not have occurred without the program and the only part of the outcome for which the program can honestly take credit– Most demanding evaluation task– Time-consuming and costly

(Rossi/Lipsey/Freeman 2004: 207)

Outline of an Impact Evaluation

1. Unit of analysis

2. Research question/hypothesis

3. Evaluation design

4. Sampling method

5. Impact indicators

6. Data analysis plan

1. Unit of Analysis

Unit of Analysis• Unit of analysis: The units on which outcome

measures are taken in an impact assessment and, correspondingly, the units on which data are available for analysis

• The unit of analysis in impact assessments is determined by 1. the nature of the intervention and

2. the targets to which the intervention is directed

• Can be individuals, households, neighborhoods, organizations, geographic areas, etc.

(Rossi/Lipsey/Freeman 2004)

What are your program’s units of analysis?

2. Research Question/Hypothesis

Hypothesis

• Hypothesis: Formal statement that predicts relationship between one or more factors and the problem under study

• Support or reject the null hypothesis

• Null = no relationship

• Test:– Compare same variable over time– Comparison between two or more groups

Can you formulate a null hypothesis for your program?

3. Evaluation Design

Evaluation Designs• Evaluation strategies:

– Comparisons over time– Comparison between groups

• Research designs:– Pre-test/Post-test designs– Time series– Quasi-experiments– Randomized experiments

Comparisons Over Time

Time

XO1 O2

Time

O4 O6O2O1 O3 X O5

Time

O3 O4XO1 O2 X X

Pretest/Post-test design

Longitudinal designs /

Time series

Effect of Intervention?

(Fisher, A A and J R Foreit Designing HIV/AIDS Intervention Studies: An Operations Handbook Population Council: May 2002, p.56)

Effect of Intervention?

(Fisher and Foreit, p.57)

Effect of Intervention?

(Fisher and Foreit, p. 57)

Effect of Intervention?

(Fisher and Foreit, p. 58)

Comparisons Between Groups

Time

XO1 O2

O4O3

Experimental group

Comparison group

Time

XO1 O2

O4O3

Experimental group

Control groupR

Quasi-experimental

design

Experimental design

Randomized Experiments• “Flagships of impact assessment”

(Rossi/Lipsey/Freeman 2004: 262)

• When conducted well, provide the most credible conclusions about program effects

• Isolate the effects of the intervention being evaluated by ensuring that intervention and control group are statistically equivalent except for the intervention received

• In practice, it is sufficient if groups, as aggregates, are comparable with regard to any characteristic relevant to the outcome

Randomization• Randomization: Assignment of potential targets to

intervention and control groups on the basis of chance so that every unit in a target population has the same probability as any other to be selected for either group

• Approximations of randomization: Acceptable if the groups that are being compared do not differ on any characteristic relevant to the intervention or the expected outcomes ( Quasi-experiments)

(Rossi/Lipsey/Freeman 2004)

Feasible?• Randomized experiments are not feasible for all impact

assessments• Results may be ambiguous if

– program in early stages of implementation– interventions change in ways experiments cannot easily

capture

• In addition, the method may – be perceived as unfair or unethical (requires withholding

services from parts of the target population) – be too resource intensive (technical expertise, time, costs, etc.)– cause disruption in program procedures for delivering services,

create artificial situation

Quasi-Experimental Designs

• Often used when it is not feasible to randomly assign targets to intervention and control groups

• Types of quasi-experimental designs: matched controls, statistical controls, reflexive controls, etc.

• Threats to validity: Selection bias, secular trends, interfering events, maturation

Threats to Validity

Threats to Internal Validity

• INTERNAL VALIDITY: Any changes that are observed in the dependent variable are due to the effect of the independent variable. They are not due to some other independent variables (extraneous variables, alternative explanations, rival hypotheses). The extraneous variables need to be controlled for in order to be sure that any results are due to the treatment and thus the study is internally valid.

• Threat of History: Study participants may have had outside learning experiences and enhanced their knowledge on a topic and thus score better when they are assessed after an intervention independent from the impact of the intervention. (No control group)

• Threat of Maturation: Study participants may have matured in their ability to understand concepts and developed learning skills over time and thus score better when they are assessed after an intervention independent from the impact of the intervention. (No control group)

• Threat of Mortality: Study participants may drop out and do not participate in all measures. Those that drop out are likely to differ from those that continue to participate. (No pretest)

• Treat of Testing: Study participants might do better on the posttest compared to the pretest simply because they take the same test a second time.

• Threat of Instrumentation: The posttest may have been revised or otherwise modified compared to the pretest and the two test are not comparable anymore.

• John Henry Effect: Control group may try extra hard after not becoming part of the “chosen” group (compensatory rivalry). • Resentful Demoralization of Control Group: Opposite of John Henry Effect. Control group may be demoralized and

perform below normal after not becoming part of the “chosen” group.• Compensatory Equalization: Control group may feel disadvantaged for not being part of the “chosen” group and receive

extra resources to keep everybody happy. This can cloud the effect if the intervention.• Statistical Regression: Threat to validity in cases in which the researcher uses extreme groups as study participants that

have been selected based on test scores. Due to the role that chance plays in test scores, the scores of students that score at the bottom of the normal curve are likely to go up, the scores of those that score at the top will go down if they are assessed a second time.

• Differential Selection: Experimental and control group differ in its characteristics. This may influence the results. • Selection-Maturation Interaction: Combines the threats to validity described as differential selection and maturation. If

experimental and control group differ in important respects, as for example age, differences in achievement might be due to this maturational characteristic rather than the treatment.

• Experimental Treatment Diffusion: Close proximity of treatment and control group might result in treatment diffusion. This clouds the effect of the intervention.

Threats to Validity Matrix

History Matura-tion

Mortality Testing Instru-mentation

John Henry Effect

Compensa-tory Equali-zation

Differen-tial Selection

One-Shot Case Study

YES YES YES - - - - -

One-Group Pretest-Posttest Design

YES YES CONT. YES MAYBE - - -

Time Series Design YES CONT. CONT. YES MAYBE - - -

Pretest-Posttest Control Group Design

CONT. CONT. CONT. CONT. CONT. MAYBE MAYBE CONT.

Posttest-Only Control Group Design

CONT. CONT. YES - - MAYBE MAYBE CONT.

Single-Factor Multiple Treatment Designs

CONT. CONT. CONT. CONT. CONT. MAYBE MAYBE CONT.

Solomon 4 – Group Design

CONT. CONT. CONT. CONT. CONT. MAYBE MAYBE CONT.

Factorial Design CONT. CONT. CONT. CONT. CONT. MAYBE MAYBE CONT.

Static-Group Comparison Design

CONT. CONT. YES - - MAYBE MAYBE YES

Nonequivalent Control Group Design

CONT. CONT. CONT. CONT. CONT. MAYBE MAYBE CONT.

Research Designs - Variations

A. Simple Designs

B. Cross-Sectional Studies

C. Longitudinal Studies

D. Experimental Designs

A. Simple Designs

• One-Shot Case Study

• One-Group Pretest-Posttest Design

• Time Series Design

X O

O X O

O O O O X O O O O

R = Random assignment of subjects to conditionsX = Experimental treatmentO = Observation of the dependent variable (pretest, posttest, interim measure, etc.)

B. Cross-Sectional Studies

Group 3

Group 1 Group 2

Comparison ofgroups. One pointin time.

Variations: Case-control study

Case-Control StudyGroup 1

(with

characteristic) Event(s)

Group 2 (without

characteristic)

Comparison ofgroups. One pointin time.

Major limitations: Cannot be sure that population has not changed since event(s).

C. Longitudinal Studies

PopulationPopulation Population

Comparison of population over time. Repeated measurements.

Variations: Panel study, Cohort study

Panel Study

Group 1Group 1 Group 1

Measures change over time. Repeated data collection from same individuals.

Major limitations: High drop-out rates pose threat to internal validity.

Cohort Study

Cohort (3)Cohort (1) Cohort (2)

Measures change over time. Repeated data collection from same cohort but different individuals.

Major limitations: Measures total change but fluctuations within cohort are not assessed.

D. Experimental Designs

Group 2

Experi-ment

Group 2

Group 1 Group 1

Pre-Test Post-Test

Compares group(s) exposed to treatment with group not exposed to treatment. Measures at two points of time.

Variations: True experimental design, Quasi-experimental design

True Experimental Design

Group 2

Experi-ment

Group 2

Group 1 Group 1

Pre-Test Post-Test

Compares group(s) exposed to treatment with group not exposed to treatment. Measures at two points of time. Research subjects are assigned randomly to treatment and control group.

Major limitations: Not feasible for all research & ethical problems.

Target population

Groups assignedrandomly.

True Experimental Designs• True experimental designs use control groups

and random assignment of participants

Variations:• Pretest-Posttest Control Group Design• Posttest-Only Control Group Design• Single-Factor Multiple Treatment Designs• Solomon 4 – Group Design• Factorial Design

Pretest-Posttest Control Group Design

• The randomly assigned experimental group receives the treatment and the control group receives no treatment or an alternative treatment

R O X OR O O

Posttest-Only Control Group Design

• Like previous but without pretest.

R X OR O

Single-Factor Multiple Treatment Designs

• Extension of Pretest-Posttest Control Group Design• Sample is assigned randomly to one of several

conditions

R O X1 OR O X2 OR O O

Solomon 4 – Group Design

• Developed by researchers that worried about the effect of pretesting on the validity of the results.

R O X OR O OR X OR O

Factorial Design

• Allows to include more than one independent variable.

• Test for the effects of different kinds of variables that might be expected to influence outcomes (gender, age, etc.).

Two Independent VariablesAB

A x B

Three Independent VariablesABC

A x BA x CB x C

A x B x C

Quasi-Experimental Design

Group 2

Experi-ment

Group 2

Group 1 Group 1

Pre-Test Post-Test

Compares group(s) exposed to treatment with group not exposed to treatment. Measures at two points of time. Random assignment not possible.

Major limitations: Not a true experiment. Threats to validity. ( Selection bias)

Target population

Groups not assignedrandomly.

Quasi-Experimental Designs• Quasi-experimental designs lack the random

assignment of experimental designs.

Variations:

• Static-Group

Comparison Design

• Nonequivalent Control

Group Design

X O-------------

O

O X O-------------

O O

Choosing an Evaluation Design

Impact Evaluation Strategy

• Comparison – Same group (over time)– Different groups

• Design balances accuracy and reliability with cost and feasibility

What is a “good enough” research design?

Research Design Flow-Chart

Research Design

ObservationalStudy

Experimental Study

Cross-Sectional Longitudinal Single Group True Experiment Quasi-Experiment

Survey ResearchParticipant Observation

Clinical ExperimentNatural Experiment

Methods Methods

Comparison Group Flow Chart

(Methodologist Toolchest, Version 3.0)

4. Sampling Methods

Sample Selection

• Sample size

• Sampling frame

• Sample selection = sampling– Probability sampling– Nonprobability sampling

Sampling Methods• Census vs. Sampling

– Census measures all units in a population– Sampling identifies and measures a subset of

individuals within the population

• Probability vs. Non-Probability Sampling– Probability sampling results in a sample that is

representative of the target population– A non-probability sample is not representative of

any population

Probability Sampling• Sample representative of the target population, large

sample size– Simple random/systematic sampling – Stratified random/systematic sampling– Cluster sampling– Experimental and quasi-experimental designs

.

Advantages• Findings representative

of the population• Advanced statistical

analysis

Disadvantages• Costly and time

consuming (depending on target population)

• Significant training needs

5. Impact Indicators

Concepts, Variables and Indicators

Example 1Example 1 Example 2Example 2 Example 3Example 3

ConceptsConcepts SizeSize Economic Economic well-beingwell-being

HealthHealth

VariablesVariables AreaArea Income per Income per capitacapita

Life Life ExpectancyExpectancy

IndicatorsIndicators Square Square kilometerskilometers

Purchasing Purchasing Power Parity Power Parity (PPP) GNP (PPP) GNP

($) per capita($) per capita

Average Average years of life years of life

if born in if born in 19701970

(Phuong Pham, Introduction to Quantitative Analysis)

Indicator Criteria1. Measurable (able to be recorded and

analyzed in quantitative or qualitative terms)

2. Precise (defined the same way by all people)

3. Consistent (not changing over time so that it always measures the same thing)

4. Sensitive (changing proportionally in response to actual changes in the condition or item being measured) 

Categorical vs. Continuous Variables

• Continuous variables– A variable that can be measured (weight,

height, age, etc.)• Categorical variables

– A variable that cannot be measured but can be categorized (ethnic group, age group, educational level, socio-economic class, etc.)

6. Data Analysis Plan

Data Analysis

• Type of variable– Categorical– Continuous

• Type of data analysis– Descriptive analysis– Hypothesis testing

Descriptive Analysis vs. Hypothesis Testing

• Descriptive data analysis– Organizing and summarizing data

• Statistical inference – Procedure by which we reach a

conclusion about a population on the basis of the information contained in a sample that has been drawn from that population

(Phuong Pham, Introduction to Quantitative Analysis)

Exercise

Exercise• Outline an Outcome and/or Impact Evaluation

for your program

• Include a description of:1. Unit of analysis

2. Research question/hypothesis

3. Evaluation design

4. Sampling method

5. Impact indicators

6. Data analysis plan