There and Back Again - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/9/... · There and...

There and Back Again:

An Impact Evaluator’s Tale

Paul Gertler

University of California, Berkeley

July, 2015

Answer these questions

What is impact evaluation?

What makes a good impact

evaluation?

Why is impact evaluation

valuable?

1

2

3

Impact Evaluation

An assessment of the causal effect of a project , program or policy on beneficiary outcomes.

Estimates the change in outcomes attributable to the intervention.

Impact Evaluation Answers

What is effect of information on hand

washing, hygiene and child health?

Does paying primary health care workers

for performance improve access & quality?

Do early childhood education programs

improve subsequent learning?

Does expansion of urban sewares reduce

open defecation & improve health?

Impact Evaluation Answers

What was the effect of the program on

outcomes?

How much better off are the beneficiaries

because of the program/policy?

How would outcomes change if changed

program design?

Is the program cost-effective?


What is impact evaluation?


evaluation?

Why is impact evaluation

valuable?

1

2

3

How are Impact Evaluations Useful?

To inform program design

As an input to funding decisions

As a means of influencing ideas


As an input to funding decisions

PROGRESA / Oportunidades (Mexico)

Inform Program Design Influence IdeasInput to Funding Decisions

Evaluation shows significant impacts on education and health →

scaled up & adopted by new presidential administration.

• Households paid to send

children to school and

regular health checkups

• First evaluation of large-

scale CCT program

Families enrolled in Oportunidades

Roof Rain Water Cisterns -- Brazil


• Northeastern states

very dry

• Collect rain from roofs

during rainy season

• Store in cistern

Evaluation:

• No significant impact on health

• Families value cisterns

• Value of house & Depression

Low Cost Pre-Schools (Mozambique)


• Pre-schools constructed in 30

villages (Save the Children)

• Volunteer community members

trained to staff schools

Evaluation shows significant impacts on health and education

indicators → scaled up to cover 600 rural communities.

7.38

3.15

26.21

17.82

5.375.552.86

14.0111.48

3.60

Physical Health Communication Cognitive

Development

Social

Competence

Emotional

Maturity

Preschool Control


To inform program design

Improving Access to Essential Medicines

(Zambia)

Influence IdeasInput to Funding Decisions Inform Program Design

58%

43%

34%

46%

22%30%

0%

25%

50%

75%

SP ACT Adult ACT

Pediatric

Baseline

Endline

Clinics receive monthly

supplies from district stores

52%48%

43%

16%

6%12%

0%

25%

50%

75%

SP ACT Adult ACT

Pediatric

Baseline

Endline

Clinics receive supplies direct

from central stores

Stock-outs are reduced under both distribution systems:

Direct distribution more cost-effective → replicated across

Zambia (World Bank, 2010)

$4.20 / day of averted stock-

out

$14.50 / day of averted stock-

out

Targeting the Poor (Indonesia)

Influence IdeasInput to Funding Decisions Inform Program Design

• Randomized field experiments

evaluating accuracy of 3

methods for targeting the

poor:

• Community-based

• Proxy means test (PMT)

• Self-targeting

PMT & community-based found to be most accurate → findings used

by Indonesian government to build registry of poorest 40% (World Bank, 2012)


As a means of influencing ideas

Conditional Cash Transfer (CCT) Programs

Input to Funding Decisions Inform Program Design Influence Ideas

Countries implementing CCT programs in

1997

Conditional Cash Transfer (CCT) Programs


Countries implementing CCT programs in

2011

Results-Based Financing (Rwanda)


• Health clinics paid to

immunize children and

encourage women to give

birth in a clinic

• Treatment group received

bonuses according to

performance;

• Control group received grant

regardless of performance

Evaluation shows RBF has significant impact on prenatal care →

results inspire RBF designs in other countries, including Nigeria(Basingra et al, 2010)

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

Sta

nd

ard

ized

Pre

nata

l

Eff

ort

Sco

re

Treatment (PBF) Facilities

Control Facilities

Baseline (2006) Follow-up (2008)

15%

Standard

deviation

increase due

to RBF

Treatment (RBF)

Facilities

Control Facilities

Results-Based Financing (Argentina)


• Plan Nacer used to provide

additional health care

coverage to women and

young children

• Federal resources allocated to

provinces based on

enrollment and health results

achieved by each province

Evaluation shows RBF has significant impact on prenatal care,

infant/ maternal mortality → results inspire RBF program in

Dominican Republic & Peru


Why is evaluation valuable?


evaluation?

1

2

How to assess impact

What is beneficiary’s diarrhea incidence of

diarrhea in last 3-days with program

compared to without program?

Compare same individual with & without

programs at same point in time

Formally, program impact is:

α = (Y | P=1) - (Y | P=0)

e.g. How much does an safe water

intervention reduce diarrhea?

Solving the evaluation problem

Estimated impact is difference between treated

observation and counterfactual.

Counterfactual: what would have happened

without the program.

Use Control group to estimate counterfactual.

Never observe same individual with and without

program at same point in time.

Counterfactual is key to impact evaluation.

Possible Solutions

Need to guarantee comparability of

treatment and control groups.

ONLY remaining difference is intervention.

Consider:

o Experimental design/randomization

o Quasi-experiments

Counterfactual Criteria

Treated & Counterfactual(1) Have identical characteristics,

(2) Except for benefiting from the intervention.

No other reason for differences in

outcomes of treated and counterfactual.

Only reason for the difference in

outcomes is due to the intervention.

2 Counterfeit Counterfactuals

Before and After

Those not enrolledo Those who choose not to

enroll in the program

o Those who were not offered

the program

Same individual before the treatment

What's wrong?

Selection bias: People choose to participate

for specific reasons1

2

3

Many times reasons are related to the

outcome of interest

Cannot separately identify impact of the

program from these other factors/reasons

Need to know…

All the reasons why someone gets the

program and others not.

All the reasons why individuals are in the

treatment versus control group.

If reasons correlated w/ outcome cannot

identify/separate program impact from other

explanations of differences in outcomes.

Possible Solutions

Need to guarantee comparability of

treatment and control groups.

ONLY remaining difference is intervention.

In this seminar we will consider:

o Experimental design/randomization

o Quasi-experiments (Regression Discontinuity,

Double differences)

o Instrumental Variables.

These solutions all involve…

Knowing how the data are generated.

Randomizationo Give all equal chance of being in control or

treatment groups

o Guarantees that all factors/characteristics will be

on average equal btw groups

o Only difference is the intervention

If not, need transparent & observable

criteria for who is offered program.

Working Smarter in IE

• Be strategic in selecting programs to evaluate

• Don’t assume costly data collection is always necessary

• Use IE to maximize program efficiency, not just impact

Impact evaluation as an operational research tool

Use administrative data when possible

Use IE’s to fill knowledge gaps & assess alternatives

for key programs

Ensuring the Impact of Impact

Evaluations

Engage early. Engage Often.

Work locally. Think globally.

• Tale of 2 Evaluations

• Involve stakeholders at every stage

• Prospective – IE part of design

• Foster relationships on the ground

with decision-makers

• Results inform decisions beyond the

borders of country studied

Water Privatization -- Argentina

o Municipal water &

sanitation

o Public versus Private

management

o Evaluation:

o Increase in water

quality

o Increase in access

by poor

o Large reductions in

child mortality

Figure 4: Evolution of Mortality Rates for Municipalities with

Privatized vs. Non-Privatized Water Services

3.5

4

4.5

5

5.5

6

6.5

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

Year

Mo

rta

lity

Ra

tes

Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities

No Influence on policy: Academic study

without stakeholders involvement

Improving Housing in Urban Slums

(Piso Firme)


Replacing Dirt Floors with Cement

Floors (Piso Firme)

• Evaluation shows significant

impacts on reducing diarrhea,

parasitic infections & improving

cognitive development

• Scaled up to 3 million households

o Moving families to new housing developments (Tu Casa)

o No Impact – canceled program

Ensuring the Impact of Impact

Evaluations

Engage early. Engage Often.

Work locally. Think globally.

• Tale of 2 Evaluations

• Involve stakeholders at every stage

• Prospective – IE part of design

• Foster relationships on the ground

with decision-makers

• Results inform decisions beyond the

borders of country studied

Messageso IE Useful for policy

o Resource allocation

o Benefit design

o Influence global ideas

o What to evaluate

o High cost programs with large #’s beneficiaries

o Little existing knowledge

o Start early – prospective

o Build control groups into rollout

o Work locally think globally

Figure 4: Evolution of Mortality Rates for Municipalities with

Privatized vs. Non-Privatized Water Services

3.5

4

4.5

5

5.5

6

6.5

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

Year

Mo

rta

lity

Ra

tes

Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities

There and Back Again - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/9/... · There and...

Documents

Transcript of There and Back Again - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/9/... · There and...