AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky...

AFRICA IMPACT EVALUATION INITIATIVE, AFTRL

Africa Program for Education Impact Evaluation

Muna Meky

Impact Evaluation Cluster, AFTRL

Slides by Paul J. Gertler & Sebastian Martinez

Impact Evaluation Methods: Impact Evaluation Methods: Randomization, IV, Regression

Discontinuity

Measuring Impact

• Randomized Experiments

• Quasi-experiments– Randomized Promotion-Instrumental

Variables– Regression Discontinuity– Difference in difference – panel data– Matching

Randomization

• The “gold standard” in evaluating the effects of interventions

• It allows us to form a “treatment” and “control” groups– identical characteristics – differ only by intervention

• Closest approximation to counterfactual

Random Assignment

• Each eligible unit has the same chance of receiving the intervention– Mimics chocolate experiment

• Allows us to compare the “treatment” and “control group”

Random Assignment vs. Random Sample

• Random Assignment– Are the observed results due to the

intervention rather than other confounding factors? (internal validity)

• Random Sample– Do the results found in the sample apply to

the general population/are they generalizable? (external validity)

Randomization

Randomization

Randomization

Random Sample

(external validity)

Random Assignment

(internal validity)

Example of Randomization

• What is the impact of providing free books to students on test scores?

• Randomly assign a group of school children to either:- Treatment Group – receives free books

- Control Group – does not receive free books

Randomization

Random Assignment

http://www.oralchelation.net/images/bs00554_.jpg

How Do You Randomize?

1) At what level? – Individual – Group

• School

• Community

• District

When would you use randomization?

• Universe of eligible individuals typically larger than available resources at a single point in time

– Fair and transparent way to assign benefits– Gives an equal chance to everyone in the sample

• Good times to randomize:– Pilot programs– Programs with budget/capacity constraints – Phase in programs

Oportunidades Example

• Randomized treatment/controls– Community level randomization

• 320 treatment communities• 186 control communities• Pre-intervention characteristics well balanced

Oportunidades Example

Variables Treatment (4, 670)

Control (2727) t- stats

Consumpti on per capi ta 233. 47 233. 4 - 0. 04

1.02 1.3

Head' s age 41. 94 42. 35 1. 20.2 0.27

Head' s educati on 2. 95 2. 81 - 2. 160.04 0.05

Spouse' s age 37. 02 36. 96 - 0. 380.7 0.22

Spouse' s educati on 2. 76 2. 76 - 0. 0060.03 0.04

Speaks an i ndi genous l anguage 41. 69 41. 95 0. 21

0.007 0.009

Head i s femal e 0. 073 0. 078 0. 660.003 0.005

Househol d at basel i ne 5. 76 5. 7 - 1. 21

0.02 0.038

Bathroom at basel i ne 0. 57 0. 56 - 1. 040.007 0.009

Total hectareas of l and 1. 63 1. 72 1. 35

0.03 0.05

Mi n. Di stance l oc-urban 109. 28 106. 59 - 1. 02

0.6 0.81

RANDOMI ZATI ON

Difference in Means

Control Treatment t-statMean CPC Baseline

233.40 233.47 0.04

Mean CPC Followup

239.5 268.75 9.6

Case 3 - Randomization

Linear Regression Multivariate Linear RegressionEstimated Impact on CPC 29.25** 29.79**

(3.03) (3.00)** Significant at 1% level


Impact Evaluation Example: Randomization vs. Other Methods

Case 1 - Before and After

Case 2 - Enrolled/Not Enrolled


Multivariate Linear Regression



Estimated Impact on CPC 34.28** -4.15 29.79**

(2.11) (4.05) (3.00)** Significant at 1% level

Other Analyses Often Lack Internal Validity

• Enrolled vs non-enrolled– The baseline characteristics will be different

because people have chosen which group they want to be in

• Before and after– There are often other interventions going on

at the same time

Measuring Impact




When Do We Use Random Promotion?

• Common scenario:– National Program with universal eligibility– Voluntary enrollment in program

• Can not control who enrolls and who does not

Randomized Promotion

• Possible solution: random promotion or incentives into the program– Information

– Money

– Other help/Incentives

• Also called– Encouragement designs

– Incentive schemes

Study Components

• Intervention– Chocolate

• Randomized Promotion – Encouragement to take chocolate

Example of Promotion Design for SATs

• What is the impact of supplementary learning material on student test scores?– Outcome

• Student test scores

– Intervention• Supplementary learning materials• (all teachers can access these)

– Randomized promotion• Letters encouraging students to use materials• (sent to randomly assigned teachers)

What Information Does Randomized Promotion Give Us?

• How effective is the treatment?– We can analyze the effect the treatment

had on the outcome in the sub-group of subjects who would not have used the intervention unless encouragement was present

How Effective is the Treatment?

• Local Average Treatment Effect

– Effect of the intervention on those who would not have enrolled unless encouraged

Encouragement Design

ENCOURAGED

Take-up = 80%

Mean outcome = 100

NOT

ENCOURAGED

Take-up = 30%

Mean outcome = 90

Change Take-up = 50%

Change Y=10

Impact = 20

Never Takeup

Takeup if Encouraged

Always Takeup



Example: Community Based School Management

• What is the effect of decentralization of school management on learning outcomes?– All communities are eligible – Community management of hiring, budgeting,

oversight– Need to write proposal

Community Based School Management

• 1500 schools in the evaluation

• Each community chooses whether to participate in program

Promotion Design

• Community based school management– Provision of technical assistance and

training by NGOs for submission of grant application

– Random selection of communities with NGO support

Community Based School Management

• Outcome – learning outcome

• Intervention– decentralization of management to

community– 1500 schools

• Promotion– NGO support– schools randomized to receive this support

Examples – Randomized Promotion

• Maternal Child Health Insurance in Argentina– Intensive information campaigns

• Employment Program in Argentina– Transport voucher

• Community Based School Management in Nepal– Assistance from NGO

• Health Risk Funds in India– Assistance from Community Resource Teams

Randomized Promotion

• Just an example of an Instrumental Variable

• A variable correlated with treatment but nothing else (i.e. random promotion)

• Again, we really just need to understand how the benefits are assigned– Don’t have to exclude anyone

Measuring Impact




Introduction

• What is the impact of monetary incentives on test scores to schools below some ranking?

• Compare the schools right above the ranking point to schools below the cutoff point

Indexes are Common in Targeting of Social Programs

• Anti-poverty programs– targeted to households below a given

poverty index

• Pension programs– targeted to population above a certain age

• Scholarships– targeted to students with high scores on

standardized test

Example: What is the Effect of Cash Transfer on Consumption?

• Intervention:– Cash transfer to poor households

• Evaluation: – Measure outcomes (i.e. consumption,

school attendance rates) before and after transfer, comparing households just above and below the cut-off point.

Example: What is the Effect of Cash Transfer on Consumption?

• Target poorest households for cash transfer

• Method:– construct poverty index from 1 to 100 with

pre-intervention characteristics• households with a score <=50 are poor

• households with a score >50 are non-poor

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

Non-poor

Poor

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

Treatment effect

Regression Discontinuity

• When to use this method? – the beneficiaries/non-beneficiaries can be

ordered– there is a cut-off point for eligibility.– cut-off determines the assignment of a

potential beneficiary to the treatment or no-treatment


Fitt

ed v

alue

s

puntaje estimado en focalizacion276 1294

153.578

379.224

2

Baseline – No treatment

0 1 ( )i i iy Treatment score


Estimated Impact on CPC

** Significant at 1% level

Case 4 - Regression DiscontinuityMultivariate Linear Regression

30.58**(5.93)

Fitt

ed v

alu

es

puntaje estimado en focalizacion276 1294

183.647

399.51

Treatment Period

Sharp and Fuzzy Discontinuity

• Sharp discontinuity – the discontinuity precisely determines treatment

– equivalent to random assignment in a neighborhood• e.g. social security payment depend directly and immediately

on a person’s age

• Fuzzy discontinuity – discontinuity is highly correlated with treatment .

– use the assignment as an IV for program participation.• e.g. rules determine eligibility but there is a margin of

administrative error.

Examples

• Effect of class size on scholastic achievement (Angrist and Lavy, 1999)

• Effect of transfers on labor supply

(Lemieux and Milligan, 2005)

• Effect of old age pensions on consumption -BONOSOL in Bolivia

(Martinez, 2005)

Potential Disadvantages of RD

• We estimate the effect of the program around the cut-off point.– the effect is estimated at the discontinuity– fewer observations than in a randomized

experiment with the same sample size– limits generalizability

• Make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: – nonlinear relationships– interactions

Advantages of RD for Evaluation

• RD allows one to estimate the effect of an intervention at the discontinuity

• Can take advantage of a known rule for assigning the benefit that are common in the designs of social policy– No need to “exclude” a group of eligible

households/individuals from treatment

Summary

Design When to use Advantages DisadvantagesRandomization •Whenever

possible

•When an intervention will not be universally implemented

•Gold standard

•Most powerful

•Not always feasible

•Not always ethical

Random Promotion

•When an intervention is universally implemented

• Learn and intervention

•Only looks at sub-group of sample


•If an intervention is assigned based on rank

•Assignment based on rank is common

•Only look at sub-group of sample

AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky...

Documents

Transcript of AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky...