WHY RANDOMIZE

March 2014

Presented by:Raul Sanchez de la Sierra

Why randomize

?

Because correlation is not causation

Example: Impact of a medicine

 

Does the medicine increase weight??

Takes the drug

Does not take a drug

Overview

1. Causal impact and counterfactual

2. Comparison groups

3. Randomization

 

CAUSAL IMPACT

AND

COUNTERFACTUAL

Causal impact of a policy

• Change due to the policy

Causal impact of a policy

Pay for performance Traditional wage

What really happened What would have happened without the policy

Real world Imaginary world

Counterfactual

Counterfactual

Counterfactual: Would would have happened (imaginary)

Problem: Will never observe what would have happened

How to measure the causal impact of a policy?

Causal impact of a policy

Pay for performance Traditional wage

What really happened What would have happened without the policy

Real world Imaginary world

CounterfactualSimilar to Counterfactual

COMPARISON GROUPS

Selecting the comparison group

• Select a group that is exactly like the group of participants in all ways except the policy

Good

Bad

Pratham’s Balsakhi program

Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program

Incorporating random assignment into the program

Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program

Incorporating random assignment into the program

The intervention Work with Pratham in 124 Municipal Schools

Hire local women (Balsakhis) from the community

Train them to teach basic literacy, numeracy

Identify lowest performing students• Balsakhi teaches them basic competencies

15

Possible comparisons

1. Compare after to before the policy

2. Compare policy group to other group

3. Compare change in policy group to change in other group

4. Other non-experimental methods

5. Randomized Experiment

Method 1: Compare after to before the policy

• Test scores after Balsakhi

• Test scores before Balsakhi

16

17

Method 1: Compare after to before the policy

Before

After Difference

Scores 24 51 +27

18

Method 1: Compare after to before the policy

Improvement could be due to other reasons related to TIME

• Problem with this comparison:

Why this may not be a good strategy

Pay for performance Traditional wage

What really happenedWhat would have happened without the policy

Before

After True counterfactual

Wrong counterfactual

20

Method 2: Compare to other group

Compare test scores of…

Children who got balsakhi

With test scores of…

Children who did not get balsakhi

21

Method 2: Compare to other group

Balsakhi

No Balsak

khi

Difference

Scores 51 56 -5

22

Method 2: Compare to other group

Balsakhi students may be different:

Examples:- Poorer- Worst performing

• Problem with this comparison:

Why this may not be a good strategy

Pay for performance Comparison group

What really happenedWhat happened to the other group

Before

After

Counterfactual

What would have happened without the policy

True counterfactual

Wrong counterfactual

24

Method 3: Difference-in-Differences

Children who got balsakhi

Compare gains in test scores of…

With gains in test scores of…

Children who did not get balsakhi

25

Balsakhi

No Balsak

khi

Difference

After 51 56 -5

Method 3: Difference-in-Differences

26

Balsakhi

No Balsak

khi

Difference

After 51 56 -5

Before 24 37 -13

Method 3: Difference-in-Differences

27

Balsakhi

No Balsak

khi

Difference

After 51 56 -5

Before 24 37 -13

Difference in Differences

+8

Method 3: Difference-in-Differences

28

Balsakhi students may IMPROVE systematically faster for other reasons than the program:

Examples:- They may start from a lower level

Problem with this comparison:Method 3: Difference-in-Differences

Why this may not be a good strategy

Pay for performance Comparison group

What really happenedWhat happened to the other group

Before

After

Counterfactual

What would have happened without the policy

30

Method 4: Regression Analysis

Children who got balsakhi

• Compare test scores at the start and at the end of the program.

• For students of the same gender, age (CONTROL)

With

Children who did not get balsakhi

31

Balsakhi students may IMPROVE systematically faster than other students of their own gender and age for other reasons than the program:

Examples:- They may start from a lower level

Problem with this comparison:

Method 4: Regression Analysis

Why this may not be a good strategy

Pay for performance Comparison group

What really happenedWhat happened to the other group

Before

After

Counterfactual

What would have happened without the policy

33

Fundamental problem:

Cannot know what makes Balsakhi student different before the program

Never fully credible

Method 4: Regression Analysis

Problem with non experiments

• SELECTION BIAS

Non-experimental Methods

• Instrumental Variables

• Regression Discontinuity

35

RANDOMIZATION

Non-random assignment

HQMonthly income, per capita

1000

500

0 Treatment Control

1457

947

Random assignment

Monthly income, per capita

1000

500

0Treatment Control

1257 1242

HQ

1. Baseline

2. RANDOMIZED ASSIGNMENT

3. INTERVENTION

3. STATUS QUO

4. Endline

TARGET POPULATION

How does randomization work?

40

Randomization at work: educationApril 06/07

Jun 05

Aug 05

Tests Tests

Pay for performance Fixed wageInitial Test

How the treated group looks now

One of the two then gets the treatment. Outcomes are then compared between those that did and did not get

treatment

In a randomized trial, two groups are formed through a lottery

(to make sure that they are comparable)

How they would have looked without treatment

This group gets the treatment

This groups does not

Randomization at work: medicineIn a randomized trial, two groups are formed

through a lottery(to make sure that they are comparable)

One of the two then gets the treatment. Outcomes are then compared between those

that did and did not get treatment

CC

The truth If project goes to easy places

If project goes to hard places If lotteries are used

T

T TT

T

TT T

T

T

T

TTC

CC

CCCCCC

C

CC

T

CT

Key advantage of experiments

Members of the groups are statistically identical

any change can be attributed to the program

43

Method 5: Randomized Experiment

• Suppose we evaluated the balsakhi program using a randomized experiment

44

Impact of Balsakhi - Summary

45

Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference

6.82*

(4) Regression 1.92

*: Statistically significant at the 5% level

Impact of Balsakhi - Summary

46

Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference

6.82*

(4) Regression 1.92(5)Randomized Experiment

5.87**: Statistically significant at the 5% level

What is the impact of this program?

Time

Prim

ary

Out

com

e

Impact

Counterfactual

Program starts

Impact: What is it?

Time

Prim

ary

Out

com

e

ImpactCounterfactualProgram starts

Impact: What is it?

Time

Prim

ary

Out

com

e

ImpactCounterfactual

Program starts