WHY RANDOMIZE
description
Transcript of WHY RANDOMIZE
![Page 1: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/1.jpg)
WHY RANDOMIZE
March 2014
Presented by:Raul Sanchez de la Sierra
![Page 2: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/2.jpg)
Why randomize
?
![Page 3: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/3.jpg)
Because correlation is not causation
![Page 4: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/4.jpg)
Example: Impact of a medicine
Does the medicine increase weight??
Takes the drug
Does not take a drug
![Page 5: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/5.jpg)
Overview
1. Causal impact and counterfactual
2. Comparison groups
3. Randomization
![Page 6: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/6.jpg)
CAUSAL IMPACT
AND
COUNTERFACTUAL
![Page 7: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/7.jpg)
Causal impact of a policy
• Change due to the policy
![Page 8: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/8.jpg)
Causal impact of a policy
Pay for performance Traditional wage
What really happened What would have happened without the policy
Real world Imaginary world
Counterfactual
![Page 9: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/9.jpg)
Counterfactual
Counterfactual: Would would have happened (imaginary)
Problem: Will never observe what would have happened
How to measure the causal impact of a policy?
![Page 10: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/10.jpg)
Causal impact of a policy
Pay for performance Traditional wage
What really happened What would have happened without the policy
Real world Imaginary world
CounterfactualSimilar to Counterfactual
![Page 11: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/11.jpg)
COMPARISON GROUPS
![Page 12: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/12.jpg)
Selecting the comparison group
• Select a group that is exactly like the group of participants in all ways except the policy
Good
Bad
![Page 13: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/13.jpg)
Pratham’s Balsakhi program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
![Page 14: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/14.jpg)
The intervention Work with Pratham in 124 Municipal Schools
Hire local women (Balsakhis) from the community
Train them to teach basic literacy, numeracy
Identify lowest performing students• Balsakhi teaches them basic competencies
![Page 15: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/15.jpg)
15
Possible comparisons
1. Compare after to before the policy
2. Compare policy group to other group
3. Compare change in policy group to change in other group
4. Other non-experimental methods
5. Randomized Experiment
![Page 16: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/16.jpg)
Method 1: Compare after to before the policy
• Test scores after Balsakhi
• Test scores before Balsakhi
16
![Page 17: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/17.jpg)
17
Method 1: Compare after to before the policy
Before
After Difference
Scores 24 51 +27
![Page 18: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/18.jpg)
18
Method 1: Compare after to before the policy
Improvement could be due to other reasons related to TIME
• Problem with this comparison:
![Page 19: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/19.jpg)
Why this may not be a good strategy
Pay for performance Traditional wage
What really happenedWhat would have happened without the policy
Before
After True counterfactual
Wrong counterfactual
![Page 20: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/20.jpg)
20
Method 2: Compare to other group
Compare test scores of…
Children who got balsakhi
With test scores of…
Children who did not get balsakhi
![Page 21: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/21.jpg)
21
Method 2: Compare to other group
Balsakhi
No Balsak
khi
Difference
Scores 51 56 -5
![Page 22: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/22.jpg)
22
Method 2: Compare to other group
Balsakhi students may be different:
Examples:- Poorer- Worst performing
• Problem with this comparison:
![Page 23: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/23.jpg)
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
True counterfactual
Wrong counterfactual
![Page 24: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/24.jpg)
24
Method 3: Difference-in-Differences
Children who got balsakhi
Compare gains in test scores of…
With gains in test scores of…
Children who did not get balsakhi
![Page 25: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/25.jpg)
25
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Method 3: Difference-in-Differences
![Page 26: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/26.jpg)
26
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Before 24 37 -13
Method 3: Difference-in-Differences
![Page 27: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/27.jpg)
27
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Before 24 37 -13
Difference in Differences
+8
Method 3: Difference-in-Differences
![Page 28: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/28.jpg)
28
Balsakhi students may IMPROVE systematically faster for other reasons than the program:
Examples:- They may start from a lower level
Problem with this comparison:Method 3: Difference-in-Differences
![Page 29: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/29.jpg)
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
![Page 30: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/30.jpg)
30
Method 4: Regression Analysis
Children who got balsakhi
• Compare test scores at the start and at the end of the program.
• For students of the same gender, age (CONTROL)
With
Children who did not get balsakhi
![Page 31: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/31.jpg)
31
Balsakhi students may IMPROVE systematically faster than other students of their own gender and age for other reasons than the program:
Examples:- They may start from a lower level
Problem with this comparison:
Method 4: Regression Analysis
![Page 32: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/32.jpg)
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
![Page 33: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/33.jpg)
33
Fundamental problem:
Cannot know what makes Balsakhi student different before the program
Never fully credible
Method 4: Regression Analysis
![Page 34: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/34.jpg)
Problem with non experiments
• SELECTION BIAS
![Page 35: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/35.jpg)
Non-experimental Methods
• Instrumental Variables
• Regression Discontinuity
35
![Page 36: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/36.jpg)
RANDOMIZATION
![Page 37: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/37.jpg)
Non-random assignment
HQMonthly income, per capita
1000
500
0 Treatment Control
1457
947
![Page 38: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/38.jpg)
Random assignment
Monthly income, per capita
1000
500
0Treatment Control
1257 1242
HQ
![Page 39: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/39.jpg)
1. Baseline
2. RANDOMIZED ASSIGNMENT
3. INTERVENTION
3. STATUS QUO
4. Endline
TARGET POPULATION
How does randomization work?
![Page 40: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/40.jpg)
40
Randomization at work: educationApril 06/07
Jun 05
Aug 05
Tests Tests
Pay for performance Fixed wageInitial Test
![Page 41: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/41.jpg)
How the treated group looks now
One of the two then gets the treatment. Outcomes are then compared between those that did and did not get
treatment
In a randomized trial, two groups are formed through a lottery
(to make sure that they are comparable)
How they would have looked without treatment
This group gets the treatment
This groups does not
Randomization at work: medicineIn a randomized trial, two groups are formed
through a lottery(to make sure that they are comparable)
One of the two then gets the treatment. Outcomes are then compared between those
that did and did not get treatment
![Page 42: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/42.jpg)
CC
The truth If project goes to easy places
If project goes to hard places If lotteries are used
T
T TT
T
TT T
T
T
T
TTC
CC
CCCCCC
C
CC
T
CT
![Page 43: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/43.jpg)
Key advantage of experiments
Members of the groups are statistically identical
any change can be attributed to the program
43
![Page 44: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/44.jpg)
Method 5: Randomized Experiment
• Suppose we evaluated the balsakhi program using a randomized experiment
44
![Page 45: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/45.jpg)
Impact of Balsakhi - Summary
45
Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference
6.82*
(4) Regression 1.92
*: Statistically significant at the 5% level
![Page 46: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/46.jpg)
Impact of Balsakhi - Summary
46
Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference
6.82*
(4) Regression 1.92(5)Randomized Experiment
5.87**: Statistically significant at the 5% level
![Page 47: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/47.jpg)
![Page 48: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/48.jpg)
What is the impact of this program?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Program starts
![Page 49: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/49.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactualProgram starts
![Page 50: WHY RANDOMIZE](https://reader035.fdocuments.us/reader035/viewer/2022081502/568166ee550346895ddb40b5/html5/thumbnails/50.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Program starts