Quasi-Experimental Methods
-
Upload
gillespie-hehir -
Category
Documents
-
view
45 -
download
0
description
Transcript of Quasi-Experimental Methods
![Page 1: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/1.jpg)
Africa Impact Evaluation Program on AIDS (AIM-AIDS) Cape Town, South Africa March 8 – 13, 2009
1
Quasi-Experimental Methods
Jean-Louis ArcandThe Graduate Institute | Geneva
![Page 2: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/2.jpg)
2
Objective
• Find a plausible counterfactual
• Every method is associated with an assumption• The stronger the assumption the more we need
to worry about the causal effect
» Question your assumptions
Reality check
![Page 3: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/3.jpg)
3
Program to evaluate Hopetown HIV/AIDS Program (2008-2012)
ObjectivesReduce HIV transmission
Intervention: Peer education
Target group: Youth 15-24
Indicator: Pregnancy rate (proxy for unprotected sex)
![Page 4: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/4.jpg)
4
I. Before-after identification strategy (aka reflexive comparison)
Counterfactual:
Rate of pregnancy observed before program started
EFFECT = After minus Before
![Page 5: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/5.jpg)
5
Year Number of areas
Teen pregnancy rate (per 1000)
2008 70 62.90
2012 70 66.37Difference +3.47
![Page 6: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/6.jpg)
6
66.37
62.9
50525456586062646668
2008 2012
Teen
pre
gnan
cy
(per
100
0)
Effect = +3.47
Intervention
Counterfactual assumption: no change over time
Question: what else might have happened in 2008-2012 to affect teen pregnancy?
![Page 7: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/7.jpg)
7
Examine assumption with prior data
Number of areas
Teen pregnancy (per 1000)2004 2008 2012
70 54.96 62.90 66.37
Assumption of no change over time looks a bit shaky
![Page 8: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/8.jpg)
8
II. Non-participant identification strategy
Counterfactual:
Rate of pregnancy among non-participants
Teen pregnancy rate (per 1000) in 2012
Participants 66.37
Non-participants 57.50
Difference +8.87
![Page 9: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/9.jpg)
9
Counterfactual assumption:Without intervention participants have same pregnancy rate as non-participants
66.4
57.5
40
60
80
100
2008 2012
teen
pre
gnan
cy(p
er 1
000)
Effect = +8.87
Participants
Non-participants
Question: how might participants differ from non-participants?
![Page 10: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/10.jpg)
10
Test assumption with pre-program data
?66.462.9
46.37
57.5
40
50
60
70
80
2008 2012
teen
pre
gnan
cy(p
er 1
000)
REJECT counterfactual hypothesis of same pregnancy rates
![Page 11: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/11.jpg)
11
III. Difference-in-Difference identification strategy
Counterfactual:
1.Nonparticipant rate of pregnancy, purging pre-program differences in participants/nonparticipants
2.“Before” rate of pregnancy, purging before-after change for nonparticipants
1 and 2 are equivalent
![Page 12: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/12.jpg)
12
Average rate of teen pregnancy in
2008 2012 Difference (2008-2012)
Participants (P) 62.90 66.37 3.47
Non-participants (NP) 46.37 57.50 11.13
Difference (P-NP) 16.53 8.87 -7.66
![Page 13: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/13.jpg)
13
66.462.9
46.37
57.5
40
50
60
70
80
2008 2012
teen
pre
gnan
cy
57.50 - 46.37 = 11.13
66.37 – 62.90 = 3.47
Non-participants
Participants
Effect = 3.47 – 11.13 = - 7.66
![Page 14: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/14.jpg)
14
66.462.9
46.37
57.5
40
50
60
70
80
2008 2012
teen
pre
gnan
cy (p
er 1
000)
After
Before
Effect = 8.87 – 16.53 = - 7.66
66.37 – 57.50 = 8.87
62.90 – 46.37 = 16.53
![Page 15: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/15.jpg)
15
Counterfactual assumption:
Without intervention participants and nonparticipants’ pregnancy rates follow same trends
![Page 16: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/16.jpg)
16
66.462.9
46.37
57.5
40
50
60
70
80
2008 2012
teen
pre
gnan
cy(p
er 1
000)
74.0
16.5
![Page 17: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/17.jpg)
17
66.462.9
46.37
57.5
40
50
60
70
80
2008 2012
teen
pre
gnan
cy(p
er 1
000)
74.0 -7.6
![Page 18: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/18.jpg)
18
Questioning the assumption
• Why might participants’ trends differ from that of nonparticipants?
![Page 19: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/19.jpg)
19
Examine assumption with pre-program data
counterfactual hypothesis of same trends doesn’t look so believable
Average rate of teen pregnancy in
2004 2008 Difference (2004-2008)
Participants (P) 54.96 62.90 7.94
Non-participants (NP) 39.96 46.37 6.41
Difference (P=NP) 15.00 16.53 +1.53 ?
![Page 20: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/20.jpg)
20
IV. Matching with Difference-in-Difference identification strategy
Counterfactual:
Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants
![Page 21: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/21.jpg)
21
Counterfactual assumption:
Question: how might participants differ from matched nonparticipants?
Unobserved characteristics do not affect outcomes of interest
Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset
![Page 22: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/22.jpg)
22
56
58
60
62
64
66
68
70
72
74
76
2008 2012
Teem
pre
gnam
cy ra
te (p
er 1
000)
73.36
66.37Matched
nonparticipant
Participant
Effect = - 7.01
![Page 23: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/23.jpg)
23
Can only test assumptionwith experimental data
Apply with care – think very hard about unobservables
Studies that compare both methods (because they have experimental data) find that:
unobservables often matter!
direction of bias is unpredictable!
![Page 24: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/24.jpg)
24
V. Regression discontinuity identification strategy
Applicability:
When strict quantitative criteria determine eligibility
Counterfactual:
Nonparticipants just below the eligibility cutoff are the comparison for participants just above the eligibility cutoff
![Page 25: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/25.jpg)
25
Counterfactual assumption:
Question: Is the distribution around the cutoff smooth?
Then, assumption might be reasonable
Question: Are unobservables likely to be important (e.g. correlated with cutoff criteria)?
Then, assumption might not be reasonable
However, can only estimate impact around the cutoff, not for the whole program
Nonparticipants just below the eligibility cutoff are the same (in observable and unobservable ways) as participants just above the eligibility cutoff
![Page 26: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/26.jpg)
26
• Target transfer to poorest schools• Construct poverty index from 1 to 100• Schools with a score <=50 are in• Schools with a score >50 are out• Inputs transfer to poor schools• Measure outcomes (i.e. test scores) before and
after transfer
Example: Effect of school inputs on test scores
![Page 27: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/27.jpg)
27
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
![Page 28: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/28.jpg)
28
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
Non-Poor
Poor
![Page 29: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/29.jpg)
29
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
![Page 30: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/30.jpg)
30
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
Treatment Effect
![Page 31: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/31.jpg)
31
Applying RDD in practice: Lessons from an HIV-nutrition program• Lesson 1: criteria not applied well
– Multiple criteria: hh size, income level, months on ART– Nutritionist helps her friends fill out the form with the
“right” answers– Now – unobservables separate treatment from
control…• Lesson 2: Watch out for criteria that can be
altered (e.g. land holding size)
![Page 32: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/32.jpg)
32
• Gold standard is randomization – minimal assumptions needed, intuitive estimates
• Nonexperimental requires assumptions – can you defend them?
Summary
![Page 33: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/33.jpg)
33
Different assumptions will give you different results• The program: ART treatment for adult patients• Impact of interest: effect of ART on children of patients
(are there spillover & intergenerational effects of treatment?)– Child education (attendance)– Child nutrition
• Data: 250 patient HHs 500 random sample HHs– Before & after treatment
• Can’t randomize ART so what is the counterfactual
![Page 34: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/34.jpg)
34
Possible counterfactual candidates
• Random sample difference in difference– Are they on the same trajectory?
• Orphans (parents died – what would have happened in absence of treatment)– But when did they die, which orphans do you observe, which
do you not observe?• Parents self report moderate to high risk of HIV
– Self report!• Propensity score matching
– Unobservables (so why do people get HIV?)
![Page 35: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/35.jpg)
35
Estimates of treatment effects using alternative comparison groups
Standard errors clustered at the household level in each round.Includes child fixed effects, round 2 indicator and month-of-interview indicators.
(1) (2) (3) (4) (5) (6)
Comparison group:
Orphans in Random sample
High/Mod. HIV Risk
households
Orphans in Random sample
High/Mod. HIV Risk
households
Orphans in Random sample
High/Mod. HIV Risk
households
ARV hh (<100 days) * Rd. 2 10.675 10.787 15.686 14.561 10.805 10.397(3.262)*** (2.720)*** (4.877)*** (3.832)*** (4.676)** (3.979)**
ARV hh (>100 days) * Rd. 2 5.808 5.316 10.930 9.302 2.503 1.652(3.133)* (2.638)** (4.467)** (3.513)*** (4.566) (4.036)
Constant 14.723 15.836 13.073 8.307 17.526 23.553(5.583)*** (4.753)*** (6.510)** (5.693) (10.406)* (7.712)***
Observations 334 424 164 210 170 214R-squared 0.86 0.85 0.84 0.87 0.90 0.86
All kids (8-18 years) All boys (8-18 years) All girls (8-18 years)
• Compare to around 6.4 if we use the simple difference in difference using the random sample
![Page 36: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/36.jpg)
36
Estimating ATT using propensity score matching• Allows us to define comparison group using
more than one characteristic of children and their households
• Propensity scores defined at household level, with most significant variables being single-headed household and HIV risk
![Page 37: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/37.jpg)
37
Probit regression results
Coefficient z-value
Single-headed household 0.8917932 3.06Amt of land owned (acres) -0.0153242 -0.83Household size 0.0060359 0.12Value of livestock owned (shillings) 9.36E-07 0.4Travel time to main road (mins.) 0.0034674 1.4Value of durables owned (shillings) -9.35E-08 -0.01House with tin roof 0.2535599 0.58House with non-mud roof 0.2180698 0.7
Household with respondent who reported high/moderate risk of having HIV/AIDS 2.76405 6.88Constant -3.250733 -4.87Observations 225Pseudo R-squared 0.5151
• Dependent variable: household has adult ARV recipient
![Page 38: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/38.jpg)
38
ATT using propensity score matching
Random Sample ARV households Difference T-statHours of school attendance Nearest neighbor matching -10.97 -3.69 7.28 1.94 neighbors=2 Kernel matching -7.82 -3.69 4.12 1.65 bandwidth=.06
Mean change between rounds 1 and 2
![Page 39: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/39.jpg)
39
Nutritional impacts of ARV treatment
Includes child fixed effects, age controls, round 2 indicator, interviewer fixed effects, and month-of-interview indicators.
(1) (2) (3) (4)Dependent variable:Sample:
ARV household * Round 2 0.315 -0.098(0.202) (0.043)**
ARV household (<100 days in rd 1) 0.570 -0.071 * Round 2 (0.277)** (0.058)ARV household (>100 days in rd 1) -0.003 -0.111 * Round 2 (0.252) (0.053)**Constant -0.498 -0.481 0.076 0.077
(0.386) (0.386) (0.082) (0.082)Observations 772 772 772 772R-squared 0.87 0.87 0.70 0.70
WHZ WHZ<=-2All children 0-5 in round 1
![Page 40: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/40.jpg)
40
Nutrition with alternative comparison groups
Includes child fixed effects, age controls, round 2 indicator, interviewer fixed effects, and month-of-interview indicators.
(1) (2) (3) (4)Dependent variable:Comparison Group: RS Mod/High Risk
ARV household * Round 2 1.038 0.521(0.733) (0.327)
ARV household (<100 days in rd 1) 1.195 0.768 * Round 2 (0.785) (0.392)*ARV household (>100 days in rd 1) 0.773 0.220 * Round 2 (0.859) (0.419)Constant 0.864 0.904 -0.339 -0.314
(1.567) (1.588) (0.819) (0.818)Observations 96 96 250 250R-squared 0.92 0.92 0.88 0.88
WHZRS Orphans
![Page 41: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/41.jpg)
41
Summary: choosing among non-experimental methods
• At the end of the day, they can give us quite different estimates (or not, in some rare cases)
• Which assumption can we live with?
![Page 42: Quasi-Experimental Methods](https://reader036.fdocuments.us/reader036/viewer/2022062400/56813460550346895d9b459a/html5/thumbnails/42.jpg)
42
Thank You