PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching •...
Transcript of PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching •...
![Page 1: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/1.jpg)
PUBL0050: Advanced Quantitative Methods
Lecture 6: Synthetic Control Method
Jack Blumenau
1 / 50
![Page 2: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/2.jpg)
Course outline
1. Potential Outcomes and Causal Inference2. Randomized Experiments3. Selection on Observables (I)4. Selection on Observables (II)5. Panel Data and Difference-in-Differences6. Synthetic Control7. Instrumental Variables (I)8. Instrumental Variables (II)9. Regression Discontinuity Designs10. Overview and Review
2 / 50
![Page 3: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/3.jpg)
Lecture outline
Motivation
Synthetic Control
Inference
Additional application
Conclusion
3 / 50
![Page 4: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/4.jpg)
Motivation
![Page 5: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/5.jpg)
Motivation
Comparative case studies have a long history in applied political science:
• Qualitative: “thick” description of the context/features of two or moreinstances of specific phenomena. Aim to describe contrasts orsimilarities across the cases and reason inductively about causality
• Quantitative: more explicitly causal, using (normally) aggregate datafrom one treated unit and a small set of control units. Often based on‘natural experiments’ where a shock affects one unit, but not others.
Unifying logic:
Systems as similar as possible with respect to as many features aspossible constitute the optimal samples for comparative enquiry.
Przeworski and Tenure, 1970, p. 32
4 / 50
![Page 6: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/6.jpg)
Quantitative Comparative Case Studies
Goal:
• Estimate effects of events or policy interventions that take place at anaggregate level
• Types of unit: cities, states, countries, etc
• Types of intervention: passage of laws, economic shocks, etc
Approach:
• Compare the evolution of an aggregate outcome for the unit affectedby the intervention to the evolution of the same outcome for somecontrol group
• e.g. Card (1990), Card and Krueger (1994), Abadie and Gardeazabal(2003)
5 / 50
![Page 7: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/7.jpg)
Quantitative Comparative Case Studies
Advantages:
• Policy interventions often take place at an aggregate level• Aggregate/macro data are often available
Problems:
• Selection of control group is often ambiguous• Standard errors do not reflect uncertainty about the ability of thecontrol group to reproduce the counterfactual of interest
Solution:
• If you don’t have a good control group: synthesize one
6 / 50
![Page 8: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/8.jpg)
Running example
Reunification of West and East GermanyWhat were the economic effects of reunification on the West Germaneconomy? Many economic historians argue that reunification had largenegative economic costs, but identification is difficult because there is noobvious country with which we can compare the growth trajectory of WestGermany. Abadie et al (2015) estimate the effects of reunification bycomparing the actual time series for West Germany with a syntheticcontrol group which provides the counterfactual.
• Outcome: GDP per capita (inflation adjusted)• Treatment: Reunification (1 for W. Germany after 1990, 0 otherwise)• Time: Years (1960 to 2003)
7 / 50
![Page 9: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/9.jpg)
What should be the control group?
What is the most appropriate control group for evaluating the effects ofreunification on West Germany in 1990?
• Geographical/cultural: Austria?• Economic: USA?• Average: OECD countries?
The choice of the control group matters!
8 / 50
![Page 10: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/10.jpg)
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West Germanyrest of OECD sample
reunification
9 / 50
![Page 11: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/11.jpg)
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyMean of OECD sample
reunification
10 / 50
![Page 12: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/12.jpg)
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyAustria
reunification
11 / 50
![Page 13: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/13.jpg)
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyUSA
reunification
12 / 50
![Page 14: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/14.jpg)
What should be the control group?
1960 1970 1980 1990 2000
010
000
2000
030
000
Year
GD
P p
er c
apita
West GermanyUSA and Austria
reunification
13 / 50
![Page 15: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/15.jpg)
What should be the control group?
Synthetic control moves away from using a single control unit or a simpleaverage of control units.
Instead we use a weighted average of the set of control or “donor” units.
Rather than assuming that either the USA or Austria are similar to W.Germany, we calculate a weighted average (the synthetic control) which ismore similar to West Germany than any individual country.
IntuitionWhen we only have a few aggregate units, a ‘synthetic’ combination ofcontrol units may do a better job of reproducing the characteristics of thetreated unit than any one unit alone.
14 / 50
![Page 16: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/16.jpg)
Synthetic Control
![Page 17: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/17.jpg)
Notation
DefinitionFor units 𝑗 ∈ 1, ..., 𝐽 + 1:• Unit 1 is the unit of interest (which receives the treatment)
• Units 2 to 𝐽 + 1 are the ‘donor pool’ or potential comparison unitsTime periods ∈ 1, ..., 𝑇 :• Pre-treatment period: 𝑡 = 1, ..., 𝑇0
• Post-treatment period: 𝑡 = 𝑇0 + 1, ..., 𝑇Potential outcomes:
• 𝑌 𝑁𝑖𝑡 = outcome for unit 𝑖 at time 𝑡 in the absence of the intervention
• 𝑌 𝐼𝑖𝑡 = outcome for unit 𝑖 at time 𝑡 when exposed to the intervention
15 / 50
![Page 18: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/18.jpg)
Notation
Estimand
𝛿1𝑡 = 𝑌 𝐼1𝑡 − 𝑌 𝑁
1𝑡 = 𝑌1𝑡 − 𝑌 𝑁1𝑡 for all 𝑡 > 𝑇0
i.e. the treatment effect on the treated unit in the post-treatment periods.
Problem
We cannot observe 𝑌 𝑁1𝑡
Why? → Fundamental problem of causal inference.
The critial question, as always, is how should we impute 𝑌 𝑁1𝑡 ?
15 / 50
![Page 19: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/19.jpg)
Imputing 𝑌 𝑁1𝑡
1. Matching
• For each time period 𝑡, find the 𝑀 ‘closest’ units to unit 1 and averagethe observed outcomes:
𝑌 𝑁1,𝑡=1 = 1
𝑀𝑀
∑𝑚=1
𝑌𝑗𝑚(1),𝑡=1
2. Diff-in-diff
• Add the average change in outcome for the control group to thetreated unit’s outcome in the pre-treatment period
𝑌 𝑁1,𝑡=1 = 𝑌1,𝑡=0 + ( 𝑌0,𝑡=1 − 𝑌0,𝑡=0)
3. Synthetic control
• Take a weighted average of the outcomes of the donor units• Weights defined by closeness to the trend of the outcome for thetreated unit in the pre-treatment period
𝑌 𝑁1,𝑡=1 =
𝐽+1∑𝑗=2
𝑤∗𝑗𝑌𝑗,𝑡=1
16 / 50
![Page 20: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/20.jpg)
Defining the synthetic control
Definition (Synthetic control)A synthetic control is a vector of weights, 𝑊 , associated with each of theavailable 𝐽 donor units.
Compare to our three examples above. 𝑊 is a vector with…
• …equal weight for each unit (OECD average)• …0 weight for all units, except Austria where 𝑤𝑗 = 1 (Austria)• …0 weight for all units, except USA where 𝑤𝑗 = 1 (USA)• …0 weight for all units, except USA where 𝑤𝑗 = .5 and Austria where
𝑤𝑗 = .5 (USA and Austria)
There are many potential synthetic controls! The goal is to select 𝑊 suchthat the characteristics of the treated unit are best resembled by thecharacteristics of the synthetic control.
17 / 50
![Page 21: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/21.jpg)
Estimating𝑊
Basic approach: compare treatment and control units in terms of theirpre-intervention characteristics 𝑋1 and 𝑋0. Put more weight on countriesin the donor pool that are similar to the treatment unit in terms of:
• Covariates that are predictive of post-intervention outcomes• Pre-intervention outcome values
The idea is to give more weight to units in the donor pool that closelyapproximate the treated unit in the pre-intervention period.
I.e. We look for weights, ��𝑗, such that
𝑌1,𝑡 ≈𝐽+1∑𝑗=2
��𝑗𝑌𝑗,𝑡 for all 𝑡 ∈ 1, ..., 𝑇0
18 / 50
![Page 22: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/22.jpg)
Estimating𝑊
For each donor unit, define a weight 𝑊 = {𝑤2, 𝑤3, ..., 𝑤𝐽+1}. We findthe values of 𝑊 by minimizing the following expression:
𝑘∑𝑚=1
𝑣𝑚(𝑋1𝑚 − 𝑋0𝑚𝑊)2
where
1. 𝑤𝑗 ≥ 0 for 𝑗 = 2, ..., 𝐽 + 1 (only positive or zero weights allowed)2. ∑𝐽+1
2 𝑤𝑗 = 1 (weights must sum to 1)3. 𝑣𝑚 is a weight that reflects the importance of the 𝑚th variable thatwe use to measure the distance between treated and control units
19 / 50
![Page 23: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/23.jpg)
Estimating𝑊
There are two components that we need:
1. 𝑤𝑗 – donor unit weights, indicating which units are most similar to thetreated unit
2. 𝑣𝑚 – covariate weights, indicating which covariates are mostpredictive of the outcome
Estimate by minimizing the mean squared prediction error for thepre-intervention periods:
𝑇0
∑𝑡=1
(𝑌1𝑡 −𝐽+1∑𝑗=2
𝑤𝑗(𝑉 )𝑌𝑗𝑡)2
IntuitionMost weight (𝑤𝑗) is put on control units which are similar to the treatedunits on covariates (𝑋1, 𝑋0) that are predictive of the outcome (𝑣𝑚) inthe pre-intervention period (𝑡 ≤ 𝑇0)
20 / 50
![Page 24: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/24.jpg)
Estimating𝑊
SC is, at heart, a sort of difference-in-differences matching estimator.
• Diff-in-diff: establish a control group which follows a parallel trend inthe absence of treatment (note we are still assuming that theparallel-ness would continue post-treatment!)
• Matching: 𝑤𝑗 calculated using observed pre-treatment observedcovariates But Abadie et al argue that SC also accounts fornon-parallel trends:
Only units that are alike in both observed and unobserved determi-nants of the outcome variable…should produce similar trajectoriesof the outcome variable over extended periods of time.
Abadie et al, 2015, p. 498
21 / 50
![Page 25: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/25.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightUSA 0.06UK 0.06Austria 0.06Belgium 0.06Denmark 0.06France 0.06Italy 0.06Netherlands 0.06Norway 0.06Switzerland 0.06Japan 0.06Greece 0.06Portugal 0.06Spain 0.06Australia 0.06New Zealand 0.06
The Synth package in R will automate this optimization problem for us.
22 / 50
![Page 26: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/26.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightAustria 0.14Japan 0.08Netherlands 0.08USA 0.07Switzerland 0.07UK 0.06Denmark 0.06France 0.06Spain 0.06Australia 0.06Italy 0.05Norway 0.05Belgium 0.04Greece 0.04Portugal 0.04New Zealand 0.03
The Synth package in R will automate this optimization problem for us.
22 / 50
![Page 27: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/27.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightAustria 0.18Netherlands 0.13USA 0.11UK 0.07Greece 0.07New Zealand 0.07Japan 0.07Switzerland 0.05Belgium 0.04Denmark 0.04France 0.04Norway 0.04Spain 0.04Australia 0.04Portugal 0.01Italy 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
![Page 28: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/28.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightAustria 0.28USA 0.10Switzerland 0.09Netherlands 0.08Denmark 0.07Greece 0.07New Zealand 0.07Japan 0.07Belgium 0.03France 0.03Norway 0.03Portugal 0.03Spain 0.03Australia 0.03UK 0.00Italy 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
![Page 29: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/29.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00
The Synth package in R will automate this optimization problem for us.
22 / 50
![Page 30: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/30.jpg)
Estimating𝑊 (Intuition)
Goal: minimize difference in outcome trend in pre-treatment period.
1960 1965 1970 1975 1980 1985 1990
010
0020
0030
0040
00
Year
GD
P d
iffer
ence
(R
eal −
Syn
thet
ic)
reunification
county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00
The Synth package in R will automate this optimization problem for us. 22 / 50
![Page 31: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/31.jpg)
Interpreting𝑊 (country weights)
Country weights in synthetic Germany:
County Weight County WeightAustria 0.42 France 0.00USA 0.22 Italy 0.00Japan 0.16 Norway 0.00Switzerland 0.11 Greece 0.00Netherlands 0.09 Portugal 0.00UK 0.00 Spain 0.00Belgium 0.00 Australia 0.00Denmark 0.00 New Zealand 0.00
23 / 50
![Page 32: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/32.jpg)
Interpreting𝑊 (country weights)
Country weights in synthetic Germany (SC and OLS):
Country Synth Reg Country Synth RegAustria 0.42 0.26 France 0.00 0.04USA 0.22 0.13 Italy 0.00 -0.05Japan 0.16 0.19 Norway 0.00 0.04Switzerland 0.11 0.05 Greece 0.00 -0.09Netherlands 0.09 0.14 Portugal 0.00 -0.08UK 0.00 0.06 Spain 0.00 -0.01Belgium 0.00 -0.00 Australia 0.00 0.12Denmark 0.00 0.08 New Zealand 0.00 0.12
Regression weights can be greater than 1 or less than zero → extrapolationoutside of the support of control units.
Extrapolation is not possible in the SC case because the weights are boundbetween 0 and 1.
24 / 50
![Page 33: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/33.jpg)
Interpreting𝑊 (assessing balance)
GDP predictor means:
Treated Synthetic Rest of OECD SampleGDP per-capita 15808.9 15802.2 8021.1Trade openness 56.8 56.9 31.9Inflation rate 2.6 3.5 7.4Industry share 34.5 34.4 34.2
Schooling 55.5 55.2 44.1Investment rate 27.0 27.0 25.9
25 / 50
![Page 34: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/34.jpg)
Interpreting 𝑣
Which variables are most important for determining the synthetic control?
Variable 𝑣GDP per-capita 0.44Investment rate 0.24Trade openness 0.13Schooling 0.11Inflation rate 0.07Industry share 0.00
The weights 𝑣1, … , 𝑣𝑘 should reflect the predictive value of the covariates.
26 / 50
![Page 35: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/35.jpg)
Break
27 / 50
![Page 36: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/36.jpg)
Causal effects
Weighting donor units leads to a synthetic unit with a similar outcometrend in the pre-intervention period as the treated unit.
Given ��, an unbiased estimator of 𝛿1𝑡 is:
𝛿1𝑡 = 𝑌1,𝑡 −𝐽+1∑𝑗=2
𝑤𝑗𝑌𝑗,𝑡 for 𝑡 ∈ {𝑇0 + 1, … , 𝑇 }
where
• 𝑌1,𝑡 is the outcome for the treated unit in post-treatment period 𝑡• ∑𝐽+1
𝑗=2 𝑤𝑗𝑌𝑗,𝑡 is the outcome for the synthetic control unit inpost-treatment period 𝑡
• 𝛿1,𝑡 is the ATT for time period 𝑡
28 / 50
![Page 37: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/37.jpg)
Causal effect I
1960 1970 1980 1990 2000
050
0010
000
1500
020
000
2500
030
000
Year
Per
−ca
pita
GD
P (
PP
P, 2
002
US
D)
West Germanysynthetic West Germany
Reunification
29 / 50
![Page 38: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/38.jpg)
Causal effect II
1960 1970 1980 1990 2000
−40
00−
2000
020
0040
00
Year
Gap
in p
er−
capi
ta G
DP
(P
PP,
200
2 U
SD
)
Reunification
30 / 50
![Page 39: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/39.jpg)
Inference
![Page 40: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/40.jpg)
Asymptotic inference in synthetic control
Standard errors from regression/t-tests are typically used to characteriseuncertainty about aggregate data:
• i.e. use a sample of restaurants in NJ and PA to estimate employmenttrends in each state
• standard errors reflect unavailability of aggregate data onemployment
So, if we use aggregate data, is there zero uncertainty? No!
• We do not have perfect information about potential outcomes, evenwhen we use aggregate data
• We have uncertainty about the potential outcome under control forthe treated unit
But, because the number of units is small in most SC applications, largesample inferential techniques are not appropriate.
31 / 50
![Page 41: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/41.jpg)
Permutation inference
Instead, we turn to an alternative inference technique: permutationinference.
1. Calculate the the test-statistic under the actual treatment assignment2. Calculate the distribution of the test-statistic under alternative
treatment assignments assuming treatment effects of zero3. Assess whether the ‘true’ test-statistic is unlikely under the null
distribution of treatment effects
Here, this implies constructing a synthetic control for every country in oursample, summarising the treatment effect, and comparing it to thetreatment effect in West Germany.
32 / 50
![Page 42: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/42.jpg)
Permutation inference (example)
What is the test-statistic here? Two components:
• Treatment effect (analogous to 𝛽 in a regression)• Pre-treatment difference between unit and SC (analogous to 𝑆𝐸( 𝛽))
For each unit calculate:
RMSE𝑗,𝑇1 = ⎛⎜⎝
1𝑇1
𝑇∑
𝑡=𝑇0
(𝑌1,𝑡 −𝐽+1∑𝑗=2
𝑤𝑗𝑌𝑖,𝑡)2⎞⎟⎠
1/2
RMSE𝑗,𝑇0 = ⎛⎜⎝
1𝑇0
𝑇0
∑𝑡=1
(𝑌1,𝑡 −𝐽+1∑𝑗=2
𝑤𝑗𝑌𝑖,𝑡)2⎞⎟⎠
1/2
Where RMSE𝑗,𝑇0characterises the lack of fit between a unit and it’s SC in
the pre-treatment period, and RMSE𝑗,𝑇1is the equivalent for the
post-treatment period.33 / 50
![Page 43: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/43.jpg)
Permutation inference (example)
Given these, the test-statistic is:
RMSE𝑗,𝑇1
RMSE𝑗,𝑇0
= Pre-intervention ‘fit’Post-intervention ‘fit’
Intuition:
• More confident that the effect is different from zero when theestimated treatment effect is larger (RMSE𝑗,𝑇1
)
• Less confident that the effect is different from zero when thepre-treatment fit with the SC is larger (RMSE𝑗,𝑇0
)
P-value: how likely would it be to observe a ratio as large as the one weactually observe if the treatment effects were zero and we picked onecountry at random?
34 / 50
![Page 44: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/44.jpg)
Permutation inference (example)
Portugal
Denmark
France
Netherlands
Japan
UK
Austria
Switzerland
Belgium
Australia
Spain
USA
New Zealand
Italy
Greece
Norway
West Germany
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
5 10 15
Post−Period RMSE / Pre−Period RMSE
35 / 50
![Page 45: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/45.jpg)
Placebos in space
1960 1970 1980 1990 2000
−40
00−
2000
020
0040
00
Year
Gap
in p
er−
capi
ta G
DP
(P
PP,
200
2 U
SD
)
West GermanyDonor countries
36 / 50
![Page 46: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/46.jpg)
Placebos in time
1960 1965 1970 1975 1980 1985 1990
050
0010
000
1500
020
000
2500
030
000
year
per−
capi
ta G
DP
(P
PP,
200
2 U
SD
)West Germanysynthetic West Germany
placebo reunification
37 / 50
![Page 47: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/47.jpg)
Additional application
![Page 48: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/48.jpg)
California’s Proposition 99
Anti-smoking legislation and cigarette consumptionIn 1988, California passed comprehensive tobacco control legislation. Thiswas a package of measures that included a tax increase, more earmarkedspending to anti-smoking health initiatives, and anti-smoking mediacampaigns. We will investigate the effect of this legislation on cigaretteconsumption in California using synthetic control methods.
• Outcome variable (Y): Per capita cigarette sales (packs)• Treatment (D): 1 for CA after 1988, 0 for all other periods/states• Time (T): 1970 to 2000
(All states which passed similar legislation are excluded from the donor pool.)
38 / 50
![Page 49: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/49.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
020
4060
8010
012
014
0
year
per−
capi
ta c
igar
ette
sal
es (
in p
acks
)Californiarest of the U.S.
Passage of Proposition 99
39 / 50
![Page 50: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/50.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
020
4060
8010
012
014
0
year
per−
capi
ta c
igar
ette
sal
es (
in p
acks
)Californiasynthetic California
Passage of Proposition 99
40 / 50
![Page 51: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/51.jpg)
State Weights in Synthetic California
State WeightUtah 0.334Nevada 0.234Montana 0.199Colorado 0.164Connecticut 0.069
41 / 50
![Page 52: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/52.jpg)
Predictor means: Real vs. Synthetic California
California Average ofVariables Real Synthetic 38 control statesLn(GDP per capita) 10.08 9.86 9.86Percent aged 15-24 17.40 17.40 17.29Retail price 89.42 89.41 87.27Beer consumption per capita 24.28 24.20 23.75Cigarette sales per capita 1988 90.10 91.62 114.20Cigarette sales per capita 1980 120.20 120.43 136.58Cigarette sales per capita 1975 127.10 126.99 132.81Note: All variables except lagged cigarette sales are averaged for the 1980-1988 period(beer consumption is averaged 1984-1988).
42 / 50
![Page 53: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/53.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
−30
−20
−10
010
2030
year
gap
in p
er−
capi
ta c
igar
ette
sal
es (
in p
acks
)
Passage of Proposition 99
Cigarette sales gap in CA (versus synthetic CA).
43 / 50
![Page 54: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/54.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
−30
−20
−10
010
2030
year
gap
in p
er−
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in all 38 states.
44 / 50
![Page 55: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/55.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
−30
−20
−10
010
2030
year
gap
in p
er−
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 20 ⋅ MSPECA.45 / 50
![Page 56: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/56.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
−30
−20
−10
010
2030
year
gap
in p
er−
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 5 ⋅ MSPECA.46 / 50
![Page 57: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/57.jpg)
California’s Proposition 99
1970 1975 1980 1985 1990 1995 2000
−30
−20
−10
010
2030
year
gap
in p
er−
capi
ta c
igar
ette
sal
es (
in p
acks
) Californiacontrol states
Passage of Proposition 99
Cigarette sales gap in states with pre-intervention MSPE < 2 ⋅ MSPECA.47 / 50
![Page 58: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/58.jpg)
Conclusion
![Page 59: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/59.jpg)
Data requirements
Synthetic control has relatively low data requirements:
• Can use aggregate data (often administrative)
• e.g. economic indicators such as GDP, current-account balance, etc;political indicators such as turnout, vote share, etc
• Causal factors can be big and important
• e.g. legislation changes, macro-shocks, etc
• Units of analysis can be large
• Countries, states, regions, etc
• Does not even require full panel data for the pre-treatment period
• Can use averages of covariates rather than full panel data (useful whencovariates do not vary yearly)
48 / 50
![Page 60: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/60.jpg)
Conclusion
The synthetic control approach…is arguably the most importantinnovation in the policy evaluation literature in the last 15 years.
Athey and Imbens, 2017
Advantages:• Builds on D&D and Matching by essentially forcing the data to exhibitparallel trends in the pre-treatment period
• Amenable to small-ish N comparisons (often easier to get data)• Clear, transparent, and easily communicable comparisons(e.g. Germany is part Austria, part USA, etc)
49 / 50
![Page 61: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/61.jpg)
Conclusion
The synthetic control approach…is arguably the most importantinnovation in the policy evaluation literature in the last 15 years.
Athey and Imbens, 2017
Disadvantages:• Inference is not straightforward! Asymptotic inference does not workwith the SC method
• Coding is not straightforward! As you will see in the seminar :)• Pre-intervention period must be relatively large for us to trust paralleltrends holds in the post-intervention period
49 / 50
![Page 62: PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:](https://reader030.fdocuments.us/reader030/viewer/2022041017/5ecad5bd600730616c181570/html5/thumbnails/62.jpg)
Next week
Instrumental Variables (I)
50 / 50