Illustrating with the Public Resource Management Program for Assam

Arnab Mukherji

Hiranya Mukhopadhyay

Conference on Impact Evaluation:

Methods, Practices, and Lessons

11 July 2012, ADB Auditorium Zone A

Applying the Synthetic Control Method

Section 1

Evaluation and the Synthetic Control Method

2

Evaluation is about Developing Parallel States of the world …

3

Before After - Restructuring Road

Implicitly: the missing counterfactual is … ?

Public Policy Evaluations …

4

Have traditionally used the Before/After Design frequently. (e.g. See RBI’s State-Wise Analysis of Fiscal Performance)

Issues

– Easy to collect time-series data

– Identification rests on pre-intervention acting as a counterfactual for post-intervention

Concern:

– Does the pre-intervention status really give a good sense of what would have happened in the absence of the intervention?

Before/After unable to disentangle other effects ….

5

FD/GSDP

Impact < 0

Observed Treatment Group

Counterfactual

FD/GSDP Program Works Program makes things

worse!

Impact > 0

Time (T) Time (T) T= Pre T= Post T= Pre T= Post

Difference in Difference tells us …

6

Observed Treatment Group

Control Group

FD/GSDP

Time (T) T= Pre T= Post

ΔC

ΔT

Impact = ΔT -ΔC

Our best estimate of

what would have

happened in the absence

of the intervention

A DiD Analysis raises …

7

A number of concerns centered on the validity of the control group:

Are the treatment and control groups similar on key dimensions?- Selection Bias

Or, do we use all untreated units in the control group symmetrically?

Strengths

– Clearly identified control group

– Omitted Variable Bias less of a concern

Needed micro data – until recently …

Synthetic Control Group- Effect of Terrorism in Basque County

8

Discuss basic question – AER 2003 paper

Critical question 1-on which dimensions would we like synthetic Basque to be identical?

Critical question 2 -Which convex combination of states can be used to construct a statistically indistinguishable version of pre-intervention Basque?

– Solve for Variable weights

Control State weights

– Goal is reproduce pre-intervention outcome trends

Statistical balance on key predictors across Basque and Synthetic Basque

Algorithm works as follows:

9

Let (YitI,Yit

N) denote an outcome in state i in year t; I, N indicate if the state received PRMA or not.

T0 is the intervention year thus, for all t in {1, …, T0-1} and for i in {1, …, N}, we want Yit

I = YitN.

Goal is identify a set of weight for each control state such that

is minimizing:

where is the set of predictors of Y for Assam and for other states and V is a diagonal matrix with specific weights for X.

An Example

Suppose there are three states Assam (Program state), Maharashtra and Tamil Nadu

There are two predictors of the outcome variable (Y) : X1 and X2

The Minimization exercise is:

Min: V1(X1A- X1

MWM)2 + V2(X2A- X2

TWT)2

Subject to WT,WM>0 and WT + WM = 1

• But we still need to determine Vs

Vs are determined by minimizing the prediction error of the outcome variable (Y) for Assam from the synthetic Assam during the pre-intervention period.

W vs. V

11

W gives us which states are to contribute to the synthetic control unit

V ensures that the chosen synthetic unit matches Assam in it it’s pre-intervention outcomes.

W is a function of V – we have a two stage simultaneous optimization problem – computationally intensive

Currently exists as a package in R – a statistical programming language.

Main Advantages

It is a data-driven process- reduces selection bias.

Unlike Propensity Score Matching, does not require large data set.

Ideal for Macro interventions.

This method provides safeguard against extrapolation.

Basque and Synthetic Basque

Section 2:

Case Study: Public Resource Management in Assam

14

PRMPA: Policy Response for Assam

15

Focus on Two Key Components

16

Key Evaluation Challenge: What would have happened in the absence of the PRMPA?

Section 3

Findings

17

Two Outcomes

18

Own Tax Revenues/GSDP

Pre- Intervention Indicators

Synthetic Control –

Karnataka, Meghalaya, Tamil Nadu, Chhattisgarh, and Mizoram

Interest Payment to GSDP

Pre- Intervention Indicators

Synthetic Control –

Meghalaya, Uttar Pradesh, Orissa, West

Bengal

Variable Assam

Synthetic

Assam

Log(GSDP) 10.594 10.587

Non-Agriculture/GSDP 0.661 0.663

Per Capita # of Factories 0.057 0.059

Variable Assam

Synthetic

Assam

log(GSDP) 10.5937 10.60007

Debt/ GSDP 29.46% 31.10%

Average Effective Interest Rate 10.45% 10.22%

Interest Payments/GSDP Ratio

19

Own Revenues/GSDP Ratio

20

Time Trends with Fiscal Deficit

21

Broad Inferences

22

With the three years of follow-on data

Short-term gains have been achieved with respect to own tax to GSDP ratio - an increase in the order of 0.35% to 1.1% in the post-intervention period, averaging about 0.71%

No evidence of short-term gains achieved with respect to a decline in interest payments to GSDP ratio

Future trends will confirm – however, it would be interesting to learn about the pattern and nature of debt swap

Longer term horizon needed for full evaluation.

Thank You

23