Post on 22-Jan-2015
description
Beyond Experiments: General Equilibrium Simulation Methods for
Impact Evaluation
Xinshen Diao International Food Policy Research Institute
J. Edward Taylor
University of California, Davis
Katmandu, Nepal, November 16-18, 2011
Outline
Why simulation approach? Why general equilibrium? What is a simulation approach? What is a local-economy general equilibrium
simulation model? Conclusions: simulation approach and its
implication to FtF
Why simulation approach? We need to look beyond experiments when…
Planning a large scale intervention (such as FtF) often requires an ex-ante assessment of its potential impact (no pilot rollout is possible)
Treatment and control groups are impracticable – Can’t randomize over large number of units – Investments (e.g., irrigation and rural road) to target certain
areas instead of individuals Economic impacts are indirect; higher-level effects
(e.g., poverty reduction and economic growth) We want to know “Why” & “if” there are impacts Multiple inputs and inter-related outcomes Impacts are heterogeneous, likely winners and losers
Why general equilibrium? - Externalities and linkages
What experimentalists call “externalities” or “control-group contamination” …GE modelers call “linkages”
Linkages transmit impacts from the treatment group to others in the same location
Can also create higher-level impacts outside the targeted locations
Transfer Policy
Transfer Policy
Treatment households adjust
External Linkages Transfer Policy
…affecting other households
External Linkages Transfer Policy
…which adjust
External Linkages Transfer Policy
…affecting still other households
What is a simulation approach? Thinking about a flight simulator
Flight simulator contains good model of mechanics and aerodynamics – If not, don’t fly with that pilot!
If we have a good model of how the local economy works, we can use it to – Simulate impacts of project, policy changes – Do an local-economy GE cost-benefit analysis – Estimate the distribution of impacts, winners and
losers, whom to compensate/provide adjustment assistance
– Experiment with project designs w/ specific goals Ex-post: We can use experimental results to
see whether the plane really flew
What is a local-economy simulation model? Recipe for simulation-based project evaluation
Understand the project or policy to be simulated – Elements of the project: E.g., cash transfer or input subsidy? Who’s the target
(i.e., treatment group)? Understand the actors and the economic system
– How is the treatment group connected with others in the zone of influence (ZOI) of the project?
– How do we model their behavior? – Sketch out a social accounting matrix (SAM) for each household group and/or
locality to reflect this Inventory existing data needs and availability to construct SAMs
– Baseline surveys fill data gaps (can modify pre-treatment surveys) Build the simulator: construct SAMs, use them to calibrate a general-
equilibrium (GE) model encompassing treatment and control groups Do simulations to evaluate high-level impacts of intervention Use the simulation results as inputs into CB or impact analysis, project
design Use experimental results for validation, recalibration of models
Examples of a local economy model: Malawi case
Challenges of this (like most) impact evaluation Three transfer mechanisms
– Input subsidy (IS) • Malawi Agricultural Inputs Subsidy Program (MAISP)
– Cash transfer (CT) • Malawi’s Social Cash Transfer Scheme, -SCTS
– Farm gate market price support (MPS) • Implemented historically
Can’t do an experiment for each of them
Challenges (continued)
Immediate indirect effects of transfers on the control group (linkages effect) – Experiments aren’t going to capture them
Heterogeneous treatment and control groups Sensitivity of outcomes to market structures
– E.g., will cash transfers create multiplier effects within households by loosening production constraints?
– Ex-post experimental evidence can help us parameterize this in the simulation model
Multiple goals of the analysis
To compare the effects of these three transfer mechanisms on incomes and welfare in rural areas – Including high-level effects, on non-beneficiary
households To assess differences in these effects across
household groups and market scenarios – The structure of the economy shapes outcomes
To understand why different transfer mechanisms produce different outcomes
Developing an economywide GE model in which…
A set of farm (and nonfarm) household models are defined
Each household model is representative of a group of households defined according to their eligibility for each transfer program
All these household models are embedded in an economywide GE model
Data Ideally, parameterize the model with data from a baseline (pre-
project) survey In this application, we had to rely on existing data…
– IHS2 (Second Integrated Household Survey) • 2004, immediately preceding the first round of the MAISP
– National agricultural production and consumption information available online from FAOSTAT
• 2003, the last completed cropping season before the IHS2 was conducted
Constructing a social accounting matrix (SAM) for each household group from the data
Nest the households within a “meta-SAM” for the ZOI (in this case, the entire rural economy)
Includes market accounts that link together the household groups
Simulations
Assumptions on market conditions matter 1. Perfect markets benchmark 2. With constrained input use 3. With unemployment 4. Combined 2 and 3
Under each type of market arrangements, simulating IS, MPS, CT separately at the given cost ($52 million)
Simulation results at the household level (perfect market benchmark) (1) (2) (3) (4) (5) (6)
Transfer Mechanism Ineligible, Non-farm households
Ineligible, Small farms
Ineligible, Large farms
Eligible for CT (ultra-poor labor-
constrained)
Eligible for MAISP (poor small-
holders)
Eligible for both CT &
MAISP Group's share of total households (%) 3 19 23 1 47 7
a) IS: Crop Inputs subsidies for eligible households
Group’s share of transfer (%) - - 93.0 7.0
Welfare (CV), % change 0.80 0.00 -0.30 0.01 5.47 4.50
Household-level efficiency - - 0.69 0.78
b) MPS: Market Price Support for Maize
Group’s share of transfer (%) 22.0 57.0 1.0 20.0 0.0
Welfare, % change -1.1 2.0 2.7 1.6 0.6 -1.9
Household-level efficiency 0.64 0.66 0.57 0.37 -
c) CT: Cash Transfer to eligible households
Group’s share of transfer (%) - 17.5 - 82.5
Welfare (CV), % change 0.0 0.0 0.0 50.8 0.0 69.7 Household-level efficiency - - - 1.00 - 1.00
Total production effects and efficiency measure under alternative market conditions
(a) (b) (c) (d)
Perfect markets benchmark
With constrained input use
With unemployment
With unemployment & constrained input
use Production effects (% change in total agricultural output)
Input Subsidy 4.0 2.3 13.4 5.0
MPS 1.0 -0.3 8.6 2.9
Cash transfer 0.0 0.8 0.0 2.0
Total transfer efficiency (welfare gain/transfer cost)
Input Subsidy 0.66 0.60 2.59 1.59
MPS 0.57 0.04 2.29 1.30
Cash transfer 1.00 1.17 1.00 1.47
Input subsidy becomes most efficient when households face unemployment and liquidity constraints
Which assumptions reflect reality?
Perfect markets benchmark seems to be overly optimistic
Effects of transfers depend on: – The elasticity of input supply – The responsiveness of wages to shifts in labor
demand – The extent to which there are cash constraints
on input demand All are likely to vary across project settings
Conclusions: Simulation approaches and FtF
Experiments have become the favored method of impact evaluation
Simulation methods will be increasingly important; and particularly important for FtF
Advantages of experiments
Verifiability – Create random treatment and control groups – Simply compare averages of outcomes of
interest to evaluate average effect of treatment on the treated
Disadvantages Experiments often are impracticable (cost, politics,
ethics) They almost never come out truly random (need for
econometrics) Control group contamination (due to GE linkages) Difficulty comparing impacts of several different
project designs Non-structural: Generally don’t tell us why
treatments have the impacts they do GE feedbacks change impacts once programs are
“ramped up”
Simulation approaches Designed to overcome these limitations of
experiments Ideal for
– Capturing higher-level impacts – Comparing alternative mechanism designs – Understanding the “Why?” – Evaluating differences in project impacts across
market environments Can be implemented before projects
The Simulation-experiment ideal Simulations: Use to evaluate likely impacts of
alternative project interventions ex-ante – Parameterize with data from baseline surveys
Carry out randomized experiment using most promising program designs
Use results of experiment ex-post to verify and (if needed) reparameterize simulation model
Use simulation model to provide a structural interpretation of experiment results (i.e., to answer the “Why?” question) – …and improve policy design
Thank You