IMPACT EVALUATION FOR POLICY MAKINGA look at LAC countries with weaker

research capacities

María Laura AlzúaHabiba DjebbariMartín Valdivia

Session: Demand and Supply of Impact EvaluationsMind the Gap: From Evidence to Policy Impact

Cuernavaca, June 15-17, 2011

Outline• Why do we need to evaluate the impact of programs?• The Evidence to Policy Process• Key Hypotheses of this study• What we do?• Some preliminary results

The need for impact evaluations• Money for foreign aid/taxpayers have alternative uses for

a set of goals• Social behavior and economic policies

– Modeling incentives, behavioral constraints and the objectives of policies/programs (Karlan & Appel, 2011; Banerjee & Dufló, 2011)

– Hidden/unexpected effects

• The challenge to identify a sound counterfactual– What would have happened to the beneficiaries in the absence

of the program?• Experimental methods• Non-experimental methods (PSM, IV, RDD, etc.)

Research questions• How many studies have been done and on what topics or

policy issues and with what methodologies (RCT, q-experimental, IV, etc.)? What are the emerging fields or sectors of interested being evaluated?

• Who is doing it? The role of local universities and research centers vis-à-vis the work by northern-based researchers and institutions and initiatives such as J-PAL, IPA, and donors and the multilaterals, etc.

• Who is funding it (3IE, Gates, MCC, the multilateral banks)? How have their project approval processes being modified to give stellar importance to impact evaluation designs?

• How are they being used to shape policy? What are the institutional arrangements that can better insert impact evaluations into policy design?

Key hypotheses• The number of programs with rigorous IEs have

increased significantly in the region, especially in countries with more capacities for research and in policy making– Institutionalization of the use of IEs for policy making has been

key

What we do?

• Collect all IE studies done in the region since 1995, and especially in the following set of countries– South America: Bolivia, Ecuador, Paraguay, Perú– Central America: El Salvador, Guatemala, Honduras, Nicaragua– Caribbean: Dominican Republic, Jamaica, Trinidad and Tobago,

Guyana

• Sources: – Bouillón & Tejerina (2007)– Most common academic papers databases: IDEAS/RePEC,

EconLit and JSTOR, and SSRN Randomized Social Experiments– Other databases: World Bank DIME, IADB OVE/Development

Effectiveness, IPA, J-PAL, 3ie, MCA, the Bill and Melinda Gates Foundation, IFPRI

What we do?

• Final selection criteria restricts the sample to those with reasonable identification strategies– We exclude studies with identification strategies based on

community characteristics as IVs, PSM without baseline, etc.

• Three case studies: Dominican Republic, El Salvador, Peru– What are the restrictions to the implementation and use of

rigorous IEs?– Are there conditions for the institutionalization of IEs? (Briceño

and Gaarder, 2009)

• Supply of training courses on IE in LAC– Sources: IPA, J-PAL, World Bank, IADB, IEN-LACEA– Who are they serving? Local policy makers , researchers