1. PROJECT IMPACT EVALUATION METHODS AKHTER U. AHMED INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE BANGLADESH POLICY RESEARCH AND STRATEGY SUPPORT PROGRAM KNOWLEDGE, TOOLS AND LESSONS FOR INFORMING THE DESIGN AND IMPLEMENTATION OF FOOD SECURITY STRATEGIES IN ASIA 14-16 NOVEMBER 2011 KATHMANDU

2. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 2 Storyline 1. What is impact evaluation? 2. How to do impact evaluation? 3. Difference-in-differences method of impact evaluation 4. How to construct a comparison group? Randomization Matching Instrumental variables Regression discontinuity design 3. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE What is an impact evaluation? An impact evaluation assesses the changes in the well-being of families or individuals that can be attributed to a particular project, program or policy Impact is the difference between outcomes (e.g., consumption, school enrollment, womens empowerment, etc) with the program and without it The goal of impact evaluation is to measure this difference in a way that can attribute the difference to the program, and only the program 4. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Why is it important? Government policymakers/implementing agencies/donors want to know if the program had an impact and the average size of that impact Understand if policies work Justification for program Understand the net benefits of the program Understand the distribution of gains and losses 5. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE What we need Compare same individual with and without programs at same point in time Problem: Every individual is uniqueeach Individual has only one existence. So, we never observe the same individual with and without program at same point in time Hence, we have a problem of a missing counterfactual of what would have happened without the program 6. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Solving the evaluation problem How about comparing impact indicators of individuals before and after the program? This is called reflexive impact evaluation But the problem is that the rest of the world moves on and we are not sure what was caused by the program and what by the rest of the world. We might pick up the effects of other factors that changed around the time of treatment So, we need a control/comparison group that will allow us to attribute any change in the treatment group to the program Difference between treated observation and counterfactual is the estimated impact 7. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Problems in constructing a comparison group Two main problems: Most social interventions are targeted Program areas differ from non-program areas in observable and unobservable ways because the program-designers intended this Individual participation is usually voluntary Participants differ from non-participants in observable and unobservable ways Therefore, a comparison of participants and an arbitrary group of non-participants can lead to biased results. This is termed as selection bias 8. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 8 Illustrating selection bias SB = 0 G=ATT SB > 0 G>ATT SB < 0 G