Slide 1

#ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 ARIANNA LEGOVINI DEVELOPMENT IMPACT EVALUATION IMPACT EVALUATION FOR REAL TIME POLICY MAKING

Slide 2

WHAT IS THE VALUE OF A POLICY? measuring HOW TO GET THE MOST OUT OF A POLICY? reaching 2

Slide 3

Policy objective Invest in what gets us results 3 Invest in the cause to get an effect

Slide 4

Evaluation objective Identify cause - effect 4 enable good policy decisions

Slide 5

55 WHY IS UNDERSTANDING CAUSALITY SO IMPORTANT? I THINK I WILL STOP USING UMBRELLAS WHEN I USE MY UMBRELLA MY SHOES GET WET It is easy to confuse correlation with causation

Slide 6

6 Invest in the cause to get results Mentorship interacts with quality of mentor to change attitudes and value for youth to act upon their life-skill training and change their behavior (Bushway and Reuter, 2002, 2007) Charismatic mentor + behavioral therapy Attitudes and values Vocational training EMPLOYMENT More motivated youth

Slide 7

Evaluation Narrow down causality by identifying a counterfactual and compare 7 WHAT HAPPENED WHAT WOULD HAVE HAPPENED TO

Slide 8

8 What is counterfactual analysis? Compare same individual with & without intervention at the same point in time Missing data Compare statistically identical groups of individuals with & without intervention at the same point in time Comparable data

Slide 9

9 Counterfactual criteria Treated & control groups Have identical initial average characteristics (observed and unobserved) So that, the only difference is the treatment Therefore the only reason for the difference in outcomes is due to the treatment

Slide 10

10 USE SCIENTIFIC EXPERIMENTS TO GUIDE POLICY DECISIONS

Slide 11

1111 Get the delivery right Subsidy for malaria treatment $350 million Sealing packages saved 16,600 under five + 2,200 adult lives in 5 years Problem: poor public sector distribution of medicines Solution: found system that reduced stockouts by 83% Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M. RANDOM ASSIGNMENT OF HEALTH DISTRICTS TO 3 DIFFERENT DISTRIBUTIO N SYSTEMS

Slide 12

12 Price a CURE $6

Slide 13

1313 Get people to do it Source: Goldstein, Thirumurthy, and Zivin HIV treatment $ 2.1 billion Reminders increase adherence to treatment by 35% Problem IE scalable solution RANDOM ASSIGNMENT OF DAILY AND WEEKLY REMINDERS TO HIV PATIENTS

Slide 14

14 Price Faiths LIFE $7

Slide 15

15 TRANSFORMATIONAL (not marginal) IMPACT

Slide 16

16 Public recognition DOUBLES performance [Ashraf et al., 2011]

Slide 17

What does dime do? 17 Transform development policy 1.Run experiments to inform decisions 2.Build agencies capacity to do this systematically 3.Draw lessons and share them face-to-face to global audiences

Slide 18

175 IEs in 47 Countries Distribution of dime s IEs by GP and CCSA (number of IEs) 18

Slide 19

1919 dime in action Inform policy design Guide mid- course corrections Inform adoption and scale-up Systematic use of evidence Train & apply Learn by doing Apply knowledge Capacity building IE Product IE DESIGN IE IMPLEMENTATION IE DISSEMINATION

Slide 20

Clear roadmap 20 Clearer roadmap WhatTo whomWhenWhere

Slide 21

2121 UNDERSTAND THE THEORY OF CHANGE Inputs Financial and human resources Activities Actions to convert inputs into outputs Outputs Tangible products Intermediate outcomes Effective access Use Compliance Performance Service quality Final Outcomes Health outcomes Lives saved Productivity WHAT (PROGRAM)HOW TO (RESPONSE) & RESULTS

Slide 22

awareness disposition Take-up ADOPTION SKILLS WILLINGNESS motivation participation Accountability quality sustainability 22 BREAK IT DOWN

Slide 23

Policy questions in dime s portfolio (% of questions) 23

Slide 24

Better capacity 24 Better capacity Research team Full time field coordinator Training More and better data

Slide 25

dime results indicators 25

Slide 26

Workshops increase participantsknowledge (pre/post test)

Slide 27

Impact of Workshop on (External Double Blind) Technical Ratings of IE proposals 27

Slide 28

dime results indicators 28

Slide 29

More effort into implementation 29 Motivation to perform Goal: this is what we are trying to achieve Control: this is how we want to do it and BTW somebody measures (and cares)

Slide 30

Better decisions 30

Slide 31

dime results indicators 31

Slide 32

IE increases disbursements IE helps projects stick to plans 32 Does IE help project implementation? 1300 World Bank projects - 100 impact evaluations (2005-2011) Legovini, Di Maro, Piza (2015) It does!

Slide 33

IE increases disbursements & closes implementation gap statistically significant at 1% level 33

Slide 34

AT WHAT COST? SOME SIMPLE COST-BENEFIT OF IMPACT EVALUATION 34

Slide 35

Is the benefit worth the cost? Zambia best supply chain IE solution: 83% reduction in stock out 21-25% reduction in malaria mortality Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2 Scaled up: Benefit ($60.7M) / Cost ($12.4M) = 4.9 Malawi commitment saving account IE solution: 22% increase in farmer production value Pilot: Benefit ($0.7M)/Cost ($0.5M) = 1.4 If scaled: Benefit ($15.6M) / Cost ($1.7M) = 9.2 35 IE PAYS FOR ITSELF A PUBLIC GOOD WORTH FINANCING

Slide 36

dime & i2i IE has real impacts on developmental effectiveness and project delivery High return on investment Spillover effects i2i opportunity to increase scale and scope Open invitation to donors and governments 36 partners CRIME LAB

Slide 37

Human Development Education Health Social Protection i2i pillars and thematic areas 37 Shared Prosperity Finance, Private Sector and Jobs Agriculture Infrastructure Governance Public Sector Governance Justice Local Development Governance Public Sector Governance Justice Local Development Climate Change Energy Environment Natural Resource Management Agriculture Transport Fragility & Conflict Reintegration Governance Gender-based Violence Urban crime & violence Gender Human Capital Economic Opportunities Voice/Agency 5 123 64

Slide 38

Discussion Which development interventions are effective and worth scaling up? Which promising mechanisms are worth testing across different sectors? No one size fits all Importance of building evidence-based institutions 38

Slide 39

#ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 Thank You! facebook.com/ieKnow #impacteval blogs.worldbank.org/impactevaluations microdata.worldbank.org/index.php/catalog/ impact_evaluation http://dime.worldbank.org WEB