Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve...
Transcript of Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve...
![Page 1: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/1.jpg)
Using Impact Evaluation to Improve Development Projects
Arianna Legovini World Bank
USG Evaluation Forum
December 2, 2015
![Page 2: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/2.jpg)
What does DIME do?
2
Transform development policy
1. Run experiments to inform decisions
2. Build agencies’ capacity to do this systematically
3. Draw lessons and share them face-to-face to global audiences
![Page 3: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/3.jpg)
ideas
1. Improve evidence-base at design stage
2. Experiment with causal mechanisms that can change results during implementation
3
![Page 4: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/4.jpg)
We evaluate the impact of packages of interventions (“what”) and experiment with mechanisms (“why/how”)
to make them work better:
IE evaluation questions in IEs with DIME Involvement
![Page 5: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/5.jpg)
(at a minimum) IE increases project disbursements & closes implementation gap
statistically significant at 1% level
5
40.8%***
-54.2%***
40.0%***
15.6%
-45.3%**
49.1%***
Rate of Disbursement(Up to Mid-Term)
Gap (Up to Mid-Term)
Rate of Disbursement- Adjusted by SectorAverage (Up to Mid-
Term)
Rate of Disbursement(overall)
Gap (overall) Rate of Disbursement- Adjusted by Sector
Average (overall)
![Page 6: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/6.jpg)
What is i2i impact evaluation to development impact?
A global platform to
generate and share IE knowledge and
transform the path of development policy
(especially in under-researched areas)
6
![Page 7: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/7.jpg)
What makes i2i different from other IE initiatives?
1. Bottom-up approach – Empower government agencies to solve their own problems/find their own solution
– Open platform for expert advice and support
– Equalize opportunities in selection for technical and financial support
2. High research quality and policy relevance – methods for causal analysis
– data quality
– attention to implementation
– analysis and peer review
– collaboration and policy dialogue to ensure policy actions
3. Generate evidence throughout policy lifecycle – policy design
– mid-course adjustments
– policy decisions on scale up/down.
7
We value equally the generation of knowledge and its application to policy decisions
![Page 8: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/8.jpg)
i2i pillars and thematic areas
8
Shared Prosperity
• Finance, Private
Sector and Jobs • Agriculture • Infrastructure
Governance
• Public Sector Governance
• Justice • Local Development
Climate Change
• Energy • Environment • Natural Resource
Management • Agriculture • Transport
Fragility & Conflict
• Reintegration • Governance • Gender-based
Violence • Urban crime &
violence
Gender
• Human Capital • Economic
Opportunities • Voice/Agency
5
1 2 3
6
Cross Cutting Themes
Human Development
• Education • Health • Social Protection
4
![Page 9: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/9.jpg)
What type of technical support does i2i provide?
• Build capacity through workshops and learning by doing
• Match researchers and operations
• Full implementation support
• In-house research
• Coordination across thematic areas in close collaboration with operational departments
– Knowledge priorities
– Operationalize results
![Page 10: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/10.jpg)
DIME i2i Workshops
No. teams
Impact Evaluation 4 Peace Mar 2014 18
Innovations for Agriculture Apr 2014 8
Impact Evaluation for Legal Empowerment May 2014 11
Innovations for Agriculture June 2014 23
Energy & Environment Oct. 2014 18
Governance Jan. 2015 27
Trade & Competitiveness & Finance May 2015 22
Transport and ICT June 2015 20
![Page 11: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/11.jpg)
Number of participating teams per region
DIME i2i workshops: 128 teams in FY15-16
Putting in perspective: The Bank approves around 400
projects a year
AFR 41%
LAC 31%
SAR 12%
ECA 6%
MENA 5%
EAP 5%
![Page 12: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/12.jpg)
Workshops increase participants’ knowledge
61
65
69
60
72
73
79
70
AGRICULTURE ENERGY & ENVIRONMENT T&C AND F&M TRANSPORT & ICT
PRE/POST TEST
Pre post
![Page 13: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/13.jpg)
56%
79%
89%
63%
77% 80%
FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT
workshop projects that submitted an EoI
note: projects with ongoing IE are excluded
![Page 14: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/14.jpg)
Impact of Workshop on (External Double Blind) Technical Ratings of IE proposals
14
2.25
2.04 2.18
2.55
2.14 2.04 2.01
1.76
2.08 2.13
1.85 1.91
FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT
Technical ratings of proposals (0-3 scale)
Attended workshop Did not attend workshop
![Page 15: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/15.jpg)
Applications received across all windows
In total 229 applications received:
• Ongoing IEs: 49
• New IEs: 180
FCS 33
Agriculture 26
Energy & Environment 27
Governance 28
T&C and F&M 35
Transport & ICT 31
![Page 16: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/16.jpg)
A THEORY OF CHANGE IMPROVING DEVELOPMENT PROJECTS
16
![Page 17: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/17.jpg)
Clear roadmap
17
Clearer roadmap
What
To whom When
Where
![Page 18: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/18.jpg)
Better capacity
18
Better capacity
Research team
Full time field coordinator
Training
More and better data
![Page 19: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/19.jpg)
dime results indicators
88%
74%
75%
36%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Client participated in IE workshop(s)
Baseline results discussed with client
IE results discussed with client
Training provided for data analysis
Capacity Building
19
![Page 20: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/20.jpg)
dime results indicators
84%
85%
48%
50%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
High-quality baseline survey
High-quality follow-up survey(s)
Improved administrative data
Availability of Data as a Public Good
Quality of Data
20
![Page 21: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/21.jpg)
More effort into implementation
21
Motivation to perform
Goal: this is what we are
trying to achieve
Control: this is how we want
to do it
…and BTW somebody
measures (and cares)
![Page 23: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/23.jpg)
dime results indicators
36%
23%
30%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Baseline informed policydesign/implementation
Adopted delivery or behavioral mechanismbased on IE results
IE results were used to motivate scale-up/scale-down of policy at national level
Quality of Policy Decisions
23
![Page 24: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/24.jpg)
In short…
• Put development in the hands of the implementing agencies
• Challenge them to improve project design and experiment with new ideas
• Provide them with the technical resources to generate great data and evidence
• Help them make smart decisions
![Page 25: Using Impact Evaluation to Improve Development Projects · Using Impact Evaluation to Improve Development Projects Arianna Legovini World Bank USG Evaluation Forum ... Transform development](https://reader034.fdocuments.us/reader034/viewer/2022042212/5eb533a106610c02e11adfb4/html5/thumbnails/25.jpg)
Thank You! facebook.com/ieKnow
#impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/impact_evaluation
http://dime.worldbank.org WEB