Development impact evaluation Dime) i2i DevelopmenT impacT...

Development Impact Evaluation (DIME)

Transforming DevelopmenT Through impacT evaluaTion

i2iDIME

TRANSFORM DEVELOPMENT

i2iDIME


Departmentfor InternationalDevelopment from the British people

Development Impact Evaluation (DIME) in the Research Group of the World Bank generates high-quality and operationally relevant impact evaluation research to transform development policy, reduce extreme poverty and secure shared prosperity.

DIME’s objective is to increase the use of impact evaluation (IE) in the design and implementation of public policy, to (i) improve development projects and public policy to deliver better outcomes, (ii) strengthen country institutions for evidence-based policy making and (iii) generate knowledge in strategic development areas.

DevelopmenT impacT evaluaTion

2 Work i2i: Transform Development


operating model

DIME runs experiments to inform decisions, builds

agencies’ capacity to use evidence systemically, and

shares evidence and lessons learned on a global

scale. It does this by:

i. utilizing a programmatic approach to using IE

knowledge for more effective policy;

ii. building capacity and matching policymakers

with researchers from the outset;

iii. assembling project-specific research teams to

implement IEs in collaboration with government

agencies;

iv. emphasizing technical quality assurance and

policy relevance; and

v. ensuring global dissemination and policy

outreach.

Where We Work

74%

Africa

East Asia &Pacific

Europe andCentral Asia

Latin America

Middle East andNorth Africa

South Asia

3%

2%

12%

1% 9%

Policy influence

Design

Strengthen economic

theory

Introduce existing

evidence

Baseline dissemination

Fine-tuning of intervention

Results of randomized control trials

+

Mid-course corrections

Implementation Analysis

Results of IE

+

Scale up or scale down

policy

3Work i2i: Transform Development


DiMe by the nuMbers

DiMe Pillars anD PrograMs

SuStainable Development Human Development equitable GrowtH, Finance anD inStitutionS

Agriculture Education & Health Trade & Competitiveness and Finance & MarketsWater, Environment & Energy Gender GovernanceTransport & ICT Edutainment Fragility, Conflict & Violence

215 projects spanning World Bank global practices and cross-cutting solutions areas

projects addressinggender issues

86of projects cover fragileand conflict situations

% 41

dissemination events in FY15 includingworkshops and master classes16

Almost

3/4 focus on casualmechanisms toimprove impactof IEs

are concentrated inthe Africa region

% 66of projects

evaluatethe overallinterventionof IEs1/4



generating and using ie knowledge for more effective policyEach DIME program is led by an economist and is supported by a team of researchers, field-based IE coordinators, research assistants, and operations and administrative support staff. All activities are implemented in coordination with the relevant World Bank Global Practice. The unit’s manager provides oversight for all DIME activities. At the programmatic level, DIME sets the overall research agenda, manages research efforts and carries out dissemination, policy outreach and fund-raising activities.

Building project-specific research teams to implement ies in collaboration with government agenciesFollowing the completion of a workshop, DIME establishes a research team to work on each IE study through its completion. Each research team includes a principal investigator and co-investigators, a field coordinator based in country to support the government team on all aspects of the IE, and one or more research assistants. DIME staff frequently serve as principal or co-investigators, and manage the overall delivery of IE research and analytical services to the client.

assuring technical quality and review relevant policyAll DIME IEs undergo an internal and external review for technical quality and policy relevance at the concept stage. Review meetings are chaired by the relevant World Bank country director, and include the participation of the government team. This further ensures buy-in from the client, the technical quality of the study, and ongoing relevance to World Bank and country policy priorities. Intermediate and final outputs are discussed and disseminated directly to the government and other partners, and feedback is incorporated into final outputs, including in World Bank working papers for submission to peer-reviewed journals.

providing access to high quality researchers for our clientsWorkshops bringing together policymakers, World Bank operations staff, external researchers, and subject experts are conducted within each program to advance knowledge within the research agenda. First, government counterparts and other partners receive instruction in IE methods and are given tools to allow active participation in the development and implementation of their IE. Second, participants are exposed to the latest available evidence in each focus area, with a view to incorporating relevant evidence in their own program and policy design. Third, each team is paired with one or more IE researchers to develop a prospective impact evaluation that will answer research questions of direct relevance to their specific program. The prospective IE becomes the basis for an ongoing program of IE research and analytical support delivered by DIME to the team.



$16Did you know? trillion in aid has been delivered since the establishment of the Bretton Woods institutions. Yet, the knowledge gap on what really works in development remains large.

policY influence of WorlD Bank impacT evaluaTions

impact evaluation helps Deliver Development projects

Does research add value to aid? Specifically, does

impact evaluation research help or hinder the delivery

of development projects?

A recent study by Arianna Legovini, Vincenzo

Di Maro, and Caio Piza (World Bank Group)

investigates the role of impact evaluations on

the quality of projects. To do this, a panel was

constructed using administrative data on the

whole portfolio of World Bank impact evaluations

and projects approved during 2005–11. By

linking information on projects’ disbursements

with information on impact evaluation, the

study investigated the relationship between IE

and project implementation using 100 impact

evaluations and 1,135 projects implemented in

114 countries across 11sectors. A difference-in-

differences approach was utilized to measure the

effect of impact evaluation on project disbursement

indicators.

Project disbursements are used as a measure

of implementation, as disbursements are made

based on evidence that the agreed activities are

being implemented by the relevant government



and supervised by World Bank project staff. Three

specific quarterly measures are utilized: (i) cumulative

disbursements relative to loan size, (ii) the gap

between planned and actual disbursements relative

to loan size, and (iii) the ratio of project to average

disbursements for all projects in the relevant sector.

To control for project selection into IE, both

institutional and idiosyncratic factors were included,

assuming the confounders are observed or time

invariant. These factors include time, sector and

country effects, project and project leader country

fixed effects, and project leader changes. If project

leaders of potentially high-disbursing projects are

more likely to select into IE, then projects with

IE may disburse faster partially due to the project

leader decision to adopt an IE. If, on the other hand,

project leaders select into IE because their projects

need additional help and support, then projects with

IE will disburse at a slower pace. The data provide

some support for the latter case: before starting

an IE, projects disbursed at the same rate or less

than those projects that never started an IE. This

suggests that more difficult projects selected into IE

to get help.

keY finDings

Doing impact evaluation has large, positive, and

significant effects on project implementation,

especially when measured against plans.

Moving from project design to implementation

is an arduous process. During the period before

midterm, or about two-thirds of the way into

projects’ implementation, projects are delayed, in

the average quarter having cumulatively disbursed

at a rate that is only around half of what they had

planned (15 instead of 29 percent of loan size). The

estimates for the effect of the IE, in this context,

have high economic significance. To midterm, the

effect size accounts for two-fifths (41 percent)

increase in average cumulative disbursements,

significant at one percent level. The planned to

actual disbursements gap is reduced by one-half

(54 percent, significant at one percent level), an

eight percentage point effect that helps realign

projects to their planned implementation path.

When compared to the disbursement path of other

projects in the same sector, IE increases project

disbursements by 40 percent, significant at one

percent level.



clienT feeDBack on ie proDucTsBackground

The World Bank’s annual Client Feedback Instrument

(CFI) is a mechanism to systematically gather

clients’ opinions on the knowledge and advisory

activities they have received from the World Bank.

These activities aim to influence decision making,

strengthen institutions, and empower clients to

implement reforms. They include economic and

sector work (ESW), technical assistance (TA), external

training (TE), and starting in 2013, impact evaluation

(IE). And since 2013 CFI survey covers 1,525 activities

and focuses on government executives (78 percent

of respondents), the group best positioned to use

knowledge services for policy impact.

ie survey results

IE products scored highly in the 2013 CFI survey.

One hundred percent of clients stated that IE work

achieved the agreed-to objectives, two-thirds of

respondents noted that IE results were used for

policy impact, and 92 percent considered the quality

of IE work as high. The quality score was driven by

client satisfaction with the high technical quality of

IE products (85 percent), timeliness (100 percent),

relevance to the client’s needs (92 percent), and

a higher level of engagement with government

counterparts (92 percent).

Compared to other knowledge services, IE

products received the highest marks for all

categories measured, including (i) use and policy

change, (ii) achieving the objectives agreed,

(iii) achieving high-quality services and products,

and (iv) accomplishing key outcomes of knowledge

services in terms of informing policy and decision

making, building consensus among stakeholders,

influencing public debate, strengthening institutional

capacity, improving networks, and generating

innovative approaches and solutions.

Morecapacity

Betterdelivery

Betterplanning

Betterdecisions

More data &evidence



i2iDIME


Departmentfor InternationalDevelopment from the British people

@ImpactEval IEKnow [email protected] www.facebook.com/IEKnow dime.worldbank.org

“

”

The importance of carrying out

impact evaluations whilst a program

is being implemented cannot

be underestimated. In SURE-P

Maternal and Child Health, we are

already learning from the baseline

survey data analysis particularly on

challenging issues such as retention

of midwives in rural areas. The

changes we make to the program

in future based on this feedback

will surely support the building of

an effective health system. We are

eagerly awaiting the follow up survey

results.

—Dr Ugo Okoli

Project Director,

Nigeria Subsidy Reinvestment and

Empowerment Maternal and Child Health

Programme (SURE-P MCH)

Development impact evaluation Dime) i2i DevelopmenT impacT...

Documents

Transcript of Development impact evaluation Dime) i2i DevelopmenT impacT...