Impact Evaluation to Development Impact - World...

Impact Evaluation to Development ImpactTransforming Development through Impact Evaluation

Annual ReportMarch 2016–March 2017

IMPACT EVALUATION TO DEVELOPMENT IMPACTiii

vACKNOWLEDGMENTS

vii–viiiFOREWORD

ix–xii1. EXECUTIVE SUMMARY

1–162. DIME MODEL FOR REAL-TIME EVIDENCE-BASED POLICY2.1 Engagement with the Global Practices. . . . . . . . . . . . . . . . . 12.2 Engagement with the Clients . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Collecting Data and Setting Up Data Infrastructures. . . . 82.4 Generating Evidence and Motivating Change . . . . . . . . . . 12

17–223. SELECTION & QUALITY ASSURANCE FOR POLICY RELEVANCE & TECHNICAL RIGOR

23–304. DELIVERING ON OUR PROMISES

31–425. HOW WE CONTRIBUTE TO DEVELOPMENT ISSUES5.1 Growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Shared Prosperity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.3 Risk and Vulnerability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.4 Governance and Accountability . . . . . . . . . . . . . . . . . . . . . . 375.5 Global Public Goods and Externalities . . . . . . . . . . . . . . . . .39

IMPACT EVALUATION TO DEVELOPMENT IMPACT ivIMPACT EVALUATION TO DEVELOPMENT IMPACT iv

43–686. ECONOMIES OF SCALE IN LEARNING6.1 Fragility, Conflict and Violence . . . . . . . . . . . . . . . . . . . . . . . 436.2 Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .466.3 Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .496.4 Climate Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526.5 Financial and Private-Sector Development . . . . . . . . . . . .546.6 Transportation and ICT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576.7 Gender. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .596.8 Edutainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

69–747. INNOVATION IN DATA QUALITY AND MONITORING OF POLICY INFLUENCE7.1 Monitoring System. . . . . . . . . . . . . . . . . . . . . . . . . . 697.2 Review of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

75–828. CASES OF POLICY INFLUENCE

83–869. COMMUNICATING LESSONS AND RESULTS

IMPACT EVALUATION TO DEVELOPMENT IMPACTv

ACKNOWLEDGMENTS

This annual report has been prepared by World Bank

DIME staff and consultants. It covers the reporting

period from March 2016 to March 2017. Contribut-

ing authors include:

Overall program management, progress, and communications:

n Arianna Legovini, DIME and i2i program

manager

n Florentina Mulaj, Operations Officer and i2i

program coordinator, DIME

n Anushka Thewarapperuma, Operations Officer

and i2i communications, DIME

n Chloe Fernandez, Research Analyst, DIME

n Rebecca De Guttry, Consultant, DIME

n Josine Umutoni Karangwa, Consultant, DIME

Fragility, Conflict, and Violence n Eric Mvukiyehe, Economist, DIME

n Marcus Holmlund, Economist, DIME

n Sarah Elise Elven, Consultant, DIME

Agriculture and rural development n Florence Kondylis, Senior Economist, DIME

n Paul Christian, Economist, DIME

n Maria Ruth Jones, Survey Specialist, DIME

n Astrid Zwager, Research Analyst, DIME

Governance n Vincenzo Di Maro, Senior Economist, DIME

n Daniel Rogger, Economist, DIME

n Bilal Siddiqi, Economist, DIME

n Michael Anthony Roscitt, Public Sector

Specialist, Governance GP

Climate Change n Aidan Coville, Economist, DIME

n Arndt Reichert, Economist, DIME

Financial and Private-Sector Development n Caio Piza, Economist, DIME

n Guadalupe Bedoya, Economist, DIME

n Aidan Coville, Economist, DIME

Transportation and ICT n Kevin Croke, Economist, DIME

Health and Edutainment n Victor Orozco, Economist, DIME

n Marcus Holmlund, Economist, DIME

Gender n Florence Kondylis, Senior Economist, DIME

n Eric Mvukiyehe, Economist, DIME


Monitoring system n Guadalupe Bedoya, Economist, DIME

n Chloe Fernandez, Research Analyst, DIME

n Rebecca De Guttry, Consultant, DIME

Collecting Data and Setting Up Data Infrastructures

n Florence Kondylis, Senior Economist, DIME


n Kristoffer Gustav Bjarkefur, Research Analyst,

DIME

IMPACT EVALUATION TO DEVELOPMENT IMPACTvii

FOREWORD

The purpose of Impact Evaluation to Development

Impact (i2i) is to change development practice

through greater emphasis on the dialectics of devel-

opment and the empirical testing of competing hy-

potheses, while leveraging international assistance

to help governments discover how they can make

best use of their overall resources. Some call it im-

pact evaluation (IE), some, science of delivery, and

others, adaptive programming. But the underlying

idea is to use operational research to generate useful

data and evidence to inform operational decisions in

real time toward greater policy effectiveness. In oth-

er words, using evidence to save and improve lives.

The core and seed funding that United Kingdom’s

Department for International Development (DFID)

provided in phase one of i2i created the dynamics

to bring a program like DIME to the forefront of the

World Bank discussions. First, it elicited an institu-

tion-wide process that generated a Bank-wide gov-

ernance for IE; second, it generated the incentives

for large and substantial participation and co-fi-

nancing; and, third and very importantly, it allowed

i2i to be present in all of the Bank’s Global Practices

and start to affect the operational culture across

the Bank and its partners- especially DFID and

Multilateral Development Banks (MDBs)-, as they

joined and contributed projects to i2i programs. As

a result, DIME is currently evaluating $8.2 billion in

the World Bank’s and $1.3 billion in DFID’s develop-

ment financing.

The problems we help address in development prac-

tice are in the conceptualization and implementa-

tion of policies and in the focus and content of de-

velopment research. These problems are common to

all development agencies. The first problem is that

policy designs are often weakly based on solid theo-

ry, that policies repeatedly (and predictably) fail the

implementation test, and that failure to generate

evidence compromises feedback loops into future

policies. Second, often development research is not

grounded in a deep understanding of reality and

research questions are not problem-driven. Third,

the traditional separation between development

research and practice compromises the quick ad-

vancement of useful knowledge generation.

We address these weaknesses by creating quality

in the engagement between researchers and practi-

tioners, and a process through which (i) data and ev-

idence is systematically generated and used through

the policy cycle and (ii) the capacity of implementer

agencies for data and evidence-based policy nur-

tured through a learning-by-doing approach. In this

report, we show that the benefits of handholding de-

velopment practice through a test-learn-and-adopt

model can affect development outcomes by large

margins. In Rwanda, earnings from farming doubled;

in Brazil, youth savings increased 11–31 percent; in

Senegal, legal delays were cut by 22–30 percent;

and in Mozambique, access to demonstration plots

increased by 20 percent. In the overall Bank portfo-

lio, we observed a 40 percent speeding up in project

implementation due to IE.

Phase one of i2i planted the seeds. The next phase

is designed to consolidate gains, expand donor part-

nerships, and continue to improve the quality of en-

gagement between research and policy.

IMPACT EVALUATION TO DEVELOPMENT IMPACT viii

First, we need to deepen and intensify the model

by investing in existing relationships to speed up

the process of problem-solving and solution-seek-

ing, thus pushing economies toward their efficient

frontier. What we learned from phase one is that,

in low-capacity and low-data environments, set up

times are long. Much time is devoted to building the

data infrastructure, and sometimes the regulato-

ry environment, before field experiments can take

place. We want to take advantage of the invest-

ments we are making across some 60 countries and

intensify experimentation and learning. This makes

eminent sense due to cost-effective use of resourc-

es and to continue building the dynamics to find

permanent and definitive solutions to development

problems.

Second, we need to work in sectors that currently

make only minor appearances in our program. Third

we need to collaborate with Global Practices in the

Bank to improve the quality of operations in the

pipeline by working systematically through project

typologies and creating an appetite for adaptive de-

signs and implementation frameworks. While it is

not always easy to distill operational learning, we

increasingly see the importance of group dynam-

ics and of operationalizing learning to change proj-

ects and address common failures in policy design.

Fourth and lastly is to test the possibility of moving

the profession from case-takers to case-givers and

propose a series of interventions to test big ideas in

development.

Arianna Legovini

Manager, Development Impact Evaluation Unit,

Development Research Group

IMPACT EVALUATION TO DEVELOPMENT IMPACTix

1. EXECUTIVE SUMMARY

OverviewImpact Evaluation to Development Impact (i2i) is a

World Bank program and global platform launched

in March 2014 with support from the United King-

dom’s Department for International Development

(DFID). It leverages international assistance to

generate high-quality and operationally relevant

impact evaluation (IE) research to transform devel-

opment policy, help reduce extreme poverty, and se-

cure shared prosperity. Its purpose is to change de-

velopment practice through a greater emphasis on

the dialectics of development and empirical testing

of competing hypotheses while embedding learning

into each element of the project cycle, from defin-

ing policy, through the decision to continue a pro-

gram or not, to the structure of the next phase of

learning.

Program components include:

n Developing and running experiments in

collaboration with government partners to

inform their policy decisions.

n Building agency capacities to do this

systematically.

n Drawing lessons and sharing them face-to-face

with direct clients and global audiences.

i2i provides support in various ways by: (i) allowing for

a more programmatic approach to evidence-based

policymaking; (ii) capacity building around a broad

set of stakeholders through policy-relevant re-

search agendas; (iii) expanding the reach of under-

developed areas in IE and evidence-based policy.

These are all critical elements to facilitate impact

and influence policy at different stages of a project

and the policymaking process. i2i has also funded

a web-based monitoring system (MyIE) that com-

plements the current advocacy and campaigning

efforts. MyIE reports on periodic progress of IEs

through the life of i2i as IEs move through different

phases of their lifecycle. These indicators showcase

the diversity of influence at all levels, which is well

beyond the typical IE results and impact.

i2i funds IEs across all World Bank sectors, covering

twelve operational sectors of the Bank across four

pillars and two cross-cutting themes. The pillars are

shared prosperity, governance, climate change, and

human development. The cross-cutting themes are

gender and fragility, and conflict and violence. Cur-

rent donor financing does not support the human

development sector, including education, health,

and social protection. i2i aims to support this sec-

tor as new donors join.

i2i adopts DIME’s operational model. DIME is a glob-

al program managed by DECIE in the World Bank

Research Group. DIME has developed institutional

structures to ensure that its IE products are relevant

and can influence the decision-making process for

development. Internally, DIME works with a cross-in-

stitutional council composed of chief economists

and directors from operational vice-presidencies.

The council provides strategic guidance. Working

groups in each global practice set learning priorities

and select IE cases. Each operational team for the

selected cases shape project design and structure

IMPACT EVALUATION TO DEVELOPMENT IMPACT x

The i2i portfolio was developed through close

collaboration with the Bank’s Global Practices,

operational teams, and client counterparts.

experiments to guide project implementation to-

ward greater effectiveness. Externally, DIME engag-

es with clients from the beginning to set research

questions, adapt policy implementation, and agree

on entry points to affect policy decisions using ex-

perimental results. This model transfers knowledge

and tools needed to support evidence-informed pol-

icymaking to country institutions through a medi-

um-term learning-by-doing approach.

Deliverables i2i has a portfolio of 131 IEs, spanning 53 countries

and covering all of the Bank’s regions and sectors.

Portfolio implementation is on track. As of January

2016, 24 percent of the portfolio has been complet-

ed. Most of these IEs have already produced final

outputs, such as working papers, publications, or

project completion reports. 43 percent of the port-

folio is in implementation phase, having passed

technical and policy relevance review and received

World Bank country director and sector manager

approval. A remaining 33 percent of the portfolio

is in preparation phase, with teams having passed

technical and policy relevance for expressions of in-

terest (EOI) and received preparation grants from

i2i to develop full technical proposals. This usually

entails research and Bank team travel to the coun-

tries to meet with the clients and work together

towards finalization of the methodology and imple-

mentation plans.

The i2i portfolio was developed through close col-

laboration with the Bank’s Global Practices, oper-

ational teams, and client counterparts. Each the-

matic program was initiated with a launch of an

impact evaluation workshop for capacity building,

aimed at developing strong IE team capacity and

collaborations between the operational staff, coun-

try governments, and researchers. From March

2014 to January 2017, i2i completed 14 workshops,

covering the following sectors: i) Fragility, Conflict,

and Violence, ii) Agriculture and Rural Development,

iii) Energy and Environment, iv) Governance, v)

Trade and Competitiveness, and vi) Transport and

ICT. Additionally, it completed sector cross-cutting

workshops focused on methods such as distribu-

tional impact analysis and innovations in big data

analytics. In all workshops combined, around 400

organizations participated and around 2000 peo-

ple were trained in impact evaluation methods.

i2i workshops serve three main purposes. They train

counterparts in IE methods to equip them with the

tools necessary for them to actively participate in

the design and implementation of their IEs. They

expose participants to latest available evidence so

that they can incorporate it in their own programs

and policy designs. And, they pair operational and

country teams with highly qualified IE researchers

to develop prospective IE proposals. Most impor-

tantly, workshops equalize opportunities to access

financial support across the development commu-

nity; by pairing policy teams with researchers they

improve the technical quality of proposals. On aver-

age, proposals submitted by teams that attend i2i

workshops receive higher technical and policy rele-

vance ratings from independent reviewers.

Policy InfluenceThe value of IE as a tool for more effective policy-

making is increasingly recognized by governments

and donors. Prospective multi-arm IEs designed to

reliably identify cause-effect relationships based

IMPACT EVALUATION TO DEVELOPMENT IMPACTxi

on counterfactual analysis can guide decisions

over which policies and programs are better able

to achieve desired objectives and how to design

these programs for maximum impact. A study

using data from Bank projects approved between

2005 and 2011 finds that projects with IEs imple-

ment development activities in a timelier manner.

Using disbursements against agreed activities as

an objective measure of implementation, the study

estimates that IE increases average cumulative

disbursements by two-fifths (40.8 percent) and re-

duces the planned-to-actual disbursements gap by

half (54 percent). The results suggest that IE is a

powerful tool to move projects from design to imple-

mentation. These projects are more likely to achieve

their objectives, supporting the idea that project

financing and IE research are complementary ser-

vices provided by the World Bank.

Further, i2i’s key feature is that its model enables

policy influence through all phases of project cycles.

The November 2016 update of the i2i Monitoring

System shows that 75 percent of the portfolio re-

ported that the IE informed design of programs

and policies, based on a clear understanding of the

underlying theory of change and highlighted areas

of uncertainty and critical assumptions. 61 per-

cent of the portfolio reported that IE evidence from

experimental testing of alternative mechanisms

was used by governments or other stakeholders to

adopt the most effective program alternatives or

to inform policy decisions. And 59 percent of the

portfolio reported that IE evidence was used to mo-

tivate scale-up or scale-down of policy.

For example, a large-scale evaluation of a school-

based financial education program in Brazil found

improved financial knowledge, attitudes, and behav-

ior of students. This resulted in the scaling up of the

program nationally by the Ministry of Education.

Another evaluation in Mozambique showed that

training and placing women in extension delivery

positions in agriculture benefits the broader popu-

lation of women in terms of technology awareness

and adoption. Results are being considered in the

design of a new agriculture extension policy in Mo-

zambique. In Nigeria, evidence and learning from

evaluations of maternal and child-health interven-

tions indicates that the lives of mothers and babies

are being saved. This is informing health policy dis-

cussions with the incoming Nigerian government.

Also in Nigeria, the results from the MTV television

drama Shuga evaluation show that the treatment

group was twice as likely to get tested, reported

fewer concurrent sexual partnerships, and reduced

gender-based attitudes and behaviors. Based on

these results, the Gates Foundation is discussing

scaling up its support for edutainment interven-

tions. In Kenya, the Patient Safety IE just recently

started the rollout of the intervention, but the IE

already reported significant contributions to poli-

cy during pre-implementation phase by enhancing

regulatory framework for health inspections and

strengthening institutional capacity through in-

spection protocols and monitoring systems.

Way Forward The last decade has seen a remarkable increase in

demonstrating value for money and effectiveness

in developmental programs run by the World Bank

and other multilateral lending and grant-making in-

stitutions. The standards for demonstrating impact

of development projects have also been raised sig-

nificantly. Rigorous and well-designed impact eval-

uation can help answer the “what” and the “how”

of economic development and help design better

policies. i2i is the largest international initiative de-

signed to systematically learn from development

experience on the basis of rigorous impact evalua-

tion. Its model is specifically designed to overcome

challenges from traditional approaches to evalua-

tion. This includes research-capacity constraints,

coordination failures and transaction costs related

to establishing researcher-policymaker relation-

ships, and a limited understanding of and the ability

to integrate IE into the implementation of policies

and programs at scale.

All i2i-supported IEs build on ongoing and complet-

ed work to create virtuous cycles of learning and

policy impact. Most importantly, by engaging the

government counterparts throughout all phases of

the IE cycle, i2i empowers governments to test in-

novations and scale up solutions. Through this ap-

proach of identifying pathways from policy to re-

sults, i2i will continue to improve accountability and

learn from development interventions and shape

the way development work is done at scale and in a

large number of contexts and practices.

IMPACT EVALUATION TO DEVELOPMENT IMPACT1

2. DIME Model for Real Time Evidence-Based Policy

2.1 Engagement with the Global Practices

Engagement with Global Practices and CCSAEngagement with the World Bank Global Practices

(GP) and Cross-cutting Solution Areas (CCSA) is a

fundamental ingredient to developing economies of

scale in learning and a strategic approach to chang-

ing development practice. GP and CCSA engage-

ment serves multiple purposes: (i) knowledge prior-

ities definition; (ii) strategic case selection; and (iii)

portfolio learning and feedback. The definition of

knowledge priorities is an iterative process that re-

flects both the composition of the project portfolio

and an evolving understanding of what is important

for GPs and CCSA to learn. They are usually reflect-

ed in selected thematic preferences in each round

of program development. Strategic case selection

is fundamental to ensure that important policies

and investments are included in each IE program

Defining Thematic Priorities in T&CThe work with T&C dates back to 2009 in the Africa region. The Director for Finance and Private Sector recognized the importance of measuring the impact of matching grant and SME service projects. These represented the bulk of the GP’s portfolio. Nine matching grant projects became the subject of IE but all of them failed to materialize for the lack of sample. This was because

the projects failed to attract firms’ interest. The problem of take-up was highlighted in the T&C IE Workshop which took place in 2015 as a central problem in private sector operations and generated awareness and willingness among project teams to rethink the way projects market and facilitate firm participation. In 2010, engagement with the business climate group expanded the program in the area of business climate reforms including simplification of firm registration procedures, tax reform, and inspection function. The evaluations of registrations procedures resulted in mild effects on registration rates and non-detectable effects on firm growth even among firms that registered their business. The limited effects of formalization on firm performance reported by those evaluations led to a reshape in the T&C approach. Formalization is no longer seen as sine qua non condition for growth of firms. The new focus shifted towards to identifying ways to make informal firms more productive. In 2016, the Trade & Competitiveness GP identified three areas of work that would move the discourse from eval-uating what projects do into stirring project designs into potential areas for firm growth: high-growth firms, firm linkages and regulatory efficiency. The evolution of T&C thematic area reflects an iterative process of learning about operational practices and the need to put emphasis on testing high-growth strategies that would help design operations in the future.

FIGURE 1: Engagement with GPs and Implementation

GP Sr. Director email

Meetings or clinics with IE experts

Workshop: Train and share evidence

Design facilitation

Call for proposals, double blind technical

review, technical committee

Concept note Bank

review

Concept note

Technical review

Step 5:Resource incentives

Step 5:Resource incentives

Step 3:Group

dynamics

Step 2:Project

feasibility

Step 1:Managerialincentives

TTL Response Preselection DesignSelection for preparation

Selection for implementation

Technical quality �2 DS Technical ratings

Technical quality �2 DS Technical ratings

Project characteristics �Operational opportunities �Budget opportunities

Project characteristics �Readiness �Nature of intervention �Thematic fit

TTL characteristics � IE awareness/ experience � Interact/incentivize �Management responsiveness

Country/ sector

strategic relevance

Strategic fit �Policy relevance �Learning opportunity

Client characteristics �Willingness �Understanding �Flexibility

TTL characteristics �Understand-ing �Flexibility

GP history �Lineage of IE program �Size of IE program

IMPACT EVALUATION TO DEVELOPMENT IMPACT 2

Furthermore, GP engagement is used to align the

priorities of Bank staff by increasing incentives for

investing in operational knowledge. These incen-

tives are both financial and non-financial to include

(i) clearly communicating managerial preferences

and incentives for project teams, (ii) developing

group dynamics and competition for excellence in

learning, and (iii) providing direct financial incen-

tives to conduct analytical work.

In practice, the level of engagement with each GP

and CCSA is endogenous to each GP and CCSA his-

tories, management interest in impact evaluation,

and whether or not i2i covered a GP or CCSA as

part of its program. GPs and CCSAs that have a

full level of engagement with DIME and i2i include

Governance, Trade and Competitiveness, Transport

and ICT, and Fragility Conflict and Violence (FCV),

each with senior management leadership, a dedi-

cated team working closely with the DIME team,

regular management consultations, co-sharing of

program costs, and active exchanges. The engage-

ments are reflected by the large shares of the i2i

portfolio in these areas. At a lesser level of engage-

i2i adopts a bottom-up and demand-driven approach, conducts high-quality and

policy-relevant research, and uses IE as a formative tool to generate

evidence throughout project lifecycles.

(as opposed to researcher-led project selection) and

that the cases reflect the priorities of the GP or

CCSA. Finally, portfolio learning is secured through

a continuous process of knowledge diffusion with

GPs and CCSAs that generates demands for sum-

maries, presentations to GP management and

knowledge boards, organization of IE events during

GF fora, etc.


ment are Agriculture, Energy, Environment, Water

and Finance and Markets. With the exception of

Agriculture with a large and striving program, lower

levels of engagement are reflected by smaller port-

folios of IE. Finally the Health, Education and Social

Protection GPs were not directly targeted for pro-

grams and their large presence reflects non-i2i fi-

nanced work or the intersection with cross-cutting

areas such as Gender and FCV.

2.2 Engagement with the Clients

Workshops to Build Capacity and Stimulate Thinkingi2i adopts a bottom-up and demand-driven ap-

proach, conducts high-quality and policy-relevant

research, and uses IE as a formative tool to gen-

erate evidence throughout project lifecycles. Early

and sustained client engagement is a core element

of i2i’s work. Engaging with government agencies

and other clients early and often ensures their ac-

tive participation in defining research questions

and designing the IE from day one. Further, this

sets the foundation to build client capacity and

empower policymakers and practitioners to exert

control over their local environments. For example,

they can use data generated during an IE to make

mid-course corrections or use the final results to in-

form scale-up decisions. Ultimately, clients become

educated producers and consumers of evidence,

whether from IEs or other types of research, and

we create local capacity for the systematic use of

IE in policymaking.

IE researchers benefit from the early building up of

relationships with policymakers and practitioners.

It affords researchers a better understanding of

idiosyncratic contexts in which policies are crafted

and implemented, allowing them to better tailor re-

search questions and design. Early engagement be-

tween researchers, policymakers, and practitioners

bridges the gap between theory and practice and

kick-starts a process of feedback loops between

various stakeholders. This is a foundation for iter-

ative learning.

The IE workshop is the vehicle to initiate, stimulate,

and strengthen this process. Thematic workshops

are carried out periodically by each i2i program.

These bring together policymakers, practitioners

and operational staff, and World Bank and exter-

nal researchers to share knowledge on research

agendas set by respective steering groups. Table

1 contains a list of workshops and the number

of persons trained over World Bank fiscal years

2014–2017.

Workshops serve four main functions. First, gov-

ernment counterparts and other partners receive

instructions on IE methods and tools. This equips

them to participate actively in development and

implementation of their own IE and become bet-

ter-informed consumers of knowledge, whether

generated through IEs or other methods. Even if the

engagement were to stop here, this alone would be

valuable as policymakers are exposed to multiple

sources of information and a better understanding

of the type of evidence needed to understand caus-

al links is vital to better decision-making.

Second, participants are presented with the lat-

est rigorous evidence in each focus area. They can

then incorporate relevant findings into their own

program and policy design. For many policymakers

and practitioners, this is a rare opportunity to up-

date technical knowledge in their fields and interact

directly with top academics and peers from other

agencies and countries. Presentation of evidence,

often in engaging formats such as seven-minute

“ignite presentations”, convey key messages in a

memorable fashion, enhances cross-country learn-

ing, stimulates creative thinking and provides ideas

that may be incorporated and tested in one’s own

IE, and illustrates the feasibility of high-quality (ex-

perimental) IEs at scale in complex settings.


TABLE 1: Workshops and Number of People Trained FY14–17

Workshops Date City Sector Objective Target Audience

# of people trained

Evaluating for Peace

March, 2014

Lisbon, Portugal

Fragility, Conflict, and Violence

Program develop-ment and capacity building

Project operational teams, government counterparts, researchers, and MDBs

95

Innovations for Agriculture

June, 2014

Kigali, Rwanda

Agriculture Program develop-ment and capacity building


126

Energy and Environment

August, 2014

Berkeley, U.S.A.

Energy and Environ-ment

Leveraging new technologies to improve measurement

Engineers, researchers, and World Bank

60


October, 2014

Lisbon, Portugal




71

Governance January, 2015

Istanbul, Turkey

Governance Program develop-ment and capacity building


143

Trade and Competi-tiveness

May, 2015 Istanbul, Turkey

Trade and Competi-tiveness

Program development and capacity building


139

Transport and ICT

June, 2015

Rio de Janeiro, Brazil

Transport and ICT



148


July, 2015 Chicago, U.S.A.


Portfolio design follow-up

Researchers and World Bank 20

Governance February, 2016

Washington DC, U.S.A.

Governance Review of evidence and to identify emerging priorities

Government counterparts, researchers, and World Bank

179

Edutainment May, 2016 Mexico City, Mexico

Edutain-ment

Review of evidence and to identify emerging priorities

Producers and researchers from leading media organizations and universities

168

Kenya Evidence Forum

June, 2016

Nairobi, Kenya

Transport and ICT

Review of evidence and portfolio design follow-up

Practitioners, researchers, and World Bank

66

Transport and ICT Follow-Up

June, 2016

Nairobi, Kenya

Transport and ICT

Review of evidence and portfolio design follow-up

Researchers, civil society representatives, government counterparts, and World Bank

52

Evidence for Agriculture

November, 2016


Agriculture Review of evidence and to identify emerging priorities

Practitioners, government counterparts, researchers, and MDBs

58

Distribution-al Impact Analysis

December, 2017


Methods Methods to evalu-ate distributional impacts of pro-grams and policies

Researchers 33


Third, each country team is matched with one or

more researchers based on compatibility of inter-

ests. Country teams are invited to workshops based

on preselection in collaboration with the relevant

Global Practices. Similarly, researchers are selected

based on their knowledge, interest, and availability

to work on long-term research collaborations with

governments. Researchers and country teams begin

their work together on the first day of the work-

shop, setting the foundation for collaboration and

compromise on what is feasible and what is ideal

for both sides. This ensures that the resulting prod-

FIGURE 2: Projects that Attend Workshops Consistently Achieve Higher Technical Scores in their i2i Proposals

Attended workshop Did not attend workshop

2.25

2.042.14

2.55

2.142.042.01

1.76

2.08 2.13

1.85 1.91

FCS Agriculture Energy &Environment

Governance T&C and F&M Transport & ICT

Technical ratings of proposals(0–3 scale)

FIGURE 3: Projects that Attend Workshops Consistently Achieve Higher Policy Scores in their i2i Proposals

Attended workshop Did not attend workshop

FCVAgriculture Energy &Environment

Governance T&C and F&MTransport & ICT

7.83 7.94 7.897.45

7.006.666.50 6.49 6.65

6.92

6.02 5.94

Policy relevance ratings of proposals by program(0–10 scale)


uct is of high technical quality, addresses pressing

policy questions directly, and contributes to an-

swering broader development questions.

While efforts are made to get this matching right

from the outset, in those cases where this is ful-

ly successful workshops serve as a marketplace

where both researchers and government teams

alike can find a suitable match. Successful match-

ing is critical for IEs as these are often multi-year

engagements. Using the workshop as a vehicle to

achieve this further equalizes opportunity for i2i

funding calls opened after workshops as well as

other funding opportunities where both technical

research quality and policy relevance count.

Fourth, IE teams develop the initial concept for a

prospective impact evaluation to answer questions

that have direct policy relevance to their program.

Providing dedicated “clinic” time for this during each

workshop day encourages teams to debate and in-

corporate relevant evidence, methods, and IE design

options discussed during the workshop week. Of-

ten, the IE process achieves its first policy influence

at this early stage by encouraging incorporation of

lessons learned elsewhere and testing of alterna-

tive policy designs (or “variations in treatment”). IE

teams are charged with presenting preliminary IE

designs on the final day of the workshop, which en-

sures that teams go home with a clear concept and

roadmap. Further, presentations are always done by

Case Study: IE 4 Peace Workshop, Lisbon, March 2014The first i2i workshop launched the IE 4 Peace program which, focuses on Fragility, Conflict, and

Violence (FCV). This was co-convened by three World Bank Group teams: DIME, the FCV Group (at

that point known as the Center for Conflict, Security, and Development), and the Latin America

and Caribbean Region’s Citizen Security Team. The workshop brought together practitioners, sub-

ject experts, and researchers to promote the strategic use of evaluation to inform policy and program

design and advance knowledge on key issues related to FCV under four themes. These themes—jobs for

resilience; public sector governance, urban crime and violence, and gender-based violence—became the

focus areas for the first two years of program implementation.

In total, 22 teams from across the world attended the workshop. The teams represented diverse coun-

tries such as Democratic Republic of the Congo, Colombia, Honduras, Russia, and Papua New Guinea.

Additionally, researchers from the World Bank and leading universities such as Harvard and MIT con-

tributed their expertise and worked with individual project teams on initial design concepts. Develop-

ment partners including DFID and USAID also participated, as did World Bank FCV and operational

staff, and research organizations such as Innovations for Poverty Action.

The first i2i call for proposals was launched following the workshop. Thirty-three eligible expressions of

interest (EOIs) were received and 14 projects were selected for funding. The selected projects included

several that had benefitted from training, experience and evidence sharing, and researcher matching at

the IE 4 Peace launch workshop. Several of these are now under implementation and incorporate ideas

first developed at the workshop. For example, after learning about the Becoming a Man program in

Chicago, the Honduras Safer Municipalities Project decided to incorporate cognitive behavioral therapy

(CBT) in a training and labor market insertion program for at-risk youth in high-violence municipalities.

The project is now testing variations in CBT delivery through its IE.


government team-members. This reinforces owner-

ship and empowers practitioners and policymakers

to present the work within their governments.

From Design to Implementation

DIME IE workshops provide a platform for collec-

tive brainstorming: to create ideas and spur further

thinking. After the workshop, consistent follow up

consolidates ideas and turns them into concrete ac-

tions which, ultimately, lead to the successful com-

pletion of one or more IEs. A critical first step is to

begin securing buy-in from a wider range of stake-

holders than the workshop’s participants. This in-

cludes decision-makers in governments, World Bank

operational and country management staff, and the

IE research teams. The objective is to form a coa-

lition where everyone has a clearly defined role, in-

cluding rights and responsibilities. This is essential if

the IE is to correspond directly to country-specific

and broader policy-learning priorities; be carried out

at scale in the context of a government program;

and facilitate the use of its intermediate and final

findings as policy and program-management inputs.

An important impetus for building such a coalition

is the possibility of seed funding from i2i. While

preparing their expressions of interest (EOI), teams

build on initial concepts developed at the work-

shops by refining key IE design details including the

types of interventions and number of treatment

arms to be tested, identification strategies, sample

sizes, key outcomes, and budgets and timelines. The

i2i EOI is often the first attempt at situating the

IE within the framework of existing knowledge and

defining how it will contribute both in its immediate

and broader contexts. It is also often the first sig-

nal of policy influence of the IE research process:

75 percent of IEs contribute to rationalizing poli-

cy design by informing the design of a particular

Case Study: The Honduras Safer Municipalities ProjectThe Honduras Safer Municipalities Project participated in the IE 4 Peace workshop. Following the

workshop, the IE team and government continued to work together to refine the IE concept. The

initial concept developed at the workshop called for the evaluation of three separate interventions

targeting three different categories of at-risk youth. This was refined to focus on two categories

of youth—those still in school and those neither in school nor with steady jobs. It was proposed that

two versions of a labor-market training and insertion program be tested for each group. The variation

between the two versions related to the intensity of the cognitive behavioral therapy (CBT) component,

which was inspired by the team’s exposure to the Becoming a Man program at the workshop. This re-

search design was written into the IE concept note, which was reviewed and approved.

After the concept note review, the project underwent several challenges which could have led to the IE’s

cancellation. First, the project was restructured and its budget reduced, leading to the decision to focus

on the second category of youth—those neither in school nor with a steady job—as they are more likely

to be both victims and perpetrators of crime and violence. Second, the intended implementing partner

withdrew support for the IE, forcing the government to make the difficult choice between continuing

with the pre-established operational modality or with the IE.

The project chose the IE. This was not the easy road. In the short run, it almost certainly entailed great-

er costs. The continued implementation of this IE is testament to the strength of DIME’s demand-driv-

en approach to knowledge generation. The IE implementation is ongoing.


intervention (or “treatment”), based on existing

knowledge, or by introducing variations in existing

interventions or entirely new interventions to be

tested. While these may have been discussed at the

workshop, their inclusion in EOIs signals a broader

consensus to go ahead.

Once seed funding is secured, IE and operational

teams work towards fully defining the research

design and documenting this in a concept note.

The concept note is subject to peer review for

both technical quality and policy relevance. Fol-

lowing quality clearance by i2i reviewers, a review

meeting is held to discuss the technical, operation-

al, and policy implications of the proposed work.

This meeting is chaired by the relevant World Bank

country management unit. This validates the policy

relevance of the proposed IE, informs a broader set

of stakeholders, and contributes to ensuring this is

factored in as part of the Bank’s support to a par-

ticular country.

The completion of the concept note review is the

final step in the IE design process and represents

a formal commitment by all parties—the govern-

ment, World Bank, external researchers, and devel-

opment partners—to conduct and complete the IE.

It is often followed by the deployment of a field co-

ordinator (if such a person is not in place already),

who serves as an in-country liaison for the research

team and whose role is to support the government

on all aspects of IE implementation; including op-

erational planning, supervision, data collection, and

dissemination.

2.3 Collecting Data and Setting-up Data InfrastructuresHigh-quality data is a hallmark of the i2i portfolio.

i2i research teams provide technical assistance for

data collection throughout the lifecycle of the im-

pact evaluation. At the initial IE workshop, teams

design a data strategy aligned with their project

cycle and discuss key points of influence. A typical

data strategy includes both in-depth surveys and

routine monitoring data.

The foundation of the impact evaluation analysis is

high-quality microdata from multi-module surveys

of a representative sample of the target popula-

tion. These surveys are completed at key influence

points during the IE lifecycle. Typically: before proj-

ect implementation (baseline survey), midway to

project completion (midline survey), and at project

FIGURE 4: Data in the IE Lifecycle

BaselineSurvey

ProjectImplementation

MidlineSurvey

Mid-CourseCorrections

RevisedImplementation

EndlineSurvey

FinalIE Results


closure (endline survey). When developing a data

strategy, the researchers discuss with the project

team how to align data collection to positively im-

pact project design and implementation.

Essential to the i2i data strategy is to provide data

early and often throughout the project cycle. This is

in sharp contrast to a project-evaluation model in

which a team of evaluators arrives after a project is

completed to assess whether or not the project was

successful. The i2i model aims to provide real-time

feedback and actionable information on how to

improve. If results are not as expected at midline,

there is the opportunity for mid-course corrections

and additional learning-by-doing.

Effecting Decisions through Better DataBaseline Surveys: While not technically necessary

for randomized control trials (the majority of the

impact evaluations in the i2i portfolio), baselines

are an excellent opportunity to provide government

ministries with high-quality sector-specific data,

which is almost never otherwise available. If timely,

this data can provide valuable input to project de-

sign and implementation.

IE Example: The Impacts and Sustainability of Irrigation IE in Rwanda provides a useful exam-ple of how baselines can influence project design. A baseline survey, conducted on a sample of farmers cultivating within the irrigation areas, provided the project team with detailed data on farm practices in the targeted area. This was well-timed to have influence: the team was in the midst of designing interventions to comple-ment the irrigation infrastructure. For example, the project team had planned to collect fees by automatic deductions from sales to coopera-tives. The data revealed that less than 2 percent of farmers made any sales to the cooperative. This forced a change in strategy. In addition, the data provided representative statistics on veg-

etable cultivation, which the project sought to promote. Realizing how few farmers had experi-ence cultivating vegetables (5.3 percent) influ-enced the structure and intensity of agricultural extension.

Midline Surveys: Implemented midway through proj-

ect implementation, at a point when the project is

expected to have achieved initial gains. The midline

survey is a critical mechanism to improve project

implementation and assure that projects meet or

exceed their development objectives. Discussions of

findings of the midline survey hinge around poten-

tial mid-course corrections, and possible new exper-

imental implementation variations.

IE Example: Rwanda Land Husbandry, Water Har-vesting, and Hillside Irrigation Project IE. There are two primary agricultural seasons in Rwan-da, known as Season A (long rains) and Season B (short rains). A midline survey found that the project had met and, in fact, exceeded its ob-jectives in increasing agricultural productivity, commercialization, and farm incomes in Season A. However, they were significantly behind tar-get on Season B. As a result, the team increased their focus on that season, shifting agricultural extension efforts and developing complementary interventions, such as an experiment testing new savings products for farmers.

Endline Surveys: Completed at the end of project im-

plementation. The objective is to capture the full

lifecycle impact of the project and measure gains

from mid-course corrections by comparing indica-

tors to the midline. The research team presents pre-

liminary findings to the project team in a dissem-

ination mission and, after incorporating feedback,

prepares an impact evaluation report and policy

brief. A primary objective of the dissemination mis-

sion is to discuss policy implications of the impact

evaluation findings, particularly opportunities for

scaling up or down. The next chapter discusses pol-

icy implications in detail.


Building a Data InfrastructureThe i2i data strategy aims to create a comprehen-

sive data infrastructure by integrating with the

project-monitoring system, utilizing existing ad-

ministrative data and testing new measurement

technologies. The end objective is to create a data

infrastructure that is informative, allows for timely

responsive action, and sustainable beyond the dura-

tion of the specific project being evaluated.

Monitoring & (Impact) Evaluation: Rather than having

parallel systems—the project’s M&E and IE data—

i2i IEs strive to fully integrate the two. Monitoring

systems are designed to use the same identifica-

tion codes as the impact evaluation, so that data

can be easily merged and compared, and key indi-

cators (for example, agricultural yield, firm profits,

and household income) are constructed in a consis-

tent manner. Each i2i IE team includes a field coor-

dinator, who is based in-country and typically sits

with government M&E staff. The field coordinator

is primarily responsible for technical assistance on

both in-depth surveys and monitoring data and fre-

quently conducts trainings on data management in

Excel and statistical software (for example, Stata,

SPSS) for government counterparts.

IE Example: The Kenya Patient Safety IE provides an excellent example of creating a data infra-structure, fully integrating monitoring and impact evaluation data, and filling an important data gap. The monitoring system includes: (i) data on planning and progress of the inspection pilots (for example, are inspections taking place?); (ii) inspec-tion results at the facility and aggregate levels for each pilot (for example, how are facilities performing in each intervention?), and (iii) third-party monitor-ing indicators to assess quality of intervention and protocol adherence (for example, what is the quality of the inspection delivered?). This customized solution then leads to the availability of timely and actionable information to identify challenges in the implementation and enhance accountabil-

ity to make mid-course corrections; without the intensive use of resources, expertise or equipment commonly absent in poor-resource contexts.

Administrative Data: Many of the i2i impact eval-

uations also incorporate existing administrative

data into their data infrastructure. Governments

typically have large amounts of existing data, but

is often in hard copy only or lacks identification

codes, inconsistently structured, or not centralized

at the national level. As a result, integrating this

data requires trips to field offices, digitization, and

painstaking efforts to merge on available variables.

However, the gains can be substantial, and it has

the positive externality of creating a useful data in-

frastructure sustainable beyond project completion.

IE Example: Building a Supportive Environment for Operation and Maintenance in the Tanzanian Rural Water Supply Subsector IE. To understand the extent of interactions between government employees (and particularly water technicians) and village citizens, the team obtained records from visitor logbooks in rural villages in the proj-ect area. After consultation with the village chief, surveyors took photos of each page of the log-book, which were then digitized. This provides a rich source of information on engagement of ru-ral communities with the government and NGOs, which had not been previously utilized or available beyond the village.

“Big” Data: The next frontier in building responsive

and sustainable data infrastructure is to establish

the most cost-effective ways to monitor project

outcomes by testing various indicators and mea-

surement technologies. Household surveys are

time-intensive and very expensive, which limits the

potential for large-scale or high-frequency data

collection. Big data has the potential to be a more

cost-effective alternative. Recent technology devel-

opments have dramatically increased data avail-

ability and processing capacity. The task at hand

is to apply these to program evaluation, and rigor-


ously test how well big data can be used to mea-

sure key outcomes of interest by comparing to gold

standard methods and carefully comparing costs.

IE Example: the Nairobi smarTTrans IE will de-velop a validated measurement framework for driving and road safety and a set of instruments and crowdsourcing methodologies that can be de-ployed in Kenya and other low-income settings. The IE builds an ICT-based monitoring system, leveraging the high penetration of mobile phones and low-cost technologies. These include: 1) vehi-cle-based technology (sensors, GPS trackers); 2) app-based interfaces for owners and drivers to learn about road safety standards; 3) direct feed-back mechanisms for drivers on performance and ways to improve their driving; and 4) a hotspot and entertainment box combined with SMS and Android platforms for consumer feedback on driver behavior. The technologies will be integrat-ed into a big data infrastructure that will provide information to improve decisions by all actors in the urban transit system, from policymakers and regulators to private insurers, operators, drivers, and riders.

Ensuring High-Quality Data

Project teams in the i2i portfolio finance all data

collection. i2i provides technical assistance to en-

sure high-quality data at each stage of the process:

developing the terms of reference for a survey firm,

designing the survey instrument, participating in

FIGURE 5: Elements of a Big Data System

Data CollectionMethods

Storage Space &Analysis Tools

ExperiencedData Analysts

BIG DATAINFRASTRUTURE

FIGURE 6: i2i Technical Assistance During Data Collection

• Design representative, well-poweredsampling strategy

• Advise on survey firm TORs

• Design questionnaire, in consultation withproject team

• Pilot questionnaire• Program CAPI questionnaire

• Review field plan• Conduct surprise spotchecks in the field• Check data quality on a daily basis, establish

feedback loops with the project team tocorrect problems

• Review final dataset for completeness• Clean data• Analyze data, according to analysis plan

developed jointly with project team• Contribute to policy briefs and reports

Survey Preparation

• Review enumerator training manual

• Participate in enumerator training

Questionnaire Design

Enumerator Training

Fieldwork

Survey Completion


enumerator training, supervising field work, and

checking data quality and alerting the team about

any issues in real time. In this way, i2i achieves

high-quality data and builds capacity for high-qual-

ity data collection within the government ministry.

Figure 6 outlines the main areas for technical assis-

tance in a typical survey.

i2i surveys are typically computer-assisted person-

al interviews (CAPI), with exceptions only in case

of concerns of enumerator safety or extreme in-

frastructural limitations (for example, electricity,

internet, and transport). The i2i team typically pro-

grams the questionnaire to ensure that program-

ming meets the highest quality standards: CAPI

technology has the potential to greatly increase

data quality, but only if carefully programmed.

All i2i surveys incorporate: automated skip codes,

range restrictions, internal consistency checks,

pre-populated identification information for fol-

low-up rounds of panel surveys, and validation

before submission (for example, that all expected

fields are completed).

The field coordinators are the primary bulwark

against poor data quality. They work closely with

the project teams and survey firms on a day-to-day

basis, with support from the IE researchers. Recog-

nizing their critical role, i2i supports an annual field

coordinator workshop, in which all field coordinators

come together as a community of practice, to be

trained on managing high-quality surveys, learn the

latest developments in best practice protocols, and

build skills on relevant survey programming and da-

ta-analysis software packages.

2.4 Generating Evidence and Motivating ChangeDIME’s ambition is to use rigorous evidence to mo-

tivate policy change in the world’s poorest coun-

tries. At a minimum, this requires connecting les-

sons from our evaluations to new policy decisions.

However, DIME’s evaluation model aims to embed

learning into each element of the project cycle,

from defining policy, deciding whether to continue a

program or not, to the structure of the next phase

of learning. Given its continuous nature, our model

is a form of ‘real-time’ learning.

In traditional evaluation models (such as that

summarized in figure 7), the evaluation is done

after the program is completed. Policies inform

designs, which are then evaluated, and the re-

sults are used to decide whether the program

should continue or not. In the best cases, eval-

uation reports aim to distill wider learning from

the efforts of evaluation, but there is no strate-

gic linkage between that learning and the poli-

cy-formation process. This could even be true if

an impact evaluation is embedded in the project.

Figure 7 represents a model where the evaluation

team is seen as separate from the implement-

ers, independently receiving data and reporting

results in the final stages.

Real-time learning implies that we should aim to

undertake a circular model, where learning is an

integral part of each stage of policy development.

Figure 8 connects learning from previous project

cycles to new ones. However, DIME’s ambition is to

embed circularity and feedback loops in each stage

FIGURE 7: The Traditional Evaluation Model

POLICY DESIGN EVALUATE CONTINUE? LEARNING


of the project cycle. From the anti-clockwise arrows

in figure 8, we see learning as feeding into policy de-

sign and also the decision as to whether to contin-

ue a program and how we should modify our plans

for the project evaluation. Similarly, discussions

around the continuation of the project are about

optimal design. Each stage of the project cycle can

be designed to feed back into any other.

Embedding such learning in each stage of the cycle

requires strong partnerships between us and our col-

leagues at the World Bank, other multilateral agen-

cies, and our government counterparts. Sections 2.1

and 2.2 explored how we build those partnerships

with our colleagues and clients. This section will

explore how we work with those partners at each

stage of the project cycle to effectively generate ev-

idence and motivate evidence-based change.

PolicyAs discussed in section 2.1, we aim to engage

with projects at an early stage of development.

Preferably, when the policies under which they

fall are still being developed. This allows us to en-

sure that policies reflect frontier evidence from

academic and policy research. This evidence

identifies both what is known, and what still

needs to be tested, so that evaluation can be

embedded into policy itself. 90 percent of DIME

projects have facilitated the embedding of fron-

tier research into government policy. This hap-

pens in both ad hoc and more systematic ways.

For example, DIME’s Fragile and Conflict States

group worked with a series of senior academics

and Bank staff to write white papers summariz-

ing frontier literature of the sector. These white

papers ensure ready access to frontier research

lessons and corresponding gaps for project

teams working in the concerned areas.

Sometimes, relevant evidence is not available in the

research literature. DIME works with government

counterparts and our colleagues to develop policies

that reflect these ambiguities. The change we hope

to inspire is a recognition in country policies that

the right path for a country is not currently obvious,

and so different interventions will be experiment-

ed with until there is sufficient evidence to choose

a single path, or expand multiple interventions to

those communities where they fit best.

We also work to generate evidence to inform policy.

In Tanzania, DIME works with the United Kingdom’s

Department for International Development (DFID)

to improve the maintenance of water infrastruc-

ture in rural areas. DFID’s approach to this project

is to pilot an ‘adaptive’ model. Rather than contract

DIME to undertake a single evaluation formulated

at the start of the project, DFID has written mul-

tiple stages of review into the contract so that

its policy on water maintenance in Tanzania can

be updated. Inspired by academic models of poli-

cy development such as ‘Problem-Driven Iterative

Adaptation’, this model ensures that DFID’s policy

constantly reflects the most up-to-date informa-

tion and insights from the field.

FIGURE 8: DIME’s Evaluation Model

POLICY

DESIGN

EVALUATECONTINUE?

LEARNING


DesignInterventions are designed, based on policy. While

policy sets the broad parameters of intervention,

there are typically large areas of ambiguity in how to

design a particular project or program. This is where

a circular, or real-time, learning model facilitates

evidence-based decision-making. Working with re-

searchers allows operational staff access to frontier

research on project design. Similarly, working closely

with operational staff allows researchers to identify

research that is ‘operationally-relevant’. Both sides

of the research-policy divide can best understand

the other’s perspectives when jointly designing a

project. That is what DIME strives to do.

For example, working closely with the Rwandan Gov-

ernment had given DIME’s agriculture team a repu-

tation for being easy to work with. The government

was keen to operationalize its agricultural strategy

and requested DIMEs support. DIME worked with

the government to design, introduce, and test inno-

vative farmer-feedback tools. Where it was not ob-

vious how to design the tools, DIME helped the gov-

ernment trial multiple modalities. DIME’s research

background also allowed for an additional twist to

be added to the project so that the evaluation could

learn about the underlying mechanisms driving the

results. It found that particular feedback tools in-

creased attendance in agricultural extension train-

ings and resulted in increased adoption of superior

farming techniques among non-users. The most

cost-effective feedback mechanism (a hotline) was

adopted and scaled up throughout Rwanda. In ad-

dition, the satisfaction data collected as part of the

evaluation convinced the government to continue

supporting a public-private partnership that had

supported the implementation. The ongoing policy

of the government reflects frontier research.

Evaluation

Embedding an evaluation into any part of the proj-

ect cycle allows for learning that can strengthen

the project and provide broader lessons for policy

around the world. Such evaluations can take many

forms, both across and within projects. In Gha-

na, DIME worked with the Office of the Head of

the Civil Service to survey all civil servants on the

bottlenecks to improved service delivery. As dis-

cussed in section 2.3, rigorous collection of care-

fully designed indicators can be sufficient evidence

to motivate change by itself. The results of the civil

servants survey showed that a series of obvious re-

forms could be implemented immediately. For ex-

ample, officials complained that organization heads

were not being monitored as was intended by the

Public Service Rules. The Head of the Civil Service

immediately fixed the monitoring system. Similarly,

in Kenya, DIME worked with the judiciary to build

a ‘Daily Court Returns Template’ to scientifically

gather and organize daily court output. This was

both legislated into national law and was a useful

data-collection tool for the IE.

DIME frequently embeds a preliminary evaluation in

its first year of engagement with a project. Using

the best available data, or that which can be col-

lected immediately, DIME researchers aim to iden-

tify whether there is an empirical basis for an evalu-

ation approach. In the Ghana case, there were areas

that all civil servants stated were working well, with

no need for reform. These didn’t seem the first

places to test for significant constraints to pro-

ductivity. There were others that required further

Many of our government and implementation partners require an answer to whether

an intervention should be continued or not.


investigation, focusing the scope of our evaluation,

and changing the reform policy of the government.

Using a multitude of impact evaluation methods al-

lows DIME researchers to shine a light on areas of

reform that the government has been considering,

and those they had not yet conceptualized. These

discussions are then the basis to identify a series

of reform options that DIME typically evaluates us-

ing a randomized control trial. In 88 percent of our

Randomized Controlled Trial evaluations, we look to

go beyond a simple understanding of which flavor

of intervention works best to why it is most suc-

cessful. In Ghana, we are now using an RCT to build

a more effective training system for public officials,

something they highlighted as a major constraint in

the survey.

ContinuationMany of our government and implementation part-

ners require an answer to whether an intervention

should be continued or not. Public-sector funding is

constrained across the world, so allocating program

budgets efficiently is of importance to the effec-

tiveness of the state. DIME aims to provide inputs

to answer this question, but with a focus on how

to generate the largest gains from a program. By

isolating the best way to deliver an intervention, we

allow ourselves to reformulate its design and evalu-

ate it on its greatest strengths.

In Kenya, DIME worked on supporting regulatory re-

form in the health sector. It supported a review of

the health inspections regime for the country. Re-

sponding to the government’s question of whether

health inspections were worthwhile, DIME research-

ers supported the passing of an enhanced regula-

tory framework for health inspections and the de-

sign of a toolkit of instruments to measure patient

safety. The database created through these efforts

allowed the team to identify multiple inspection

regimes that could be trialed in an IE framework.

Thus, the project is now looking at how to maximize

the impact of patient monitoring.

LearningThis subsection of the report has outlined how

evaluation teams can organize projects to learn in

‘real-time’. There is often a point in an evaluation

cycle that is seen as an opportune time to reflect

substantively on the learning process and aggre-

gate lessons to date. In the traditional model, this

would be the point at which the evaluation report

was produced. In the DIME model, this can be at

various points throughout the lifetime of a single

evaluation or on completion of multiple comple-

FIGURE 9: IE on Real-Time Learning

IEs that provided:

Better Data Improved M&E57%

High-Quality88%

High-QualityEndline

96%

Access toKnowledge

Program LaunchWorkshops

63%

BaselineDiscussions

69%

IE ResultsDiscussions

92%

At Design At Implementation For Continuation. . .

mentary evaluations. DIMEs organization around

thematic groups allows it to strengthen the lessons

of any single evaluation by relating it to the learn-

ing from a range of others.

Working closely with the Nigerian Government on

its healthcare sector for almost a decade, DIME

undertook real-time learning in partnership with

the Ministry of Health. We introduced variations in

policy design informed by existing evidence on in-

terventions in the sector. Generating a wealth of

microdata, DIME supported the delivery of routine

program elements (such as working to improve the

timeliness of payments to frontline health work-

ers) and fed the resulting data into decision-mak-

ing across the health sector. However, to capital-

ize on the multitude of evaluations in the sector,

DIME organized an ‘Evidence and Action’ workshop

that presented senior stakeholders with the results

of all the evaluations in one go. This facilitated a

‘sector-wide approach’ to evaluation that enhanced

the learning of any single evaluation by allowing ev-

idence from one project to be cross-checked and

validated by another. DIME also benefited from this

exercise by allowing us to conceptualize the struc-

ture of learning that would best fit our next tranche

of work with the Nigerian Government.

Undertaking policy making in the developing world

can be a daunting experience. DIME has strived to

enable our partners to have access to better data,

knowledge, and learning at each stage of the proj-

ect cycle. Providing continuous feedback to imple-

menters that we work with also means that we are

learning in real time ourselves. Our model is chang-

ing and evolving with each new partnership and

evaluation that we undertake.


3. Selection and Quality Assurance for Policy Relevance and Technical Rigor

As described in previous chapters, DIME’s goal is

to increase the use of impact evaluation evidence

in the design and implementation of public policy

across the developing world. It works with programs

at scale on game-changing issues to answer policy

questions that clients identify and to increase de-

velopment outcomes. The highest technical quality

and policy relevance is ensured at all stages of proj-

ect cycles, while maintaining a flexible environment

for adaptation based on real-time evidence and

changes in context and client needs.

How does DIME achieve this? It does so through an

innovative operating model with a bottom-up ap-

proach, transferring IE knowledge and tools to clients

and matching them with internationally renowned

technical experts to deliver the highest quality prod-

ucts of policy relevance (see chapters 2.1–2.4). Client

engagement from the early phases of the design

ensures relevance of policy questions, government

buy-in, feedback loops, and policy action through al-

most all phases of the project cycle. DIME projects

report policy influence at baseline, during implemen-

tation, and scaling up or down based on final results

(see chapter 8 on Cases of Policy Influence). This

approach, which is significantly different from con-

ventional development research, has defined DIME’s

model over the last 12 years.

But why this approach? Because it overcomes

critical challenges of traditional evaluation and re-

search. As chapter 2.4 discusses, to be done well,

FIGURE 10: DIME’s Operating Model

Syst

emat

ic o

f evi

denc

e

Inform policydesign Identify

knowledge gaps &Set sector priorities

Build capacitythrough IEworkshops

Open callsfor proposals

Traininggovernment

clients and othersin IE methods

Guide policyimplementation,make mid-course

corrections

Review &Select proposals

Provide financial& technical

support for IEs

Monitorimplementation

Match teamswith researchers

and subjectexperts

Provide policyfeedback to

inform adoptionand scale-up

Conduct analysis& Generate

results

Disseminatefindings

Improveeffectiveness of

development policy

Generate actionable results

and empowergovernments

Capacity building

1

5

2

6

3

4

8 97


IE requires strong technical expertise. This is why

researchers tend to secure most research funds

that come through various development channels.

However, due to a lack of client engagement in the

design of research, policy questions addressed and

results are often questionable in terms of their

policy relevance and there are usually no feedback

loops to generate policy action. DIME was built pre-

cisely to overcome this barrier and serve as a bridge

between researchers and policymakers. Its the-

matic programs start by defining broader research

priorities through close consultation with the re-

search community and development partners, in-

cluding World Bank’s global practices and regions

(see chapter 2.1). It then identifies operationally

relevant programs that enable impact evaluations

around critical policy issues. It engages operational

teams and government counterparts; starting with

a workshop to build capacity, raise awareness, and

form partnerships with technical experts, setting

the foundation for longer-term collaborations and

policy influence (see chapter 2.2-2.4).

Developing the Program in Energy and EnvironmentDIME’s Energy and Environment program began in June 2014, following an iterative approach

to balance the objective of addressing prioritized knowledge gaps with operational realities. The

process included direct engagements with the Bank’s Climate Change Cross-cutting Solutions

Area, Energy and Extractives Global Practice, Environment and Natural Resources Global Practice,

Water Global Practice within the World Bank, the Global Environment Facility (GEF), the Climate Invest-

ment Funds (CIF), the Energy Sector Management Assistance Program (ESMAP), and DFID (including

the evaluation department and climate-change teams).

These partners actively contributed to the identification of evidence gaps, mostly from an operational

standpoint, and a portfolio of potential projects aligned with those gaps. The main research partner-

ship was with the University of Chicago led by John List (Chairman of the Department of Economics).

Other collaborations on measurement and technology include partnerships with the University of Cali-

fornia at Berkeleyand the Centre for Effective Global Action (CEGA). These partners contributed to the

knowledge agenda by summarizing and discussing the current stock of academic literature and future

research priorities.

A kick-off workshop, held jointly with CEGA in August 2014 in Berkeley, brought together engineers, econ-

omists, and World Bank counterparts to explore leveraging new technologies to improve measurement

in energy and environment projects and research. A follow up measurement workshop, with a focus on

innovative measures for climate resilience, was held in June 2015. The main workshop, and official launch

of the program of work in Energy and Environment, was held in Lisbon in October 2014 bringing together

19 project teams (financed through DFID, the GEF, CIF, IDA, and IBRD) and 28 researchers from 11 aca-

demic institutions to refine research opportunities based on project interest and operational feasibility.

Representation from governments typically included high-level policymakers, project implementers,

and/or monitoring and evaluation specialists. For instance, from Bangladesh, participants were the

Project Director, the Project Deputy Director, and the Executive Engineer of the government’s Rural

Electrification Board of the Bangladesh Rural Electricity Transmission and Distribution project. From


At the project level, the DIME model generates

evidence throughout the policy lifecycle. As chap-

ter 2.4 describes, at the design phase, DIME IEs

strengthen economic theory of interventions and

make improvements based on existing evidence.

During baseline and follow-up, data is used to stim-

ulate policy dialogue and sometimes to make mid-

course corrections. During implementation, the

IE strengthens client M&E systems and develops

high-quality survey instruments and data-collec-

tion protocols. During analysis and results, teams

fine tune country policies and programs based on

evidence, often motivating scale-up or down. Re-

sults are also shared more broadly with the inter-

national development community through pub-

lications, seminars, workshops, and face-to-face

interactions.

Applications for IE work are submitted to DIME

through calls for proposals, usually announced after

DIME workshops and targeting teams participating

in workshops as well as the development commu-

nity at large. All submissions, both at the EOI and

concept note stage, undergo rigorous technical and

policy relevance reviews. External technical experts,

identified for their IE and subject matter expertise,

score proposals on a set of technical criteria through

a blind review process. Internal Bank GP and regional

focal points score the proposals on policy relevance

and feasibility of implementation. All submissions,

review scores, and ranking of the submissions by

scores are submitted to the i2i technical commit-

tee. The technical committee, which comprises se-

nior and lead economists from the Bank’s Research

Group, makes selection decisions based on technical

the Argentina Renewable Energy in Rural Markets Project, representation included the coordinator gen-

eral, electrician engineer, and M&E specialist from the Energy Secretariat of the Ministry of Energy and

Mining. During the week-long workshop, project teams spent time with researchers to develop proposals

relevant to their specific projects. They presented their achievements on the final day of the workshop.

Two months after the workshop, in July 2015, researchers participating in the program were invited to

present initial designs and receive feedback during a session chaired by John List and supported by Mi-

chael Greenstone (Director of the Energy Policy Institute) at the University of Chicago. The first round

of projects presented concepts after initial scoping missions that led to the refinement and finalization

of concept notes. Nine projects developed through this process are currently ongoing, seven passed

the CN stage and are currently in the implementation phase. Client engagement continues through all

stages of the impact evaluations. Examples include joint analysis and discussion of the baseline data

to develop technical capacity in the government and inform intervention design.

FIGURE 11: i2i Review Process

IE teams submitproposals

Technical committeeapproval

Bank's review andmanagerial approval

Monitoring andreporting

External technicalreview

Policy relevancereview


and policy score, overall feasibility of implementa-

tion, and capacity to target important knowledge

gaps. The teams that pass the EOI stage receive

a preparation grant from i2i ($25,000) to devel-

op a full technical proposal. The teams that pass

the concept note stage receive an implementation

grant from i2i ($150,000 over three years).

After the concept notes receive i2i approval, they

undergo the Bank’s internal quality assurance pro-

cess, involving a separate review meeting, chaired

by country or Global Practice manager, and incor-

porating review feedback from at least two peer

reviewers, usually a subject matter expert and an

operations expert. This process of combining i2i’s

and the Bank’s review processes ensures technical

quality, buy in from the client, and ongoing rele-

vance to World Bank and country policy priorities.

In some cases the Bank’s regions rank a proposal

highly on policy relevance and there is a strong com-

mitment from the client to do the evaluation, but

the technical evaluation from the external review

does not meet i2i technical standards. Here, DIME

provides technical expertise to build capacity and

revise the design. For example, Colombia Mobile Vic-

tims Unit Impact Evaluation, selected through the

Fragility, Conflict and Violence funding window, re-

ceived very high policy relevance ranking but did not

pass the i2i technical review. With demand from the

Bank project team and clients, DIME assigned two

researchers to this team to think through the criti-

cal aspects of the design and revise the methodol-

ogy. The revised proposal was submitted to i2i for a

second review through external reviewers and then

went back to the technical committee, after which it

was approved and included in the i2i portfolio.

During implementation, i2i-supported IEs report an-

nually on progress and potential channels through

which they affect capacity and influence policy

decisions throughout the cycle of implementation

(see chapter 7.1 monitoring). On completion, final

product reports and working papers are submit-

ted to i2i for technical review, following a similar

FIGURE 12: Bank’s Review Process

IE teams submit proposals IE team revises andresubmits

Decision meeting chaired by manager

or country director

Peer review by at leasttwo internal reviewers

Wider circulation in the GPand region for feedback

FIGURE 13: Quality Assurance during Project Cycle

1Policy Design

2Baseline and

Follow-up

3Implementation

4Analysis and

results

• Concept Notetechnical andpolicy relevancereview

• Technical reviewof all surveyinstrumentsand protocols

• Annual in-depthprogress updatethrough MyIEmonitoring system

• Final product/IEreport and papertechnical review


review process from the concept note stage. After

i2i clearance, teams proceed with the Bank’s review

of the final product, followed by delivery to client

and publication. Crucially, all data is to be made

available for public use through the World Bank’s

Microdata library.

Review Criteria for Expressions of Interest n Clarity of research questions and potential to contribute to evidence gaps

n Prioritized project components/interventions have logical pathways to intermediary and final

outcomes

n Credible identification strategy for each research question

n Potential for learning, for example, by including multiple treatment arms

n Clearly defined targeting and recruitment of participants, and adequate number of participants to

implement proposed analyses

n Feasibility of implementation (sample size, intervention, selection of beneficiaries, and country

context)

n Evidence of partner engagement and support

n Potential to influence the design and/or prioritization of current and future development

interventions

n Potential to influence policy design and/or scale-up.

Review Criteria for Concept Notes n Hypotheses and research questions clearly linked to the theory of change and relevant to

important research and/or policy questions

n Main outcomes of interest are relevant to answering research questions and are feasibly gathered

n Evaluation design and sampling strategy:

¡ identification strategy well explained and defines a credible counterfactual

¡ design presents no ethical issues, or if it does, mitigation measures are highlighted

¡ sufficient detail on sample size/power calculations provided for each of the primary research

questions, given available data.

n Details on data collection instruments; the data collection strategy is thought out and feasible and

includes information on ethical clearance

n IE management, research team, and implementing partners have sufficient capacity to carry out

the proposed research and proposed budget is realistic and represents research value-for-money.

Additional Requirements before i2i Funds are Transferred n Co-financing of at least USD 10,000 per year (for each year of i2i support)

n Trial registration of the research design prior to initiating a study

n Ethical approval of the IE if it involves human subjects

n Ethical training certification for researchers.


4. Delivering on Our Promises

i2i has met and over-delivered its targets for FY17.

The i2i results framework in annex 1 shows prog-

ress towards all targets as agreed with the donor.

The rest of this chapter provides a description of

some of the main deliverables of IE products, ca-

pacity-building workshops, and policy influence.

PortfolioThe i2i portfolio has a total of 131 IEs across 53

countries, covering all regions and i2i thematic ar-

eas. Figure 14 below shows the organization of i2i

topics around thematic pillars. All targets for re-

gional and thematic area distribution have been

met. The portfolio was developed through close

collaboration with the Bank’s Global Practices and

calls for proposals during the first two years of i2i

implementation. In FY15, four calls for proposals

were completed for new IEs in i) Fragility, Conflict

and Violence, ii) Agriculture, iii) Energy and Environ-

ment, and iv) a separate call for ongoing IEs across

FIGURE 14: i2i Thematic Pillars

1

5

2

6

3 4SharedProsperity• Finance and

Private Sector• Agriculture• Infrastructure

Governance• Public Sector

Governance• Justice• Local Development

ClimateChange• Energy• Environment• Natural Resource

Management• Agriculture• Transport

HumanDevelopment• Education• Health• Social Protection

Gender• Human Capital• Economic

Opportunities• Voice/Agency

Fragile andConflictSituations• Economic/social

reintegration• Governance• Gender-based violence• Urban crime and violence

Cross-Cutting Themes


sectors (including proposals at various phases of

IE implementation). In FY16, three additional calls

were completed for new IEs in i) Governance, ii)

Trade and Competitiveness and iii) Transport and

ICT (see chapters 2.1, 2.2, and 2.4 for more back-

ground on thematic program development, work-

shops, and selection of proposals). Figures 15 and

16 show the distribution of the portfolio by region

and thematic areas.

In addition, targets for gender and fragile and

conflict effected situations have been met as well.

Gender and FCS are core themes of i2i work. As

figures 17 and 18 show, at least 17 percent of the

portfolio evaluates a gender-specific intervention

and 57 percent conducts disaggregated gender

analysis. 17 percent of the portfolio is in FCS coun-

tries, and 28 percent in the FCS sector.

In terms of design, 73 percent of the IE portfolio

adopts experimental methods, 14 percent uses both

FIGURE 15: Distribution of the i2i Portfolio by Region

IEs by RegionNumber (percentage) of i2i IEs

AfricaLatin America and

the Caribbean

South Asia

East Asia and PacificEurope and

Central AsiaMiddle East and

North Africa

Global

69 (53%)

24 (18%)

17 (13%)

9 (7%)

8 (6%)

3 (2%)

1 (1%)

The sum percentages may not be equal to 100% due to rounding off.

FIGURE 16: Distribution of the i2i Portfolio by World Bank Global Practice

IEs by Global PracticeNumber (percentage) of i2i IEs

Governance

Agriculture

Trade andCompetitivenessSocial Protection

and Labor

Transport and ICTs

Water

Finance and Markets

Energy andExtractives

Social, Urban, Rural,and Resilience

Health, Nutritionand Population

Education

Poverty

The sum percentages may not be equal to 100% due to rounding off.

4 (3%)

2 (2%)

Environment andNatural Resources

36 (27%)

20 (15%)

19 (15%)

12 (9%)

11 (8%)

7 (5%)

7 (5%)

5 (4%)

5 (4%)

2 (2%)

1 (1%)

FIGURE 17: Distribution of the i2i Portfolio by Gender

IEs Including a Gender AnalysisNumber (percentage) of i2i IEs

IEs Evaluating a Gender-SpecificIntervention

Number (percentage) of i2i IEs

IEs Falling under the GenderCross-Cutting Solution Area

Number (percentage) of i2i IEs

30 (23%)

98 (77%)73 (57%)

56 (43%)

107 (83%)

22 (17%)

Yes No


experimental and non-experimental, while another

14 percent has non-experimental design. In terms

of multiple treatment arms, which test project in-

novations, around 65 percent include more than

one treatment arm.

Progress of portfolio implementation is also on

track and i2i meets its target for the number of

competed IEs. As of January 2016, 24 IEs reported

as completed and have final outputs such as work-

ing papers, publications, or project completion re-

ports. There are an additional seven IEs that report

having completed the analysis and working towards

producing the final output. Further, 43 percent of

the portfolio is in the implementation phase. These

have passed technical and policy relevance reviews

from i2i, completed World Bank concept note review,

and received country director and sector manager

FIGURE 18: i2i Portfolio in Fragility, Conflict, and Violence

IEs Fragile and Conflict-affected SettingsNumber (percentage) of i2i IEs

IEs Fragile and Conflict-affected CountriesNumber (percentage) of i2i IEs

36 (28%)

22 (17%)

108 (83%)

94 (72%)

Non-FCS Countries FCS Countries

Non-FCS Sector FCS Sector

FIGURE 19: Distribution of the i2i Portfolio by Evaluation Method

IEs by Evalution MethodNumber (percentage) of i2i IEs

18 (14%)

18 (14%)

95 (73%)

Experimental Non-experimentalBoth experimental and non-experimental

FIGURE 20: Distribution of the i2i Portfolio by Number of Treatment Arms

IEs by Number of Treatment Arms*Number (percentage) of i2i IEs

32 (36%)29 (32%)

21 (23%)

8 (9%)

>3321*Applicable after CN review


approval. The remaining 33 percent of the portfolio

is in preparation phase. Here, teams have passed

technical and policy relevance for expressions of in-

terest and received preparation grants from i2i to

develop full technical proposals. This usually entails

research and the Bank team travels to the country

to meet with the client and work together to final-

ize methodology and implementation plan.

Workshops for Capacity-Building and Dissemination In addition to supporting IE products, the i2i pro-

gram is committed to delivering four IE workshops

per year. As table 1 in section 2.2 shows, the i2i pro-

gram completed 14 workshops between FY15–FY17.

During the first two years of i2i implementation,

workshops focused on thematic program develop-

ment, capacity building, and portfolio selection. Au-

dience in these workshops comprised researchers

and subject experts from leading universities, gov-

ernment policymakers, operational staff from the

World Bank, and representatives from other Mul-

tilateral Development Banks and donors. Following

workshops, calls for proposals were organized to

invite teams to submit proposals for IEs in select

thematic areas. During this last FY, the nature of

workshops shifted from program development and

capacity building of government counterparts to

strengthening the research designs of the selected

portfolios in each thematic area and reviewing and

disseminating evidence. The target audience there-

fore shifted towards the research community, sub-

ject experts, and MDB operational staffs.

As the table below shows, all targets were met

for the number of workshops completed, people

trained, participation of organizations, and dis-

semination to policymakers. People trained in i2i

workshops and by i2i project teams form the i2i

network, which today includes close to 2000 rep-

resentatives and over 400 different organizations

(donors, MDBs, government agencies, academic in-

stitutions, and NGOs).

Policy Influencei2i IEs affect policy through all phases of project cy-

cle. Chapter 2.4 describes this model in more detail.

Data from November 2016 from the i2i Monitor-

ing System shows that 75 percent of the portfolio

reported having informed design of programs and

policies, based on a clear understanding of the un-

derlying theory of change and highlighted areas of

uncertainty and critical assumptions. 71 percent of

the portfolio reported that IE evidence from experi-

mental testing of alternative mechanisms was used

by governments or other stakeholders to determine

the most effective program alternatives or to in-

form policy decisions. And 59 percent of the portfo-

lio reported that IE evidence was used to motivate

scale-up or scale-down of policy.

TABLE 2: Workshops and Dissemination

Workshops and Dissemination

Target FY17 Actual Completed

Workshops (capacity building, knowledge sharing, program follow-up)

4 5 +

Number of people trained in i2i workshops or by i2i-supported teams

700 1944 +

Number of organizations 50 414 +

Dissemination events to policymakers 25 128 +

World Bank seminars 30 51 +


It is worth noting that the i2i IEs included in this

survey are at different stages of their lifecycles.

Some of the policy indicators reported are highly

dependent on where in the cycle they are located.

For instance, only IEs that already have results are

asked questions related to whether the results were

discussed with the client or did the results motivate

scale-up/scale-down of a policy. We divided the IEs

according to their lifecycle into three phases: Phase

1 refers to the beginning of the IE, before implemen-

tation starts; Phase 2 refers to the time between

implementation of the intervention and before the

IE results are available; and Phase 3 refers to after

IE results are available.

Capacity Building and Dissemination Events

The following set of indicators aims to capture

the extent to which the IEs assisted clients or oth-

er counterparts in building capacity for policies or

programs, or for client staff in general. We present

three measures of engagement with the client to

develop skills or feed evidence into policy:

n Trainings provided: Over 660 people were

trained by IE teams on general monitoring, data

analysis, and other topics. This took place across

30 different training events.

n Discussion of baseline results and final re-sults with clients: About 69 percent of baseline

results were discussed with clients. Further, 92

percent of IE teams with final results discussed

them with clients. This represents, respectively,

1641 and 1515 people over 44 and 32 events.

n Task force type meetings: About 29 high-level

meetings were organized to align counterparts

and broader sets of stakeholders.

n Presentation to non-clients: IE results were

presented 52 times to non-clients, which rep-

resents a total of 2060 people.

Quality of Data

In this set of indicators, we aim to measure whether

the IE played a role in influencing the quality of the

data collected and used. We include three indica-

FIGURE 21: i2i IEs Add Value throughout the Project Cycle

Inform policy design75%

Guide mid-coursecorrections

61%

Inform adoptionand Scale-up

59%

FIGURE 22: Number of People Trained in i2i IEs FIGURE 23: Number of Events in i2i IEs

Capacity BuildingNumber of people trained in i2i IEs

Data Training

Other Training

211

455

Data Training

Other Training

Capacity BuildingNumber of events in i2i IEs

14

20


tors related to M&E and generation of data across

the IE, which in many occasions results in the first

comprehensive data on the topic in the country or

region where the IE takes place.

n Improved counterpart M&E*: Over 57 percent

of teams of IEs after concept note report that

the IE data requirements led to improvements in

monitoring and evaluation (M&E), data collection,

and/or reporting activities of the counterpart(s).

n High-quality baseline survey**: About 88

percent of i2i IEs provided high-quality baseline

surveys (including covariates with sufficient

sample sizes and representative of policy-

affected populations), thus creating evidence for

policymaking even before the IEs start.

n High-quality follow-up survey***: About 96

percent of i2i IEs provided high-quality follow-

up surveys (including covariates with sufficient

sample sizes and representative of policy-

affected populations).

Quality of Policy Decisions

This set of indicators aims at capturing whether

the impact evaluation has influenced policy deci-

sions in several ways, including at the beginning

through program design and later as the IE pro-

duces evidence through new data or IE results. We

present the following four indicators:

n Rationalized policy design*: About 75 percent

of IE teams report that the IE improved design

based on a clear understanding of the underly-

ing theory of change (causal links between the

intervention components and the outcomes) and

highlighted areas of uncertainty and critical as-

sumptions.

n Influenced others**: 45 percent of i2i IEs in-

fluenced the design or implementation of other

projects outside of the IE itself.

n Baseline informed policy design/implementa-tion**: Around 69 percent of IE teams report

that baseline data was used by governments and

other stakeholders to stimulate policy dialogue

and/or help identify problems and solutions.

FIGURE 26: Improved Data Quality (Percentage of i2i IEs)

High QualityFollow-up Survey** 96%

ImprovedCounterpart M&E*

High QualityBaseline Survey**

57%

88%

* : applicable after Concept Note review** : applicable after baseline results*** : applicable after final results

FIGURE 24: Number of Participants in i2i IEs FIGURE 25: Number of Events in i2i IEs

Presentation tonon-clients 2060

1641

1515

496

Presentation of baselineresults

Presentation ofIE results

Task Force type meeting

Dissemination EventsNumber of participants in i2i IEs

52

44

32

29

Presentation tonon-clients

Presentation of baselineresults

Presentation ofIE results

Task Force type meeting

Dissemination EventsNumber of events in i2i IEs


FIGURE 27: Policy Indicators by Phase (Percentage of i2i IEs)

Rationalized design*

Baseline informeddesign**

Informed Adoption**

Scale-up or Down***

75%

69%

32%

59%

Phase 1 Phase 2 Phase 3

* : applicable after Concept Note review** : applicable after baseline results are available and discussed*** : applicable after final results are available and discussed

n Government or other stakeholders adopted causal mechanism(s) based on IE results***:

32 percent of IE teams report that IE evidence

from experimental testing of alternative mecha-

nisms was used by governments or other stake-

holders to determine the most effective program

alternatives or to inform policy decisions.

n IE results were used to motivate scale-up/scale-down of policy at national level***: Over

half of IE teams in Phase 4 report that the IE

produced evidence of sufficient (or insufficient)

effectiveness of the intervention in achieving de-

sired outcomes and were used by governments

and/or other agencies/stakeholders to motivate

scale-up (scale-down) of policy.


5. How We Contribute to Development Issues

efficiency due to resource misallocation and/or

behavioral biases.

In this context, technology change encompasses any

shock in the production process that leads to higher

output, given the inputs available. That shock could

be caused by, for instance, better use of land, bet-

ter trained employees, better managerial practices,

reduction in red-tape costs, and change in organi-

zational culture. Technology change and, thus, pro-

ductivity growth goes hand in hand with technology

adoption. Shifts in the production frontier assume

factors of production are already used optimally.

The reality shows that this is more the exception

than the rule. DIME’s portfolio reflects that.

DIME’s Approach

The DIME team uses rigorous evaluation methods

to test different policies and interventions aimed

at increasing productivity and growth. Even though

several impact evaluations in DIME operate at a

micro level, the close partnership with both oper-

ational teams at the World Bank and government

counterparts help with scaling up good practices.

DIME’s endeavor then plays a crucial role in build-

ing knowledge, improving program designs inside

and outside the Bank, and the quality of policy

recommendations.

Currently, there are 31 IEs in 26 different countries

that are looking at issues related to productivity

growth. $1.7 billion has been allocated to this agen-

da through World Bank lending operations and $20

million has funded IE research.

5.1 Growth

Context Productivity is the key driving factor for long-

term and sustainable growth. The empirical re-

search, specifically at macro level, has mostly

emphasized the determinants of productivity

growth. Empirical research has mapped innova-

tion, human capital, and institutions as some of

the key determinants of the (total factor) pro-

ductivity (Syverson, 2011). However, much less

is known about how to increase productivity. In

other words, how to improve skills of the working

force? What reforms and regulations enhance

business environment? And, what incentives are

needed to spur innovation? The how to design ef-

fective policies question is key, given that many

developing countries struggle with low levels of

labor and total factor productivity (The World

Bank Enterprise Surveys, 1/2013).

In reality, to design effective interventions that in-

crease productivity, one needs to (1) identify ways

to increase efficiency of factors of production that

were idly used or misallocated and (2) find alter-

native combinations of factors of production that

result in higher growth potential.

Productivity growth can occur via both efficien-

cy gains (‘catch up’ effect)–changes in the pro-

duction process so as to help firms move closer

to the efficient production frontier–and shift in

the production frontier. In many cases, adoption

of new technologies is required to overcome in-


The current portfolio is structured in way that al-

lows the DIME teams to test different strategies

aimed at increasing productivity in both rural and

urban settings and among small and medium-sized

firms. The current DIME portfolio accommodates

these two sets of approaches dealing with produc-

tivity and growth.

For instance, in Bangladesh, Malawi and Mozam-

bique we are testing the impact of agricultural ex-

tension to leverage gender and social dimensions of

communication to disseminate a new technology.

In Brazil, two IEs are testing different ways of im-

proving management practices of small and medi-

um-sized firms. In one experiment (Sebrae-Parana),

small firms interacting on an electronic platform

will be randomly assigned to receive feedback in-

formation on their business practices. We will see

how they perform, compared to their peers, in the

adoption of best practices.

In the other Brazilian IE (Banco do Nordeste), me-

dium-sized firms randomly assigned to receive

feedback information, will be randomly split to

receive intensive on-the-job training and moni-

toring visits from consultants for six months to

help them adopt best business practices. Also in

Brazil, we tested the impact of a large finan-

cial literacy pilot program on knowledge and

adoption of improved financial decisions by high

school students. We found positive effects that

were magnified when parents were also provided

some training.

Other IEs in the portfolio are expected to affect pro-

ductivity through both efficiency gains and change

in the frontier. For example, in the Georgia IE, we are

exploring the rollout of broadband internet across

the country to test the impact of the broadband

expansion on firm performances, and whether an

intervention combining training on e-commerce

and demand shocks on first online orders increase

firms’ access to markets.

In Kenya, Mozambique, Nepal, and Rwanda we are

testing the impact of different irrigation interven-

tions on returns to on-farm investments and land

productivity. The interventions per se will also re-

duce exposure of small farmers to climate shocks

and thus incentivize investments in riskier (higher

returns) projects.

5.2 Shared ProsperityDIME’s research contributes to the theme of shared

prosperity in several distinct ways. Shared pros-

perity is a key goal of the World Bank, which has

twin objectives of ending poverty and raising the

incomes of the bottom 40 percent. The rationale

for this is based on the idea that we care not just

about the mean of a distribution, or the percent

below a certain threshold (such as the poverty line),

but also about the distribution across the popula-

tion. A given mean level of income can be consistent

with divergent levels of poverty depending on the

distribution of income within a society. Moreover,

issues of inequality have reemerged on the intellec-

tual agenda in recent years in both developing and

developed societies (Piketty 2014, Milanovic 2016).

In the development sphere, shared growth is now

seen as critical, not just for poverty reduction, but

also for the stability and legitimacy of institutions.

And inequalities are persistent, not just across in-

The DIME team uses rigorous evaluation methods

to test different policies and interventions aimed at increasing productivity and

growth.


come groups, but between marginalized groups,

castes, ethnicities and across gender.

DIME’s contribution to this research agenda can be

organized conceptually around two fundamental re-

search questions:

1. Pervasive market failure suggests opportunities to

improve efficiency AND promote equity–but which

interventions work?

2. When are the objectives of promoting equity and

efficiency/growth complements and when are they

substitutes?

We can then further break this down into a series

of sub-research questions:

n Sub-research question 1: How can we reduce

physical (transportation) and soft (information,

networks) barriers for household and firms to

overcome social and spatial inequalities?

n Sub-research question 2: How can government

services be targeted to sustainably expand

access to marginalized populations?

n Sub-research question 3: How do social norms

contribute to gender inequality and what are the

growth and equity opportunities in overcoming

these inequalities?

Theme 1: Connecting People and Firms to MarketsThe importance of physical linkages to markets is

a central issue in development economics. Recent

contributions (Wantchekon and Stansig 2015)

have underlined the magnitude of this in develop-

ing countries. They estimate that transport costs

are the main drivers of poverty in Africa. Beyond

physical barriers, poor households and firms are

isolated from relevant information and face large

entry barriers to entrenched networks, exacerbating

inequality. An example of a policy response to this,

being evaluated by DIME, is the question of rural

road rehabilitation. While rural road projects are ex-

tremely common, there is relatively little evidence

about the returns to rural road construction. This

is at least in part because purposive targeting is

a major component of road project design, making

it difficult to construct a plausible counterfactual

scenario or control group.

DIME has two impact evaluations that will enable

progress on this critical issue. The first is the rural

feeder roads, transport costs, and local welfare in

Rwanda, which will use an Interrupted time series

identification strategy to estimate the impact of

road rehabilitation on import and export prices and

goods availability, land value, migration. The second

is a randomized rural road rehabilitation in Peru,

which may be the first experimental estimates of

the impact of rural roads. It will focus on income/

consumption gains and the use of public services

and gender-specific impacts.

Beyond rural roads, other projects in this area focus

on a) connecting firms to improved supply chains

(South Africa), b) connecting youth to job oppor-

tunities (Togo, Mauritania), c) connecting farmers

to markets (Kenya, other agricultural IEs), and d)

connecting workers in (urban) peripheries to urban

cores (Tanzania, Kenya, Colombia, Brazil, China, Ni-

geria, Nicaragua, Ethiopia).

Theme 2: Access to Government Infrastructure and ServicesA second theme relates to access to government

infrastructure and services. Most current evidence

focuses on the impacts of service provision (water,

sanitation, electricity, etc.), but there is relative-

ly limited focus on efficiency/equity issues. At the

same time, huge inefficiencies exist (for example,

$89 billion worth of electricity and 32.6 trillion liters

of water lost each year to theft and low-quality in-

frastructure). Private delivery of basic services may

be more efficient than public delivery (for example,

Galiani et al. 2005), but this raises equity concerns.

The marginal costs of expanding services to urban

and rural poor is often higher than willingness to


pay (WTP), and utilities face fiscal pressure and un-

certainty. Even when services are delivered to poor

areas, the transitory nature of urban slums means

that this may simply result in crowding-in higher

income populations, rather than providing targeted

support to the most needy. Therefore two key poli-

cy questions are: (i) how can we effectively expand

services to the poor and is public intervention jus-

tified?; and (ii), even when services are expanded

to poor areas, are the intended recipients the ones

who benefit?

Ongoing DIME projects in this area provide sub-

sidies to promote expansion of energy (Kenya,

Argentina), broadband (Georgia, Mauritania),

and water/sanitation (Kenya) services. Varying

subsidy levels allows each study to trace out the

demand curve for each service and the result-

ing benefits that accrue through access to the

service. This provides the information needed to

estimate the implied weighting of benefits/equal-

ity needed to justify intervention. Other work in

Kenyan slums takes this a step further. It mea-

sures changes in rent as well as in and out-migra-

tion resulting from provision of sewerage connec-

tions to households to estimate the effect of the

intervention on housing markets and resulting

gentrification.

Theme 3: Gender Equality and Social Norms A final area of focus is gender inequality and so-

cial norms. In this realm, taste-based discrimina-

tion and social norms limit the participation and

productivity of marginalized groups (gender, race,

ultra-poor, etc.). There is strong underlying theory

and empirical evidence across multiple sectors that

this leads to both inefficient and inequitable out-

comes (for example, Sen 1992, 2001; World Bank,

2015). DIME IEs focus particularly on gender and

ask: Where and how big are the market failures result-

ing from discrimination and what interventions work to

reduce them?

What is the impact of gender discrimination in

agricultural settings? One impact evaluation of

extension services in Malawi (Bin Yishay, Jones,

Kondylis, and Mobarak, 2016) shows that women

adopt new technologies better and retain knowl-

edge as well as (or better than) men, but are less

trusted as teachers. A second IE focuses on which

interventions work to reduce harassment of women

on public transport, in the context of women-only

train cars in Rio. This study estimates a) compli-

ance with the women-only law; and b) willingness to

pay for female-only cars and finds that willingness

to pay is close to zero, but that this is strongly (and

positively) associated with enforcement of the law.

The ongoing study also has planned experiments on

norm-shifting and efforts to increase compliance.

Other projects in this area focus on the impact of

providing legal aid for poor women (Jordan); female

leadership of SMEs (Haiti); transport impact on

women’s access to health and education services

(Peru); and the impact of factory jobs on young

women’s well-being (Ethiopia).

5.3 Risk and VulnerabilityIn addition to targeting poverty, we must con-

sider the extent to which the circumstances of

individuals, households, communities, and coun-

tries fluctuate over time. Indeed, variation in the

face of an otherwise sustainable, even comfort-

able livelihood can mean the difference between

well-being and destitution. Dercon and Shap-

iro (2007) estimate that 30–60 percent of the

world’s poor live in transitory poverty, meaning

that they may move in and out of poverty ac-

cording to fluctuations in their situation. For the

chronically poor, unmanaged negative shocks

have long-term negative consequences (Dercon

and Hoddinott, 2005).1

1 Deaton and Hoddinott (2004) show that adults exposed to drought in childhood experience permanent consequences in terms of stunting as well as educational and labor market outcomes.


Sources of Risk Risk can come from a multitude of sources, some

unique to individuals and households (idiosyncrat-

ic) and others common to a whole area (covariate).

Negative shocks are likely to be most detrimental

to those living just within their means, since such

individuals can afford to lose the least. The 2014

World Development Report ‘Risk and Opportunity’

finds that countries with high incidences of pover-

ty are some of the least prepared to deal with the

risks that threaten so many livelihoods: Sub-Sa-

haran African countries are the least prepared to

manage risk, followed by southern and southeast

Asia and Latin America. For this reason, research

that seeks to improve risk mitigation and man-

agement in the developing world is vital for pover-

ty reduction.

Constraints to Risk ManagementSome barriers to risk mitigation and management

are within the control of individuals, while others

relate to broader issues such as missing markets,

absent institutions, and social or economic ex-

ternalities. In this section, we focus more on the

former issue: helping individuals to mitigate and

manage their own risks over time (see sections 5.4

on Governance and Accountability and section 5.5

on Global Public Goods and Externalities for de-

scriptions of work that seeks to tackle the latter

constraints). In particular, risk and vulnerability re-

search within DIME seeks to alleviate the following

three constraints:

n Cognitive and behavioral bias/failure: How

information asymmetries and behavioral biases

can be reduced/corrected for.

n Resource constraints: How to best unlock the

productive potential of poor and vulnerable

groups.

n Collective action failures: How to resolve

incentive and coordination failures between

multiple individuals.

Fifteen economists at DIME currently contribute

research to issues of risk and vulnerability through

36 impact evaluations in 23 countries across Afri-

ca, Asia, and Latin America. Indeed, the distribution

of DIME’s risk and vulnerability research is con-

FIGURE 28: Global Risk Preparedness

Source: WDR 2014, ‘Risk and Opportunity’.


centrated in the geographies with lowest risk pre-

paredness, as diagnosed by the 2014 World Devel-

opment Report. The risk and vulnerability portfolio

represents around $33 million of research across

projects of a total value of $2.1 billion.

1. Informational Asymmetries, Behavioral and Cognitive BiasesInformational asymmetries lead to sub-optimal risk

management, since individuals must either base

their decisions on expected returns or expected

constraints (depending on the type of information

that is limited). In the event that expectations

deviate greatly from reality, or when the possible

options are very variable, this can lead to sub-opti-

mal individual decisions and socially inefficient out-

comes. On the other hand, individuals may also act

inefficiently despite seemingly having all the infor-

mation they need to choose the most efficient ac-

tion. This may be a result of poor interpretation of

available information or of a behavioral bias (such

as time inconsistent preferences), making the most

efficient solution less appealing at the time of deci-

sion-making (Duflo, 2006).

Interventions that seek to combat information

asymmetries or mitigate behavioral and cognitive

biases can help mitigate the risks faced by vulner-

able populations. DIME research targets both these

areas to answer questions such as: “How can we

harness mass media to provide information to em-

power individuals to make better decisions relating

to sexual behavior, human rights and democracy

promotion, and citizens’ rights to transitional jus-

tice in post-conflict contexts?” The program also

evaluates the effectiveness of interventions to mit-

igate behavioral biases such as time inconsistency

and present biases, asking questions such as: “How

expanding choice sets, such as by encouraging at-

risk youth to invest time in marketable skills or

through soft skills and cognitive behavioral therapy,

can reduce the propensity of such youth to partic-

ipate in crime?”

2. Resource ConstraintsNegative shocks can push households into poverty

traps (Sachs, 2005). For example, in the case where

an indivisible investment is needed to reach a cer-

tain level of productive capacity, households with

FIGURE 29: Distribution of Risk and Vulnerability Research: DIME

Source: DIME Internal.


resources below this critical amount are unable

to transition from poverty. Resource injections to

move individuals over critical thresholds and social

safety-net schemes to prevent falling below these

are well-known approaches to resolving poverty

traps, though whether they do so in a sustainable

manner and how best to design and target such

schemes remain open questions. DIME’s work in

this area considers both approaches, asking: “How

to break poverty traps for the ultra-poor in the con-

text of high unemployment, to enable households

to support their own trajectory from poverty?” and:

“How best to design social safety-net programs to

prevent poverty traps, integrating productive safety

nets to foster long-run sustainability”.

3. Coordination Failures and Collective Action Problems

Coordination failures and collective action problems

may occur in contexts where the external conse-

quences of one’s actions do not factor in individu-

al decision-making or in the case where the action

chosen by other individuals is uncertain. For exam-

ple, in the case of disease prevention, an individual

deciding whether or not to have a particular vacci-

nation may not take into account the possible con-

tagious nature of an illness in the event of suffering

from it. By ignoring the effects their illness could

have on others, individuals underestimate the cost

of no vaccination and fewer are vaccinated than

might be socially optimal.

DIME’s risk and vulnerability research both evalu-

ates attempts to convey the marginal social cost

of actions with negative externalities and to help

individuals collaborate to reduce risks in the polit-

ical or social sphere. In the context of externali-

ties, research considers: “How informal institutions,

such as extended family networks, can be mobi-

lized to help individuals internalize the social cost

of disease?” When it comes to collective action

failures, DIME research asks, “To what extent fa-

cilitating information sharing, such as using public

recognition as a non-monetary incentive, can help

individuals to cooperate to reduce risk?”

5.4 Governance and AccountabilityA society’s institutions, or rules of the game, are

critical determinants of its development outcomes.

The design of a society’s governance and account-

ability structures leads to huge differences in devel-

opment.2 As such, large-scale development efforts

require an improved understanding of institutions

and the policy interventions that can change them.

Governance and accountability have taken center

stage in the search for institutions that underpin

development. For example, property rights have

frequently been argued as being the fundamen-

tal building block to an effective economy. Such

institutions span the public and private sectors,

and are present at each level of socio-economic

organizations.

At the theoretical level, a rich body of micro-the-

ory forged around contracting and principal-agent

models provides a framework for understanding

governance relationships. In its simplest formula-

tion, the principal-agent problem is about designing

contracts that can induce an agent (an employee, a

child, a subordinate division, or an aid contractor) to

perform a task as required by a principal. This could

be within a firm or government office or between a

community and their health workers. It is relatively

straightforward to design effective contracts when

effort and/or outcome are observable and there is

no uncertainty on how effort is transformed into

outcomes. In real world applications, problems arise

when neither effort nor outcomes are directly ob-

servable. Theoretical contributions have analyzed

these trickier situations at great length, including

2 Seminal contributions on this are those of North, Besley, and Person, and Acemoglu and Robinson.


the seminal work of recent Nobel Prize winners

Bengt Holmstrom and Oliver Hart.

However, micro-empirical evidence has not kept up

with theory. The evidence base on many governance

institutions is limited. The approach of the DIME re-

search program in Governance and Accountability

is to start from the classical principal-agent frame-

work to produce evidence on the impact of gover-

nance and accountability mechanisms that are key

to development outcomes. As these mechanisms can

allow self-interested agents and principals to reach

a more cooperative equilibrium, our approach, ulti-

mately, also explores novel areas of collective action.

In particular, we focus on information, regulation,

monitoring, and enforcement mechanisms. Also, we

study governance and accountability problems with

an approach that recognizes their systemic nature.

Finally, we study the principal-agent model through

multiple lenses, which span theory, measurement,

and intervention.

Information, Regulation, Monitoring, and Enforcement

These mechanisms act as commonalities across

our research program. We study them in a variety

of contexts to assess whether they can be effective

in addressing one (or more than one) dimension of

the principal-agent problem. Shifting societal rules

and norms can affect the observability of effort

and outcomes and the type of agents that enter

into contracts with the public sector. Regulatory

reforms—that introduce new procurement process-

es (in Bangladesh, Brazil, Colombia, and The Philip-

pines) and introduce minimum patient and building

safety (in Kenya and Peru)—are studied to ascer-

tain whether they change the outcome (in terms

of quality and efficiency of the service delivered) of

the principal-agent game.

Providing information to key actors is predicted

to reduce information asymmetry, which is one of

the main issues within the principal-agent frame-

work. The research program is testing the impact of

providing information on court efficiency to judges,

registrars, clerks, and users (in Azerbaijan, Kenya,

and Senegal) as well as information on public works

performance to politicians and senior and junior bu-

reaucrats (in Pakistan). More direct mechanisms to

increase observability of effort and outcomes work

through monitoring and enforcement devices. Sev-

eral ways of tracking provider performance for bot-

tom-up accountability are being tested as part of the

research program (in Angola, Burkina Faso, Cambo-

dia, Democratic Republic of the Congo, and Nigeria).

Systemic-Wide Analysis Nature of Governance and Accountability

The systemic nature of collection action problems

requires a system-wide approach to analyze them.

Predicted solutions from theory (increase informa-

tion or monitoring) typically apply only to specific

elements of the system. There is growing evidence

on these elements of the system (for example, au-

dits and monitoring, rewards and warnings) but still

scarce evidence on the system and market equilib-

rium (price, quality, supply) effects of interventions.

A system-wide analysis is understanding and

aligning incentives in markets for road safety and

patient safety in Kenya. In particular, the Kenya

research program is studying, within the same

market, the impact of shifting formal and infor-

mal rules (regulation and enforcement of patient

safety and road safety), of providing information

(on clinic performance to clinic managers and pa-

tients and on driver safety to drivers, transport

company owners and riders), and of enabling bot-

tom-up and top-down accountability devices for

clinics and patients, and bus owners and drivers.

In addition to partial equilibrium impacts on ser-

vice provider performance, the research program

will study market outcomes in terms of prices and

supply indicators.


Theory, Measurement, and InterventionOur research program explores the principal-agent

model through multiple lenses. These include: (i)

theory, which provides frontier predictions that can

be tested by our research program; (ii) a focus on

measurement to improve observability of efforts/

actions and performance and provide data for

monitoring/diagnostics; and (iii) an emphasis in de-

signing interventions/treatments that are embed-

ded in theory and test mechanisms.

We test theory predictions in Tanzania as we

study the role of characteristics of organization

structures on solving incomplete contracting in

infrastructure maintenance and the impact of

performance-based incentives to improve sub-

national service delivery under weak monitoring

capacity. Specific treatments underlying these

IEs stem from the same theoretical framework.

They include co-production organization in in-

frastructure maintenance (to solve incomplete

contracts) and performance based-grants to

subnational governments (to solve observabil-

ity of agent outcomes). On the measurement

side, our research program is producing globally

comparable micro-data on civil servants, devel-

oping systems to track subnational government

capacity, and court measurement systems to

track judiciary performance (Kenya, Senegal,

and Tanzania).

5.5 Global Public Good and ExternalitiesEconomies often fail to reach their full productive

potential because of coordination failures. In the

field of policy analysis, the Samuelson Rule states

that all production and consumption decisions

should be made so that the sum of benefits from

that good experienced by all people who consume

it equals the total cost borne by all actors who ex-

perience that good. Unfortunately, the optimal out-

come is often not achieved when the good in ques-

tion has the features of a public good, a common

good, or when either production or consumption of

a good creates externalities.

Public goods are goods whose consumption is

both non-rival and non-excludable. These features

create unique challenges for pricing and produc-

tion incentives, because those who pay the cost of

producing the good do not experience the benefit.

A classic example of a public good is broadcast

media. Once media such as an educational TV pro-

gram is produced by a government or foundation,

it is difficult to prevent other agencies from using

it (especially when copyrights are difficult to en-

force, which is the case in many developing coun-

tries) and anyone with a TV from viewing it for

free. But in order to incentivize the production of

high-quality educational TV content, co-financing

by other agencies or viewership fees need to be

collected.

Common goods are those whose consumption is

rival but non-excludable. An example of a common

good is water from an unregulated river. Any given

farmer can freely take water from a river to irrigate

crops, but if she does not take into account how

much her use affects other’s availability, water may

be overused.

Economies often fail to reach their full productive

potential because of coordination failures.


Finally, goods with externalities are those where

the full benefits or the costs of consuming a good

are not fully borne by those who choose to consume

the good. Externalities may be positive or negative.

An individual’s decision to take actions to prevent

HIV infection reduces the risk of passing on infec-

tion to everyone around them, a positive external-

ity. Conversely, a decision to engage in criminal vi-

olence imposes negative externalities by diverting

resources toward policing and creating incentives

for vulnerable people to take costly actions to avoid

victimization.

Goods with the features of public goods, common

goods, or externalities appear in many contexts.

The coordination failures that cause them to be

produced and consumed inefficiently underlie many

of the common constraints that inhibit develop-

ment. Many contexts where non-rival goods or ex-

ternalities arise create justification for governance,

and can be addressed though properly structuring

the incentives of individual agents to align with so-

cial welfare as described in the previous sub-sec-

tion. But solutions to these problems can also lie in

addressing the constraints to market structures or

human behavior that interfere with efficient provi-

sion of goods. DIME is working to uncover the solu-

tions that best tackle these constraints.

Information as a Global Public GoodInformation is a unique type of public good that

has large positive, non-rival benefits, but is often

under-produced because producers are not able to

capture returns experienced by users and consum-

ers of information goods. Information goods may

be under supplied because of high production costs

and free-riding issues in their consumption, even

in the public sector. Examples are mobile apps or

edutainment programming, which can be costly to

produce. For these goods public investments are

difficult to be justified by any one government or

development agency without co-financing of other

agencies that may also benefit from these infor-

mation goods. As a result, despite the potential of

information goods in reaching millions of individuals

at low marginal costs—through mobile, TV, radio or

other communication and media outlets—they re-

main undersupplied.

In this theme, the impact evaluations address im-

portant knowledge gaps on effective ways to use

mobile technologies that monitor and promote effi-

cient use of resources and the cost of under-mon-

itoring; the relative effectiveness of different mass

media outlets (for example, TV, radio, print) to pro-

mote adoption of new technologies; and the role

that entertainment-education can play in address-

ing global challenges and epidemics, such as HIV/

AIDS and gender-based violence.

The knowledge agenda aims to unpack the causal

mechanisms from media exposure to individual and

community-level impacts on knowledge, attitudes,

and behaviors. In one example, an evaluation in Mo-

zambique measures the value of monitoring water

use within and across irrigation schemes to improve

efficiency. An evaluation in Nigeria has demonstrat-

ed the potential for an MTV-produced soap opera

to change social norms and behaviors. Section 6.8

provides greater details about DIME’s edutainment

research program.

Finally, goods with externalities are those where the full benefits or the costs of consuming a good are not

fully borne by those who choose to consume the good. Externalities may be positive

or negative.


Managing the CommonsRecognizing that many crucial economic goods are

common pool resources, DIME evaluations demon-

strate and test the strategies for creating and

maintaining institutions that manage common pool

resources.

DIME’s work on this area seeks to test and demon-

strate which institutional structures and incentives

are effective in managing common goods. There are

many examples where individuals either over- or

under-perform an action relative to the social op-

timum because they do not take into account the

fact that their own use reduces the available re-

sources for others. DIME’s evaluations focus first on

the question of identifying which resources are sub-

ject to this problem. For example, for managing for-

ests in Burkina Faso, by varying institutional struc-

tures across communities, we measure how much

deforestation is due to failure to take account of

individual costs for others when making decisions

about collecting firewood.

In contexts where a given resource is known to be

subject to the problems arising from managing

common pool resources, such as irrigation, DIME

tests the question of why existing institutions es-

tablished to manage the problem fail. For example,

in Rwanda a project varies the user fee paid by ir-

rigation users to assess whether these fees are set

too high for sustainable participation in water-user

groups. Finally, DIME works to design and test in-

novations to institutions to manage the commons

to improve these institutions. For example, a project

in the Democratic Republic of the Congo is test-

ing accountability mechanisms for service providers

working in conflict-affected areas to assess wheth-

er these reforms change the private incentives to

participate in conflict.

Private Incentives for Adopting Pro-Social Activities and Conservation Technologies DIME research focuses on uncovering and mitigat-

ing the behavioral constraints that prevent people

from adopting technologies that would be privately

profitable to them, but also have non-rival attributes

that create large public externalities. This situation

is particularly apparent in the context of production

or consumption decisions that may be just prof-

itable or incentive-compatible for an individual or

firm to adopt, but also entail large public benefits.

For example, environmental conservation. Govern-

ments may need to provide incentives to individuals

or communities to adopt conservation technologies

with large social benefits. This can be the case even

when technologies may be profitable, from market

failures (for example imperfect information about

the technologies’ benefits, unclear property rights)

to financial constraints faced by households or

firms to psychological limitations that are increas-

ingly being studied by behavioral economists. For ex-

ample, present bias may prevent people from prop-

erly weighing the long-term benefits or savings of

adopting new technologies, even in the cases where

they can afford it and would be better off in the long

term by adopting these technologies.

The knowledge agenda in this theme covers opera-

tional research questions such as optimal subsidy

schemes for purchasing new technologies, such as

solar lamps in Argentina; and selecting the most

cost-effective methods to diffuse new information,

such as new cultivation strategies among farm-

ers in Bangladesh. A key question in this domain

is whether the policies that improve take-up of

pro-social technologies in the short term will lead

to sustained use and maintenance in the long term.

This question is being explored in Mexico and Ar-

gentina, where upkeep of pro-environment actions

is being assessed after the initial policies encourag-

ing take-up are withdrawn.


6. Economies of Scale in Learning

6.1 Fragility, Conflict, and Violence

Context Around 2 billion people live in countries affected by

fragility, conflict, and violence. Poverty rates are 20

percent higher in countries affected by repeated cy-

cles of violence. By 2030, an estimated 46 percent

of the world’s poor will live in areas characterized

as fragile or conflict-affected. While trends show

that poverty is declining across much of the world,

countries affected by conflict are falling behind.

Development challenges in fragile, conflict, and vio-

lence (FCV) settings transcend national boundaries

through the displacement of populations, spread

of disease, reduced trade, and increased organized

crime and terrorism. Further, high levels of fragility

and violence exist in countries otherwise considered

relatively stable. Many countries in Latin America

and the Caribbean, for example, suffer levels of vio-

lence comparable to those in the most conflict-rav-

aged states.

The international community is committed to as-

sisting communities emerge from conflict, sustain

peace, and resume growth. The World Bank, bilat-

eral, and other multi-lateral donors invest billions

of dollars a year to help achieve peace and build

states. The evidence base for designing such pro-

grams is, however, sparse, especially with regards

to rigorous evaluations aiming to identify what

works, and how, to reduce fragility, conflict, and vi-

olence. This knowledge vacuum impedes our ability

to design effective interventions to promote pov-

erty reduction and welfare improvement in FCV

settings.

Rigorous evaluation of policies targeting FCV issues

is therefore of paramount importance, all the more

so as the volume of resources from the World Bank

and other development partners towards such

settings increases (under IDA18, for example, the

World Bank is set to significantly increase financ-

ing for FCV-related issues). Not only is developing

this evidence a priority, but experience to date

shows that, even with the amplified challenges of

working in FCV environments, rigorous evaluation in

such settings is possible. IEs have been conducted

in diverse FCV countries such as Afghanistan, DRC,

Colombia, Cote d’Ivoire, Sri Lanka, Rwanda, north-

ern Nigeria, and Liberia.

FCV Impact-Evaluation Program

In March 2014, DIME and partners inside and out-

side of the Bank launched the Evidence for Peace

(E4P) program. Its overall goal is to assess evidence

gaps in FCV responses and generate improved

knowledge about how to best support FCV clients

to deliver the results so critically needed for citizens

to gain confidence in the path out of conflict.3 E4P

was the first IE program to benefit from a launch

workshop under i2i, followed by the first open call

3 Internal World Bank partners include the Fragility, Conflict, and Violence (FCV) Group-CCSA and the Latin America’s Citizen Security Team. External partners include the Initiative for Impact Evaluation (3ie) and Innovations for Poverty Action (IPA). The original initiative included: (i) development of a scoping paper, based on an evidence ‘gap map’ that identifies the status of the evidence base to highlight priority questions for future research; (ii) design, implementation,


for proposals for impact evaluations addressing the

program’s four initial themes: jobs for resilience,

public-sector governance, urban crime and violence,

and gender-based violence.

Today, the program includes 36 IEs across 21 coun-

tries. The portfolio represents around $30 million

of research across projects of a total value of $2.1

billion. Further, a series of white papers synthesizing

the state of the evidence in each of the four target

themes and proposing priority IE research topics

will be completed and disseminated in 2017.

An important development during the last program

year has been the redefinition of the E4P focus

areas around broader development issues encom-

passing and building on the original four program

themes. Four key research areas have been defined:

(i) basic service delivery in weak states; (ii) job op-

portunities for at-risk youth; (iii) breaking poverty

traps and vulnerability; and (iv) the political econo-

my of post-conflict reconstruction. This redefinition

was conceived with a view to strengthen linkages

with other i2i areas and with World Bank Global

Practices, beyond the FCV group. Work under each

key research area is summarized below.

Basic Service Delivery in Weak States Strong institutions that provide quality services to

citizens are a necessary condition to support coun-

tries to move out of fragility, conflict, and violence.

Yet, these are the settings in which institutions are

likely to be the most eroded and dysfunctional. Here,

the program focuses on civil-service reforms as well

as the rebuilding of government capacity and ac-

countability systems, to improve our understanding

about what works to develop effective governance

structures where it is most needed and, perhaps,

the state is least capable.

Work towards service provision in weak states is

ongoing in Liberia, Zimbabwe, and the DRC. For ex-

ample, work in the DRC seeks to attract talented

individuals to work in the country’s civil service, in-

vestigating the effect of deployment outside one’s

home region on breaking the patronage systems

common in the government.

Improving Job Opportunities for At-Risk YouthYouth in FCV contexts are often left with few mar-

ketable skills and little opportunities to cultivate a

sustainable livelihood. They can become vulnerable

to involvement in conflict, illicit activities, or violent

crime. Work in this area investigates ways of break-

ing this cycle of poverty and violence through hard

and soft-skills training, psychosocial therapy, and

labor market-insertion programs.

Our program is currently seeking to expand the

opportunities for at-risk youth in Cote d’Ivoire,

Honduras, Liberia, and Nigeria. For example, work

in Honduras looks to break cycles of crime and vi-

olence through a temporary jobs program aiming

to provide at-risk youth with the hard and soft

skills needed to succeed in the labor market. A

focus on cognitive behavioral therapy as well as

traditional skills-based training aims to provide

recipients with the practical and emotional re-

sources to earn a sustained livelihood in a chal-

lenging setting.

Breaking Poverty Traps and Cycles of (Gendered) VulnerabilityIn FCV contexts, support for vulnerable groups is

often lacking and both their immediate and long-

run needs are overlooked and their productive ca-

pacity ignored. This can lead to a perpetuation of

poverty traps and cycles of vulnerability. Further,

evidence shows that poverty-induced vulnerabili-

and dissemination of impact evaluations funded within the Bank and through a new external funding window; (iii) development of a framework to improve the quality of analytical work on FCV and of tailored methodologies for evaluation/data collection in FCV settings; and (iv) hands-on training in impact evaluation and the creation of communities of practice for knowledge sharing.


ties tend to disproportionately affect women and

children. Our research work in this area builds on

the ultra-poor literature to understand the poten-

tial of interventions geared toward breaking pov-

erty traps and addressing systemic vulnerabilities.

It also considers the effectiveness of social safe-

ty-net programs to support such individuals as well

as big-push interventions, which seek to provide a

productive livelihood and exit from poverty traps in

the long run.

Research is currently ongoing in Afghanistan, Co-

moros, the DRC, Egypt, and Tunisia. In Comoros,

research examines the implications for expenditure

and intra-household resource allocation of assign-

ing cash-for-work safety-net schemes to women.

Our work measures differences in investment in

children, an area of central importance for breaking

cycles of poverty in fragile settings. It also investi-

gates the potential of norms-shifting and targeted

interventions to eradicate child labor and address

gender-based violence.

Political Economy of Post-Conflict Reconstruction

E4P research in this area focuses on understand-

ing the drivers and perpetuators of conflict and on

evaluating strategies designed to address these.

Postwar societies are often confronted with a wide

range of issues—including information asymmetries

between elites and masses, low levels of inter-per-

sonal coordination, social dislocations, and security

and mobility constraints—that prevent a rapid re-

turn to stable social and political orders. Some of

these are root causes of the conflict in the first

place or conflict drivers that sustain FCV cycles.

Work in post-conflict reconstruction currently fo-

cuses on Liberia, Sierra Leone, and the DRC. For

example, research in Liberia seeks to overcome

informational asymmetries and collective action

problems in rural areas by providing groups of wom-

en with access and a safe space to listen to unbi-

ased political radio broadcasts by United Nations

peacebuilders. Results show that overcoming these

barriers can increase female political participation

in many forms.

Going ForwardOver the past two years, the FCV IE program

has contributed to generating valuable knowledge

on what works, and why, to resolve challenges in

key FCV areas. In addition, we have undertaken a

stock-taking exercise of the existing evidence in the

program’s initial four target areas in an attempt to

contribute global public goods over and above indi-

vidual studies. The white papers produced from this

exercise, alongside practical knowledge generated

over the past years through ongoing IEs, will help

us define an ambitious, integrated approach to fu-

ture research, which will maximize the possibility for

linkages and broad lessons.

The program has now entered a new phase of con-

solidation and expansion focusing on at least four

priorities:

n Engaging stakeholders on findings from the white papers: As already noted, the FCV IE

program has produced four draft white papers,

one for each of the original thematic areas,

which summarize the current body of evidence

and policy implications and highlight future

research directions. In the next phase, we are

planning a series of events to discuss and finalize

these papers and engage with stakeholders

from the World Bank, governments, academic

institutions, and other development partners to

share the lessons learned, to promote broader

dissemination of findings, to explore further

research synergies, and to launch a new series

of IEs. This process will begin with a workshop

in FY17.

n Strengthening analytical and IE work of our Gender/GBV and FCV portfolio: Our work on

gender/GBV and FCV remains limited. To expand


on this, we will seek to deepen the gender impact

of existing projects and to draw comparisons

across settings by adding similar interventions

in a number of contexts. In particular, we will

leverage existing interventions in the area of

‘Breaking Poverty Traps and Vulnerability’ to

study the potential for deepening the effect

of temporary safety-net programs through

business grants to females to provide productive

livelihood opportunities in the long term. We

will also research how best to tackle norms

surrounding GBV in the context of these

programs.4

n Launching new IE window(s) in emerging priority areas: As well as deepening the impact

of existing projects in current thematic focus

areas, we seek to expand our work into new

FCV-relevant areas. We will create a working

group in order to establish a formal process of

identifying knowledge gaps in priority areas. We

will then seek new collaborations to broaden

our research in the most considered way

possible. For example, we plan to work with the

FCV Group and its partners on a new initiative

towards understanding and addressing issues

related to forced displacement, both for the

displaced themselves and for destination

communities.

n Expanding collaborations with DFID country programs in overlapping priority areas: Thus

far we have had successfully collaborated with

DFID to carry out joint IEs on priority country

programs that fit with the current thematic

priorities. (We have completed IEs on the

political economy of post-conflict reforms in the

DRC and have ongoing IEs in Nigeria focusing

on employment of marginalized youth.) We plan

to strengthen and expand such collaborations

and, depending on demand, identify new priority

areas for further collaboration on IE work.

6.2 Agriculture

Urgent Need for Evidence in AgricultureThe Sustainable Development Goals (SDGs) note

that agriculture is the single largest employer in

the world. Globally, 40 percent of the population

earns its income from agriculture. The SDGs urge

the international community to make the invest-

ments needed to double agricultural incomes of

small-scale food producers.5 Astonishingly, little

evidence exists to rigorously inform the invest-

ments needed to meet this urgent goal. For ex-

ample, a 2015 systematic review on the effects of

training, innovation, and technology on smallhold-

er productivity in Africa identified only 19 studies

that met the scientific standard to be included in

the review, making it impossible to assess which

interventions yield the highest returns.6 The gap

between urgent need for action and the evidence

available to inform such action is therefore great-

er in agriculture than in many other sectors in

development.

Agricultural development is crucial, not only for pov-

erty reduction but for many other SDGs as well.

Ending hunger and improving nutrition for the 13

percent of people in the developing world, who are

hungry, requires restructuring the agricultural val-

4 For instance, we have a new multi-country initiative looking to target women who graduate from the livelihood interventions we are evaluating with some additional capital (through an unconditional cash transfer), with the view to promoting entrepreneurship and enhancing sustained livelihoods. We are also working with internal and external partners to incorporate GBV-prevention pilots into large social-protection programs we are already evaluating in countries such as DRC, Egypt, and Tunisia. Because such programs are typically implemented in poor communities with a high incidence of GBV, they figure as promising GBV interventions. Additionally, we are working on partnerships with NGOs such as Promundo, International Rescue Committee (IRC), Search for Common Ground or Women for Women International (WfWI) that are conducting innovative gender-empowerment and/or GBV-prevention programing to develop and implement joint IEs in critical areas.

5 http://www.un.org/sustainabledevelopment/hunger/.6 http://www.campbellcollaboration.org/lib/project/310/.


ue-chain; from farmers who grow food to retailers

who sell it to consumers. As a sector contributing

both carbon emissions and capture and uniquely

susceptible to climate and extreme weather, agri-

cultural innovations can offer solutions to climate

change through both mitigation and adaptation.

DIME builds evidence on the innovations that best

address all of these challenges through its agricul-

ture portfolio.

Policy-Driven Evaluation Design

Many of DIME’s impact evaluations in agriculture

were launched following a workshop on Agriculture

Innovations held in June, 2014 in Kigali, Rwan-

da. Ahead of this event, the Africa Region of the

World Bank organized a high-level meeting of de-

cision-makers from ministries of finance and agri-

culture, researchers, and other policymakers to set

priorities for research. The June event then took the

resulting recommendations to a gathered set of

policymakers, project staff, and researchers to em-

bed research questions and designs into the project.

This model of involving policymakers from the ear-

liest stages of designing evaluations and building

the evaluation directly into projects ensures buy-in

from projects and immediate policy relevance of re-

search findings.

In November, 2016, DIME convened a conference on

Evidence for Agriculture to share findings from on-

going and completed evaluation in the agriculture

portfolio and identify emerging priorities for eval-

uations in the sector. This event engaged partici-

pants from fifteen institutions, including university

researchers, policymakers from governments and

multinationals, and donor agencies.

Active Impact Evaluations

The AADAPT portfolio includes more than 20 im-

pact evaluations in 12 countries across Africa,

South Asia, Latin America, and the Caribbean. The

evaluations are distributed across five knowledge

gaps identified as constraints to the design of ef-

fective agriculture policy. These topical areas of fo-

cus are highlighted in table 3.

Improving the State of Knowledge and ImplementationDIME produces rigorous evidence on under-studied

issues relevant to agricultural policy. This advances

knowledge that can be used to design policies to

improve productivity in the sector that provides the

largest source of income and jobs for the world’s

rural poor. DIME has already produced rigorous

research on how to adjust extension programs to

optimize knowledge diffusion, the relationship be-

tween land rights and technology adoption, and the

role of gender in learning about technology.

DIME’s model changes the way that agriculture

programs operate throughout every stage of the

impact evaluation, from establishing comprehen-

sive data-collection platforms to monitoring roads

in Rwanda, to changing the way that recipients

are selected for irrigation investments in Mozam-

bique, and providing conclusive evidence on the

most cost-effective arrangements for extension

programs in Malawi. DIME’s research from the ag-

riculture portfolio influences policy directly through

intensive interaction with partners from govern-

Many of DIME’s impact evaluations in agriculture were launched following a workshop on Agriculture

Innovations held in June, 2014 in Kigali, Rwanda.


ments and multinationals. Further, it often appears

in working paper and top journals in the field of

development economics.

Going ForwardFollowing the stock-taking and knowledge-sharing

event in November 2016, the AADAPT program will

launch the next round of impact evaluations to in-

fluence both agricultural policy and the agricultural

research agenda. This process will include four sets

of activities.

Consolidating evidence into policy-relevant mes-saging: Almost all evaluations have now conduct-

ed a baseline survey and about half of those have

recently completed follow-up surveys. Another 25

percent of evaluations is expected to graduate to

follow-up stage in the next year. As the program is

maturing, we fill focus on consolidating policy les-

sons and ensuring dissemination of those findings

within the local government, across the portfolio,

and the wider development community. In Janu-

ary, 2017, Florence Kondylis gave a Policy Research

Talk at the World Bank on the topic of Retarget-

ing Investments in Agriculture. She distilled the

learning from the first eight years of DIME’s work

demonstrating strategies for enhancing agricul-

tural productivity. This talk provides the basis for

engaging policymakers in the World Bank and be-

yond on frontier issues in agricultural research and

innovation.

Focusing on emerging priority areas: In consul-

tation with the Agricultural Global Practice, the

AADAPT team have identified areas where prac-

titioners within the Bank feel that additional fo-

cus is needed. One example of a new approach is

understanding complementary investments and

TABLE 3: DIME’s Work in Key Learning Areas

Knowledge Gap Example Evaluation QuestionActive IEs in this

Area

Commercialization: What are the public investments needed to ensure that farmers have access to markets and receive fair prices for their products?

Can matching grants for inputs and equipment combined with training on business planning improve competiveness in markets?

Haiti, Brazil, Liberia

Financial Constraints: How do financial barriers and institutional constraints prevent farmers from making profitable investments? What are the simple interventions that can overcome these constraints?

Can financial education tailored to farmers increase savings and increase investments in inputs?

Rwanda, Benin, Haiti, Uganda,

Rural Infrastructure: Are large infrastructure investments always profitable? Beyond construction, how can we ensure sustainability of investments by building effective users groups to manage the infrastructure?

Is there a productivity trade-off when prioritizing smallholder participation in irrigation programs?

Rwanda, Mozambique, Kenya, Nepal

Information: Are farmers aware of the productivity gains to be realized from adopting new technologies and methods? If not, what are the most efficient ways to help them learn about these opportunities?

Which approaches to crop demonstration lead to the highest adoption of improved crop varieties?

Bangladesh, Mozambique, Malawi, Rwanda, Nepal

Natural Resource Management:How can we encourage rural communities to manage and protect natural resources such as forests, clean water, and soil, while supporting livelihoods that rely on these resources?

What is the role of Payment for Environmental Services (PES) in Reducing Emissions from Deforestation and forest Degradation (REDD+) in Burkina Faso?

Ghana, Burkina Faso, Brazil


goals in agricultural programs, particularly those

related to nutrition, social protections, and climate

change. Ongoing evaluations in Nepal and Rwanda

seek to uncover how nutrition interventions and

social-protections interventions can enhance the

impact of agricultural productivity-enhancement

efforts. A second area is a push toward expanding

the commercialization portfolio. Newly-launching

projects in Senegal and the DRC will focus on val-

ue-chain interventions, warehousing, and market-

ing as channels to translate agricultural productiv-

ity into income.

Launching the next wave of agriculture IEs: ln the

fall of 2017, DIME will again convene policymakers

currently implementing impact evaluations to share

final findings and learning and to engage projects in

the early phases of implementation to launch the

next wave of evaluations.

6.3 Governance

The ieGovern ProgramGovernment organizations provide essential public

services in key areas such as health, education, and

infrastructure. The size of the public sector is espe-

cially large in developing countries. However, key de-

terminants of effective governments are still largely

unknown. For instance, which factor is most important

to ensure effective delivery of public goods? Or, which

mechanisms ensure a more transparent and account-

able public procurement process? These are still un-

answered policy research questions. Governance

reforms are often long term, complex, and difficult

to measure. Rigorous evidence on what works in the

sector is, therefore, in short supply. In fact, the gov-

ernance field represents less than 3 percent of reg-

istered impact evaluations.7

DIME and the Governance Global Practice launched

the ieGovern program in 2013 to produce rigorous

evidence to improve governance project results and

to push the frontier of available evidence on what

works in governance reform. To date, the program

has a portfolio of 31 IEs across the world that study

four main themes: (i) civil service reform, (ii) public

financial management (tax and procurement), (iii)

justice, and (iv) decentralization/subnational pub-

lic-sector management.

Over the last year, the program has reached matu-

rity: most IEs in the portfolio have passed concept

note stage and are currently being implemented.

A conceptual framework centered on mechanisms

of incentives, demand-side and top-down account-

ability, relaxing of constraints, and delivery mech-

anisms has been firmly established (see also the

Governance and Accountability section in chapter

5.4). Lastly, important initiatives related to ieGov-

ern have been launched: the Bureaucracy Lab and

the Research Flagship report of the Global Tax

Team (see more details below).

The Bureaucracy Lab (Civil-Service Reform)

IE research has mainly focused on studying perfor-

mance incentives for frontline staff—such as teach-

ers, nurses, and doctors—that address, for instance,

problems of absenteeism or underperformance. To

go beyond this, IE work in the civil service-reform

pillar focuses on research questions related to those

civil servants who work in core government minis-

tries, such as ministries of finance and education,

and bear the responsibility for designing a country’s

policies, collecting its taxes, and so on. Key policy

questions being studied include how to motivate

public-sector workers to perform better with differ-

ent (monetary and mission-based) incentives (Libe-

ria and Pakistan), how to improve the governance

of maintenance of public infrastructure (Tanzania),

how streamlined information flows within the pub-7 3ie Impact Evaluation Repository (http://www.3ieimpact.org/

evidence/impact-evaluations/), accessed 1/14/2016


lic sector can improve project performance (Pa-

kistan), and how a public-private partnership can

facilitate access to public services for marginalized

groups (India).

IE work on civil-service reform fits into a broader

research program called The Bureaucracy Lab, which

is an initiative co-led by DIME and the Governance

Global Practice. The Lab is creating improved ad-

ministrative data on the characteristics of public

officials and their organizations to inform the op-

erational design of public-sector organizations. In

addition, the Lab is undertaking experimental work

with large-scale surveys of civil servants to generate

an evidence base on how to survey civil servants ef-

fectively. The Bureaucracy Lab is also working with

academic anthropologists and sociologists to create

a detailed picture of civil services across the world.

Each of these elements uses the ieGovern program

as a platform for research, while providing inputs

that feed back into the design of the evaluations.

Tax and Procurement

Public Financial Management (PFM) reforms are

a core of support in client countries, by the World

Bank and other donors for a long time. Yet only few

IEs exist on the effectiveness of different PFM sys-

tems. The IE research work under ieGovern tried to

fill this gap with several IEs in the PFM subsectors

of tax and procurement. Research questions being

addressed include the impact of the adoption of

e-procurement systems on competition and mar-

ket entry of new firms, prices and value for money

of government purchases (Bangladesh and Brazil),

how centrally coordinated framework agreements

affect the procurement process and quality of

services procured (Colombia), and how behavioral

nudges and facilitation measures can affect will-

ingness to pay taxes and tax compliance (Tanzania

and Colombia).

The ieGovern portfolio of tax research has helped

spark the creation of a broader research program

embedded in the Governance Global Practice fo-

cused on Innovations in Tax Compliance. The objective

of the research program is to influence the design

of World Bank tax operations through the develop-

ment of a multifaceted approach to improving tax

compliance. This approach explores strategies that

are both i) technically appropriate and ii) lever the

Bank’s broader governance operations to engage

citizens and progressively build trust, reciprocity,

and support for tax compliance.

To do this, the project will develop a framework

that holistically looks at enforcement, facilitation,

and trust as key mechanisms to improving tax

compliance. The work recognizes that technocrat-

ic reform focused on enforcement and facilitation

remains essential, but more substantial and long-

term improvements are ultimately likely to depend

on building a relationship of mutual trust between

government and taxpayers. The project will serve

as a convening force for research partnerships

both inside the Bank and with outside academic/

research institutions, including International Centre

for Tax and Development (ICTD), Institute for Fiscal

Studies (IFS), and Chr. Michelsen Institute (CMI).

JusticeAn efficient, fair, and accessible justice system is

consequential for peace and security, encourages

investment and growth, and is fundamental to no-

tions of citizenship and trust in government. Yet,

there is little empirical research in justice-system

reform, in large part because data is not easily avail-

able. In recent years, however, governments around

the world have embraced electronic case-manage-

ment systems and used innovative technologies to

expand access to justice. Leveraging the Bank’s re-

lationship with governments, the ieGovern work on

justice is uniquely positioned to take the lead in jus-

tice research. It is evolving into a broad research pro-

gram to both establish a global data infrastructure

for the justice sector and develop a global program

for understanding the economics of justice reform.


The ieGovern program is currently focused on (i)

strengthening administrative data collection and

developing case-management systems capable of

producing high-quality data (Kenya and Senegal),

(ii) developing an empirically validated measure-

ment framework for justice research, (iii) using

high-frequency case data to understand the impact

of justice reforms (Croatia and Senegal), and (iv)

setting up the foundation for future experimenta-

tion in justice (Azerbaijan and Kenya). Going for-

ward, the program aims to establish a global depos-

itory of administrative and survey data on justice

and identify additional priority countries in which to

experimentally and iteratively test the impacts of

new justice reforms.

Subnational PSM/ Decentralization

Transferring power and responsibilities to local enti-

ties is a very popular reform for many countries, in-

cluding in OECD, middle-income, and poor countries.

However, the evidence base of how decentralization

reforms fare in practice has not kept up with the

number of reforms. Unanswered research questions

include how to measure and incentivize the per-

formance of local governments, how to deal with

potential elite capture at the local level, and how

to ensure local governments have sufficient capac-

ity to handle increasing responsibilities and collect

their own revenues.

Our research programs explore several dimensions

of the decentralization puzzle. In Cambodia, an IE is

testing how to harness social-accountability inter-

ventions to improve service delivery of local govern-

ments. The impact of demand-side actors such as

community officers and community-based organiza-

tions in making local governments more accountable

is being studied in Burkina Faso and Solomon Islands.

Transfers of resources from central government to

local governments based on their institutional per-

formances are being tested in Tanzania to assess

whether this program-for-results type of incentive

scheme is effective in improving local service delivery.

Highlights of Future ActivitiesSeveral key items will be launched or produced in

the next year. As part of the Bureaucracy Lab, we

plan to publish a global data-set on public sector

wage bill, public-private employment comparisons,

and public sector compression ratios. In addition,

surveys of civil servants in Liberia, Tanzania, and

Guatemala will be completed. Overviews of the lit-

erature on the anthropology of bureaucracy and a

review of civil-servant surveys are also planned.

The justice program aims to establish a global

depository of administrative and survey data on

justice and identify additional priority countries in

which to experimentally and iteratively test the im-

pacts of new justice reforms.

Two new seminar series are in the pipeline: one fea-

turing task team leaders and IE research teams

that highlights how the i) design of an impact eval-

uation and ii) embedding it within the project is pro-

moting adaptive learning at various stages of the

project lifecycle and another on aspects of bureau-

cracy in the developing world.

An efficient, fair, and accessible justice system is consequential

for peace and security, encourages investment and

growth, and is fundamental to notions of citizenship and trust

in government.


6.4 Climate ChangeWhat is the right balance between meeting the

World Bank’s objective of eradicating poverty while

limiting environmental consequences? The Sustain-

able Energy for All initiative aims to achieve uni-

versal access to sustainable energy, but, this would

mean providing electricity access to over one billion

new people. Globally, approximately two-thirds of

greenhouse gas emissions already come from en-

ergy extraction and use (IPCC 2014). Agriculture is

the largest sector in many developing country econ-

omies, but deforestation contributes between 10

and 17 percent of annual carbon emissions (Samii

et al., 2014).

While we are on target to eradicate extreme pover-

ty by 2030, we are also on target to increasing the

Earth’s temperature to irreversible levels that are

anticipated to have far-reaching long-term conse-

quences on economic growth, vulnerability, and the

environment. While the problem of climate change

is fundamentally a global collective action chal-

lenge, there are important program-level activities

and insights that can help us mitigate its effects

and strengthen resilience.

The initiation of an Energy and Environment (E&E)

program was motivated by the dearth of rigorous

impact-evaluation evidence in these sectors and

the influential role they play in poverty alleviation

and climate change. In 2011, the World Bank had 12

ongoing or completed impact evaluations in both

energy and environment (compared to over 100 in

education), despite the fact that these programs

consist of almost 20 percent of the World Bank’s

lending portfolio.

E&E Impact Evaluation Program

The E&E program was launched in Lisbon in Octo-

ber 2014. It brought together 19 project teams (fi-

nanced through DFID, the GEF, CIF, IDA and IBRD)

and 28 researchers from 11 academic institutions

to refine research opportunities based on project

interest and operational feasibility. This was com-

plemented by a parallel set of workshops focused

on measurement opportunities in the sector.

The first workshop, held jointly with CEGA in Au-

gust 2014 in Berkeley, brought together engineers,

economists and World Bank counterparts to ex-

plore leveraging new technologies to improve mea-

surement in energy and environment projects and

research. A follow-up Berkeley measurement work-

shop, focused on innovative measures for climate

resilience, was held in June 2015. To strengthen the

economic theory underpinning each IE, a research

workshop was then held in Chicago under the lead-

ership of John List and Michael Greenstone. A set

of project teams were given a chance to present

their current design and receive critical feedback

from leading academics within the E&E research

team to ensure the work is able to maximize its

contribution to the global knowledge agenda.

The original set of 12 impact evaluations was se-

lected from a funding window in Jan 2015 that laid

out the research program focusing on two pillars:

(i) environmentally sustainable electricity supply,

access, and efficiency; and (ii) natural resource and

sustainable land-management issues, with a focus

on incentive schemes, governance, and vulnerabili-

ty. Transport, industrial pollution, and urban devel-

opment relate directly to energy and environment

and present major development challenges with

important economic and environmental implica-

tions. However, these topics are addressed by other

programs.

The research agenda has benefited from direct en-

gagements with the World Bank’s Climate Change

Cross-Cutting Solutions Area, Energy and Ex-

tractives Global Practice, Environment and Natural

Resources Global Practice, Water Global Practice

within the World Bank, the Global Environment Fa-

cility (GEF), the Climate Investment Funds (CIF), the

Energy Sector Management Assistance Program

https://www.ipcc.ch/report/ar5/syr/

http://search.proquest.com/openview/f3e916ccdc7688fe15517c943b4fc557/1?pq-origsite=gscholar&cbl=2040247


http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment

http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment

http://cega.berkeley.edu/events/cega-dime-workshop-on-measuring-development-energy-environment/

http://cega.berkeley.edu/events/cega-dime-workshop-on-measuring-development-energy-environment/


(ESMAP), and DFID (including the evaluation de-

partment and climate-change teams). Work under

the main themes is summarized below.

Topic 1: Energy Access, Reliability, and EfficiencyThere is a growing body of evidence on the impacts

of energy access on health, education, and produc-

tivity. But, important questions remain about the

cost-efficiency of investments to balance the cov-

erage (extensive margin) and per-connection avail-

ability (intensive margin) of providing electricity.

The average annual energy consumption of elec-

trified households in Kenya is 20 times less than

the average American household. Therefore, under-

standing the demand and impact of different tiers

of access from solar lanterns (Tier 1) through to full

grid access (Tier 5) becomes an important concern

to help governments efficiently allocate resources.

There is an important trade off—while lower tiers of

access may provide less opportunities for economic

growth (for example, being unable to power large

appliances and machines), the benefit of lower in-

vestment costs and easier expansion may outweigh

this concern.

We explore this question programmatically by

looking at the demand for, and impacts of differ-

ent levels of energy access in Senegal (solar lan-

terns), Argentina (solar home-systems) and Kenya

(grid connections). We offer different subsidies in

the various projects to elicit demand curves for so-

lar-energy products and produce evidence that will

help policymakers learn how to set efficient subsidy

levels to balance expanding access with fiscal sus-

tainability for service providers.

Moving beyond access, there is emerging evidence

that focuses on the importance of providing reli-

able electricity for industrial development and the

effects of rural electrification on household welfare

finding strong negative effects on firm revenues

and producer surplus. The program is currently

working with two large infrastructure investment

projects in Nepal and Bangladesh—both of which

face acute energy constraints—to understand the

impacts of improving reliable energy access by re-

habilitating and expanding transmission lines and

upgrading grid substations.

Topic 2: Incentivizing Sustainable Land Use and Natural-Resource ManagementThe overuse of natural resources can be a result

of externalities, unclear property, or high discount-

ing of the future. Since natural-resource manage-

ment has both local and global implications, find-

ing the right interventions and policy approaches

to address these issues presents a challenge. For

instance, a systematic review on the effective-

ness of one of the most common policy interven-

tions used to overcome coordination failures—the

creation of decentralized forest management

groups—found limited evidence of reduced defor-

estation rates and could not reject the possibili-

ty that these programs have negative economic

consequences (Samii et. al., 2014). The objective of

this program is to generate knowledge on effective

ways to address the causes of unsustainable use

of natural resources.

One common intervention to address the external-

ities associated with sustainable forest and land

management is Payment for Ecosystem Services

(PES). The program includes four PES projects that

offer financial incentives to landholders to reduce

deforestation and promote sustainable land man-

agement in Uganda, Burkina Faso, Ghana, and Mex-

ico. Here, we explore the role PES incentives play in

reducing deforestation, but also whether alternative

livelihood options may help ensure the economic

well-being of beneficiaries and increase the sustain-

ability of these programs.

We also explore the dynamics associated with in-

centivizing long-term behavior change. For instance,



in Uganda we explore the impact of PES after in-

centives are removed. Does deforestation remain

low, return to pre-intervention rates, or increase to

catch up with total deforestation in control areas?

Each scenario has plausible justifications, but re-

sults in very different interpretations around the

overall role that PES schemes can play to mitigate

our impact on the climate.

Going ForwardSince the program began two years ago, the focus

on using rigorous evidence in the energy and envi-

ronment sectors has only modestly increased. It still

requires a more concerted effort to catch up to oth-

er evidence-led sectors like education and health.

The mapping of evidence to development projects

is currently skewed in favor of subtopics that are

more amenable to impact evaluation. While the im-

pacts of energy access have been a preoccupation

in current economic literature, the reality is that the

vast majority of development funds are directed to-

wards generation and supply.

Tackling questions on the drivers of energy avail-

ability to connected customers will be the primary

focus of the energy agenda, moving forward. This is

more aligned with the major development challeng-

es in the sector. The program aims to work with

utilities and other service providers to explore the

interplay between pricing, service delivery guaran-

tees, billing and payment schemes and enforce-

ment. This will help identify the bottlenecks and

associated solutions to optimally utilize electricity

infrastructure and provide reliable energy to house-

holds and industry.

For environment topics, we aim to expand the fo-

cus area beyond financial incentives, to also include

co-management practices and regulatory influenc-

es to better represent the major development tools

available to practitioners and policymakers.

6.5 Financial and Private-Sector DevelopmentProductivity is the key driving-factor for long-term

sustainable growth. Empirical research, specifically

at macro level, has mostly emphasized the deter-

minants of firms’ productivity. Much less is known

about how to increase firms’ productivity (Syver-

son, 2011). In reality, this challenge has to do with

(1) identifying ways to increase efficiency of factors

of production that are used idly, and (2) find in-

novative ways of combining factors of production

so as to increase growth potential. This question is

paramount, given that most developing countries

struggle with low levels of labor and total factor

productivity (The World Bank Enterprise Surveys,

1/2013).

The FrameworkThe Trade and Competitiveness (T&C) agenda in

DIME is structured to test and identify effective

ways of increasing firms’ productivity through both

efficiency gains and shifts in the production fron-

tier. Efficiency gains are understood as changes in

the production process to help firms move closer to

the efficient production frontier. The two assump-

tions underlying suboptimal allocation of inputs

are: (1) market imperfections and/or behavioral bi-

ases—such as misperception of returns associated

with a given business practice, lack of motivation

to adopt better production process (Gibbons and

Henderson 2012; Nguyen and Nguyen 2016); and

(2) organizational barriers that prevent firms from

adopting new technologies (Atkin et al., forthcom-

ing at QJE) and using inputs optimally. In this light,

adoption of new technology is key to firms’ (and

economic) growth.

In this context, technology change encompasses

any shock in the production process that leads to

higher output given the inputs available. That shock

could be caused by, for instance, better trained em-

ployees, use of better managerial practices, use of


cheap credit lines in credit-constrained firms, and

business registration to access public services that

are only made available to formal firms. Technology

change and, thus, productivity growth go hand in

hand with technology adoption.

Shifts in the production frontier occur only when

factors of production are already used optimal-

ly, and could result from improvements in busi-

ness regulations, innovation, and infrastructure. It

is worth mentioning that interventions aimed at

moving out the production frontier are more dif-

ficult to evaluate with randomized controlled trials

for two main reasons. First, regulation policies usu-

ally involve changes in the current legislation and

such decisions are usually taken at higher levels and

are not necessarily theoretically grounded. Second,

those policies are highly likely to have general equi-

librium effects (spillovers).

The T&C Program

Some Background

Even though the T&C agenda in DIME accommo-

dates both efficiency gains and growth to increase

firms’ productivity, the projects in the pipeline tend

to be more concentrated on testing policies aimed

at raising firms’ productivity through the former. It

hasn’t always been like that.

In 2010, DIME and the T&C Impact Program (now

called ComPEL) at IFC co-organized the first IE

workshop that focused on interventions to improve

firm capabilities, such as matching grants and

training programs. Several training programs were

rigorously evaluated since then, but, almost all eval-

uations found null effects on jobs creation and firm

productivity (McKenzie and Woodruff 2012). Nine

matching-grants programs could not even be eval-

uated, because very few firms took up the program

(Campos et al. 2013). We learnt that supply-side

interventions should pay closer attention to con-

straints on the demand side, which can hinder in-

tervention participation and, thus, adoption of new

technologies.

Three years later, in 2012, DIME and the T&C Im-

pact Program co-organized a second T&C IE work-

shop. The objective was geared towards generat-

ing rigorous evidence on how to help/nudge small

informal firms to formalize their business. The

prevalent view was that reducing business regis-

tration costs (formalization) and simplifying tax

were necessary conditions to the growth of small

firms. The overwhelming evidence suggested that

most small firms do not want to formalize their

business, even when registration costs are fully

subsidized.

Even among firms that do formalize, the impact on

performance indicators (for example, sales and rev-

enues) are either small or null (Bruhn and McKenzie

2014). The evidence generated by the rigorous eval-

uations led to a change in IFC’s approach towards

small informal firms. It no longer sees formalization

as stepping stone for growth of informal firms. The

challenge now is how to make informal firms more

productive.

Current Portfolio

Improvements in the investment climate are still

central to the Bank’s agenda, but the projects in

T&C are now looking towards better understand-

ing issues that are under firms’ control and beyond

Find innovative ways of combining factors of

production so as to increase growth potential.


firms’ control, to better attack the constraints

firms face to grow.

A good example on how to better tackle intervention

participation while targeting supply is that of our IE

in Brazil. Medium-sized firms will be randomly split

to receive information, information plus training,

and a placebo intervention. As evidence shows that

take-up of training programs is low, this IE aims at

measuring participation incentives. One group will

be offered 40 percent subsidy on training costs in

addition to information, while the second will be of-

fered 80 percent subsidy plus information. Unlike

most training programs, the training will be provid-

ed at firm location and consultants will pay weekly

visits to monitor the adoption of good practices. In

addition to testing the impact of different subsidy

rates on training take-up, this design will help the

implementing institution (a regional development

bank) identify how much firms are willing to pay for

a training program, but also if firms misperceive the

actual returns.

Another example of how we can face firm growth

constraints, is of another training-oriented IE, this

time in Georgia. In this case, the training will focus

on e-commerce and be offered to a random sample

of small firms. This is to ensure that they have the

basic skills to compete in the market and gain ac-

cess to markets. However, to also address demand

constraints, half of the treatment firms will then

be randomly assigned to receive additional step-

by-step trainings on product branding, marketing,

and various other online services. Access will be

granted through a self-selection process, where

firms will have to reach a specific goal at the end of

each level (for example,. receive ten online orders),

before continuing to the next step. This competi-

tion type process aims at increasing the odds for

firms to access a broader consumer market. The

idea is to develop some record on service quality

and measure if firms are more likely to receive new

customer orders.

Overall, the current T&C portfolio accounts for 26

IEs, distributed across 22 countries—nine in the

preparation phase, 11 ongoing, and six completed.

The total estimated budget is $14 million, of which

25 percent is funded through i2i. Over half the pro-

gram (16 IEs) evaluates World Bank projects, repre-

senting a total of $243.7 million in loans. In terms

of outputs, the T&C teams have produced six re-

ports, eight working papers, and two publications.

Going Forward The lessons generated so far have substantially

shaped the current T&C Global Practice (GP) agen-

da. A more holistic approach has been put in place

since the last IE workshop, held in Istanbul in May

2015. It has been maturing since then. The ComPEL

and DIME teams are pushing projects that were se-

lected for funding support (September 2015 call)

to take into account some of the cross-cutting

themes that were identified as major issues in pre-

vious IEs such as: (i) potential spillovers occurring

in the market; (ii) low take-up rate of input-based

interventions; and (iii) the importance of improving

intervention targeting.

Those cross-cutting issues gained momentum in

the T&C program and played a critical role in the

selection of projects that will attend the next IE

workshop in Mexico City between February 27 and

March 2, 2017. Three priority areas were recently

identified by the T&C GP as strategic for knowledge

generation through rigorous impact evaluations:

1. Firms’ access to markets and spillovers.

2. Identification and support to high-growth small

and medium-sized enterprises (SMEs).

3. Regulatory efficiency.

There is an increasing focus on how to improve

firms’ linkages to both consumers and larger firms

in the global value-chain. Interventions will then

have to look carefully at both supply (for example,

training and matching grants) and demand-side/or


institutional constraints (for example, technology

adoption and diffusion, access to markets—busi-

ness-to-business and business-to-consumers, and

regulatory environment) to (i) maximize the chanc-

es for intervention success at least on implemen-

tation grounds, and (ii) increase the odds of policy

effectiveness.

6.6 Transportation and ICTDIME’s ieConnect impact-evaluation program seeks

to generate rigorous evidence on the impact of

large-scale development investments in the trans-

portation and ICT sectors. Transportation and as-

sociated infrastructure investments are critical in-

vestment sectors for developing countries. A large

percentage of lending by the World Bank and of de-

velopment finance and grants from other donors is

aimed at this sector. But, there is relatively limited

evidence of their impact through rigorous experi-

mental or quasi-experimental evaluation.

The goal of the ieConnect program is to generate

impact-evaluation evidence on transport policy and

investments, including indirect benefits, at suffi-

cient scale to substantially improve the evidence

base for policy making in selected evidence gap

areas. DIME’s ieConnect program, which was de-

veloped in collaboration with the Bank’s Transport

and ICT Global Practice and DFID, aims to fill this

gap by linking projects with research teams and en-

abling them to develop innovative and rigorous im-

pact-evaluation designs. The focus is on identifying

and estimating the impact of transport and ICT

investments themselves, as well as developing and

testing ancillary interventions that can maximize

their impact.

The program targets the subthemes of roads and

transport corridors, urban mobility, road safety, and

information and communications technology (ICT)

infrastructure. There are currently 11 impact evalu-

ations in progress or under preparation. Following

a planned expansion of the program for ieConnect

phase 2, there will eventually be 22 experiments in

the portfolio.

The ieConnect program was launched in Rio de

Janeiro at the ieConnect for Impact workshop in

June 2015. A call for proposals was made in Sep-

tember 2015 and projects were selected on a com-

petitive basis in December 2015. Work began in

earnest in 2016 on concept note development and

IE implementation. Over the course of the year,

there was important progress on the projects that

emerged from this original workshop and call for

proposals.

A key milestone occurred in June 2016. An in-

tensive technical quality/IE design-strengthening

and capacity-building workshop for East Africa

transport IEs was held in Nairobi. It was attend-

ed by a number of transport IE teams (Rwanda

rural roads IE, Ethiopia expressway IE, Nairobi

SmarTTrans IE, Nairobi BRT IE, and the Dar es

Salaam BRT IE), external academic experts, civ-

il-society representatives, Kenyan government

counterparts, as well as a group of World Bank

transport-corridor projects, which could become

candidates for ieConnect IEs as the program is

expanded in the future. The workshop enabled

strengthening of selected IE designs via targeted

technical assistance from other IE teams and ac-

ademic experts, initial preparatory work on poten-

tial future transport IEs, and continued capacity

building of government and project counterparts

on impact-evaluation methods.

The original IE work from the Rio workshop was

supported by seed funding from DIME’s main i2i

program, but the program was initiated with the un-

derstanding that the scale and data requirements

of transport IEs would necessitate the creation of

a separate, transport-specific sectoral IE program.

This expansion of the program, beginning in 2017,

will be possible thanks to DFID’s ieConnect phase


two grant, which was developed over the first half

of the year and submitted in July 2016. The grant

was finalized over the second half of 2016, setting

the stage for the kick off of ieConnect phase two

in early 2017.

Beyond these programmatic developments, the fol-

lowing are IE design and implementation highlights

from ieConnect’s work in 2016:

n In Rio de Janeiro, the IE team studying the im-

pact of women-only gender-segregated metro

cars has generated preliminary results, using ex-

perimental variation in pricing to show low will-

ingness-to-pay for the female-only cars, which

may be explained by low compliance with gen-

der-segregation policy by men.

n In Rwanda, the rural roads impact evaluation

team has completed baseline data collection in

all study communities, and has begun end-line

data collection in a subset of communities. In

addition, ongoing high-frequency data collec-

tion on prices and availability of goods in local

markets is in process. This will ultimately allow

estimation of the impact of roads on compre-

hensive measures of household welfare.

n In Peru, a concept note has been approved for

what is, to our knowledge, the first randomized

impact evaluation of a rural roads-rehabilitation

program. The IE focuses on economic impacts of

road rehabilitation and also includes a substan-

tial gender focus, examining the link between

road rehabilitation and women’s access to edu-

cation and health services.

n In Dar es Salaam, the Bus Rapid Transit System

impact evaluation has fielded a baseline survey

in early 2015, prior to the system’s Line 1 open-

ing in May 2016. An end-line survey for phase 1

of the project is planned for 2017. The IE plans

to cover successive BRT line as they open in the

coming years.

n In Nairobi, the concept note was approved for

the SmarTTranS impact evaluation, which will

test the effect of shifting information and in-

centives for safe driving in Nairobi’s informal bus

transit (matatu) system. The project team has

begun developing a state-of-the-art technolo-

gy and big-data system for real-time monitor-

ing and an empirically-validated measurement

framework for road safety.

n In Ethiopia, the Expressway Development Sup-

port IE (still in design phase) is working to ex-

pand its scope beyond evaluating the impact of

expressway construction from Mojo to Hawassa.

The IE team is seeking to study the joint effect

of these transportation improvements with the

impact of major industrial investments such as

the Hawassa Industrial Park, which will employ

thousands of young workers in light manufac-

turing in rural Ethiopia.

n Several other projects are also still in the design

phase, due to either significant redesign (Mau-

ritania) or changes on the client side (Bogota,

Rio de Janeiro). Others, such as the Tunisia ICT

IE, have been dropped because associated World

Bank lending operations were cancelled.

Finally, an important focus of effort for the team

in 2016 has also been preparations to scale up ie-

Connect’s work program in 2017 for phase two.

Additional phase two resources will be used to a)

continue pushing existing IEs to concept-note com-

pletion and IE implementation, and b) to develop

follow-up/additional experiments in phase 1 IE

sites, as well as to develop several completely new

projects. We are also working to expand the team

to deliver this expanded work program by hiring

economists (2), a data specialist (1), an operations

officer (1), a research analyst (1), and an adminis-

trative assistant.

Areas of focus in phase 2 will expand to include

more gender-specific interventions and gender-dis-

aggregated data collection, additional work on frag-

ile and conflict (FCV) or conflict-affected settings,

additional work on transport corridors, greater re-


gional focus on South Asia (outside of India), and

increased collaboration with other MDBs including

the Asian Development Bank and the African De-

velopment Bank.

6.7 GenderGender equality for development is a core theme

of i2i work and cuts across all thematic areas.

We identify two areas of growth for the current

i2i gender analytical agenda: design and test gen-

der strategies in areas where gender-specific con-

straints have been identified; and build the evidence

on gender-specific market failures in underserved

areas of impact-evaluation practice. The themat-

ic coverage of the i2i gender program seeks to fill

the gaps in four areas identified in the 2012 World

Development Report, “Gender Equality and Develop-

ment”: (i) human capital, (ii) economic productivity,

(iii) access to finance, and (iv) empowerment. Over-

all, over half of the current i2i portfolio is planning a

gender-disaggregated analysis, while 17 percent of

i2i-supported IEs are testing interventions tailored

to address gender issues.

i2i supports rigorous evaluations of policy actions

that look to relax supply-side constraints (for ex-

ample, improving service delivery for clean water,

sanitation, and maternal care) as well as market

and institutional constraints (for example, reducing

systematic differences in earnings). In underserved

IE research areas, the i2i program places emphasis

on documenting gender constraints in the context

of infrastructure investments and governance, with

special focus on transport, electoral participation,

and women’s labor market participation in FCV

settings. The research agenda evolves with i2i’s

portfolio, fueling iterative learning. i2i operational-

izes this vision by providing technical and financial

assistance to policymakers to identify relevant gen-

der issues, designing appropriate policy action, and

testing their impact to motivate scale-up, scale-

down and, new testing.

Addressing Human Capital Gender Gaps through the LifecycleGender gaps in human capital are well-document-

ed, but there is little evidence on how best to close

those gaps. The i2i research agenda focuses on how

to design interventions that address women’s ac-

cess to and use of health services and education,

and reduce women’s vulnerability to shocks that

disrupt human-capital acquisition. An IE of a voca-

tional training program in Malawi (Cho et al, 2015)

found that family obligations limited participation

and resulting skills development for young women.

Another IE testing the impact of a business literacy

course for female micro-entrepreneurs with rela-

tively low education in five different states in Mex-

ico, finds significant improvements of managerial

skills (Iacovone et al., forthcoming). As a result, nine

new states in Mexico have submitted proposals to

expand the program to their states.

New IEs in Nigeria are testing supply and de-

mand-side interventions to increase women’s

access and use of medical antenatal and birth

services, and community-level interventions to in-

crease uptake of malaria-prevention technologies

and increase accessibility of anti-malarial drugs.

One specific study in Nigeria, measuring the impact

of entertainment education through soap operas

on attitudes and behaviors about safe sex and HIV

testing, finds positive impact on both outcomes

(Orozco et al., forthcoming). The study also shows

Gender equality for development is a core

theme of i2i work and cuts across all thematic areas.


that, given the popularity of soap operas among

poorer and less educated households, they can be

used to positively alter attitudes and behaviors

of millions of individuals at very low costs around

many development issues.

Economic OpportunitiesWomen’s access to economic opportunities is undermined by their lower access to production inputs. Female farmers have less access to in-formation, as agricultural extension networks are dominated by men. In Malawi (BenYishay et

al, 2016) and Mozambique (Florence Kondylis et

al., 2014), i2i IEs showed that women can make effective extension partners: they are at least as good as men at encouraging adoption of im-proved technologies.

Throughout Sub-Saharan Africa, women are dis-

proportionately limited in their land ownership and

transfer rights. An IE in Benin showed that land

demarcation increases soil fertility investment in

female-managed landholdings, shifts household

decision-making, and reduces spousal conflict

(Goldstein et al, 2015). An IE in Kenya tested a

recent policy innovation, known as “microfranchis-

ing”, which provides unemployed participants with

a proven business model and the specific capital

and business linkages based on the hypothesis that

many unemployed youth would like to be generating

income, but lack both experience to be competitive

and the financial and human capital (Owen Ozier et

al., forthcoming). Early results from the study found

that, for young women, the program increased

self-employment. This is an important finding con-

sidering around 55 percent of urban women in Ken-

ya aged 15 to 25 are unemployed.

Another IE in Afghanistan is measuring the impact

of a program aimed at lifting the poorest out of ex-

treme poverty by providing a way to transition into

sustainable and profitable economic activities and

linking them with microfinance programs (Aidan

Coville et al., forthcoming). It applies the program in

a setting where female labor force participation is

among the lowest in the world (15 percent) and has

a strong focus on supporting female-headed house-

holds, tackling multiple constraints simultaneously

to provide households with a big push out of ex-

treme poverty.

Further, large parts of populations in developing coun-

tries do not have access to essential social services.

In Comoros Island, for example, an i2i IE is testing

the effects of temporary employment cash-for-work

program on social and economic outcomes of poor

households, and whether they vary according to gen-

der within household (Mvukiyehe et al., forthcoming).

Access to FinanceAccess to productive assets constrains women’s economic opportunities, whether women farm-ers or micro-entrepreneurs. An i2i IE in Rwanda

tested introduced targeted and pre-commitment

savings accounts. Initial findings show that wom-

en are more likely to earmark their savings to buy

durable goods, relative to men who invest in ag-

ricultural inputs, suggesting that intra-household

bargaining over resources plays an important role

in women’s investment decisions (Jones et al,

forthcoming).

In India, an IE showed that women that partici-

pated in a women’s empowerment and rural live-

lihoods program had improved access to loans, ac-

cumulated assets, and invested in education, which

Women’s access to economic opportunities is undermined by their

lower access to production inputs.


further made them feel more empowered (Shah et

al., forthcoming). Early results from an ongoing IE

in Benin examining several incentive mechanisms

to attempt to get business owners to formalize

found that male business-owners formalized much

more than female business-owners (McKenzie

et al., forthcoming). Ongoing analysis is exploring

the reasons for lower formalization amongst fe-

male-owned firms.

In Dominican Republic, an ongoing IE is studying

the impact of financial literacy and job skills, es-

pecially benefiting women, on household-finances

management, savings, credit, use of formal-sector

financial products, ability to search for, obtain, and

retain formal employment, management of small

businesses, new businesses opened, and income

levels (Xavier Gine et al., forthcoming). An ongoing

IE in Malawi on identification and fingerprinting,

a topic that is at the heart of the development

agenda, is testing whether requiring fingerprint au-

thentication for transactions alleviates access to

credit more for females and improves repayment

more for those that borrow (Gine et al., forthcom-

ing). This is also important because it would make

it impossible for male relatives to seize control of

women’s assets on the death of the husband, as is

common in Malawi.

Promoting Women’s Empowerment and Agency for Economic Development

A growing body of evidence shows that placing wom-

en in the center of the development agenda can in-

crease efficiency in the management of institutions

and resources. Also, female leaders can have benefi-

cial impacts on social norms. The i2i research agen-

da focuses on using gender empowerment to com-

bat domestic violence, testing interventions such as

cash transfers and active labor-market policies to

economically empower women, and role of law and

justice in achieving gender equality, among others.

An ongoing IE in Azerbaijan, for example, tests the

extent to which free legal aid leads to greater le-

gal empowerment, improved dispute resolution, and

higher welfare from reclaimed income and benefits,

more stable household settings, productivity gains,

and a gradual move away from discriminatory

norms and practices (Bilal Siddiqi et al., forthcom-

ing). In Pakistan, an IE is evaluating the impact of

women-inclusion mandates and ratification in vil-

lage-level grant management, which imposes an

inclusion mandate that 50 percent of individuals

organized in a village have to be women (Gine et

al., forthcoming). As results come in, we will know

whether having more women in these village-level

bodies changes the composition of projects that

are funded and leads to a better overall allocation

of resources. Another IE in India is testing whether

privately-run kiosks offering access to government

services under the Right to Public Services Act al-

low women greater access to basic services, and

whether this changes their attitude (Daniel Rogger

et al., forthcoming).

Gender in Underserved Research AreasA notable opportunity for the i2i portfolio is to make

a dent in understanding gender issues in under-

served areas. Recent progress on this front includes

the transport sector, and economic and electoral

participation in fragile settings.

Transport

Reducing transaction costs by improving trans-

port infrastructure has the potential to change the

way women access markets. In Ethiopia, a large

expressway construction is combined with the

development of a large industrial zone. Since the

large majority of employment in the industrial zone

will be of young women, this will be an opportunity

to study the effect of a large labor market shock

(60,000+ jobs over a period of several years) on

young women’s economic and social outcomes in


the vicinity of the zone. A complementary interven-

tion will be set up to experimentally study the role

of skills, information, and access to employment

opportunities.

In Peru, an intervention to promote women’s ac-

cess to health services and education is being eval-

uated in the context of a rural road-rehabilitation

project. In Brazil, a new IE on gender-segregated

public transport tests the extent to which gender

segregation is beneficial for women (Kondylis et

al., forthcoming). Harassment in public transport,

and sometimes even risk of rape, limits women’s

movements, activities, and employment in many

developing countries. Results from this study are

expected to inform policies going forward on public

transport-systems in cities worldwide.

Electoral Participation in FCV Settings

Despite recent policy efforts to increase women’s

participation and representation in politics, sig-

nificant gender gaps remain. Less than 10 per-

cent of the world’s countries have a female head

of state and fewer than 30 countries have reached

the target of 30 percent female representation in

parliament. Further, women continue to have low-

er electoral participation rates than men and their

voting choices are often influenced by powerbro-

kers or household heads (Giné and Mansuri 2011;

Tripp 2001; Geisler 1995).8 Gender gaps in political

participation are especially pronounced in war-torn

settings, where women tend to disproportionately

bear the consequences of conflict (Buvinic et al.

2013; Sow 2012; Rehn and Sirleaf 2002). While

there are individual country cases where women’s

representation in governing bodies have increased

in the aftermath of civil war, such representation

has not necessary translated into their efficacy in

voicing policy preference or interest (O’Connell 2011;

Tadros 2011; Hogg 2009).

What explains gender gaps in political partici-

pation? A growing number of DIME IEs investi-

gate the effects of information-provision inter-

ventions and a variety of delivery mechanisms

designed to remove or circumvent these con-

straints. The underlying premise of these in-

terventions is that since information provision

can occur relatively quickly and at lower cost,

interventions designed to provide information

can potentially address the lack of awareness,

thereby promoting political participation (Gine

and Mansuri 2010; Kumar 2001).

Consistent with this intuition, a DIME impact

evaluation in Liberia investigates the positive

effects on the political attitudes and voting be-

haviors of rural women when they are provided

access to United Nations elections-related radio

programs. The results point to significant effects

of the intervention on women’s political participa-

tion, both on national and local levels. Worrying-

ly, though, the study finds no evidence of effects

on women’s political efficacy and empowerment

outside of the electoral context, suggesting the

need to complement such brief interventions with

more sustained interventions that tackle slow-to-

change constraints (on supply and demand sides)

that might be embedded in prevailing social struc-

tures and norms.

DIME is evaluating a number of interventions that

do just that. In Zimbabwe, for example, a DIME IE

tests the effects of an intervention designed to

reform village-level governance via horizontal pres-

sure on gender inclusion and empowerment. Like-

wise, in Liberia, another DIME IE investigates the

extent to which a nine-month civic education in-

tervention that provides men and women a forum

for monthly deliberation on governance, rights, and

gender-equality issues, helps narrow gender gaps in

political participation.

8 Evidence from recent public opinion surveys across African countries suggests that 36 percent of female respondents are not interested in politics and that 39 percent never discuss politics. 22 percent and 24 percent were reported for men, respectively (Afrobarometer, 2008, reported in Bleck and Michelitch 2011).


Economic Participation in FCV Settings

Women’s access to productive assets and agency

over the household’s economic decisions is even

lower in fragile contexts. Yet, women’s access to re-

sources is particularly impactful on human-capital

investments that can help poor children in tough

places get out of poverty (Duflo 2003). i2i is sup-

porting a number of studies that aim to provide

economic opportunities to poor women in fragile

states.

In DRC and Indonesia, i2i is supporting the eval-

uation of unconditional ‘business grants’ to wom-

en for the creation of sustainable livelihoods and

for long-term poverty alleviation. A social-network

treatment will also be tested, in which participants

join a series of workshops from female mentors

that focus on building links between individual busi-

ness-owners. The relative impact and complemen-

tarities across these interventions will be captured

by the experimental design. In Tunisia, an IE is

testing the effect of capital injections to comple-

ment a more traditional income-support program

that supports the unemployed through short-term

employment opportunities. Focusing on vulnerable

women, the impact of this additional intervention

on long-run consumption and labor-market out-

comes will be compared to the outcomes of those

who merely participate in short-term labor-inten-

sive works (Mvukiyehe et al., forthcoming).

In Liberia, activities will be centered on developing

diagnostic studies, designing materials and mi-

cro-interventions, and testing these materials and

micro-interventions as part of the Liberia Youth

Opportunities Project (LYOP). The diagnostic stud-

ies will focus on understanding the concrete actions,

behaviors, and decisions that influence women’s

access to male-dominated trades and explore the

underlying preferences, information, and assump-

tions (conscious or otherwise). The goal is to step

back and research actual needs and obstacles on

the ground before jumping to conclusions about

solutions. It will pay special attention to considering

prevention and mitigation strategies for potential

unintended consequences of supporting females in

non-traditional or male-dominated fields.

Once the diagnoses have been carried out, the next

step will be to design and develop materials and

interventions to address the identified constraints

and safely support vulnerable females in these sec-

tors. A critical last step will be test and refine the

FIGURE 30: i2i IEs with Gender Component

Yes No

IEs Including a Gender AnalysisNumber (percentage) i2i IEs

73 (57%)

56 (43%)

IEs Evaluating a Gender-Specific InterventionNumber (percentage) i2i IEs

107 (83%)

22 (17%)

IEs Falling under the GenderCross-Cutting Solution AreaNumber (percentage) i2i IEs

98 (77%)

30 (23%)


material and interventions before taking them to

scale in the project, including in the impact evalua-

tions each project intends to undertake.

Portfolio overviewBased on current donor funding, the i2i portfo-

lio consists of 131 IEs spanning 53 countries and

covering all of the i2i thematic areas. At least 17

percent of the portfolio evaluates a gender-specific

intervention and 57 percent conducts disaggregat-

ed gender analysis.

6.8 Edutainment

Background Every year, the World Bank and client governments

invest millions in behavior-change campaigns across

almost all development sectors. However, many of

these campaigns are unconvincing, lack inspiring

narratives, and are communicated through out-

moded and uninteresting outlets such as billboards

and leaflets. Systematic reviews of these cam-

paigns, from risky sexual behavior to handwashing,

consistently show little or no effect on behavior, es-

pecially in the long term.

There is an unprecedented opportunity to use en-

tertainment media to change the lives of billions of

people, especially in urban areas.9 Entertainment

education or edutainment can be a game-changer

for development. Unlike traditional behavior-change

campaigns that convey abstract concepts and can

become repetitive quickly, educational narratives

are easier to follow and remember than abstract

information. Characters in mass media have the

power to be role models, inspire audiences to en-

gage in new thinking about “what is possible”, and

change the perception of what is “normal” and so-

cially acceptable behavior.

The 2015 and 2016 World Development Reports

respectively highlighted the untapped potential of

entertainment education and mass media in de-

velopment practice. However, the evidence base re-

garding the effectiveness of entertainment media

remains thin, especially to advise the scale up of

9 While access to TV and radio is almost universal in developing countries, consumption of internet entertainment is prevalent in urban areas. According to the 2016 World Development Report, last year there were 3.2 billion Internet users in the world and 8.8 billion Youtube videos were watched every day.

10 http://blogs.worldbank.org/voices/newest-weapon-against-hivaids-africa-mtv — https://www.youtube.com/watch?v=0mCv5U5LRG4 https://www.bloomberg.com/news/videos/2015-01-20/using-data-entertainment-to-combat-hiv-stigma — http://www.hollywoodreporter.com/news/mtv-shuga-viewers-twice-as-900752.

MTV Shuga: A Dramatic EvaluationProduced by the MTV Staying Alive Foundation, Oscar-winner Lupita Nyong’o starred in the first

two seasons of the show. MTV Shuga is a television drama that targets African youth. The show

is broadcast in over 70 countries, reaching over 500 million people worldwide. In the eight-month

follow-up survey, the DIME study finds that the treatment group was twice as likely to get test-

ed, reported fewer concurrent sexual partnerships, and reduced attitudes and behaviors related to

gender-based violence. Among female viewers, chlamydia infections were halved. These are substantial

impacts, especially in light of the limited effects found in other HIV behavior-change trials. The study de-

sign and preliminary results have been discussed in TEDx talks, Bloomberg TV, The Hollywood Reporter,

and WB blogs, among other media outlets.10


entertainment media as a development tool across

different sectors. There is a lot to learn about the

best way to maximize the impact and minimize

unintended consequences of entertainment me-

dia, a powerful tool that is largely untapped for

development. DIME is starting to expand this evi-

dence base with ongoing experimental evaluations

that explore the relative effectiveness of radio

spots versus printed narratives to promote adop-

tion of solar lanterns in rural Senegal; the use of

a Nollywood (the Nigerian film industry) movie to

promote financial savings, and of the MTV Shuga

drama to reduce risky sex and gender-based vio-

lence in Nigeria.

A Multi-Sectoral Program The Entertainment-Education program was

launched in May 2016 to explore the use of en-

tertainment-education and, more generally, how

mass media behavior-change campaigns can be

designed to change perceptions of social norms,

achieve adoption, and sustain healthier behaviors.

A multi-sectorial tool, the knowledge agenda of the

first phase, focuses on edutainment applications to

promote social-norms shifting and behavior change

in reproductive health, gender equality, early years

education, water and hygiene, and violence preven-

tion, including gender-based violence. Thus, it aims

to contribute to the achievement of Sustainable De-

velopment Goal 3 - Good Health and Well-being, 4

- Quality Education, 5 - Gender Equality, 16 - Peace,

Justice, and Strong Institutions, and, more broadly,

of SDGs 1 - No poverty.

Impact evaluation studies being implemented or

planned explore topics like: the use of a Nollywood

movie to promote financial savings among entre-

preneurs (SDG 1); the impacts of the MTV Shuga

drama on risky sexual behavior and gender-based

violence (SDG 3, SDG 5, and SDG 16); the use of so-

cial-norms campaigns to encourage families to en-

roll girls in primary school (SDG 4 and SDG 5); the

relative effectiveness of radio spots versus printed

narratives to promote adoption of solar lanterns in

rural areas (SDG 7); the impacts of including enter-

tainment education in in-school life-skills programs

to reduce bullying and to prevent drug and alco-

hol consumption among young people (SDG 3 and

SDG 16).

This new program is being rolled out in the major

entertainment hubs of the world and is supported

by different Bank units and leading media hous-

es and research centers of edutainment from the

“Hollywoods” of the world, including MTV Staying

Alive Foundation, USC Hollywood Health & Society,

UCLA Global Media Center, Children’s Film Society

of India, the Asian Center for Entertainment Edu-

cation, and the TV networks Televisa (Mexico) and

Rede Globo (Brazil).

The impact evaluations of the first phase also ad-

dress important questions regarding the indirect

or spillover effects of mass media on communi-

ty members that may have heard about the pro-

gram messages through their friends; as well as

the role that social networks have in disseminating

and magnifying potential impacts. Finally, the im-

pact evaluations also study how best to reinforce

edutainment messages through new interactive

technologies, from mobile messaging to social me-

dia outlets to videogames.

DIME “Narrating Behavior Change” Workshop The official launch of the “Narrating Behavior

Change” program took place during a DIME im-

pact-evaluation workshop, jointly conducted with

the Inter-American Development Bank. The event

brought together 22 project teams from Lat-

in America, Sub-Saharan Africa, and South Asia,

and producers and researchers from leading media

organizations and universities to design the next

generation of impact evaluations of entertainment

media and behavior-change campaigns. The work-

shop outlined the evidence base and knowledge pri-


orities and through clinics, allowed project teams to

work with researchers to develop interventions and

evaluation proposals relevant to their projects.

Before-after comparisons show the workshop im-

proved knowledge of participants. In a seven-point

scale, the average score increased from 4.17 to 4.46

or 6.9 percent. As expected, individuals who scored

low during the pre-test, benefited the most from

the workshop: the scores for those that scored un-

der 4 in the baseline test increased from 3.12 to

3.75 or 20.4 percent. Over 90 percent of partic-

ipants reported being satisfied with the technical

content and to have learned what works and what

doesn’t to measure the impact of a program. The

positive feedback was also reflected by positive

emails from participants:

“The workshop was a huge learning curve for

me and I have come back to India enlightened

and feeling super confident about the next

stage of our work here, Vinta Nanda (India), Managing Director, Asian Center for Entertain-

ment Education and CEO, The Third Eye.

“Let the magic that we all witnessed last week

transform the world with all these projects,

ideas, connections, and impact evaluations,”

Lorena Guillé-Laris (Mexico), Director, Cinepolis

Foundation.

“I learned a lot and now I have the challenge to

share all ideas with my colleagues here at Ro-

berto Marinho Foundation. Today, it is so rare

to participate in a meeting of so high a level,

with so many interesting people from differ-

ent places and backgrounds. Congratulations!”

Monica Pinto (Brazil), Development Manager, Ro-

berto Marinho Foundation.

“Thank you for a wonderful gathering of the

most interesting leaders in EE and for pro-

voking stimulating discussions in plenary

and small teams throughout the week. I hope

many new EE collaborations will take place as

a result, and I look forward to working with

you all!” Sandra de Castro Buffington (USA), Di-

rector, UCLA Global Media Center.

Going ForwardDespite being launched a year ago, the Edutain-ment program has generated important knowl-edge in the field of mass-media entertainment.

DIME has or will soon have three published papers of

edu-tainment interventions to i) promote financial

literacy and savings among entrepreneurs in Lagos;

ii) reduce risky sexual behavior and gender-based

violence among youth in Nigeria; and iii) adopt solar

panels in rural Senegal. In addition to studying the

effectiveness of edutainment across sectors, these

advanced evaluations study the effectiveness of

different mass-media outlets (that is, movies, TV

series, radio spots, and printed material). Study re-

sults have been presented in academic, policymak-

er, and producer circles. As mentioned above, the

results have received media coverage beyond devel-

opment outlets.

The program team is raising funds to open a window that will support a new generation of edutainment research required to introduce edutainment into development mainstream. The

window will focus on innovations that can poten-

tially promote and sustain behavior change among

the largest number of individuals. Thematically, the

window would support research projects in the fol-

lowing sub-themes: Sex in the city, Stopping vio-

lence, Empowering men and women, Keeping clean,

and Playful learning. Table 5 provides a list of ongo-

ing and likely edutainment projects. Over the next

two years, the program will use its existing part-

nerships with development partners and leading

media houses to expand its research in the major

entertainment hubs of Brazil, Mexico, Nigeria, and

India, countries with large populations that produce

for their respective regions. This should facilitate

translating research evidence into development and

industry strategies for global impact.


TABLE 4: List of Ongoing and Likely Edu-tainment Projects

Sub-theme Country Region Development ChallengeIntervention Evaluated IE Objective

General behavior change

Nigeria AFR The growth of small enterprises in developing countries is often restricted by poor financial and debt management of entrepreneurs.

Promoting financial literacy and savings of entrepreneurs through the Nollywood movie “Story of Gold” in Lagos.

Evaluate immediate and medium-term effects of movies on financial literacy, savings, and responsible borrowing.

General behavior change

Senegal AFR Many rural communities in Africa lack access to the electric grid. Solar panels provide clean light and energy.

Promoting adoption of solar lanterns through radio spots and printed narratives in rural Senegal.

Contrast the relative effectiveness of radio “narrative” spots and print media in awareness and adoption of solar lanterns.

Sex in the city

Brazil LAC Bahia has one of the highest teenage pregnancy rates in Brazil.

Combining live skills and edu-tainment to reduce teenage pregnancies in Bahia.

Measure the impact of a comprehensive life-skills program and the additional impact of edutainment.

Sex in the city

Kenya AFR Stigma against people living with HIV remains a challenge, contributing to lower HIV testing.

Mixing soccer, social media, and community screenings to reduce stigma and promote HIV testing.

Evaluate impacts of behavior change campaigns during national amateur soccer league on attitudes, HIV testing, and risky sexual behaviors.

Sex in the city; Stopping violence

Nigeria AFR Two million people get infected with HIV every year, disproportionally affecting African youth. Gender violence is prevalent worldwide.

Stopping HIV and gender-based violence through the television drama “MTV Shuga”.

Evaluate impacts of TV dramas on knowledge, attitudes, and behaviors of risky sex and gender-based violence.

Stopping violence

Mexico LAC Mexico has the highest bullying rates in OECD countries. Most bullying takes place in schools.

Stopping bullying in middle schools with edu-tainment and social media approaches.

Measure the programs’ impacts on bystanders self-efficacy to intervene behaviors towards bullying.

Stopping violence

Brazil LAC The government adopted new regulations against sexual harassment, common in public spaces.

Stopping sexual harassment and GBV in transport and public spaces through social-norm campaigns.

Fill evidence gap about how to combine different media outlets to reduce sexual harassment in public spaces.

Empowering men and women

Niger AFR Child marriage and fertility rates in Niger are among the highest in the world, restricting economic and human development.

Stopping child marriages and reducing fertility rates through “husband” clubs and behavior-change radio campaigns.

Fill evidence gap of gender social-norm campaigns targeted at men and the use of radio campaigns to change attitudes and behaviors around fertility.

—continued


TABLE 4: List of Ongoing and Likely Edu-tainment Projects

Sub-theme Country Region Development ChallengeIntervention Evaluated IE Objective


India SAR Many young women have low aspirations and are forced to work under exploitative conditions in spinning mills in Tamil Nadu.

Changing aspirations and stopping bonded labor through community screenings in Tamil Nadu.

Evaluate the programs’ impacts on aspirational alternatives of young women and parents and attitudes towards bonded labor.


Nigeria AFR Eight million girls in northern Nigeria are not in primary school due to social norms against female education.

Getting girls in primary school and reducing child social-norm campaigns and conditional cash transfers (CCT) in northern Nigeria.

Contrast relative effectiveness and social-norms campaigns on school enrollment, retention and completion rates, and child marriage rates.

Keeping clean

Ghana AFR Due to social norms and bullying, many teenage girls don’t go to school during their periods, affecting their self-esteem and educational achievement.

Behavior change campaigns for menstrual hygiene management and educational engagement in middle schools in Accra.

Fill evidence gap on how to maximize water infrastructure investments with behavior change.

Playful learning

Colombia LAC Access to quality school-readiness programs is limited in Colombia and developing countries.

Improving school readiness and healthy habits of young children through digital entertainment in Medellin.

Study the effectiveness of Fun Academy’s digital platform, which brings together teachers, parents, and students through homework and play.


7. Innovation in Data Quality and Monitoring of Policy Influence

7.1 Monitoring System and ResultsMind MyIE is a monitoring system that aims to

give an overview of all i2i and/or DIME-support-

ed impact evaluations (IE). It is a user-friend-

ly cost-effective and low-maintenance data-

base-management system and M&E software

to track status and collect information for each

impact evaluation. It gathers data on all IEs: pro-

file and status, evaluation design, data-collection

aspects, monitoring and quality indicators, coun-

terpart details, influence on programs and policies

and, finally, produced documentation. It provides

over two hundred indicators or characteristics as-

sociated with each IE.

An Innovative PlatformThe MyIE Monitoring System collects over two hun-

dred indicators for each registered IE. In this sense,

it serves as a monitoring and evaluation instrument,

facilitating production of reports for intra-unit (DE-

CIE/DIME) monitoring purposes, inter-unit com-

munication (for example, World Bank Group Global

Practices), and outside reporting such as for donor

results framework.

More than just a reporting tool, the system is unique

in its approach and serves as a knowledge-generat-

ing platform. By collecting data on how IEs feed into

project design, support capacity building, influence

outside projects, and/or motivate scale up or down

of an intervention, the MyIE system was developed

with the objective to better inform policy decisions,

while focusing on research with “impact”. With

this view, the series of produced summary statis-

tics were created with a problem-based approach,

helping to understand the challenges and lessons

learned from the DIME/i2i portfolio. The system’s

outputs can be delivered in various dimensions,

supplying information on levels such as regions,

timeline, budget, evaluation designs, project lifecy-

cles, and themes. Finally, its added value is also ac-

FIGURE 31: Mind MyIE Framework

INPUT

Indicators on:

Evaluation Design

Policy Influence

Capacity Building

MIND MyIESYSTEM

OUTPUT

Reporting System

Knowledge Platform

Personal Dashboard


counted for in its efficiency and transparency gains,

the system delivering automatic reports with just

one click.

Features and ContentThe system asks of Task-Team Leaders (TTLs), or

other authorized respondents, to self-report during

the entire lifecycle of their respective portfolios. The

data is collected annually for all ongoing IEs and

currently translates into 172 i2i and/or DIME IEs,

of which 76 percent are DFID/i2i-funded. The lat-

ter thus encompasses the 131 IEs presented in this

annual report. A complete list of all collected indi-

cators can be found in the Indicators Section. These

are divided into eight sections: IE profile, Evaluation

Design, Data Collection, IE Monitoring and Quality

Indicators, DIME Involvement, Counterpart Details,

IE Influence on Program/Policy, Documentation and

Research Outputs.

Through the IE questionnaire tab (“Manage my Proj-

ects” in the system), users have access to a Reports

Section. It consists of graphs and tables for each of

the following:

n Map of IE distribution across the world

n IEs across GPs and sub-themes

n IEs across regions

n IEs across IDA countries

n IEs by lifecycle

n IEs by duration

n IEs involving gender components or analysis

n IEs in fragile and conflict-affected settings

n IEs and main counterparts

n IE budget distribution

n IE secured funding

n IEs across evaluation methods

n IEs across number of treatment arms

n IE data-collection rounds and response rates

n IEs that have ethical clearance and/or study

registry

n IEs that influence project design or

implementation

n IEs that generate evidence used to support proj-

ect adoption, scale-up, scale-down, continuation,

or cancelation decisions.

n IEs that have contributed to improving the cov-

erage, quality, delivery, output/outcomes, or cost

of the program or other interventions outside

the IE

n Number of previously-ongoing impact evalu-

ations completed and reported (for example,

as working papers or policy briefs published

online)

FIGURE 32: Mind MyIE Phases

Design Phases

CURRENT:

Enables Task Team Leaders to enter a large set of indicators on their IE portfolio

MEDIUM TERM:

Will also serve as a resource platform, giving users access to their personal dashboards

LONG TERM:

Incorporate all the World Bank’s IEs to the system

1 2 3

https://wb-dime-web-01.dc1.rhythmic.net/WorldBankPPM/indicators


Moreover, users currently have the option to enroll

in a MyIE competition, which is divided into three

categories: Best Concept Note Technical Rating,

Most Innovative IE, and Best Pictures.

What’s Next?In the short term, the MyIE monitoring system will

add to the resource aspect of the platform by giv-

ing users access to their own personal dashboards.

This will reflect the same indicators that are in the

Reports Section, but on a portfolio level. The disag-

gregated data will be presented relative to the over-

all average statistics, giving users an indication as

to where they stand in relation to other portfolios.

Additionally, a databank is currently in production.

This will allow users to directly download data from

the system, conditional on the latter being public.

Finally, in the long term, the system will have a

public-access feature, where unregistered users will

be able to obtain aggregated descriptive statistics

and access to documentation for completed evalu-

ations. Moreover, the final objective is for MyIE to

serve as a recording system for all World Bank-re-

lated impact evaluations.

7.2 Review of DataA new i2i data initiative, “DIME Analytics”, pro-

vides data-quality assurance to all i2i impact eval-

uations. DIME has spent the past seven years col-

lecting multi-module household surveys across all

regions, perfecting methods and protocols. Since

2009, DIME has supervised more than 208 sur-

veys.11 In the past year (2016), DECIE staff super-

vised 37 surveys (15 baselines and 22 follow-up

surveys), representing over $7.5 million in cli-

ent-executed funds. However, there is no platform

for sharing knowledge and standardizing protocols

across research teams. Moreover, the knowledge

is staying in-house, failing to reach researchers

and policymakers beyond the DIME team and ac-

ademic and policy partners. To bridge this gap, we

created DIME Analytics, which has the following

objectives:

1. Create a publicly-accessible DIME Wiki to ensure

that protocols, guidelines, and training materials

are available to all and remain up-to-date in a

rapidly-changing field.

2. Develop and implement a Survey Review process

to ensure that all i2i-associated surveys are fol-

lowing established best practices.

3. Advance the knowledge frontier on Survey Meth-

ods by identifying and capitalizing on opportuni-

ties to test new technologies for data collection

and cross-cutting research opportunities.

The DIME Wiki and the Survey Review go hand-in-

hand, as standardizing data quality across the i2i

portfolio requires both better resources for manag-

ing a high-quality survey and an accountability sys-

tem to ensure that best practices are implemented

and final data gets publicized.

DIME Wiki: a public good, targeted at all researchers

and M&E specialists at the World Bank, donor insti-

tutions, NGOs, and governments. It creates unique

value by providing a ‘one-stop shop’ for resources on

all phases of the survey process, from questionnaire

design to publication of the final dataset (see figure

33). It features both content developed by DIME and

links to existing resources.12 The Wiki structure of

interlinked articles perfectly suits the type of body

of knowledge we want to disseminate as there is no

starting point common to all users. It also has the

advantage of being familiar to future users around

the world thanks to Wikipedia. Articles with links

11 Data from the myIE monitoring system, this only refers to surveys associated with the i2i trust fund, and is a lower bound estimate for DECIE as a whole.

12 For example, methodological research from the Living Standards Measurement Survey team at the World Bank or impact evaluation resources from Innovations for Poverty Action or the Abu Jamal Poverty Action Lab.


to detailed further explanations of concepts men-

tioned will make this resource equally accessible to

new and advanced users. Most importantly, unlike

a paper manual, it is a dynamic, searchable, interac-

tive platform with a community of users.

Survey Review: We are currently piloting the review

process for six IEs with surveys planned in the

first half of 2017. The review process is a mecha-

nism to identify common weaknesses, link teams

with existing resources on survey best practices,

and create accountability to follow best practices

and generate the best data. It is not a one-time

review, rather a dynamic and ongoing engagement

with the IE team, with review of key materials trig-

gered at specific milestones in the survey-prepa-

ration process. The review will start a minimum

of four months ahead of intended data collection.

Materials will be approved at each key milestone

in the survey-preparation process: sampling strat-

egy, Terms of Reference for survey firm, survey in-

strument(s), fieldwork supervision, and data-qual-

ity checks.

Survey Methods: The review process will allow us to

identify opportunities for methodological research,

both in terms of testing new technologies for data

collection (for example, satellite data, unmanned

aerial vehicles, crowdsourcing) and rigorously test-

ing options for survey planning and design. Among

the survey resources under development are ‘gold

standard’ modules, based on the most recent meth-

odological research and high-quality programming.

Offering off-the-shelf gold standard survey modules

will enhance data comparability. This opens up new

possibilities for portfolio management, comparing

outcomes across space and time, and re-defining

their development targets and project design based

on solid evidence.

We anticipate that the DIME analytics will bene-

fit the i2i portfolio, but will also create high-value

public goods, with particular benefit for govern-

ment counterparts in the line ministries collabo-

rating on DIME IEs. We experience high demand

for training in data generation and analysis from

government counterparts in line ministries. Our

workshops are an excellent introduction to im-

pact evaluation and research methods, but teams

need further resources to implement high-quali-

ty data collection. Offering survey methods and

resources through an open, interactive platform

with accessible off-the-shelf solutions to the

most common survey design and implementation

roadmap will contribute to filling these gaps in

coverage.

FIGURE 33: Impact Evaluation Data Lifecycle

Sampling Questionnairedesign

Piloting Questionnaire programming

Survey firm procurement Field plan Enumerator training Data quality monitoring

Data management &cleaning

Data analysis Review of analytical code Publication to microdatacatalogue


Off-the-Shelf Solutions for High-Quality Surveys

The survey resources that will be developed by our team will include the following:

Manage High-Quality Surveys: Field Coordinator Training Workshop n Knowledge aggregation process: We have developed a well-regarded and highly in-demand training on

managing successful impact evaluations. Every year, the workshop grows significantly in terms of

both participants and content. The most recent workshop (June 2016) included 30 sessions, 16 of

which were interactive lab sessions, and 50 participants (the course filled to maximum capacity well

before the start). Much of the training is focused on data collection and analysis, and applies to all

IE surveys.

n Impact on quality of future surveys: Participants learn best practices in sampling for project surveys,

procuring survey firms, supervising data collection, and cleaning and analyzing household survey

data. They will be introduced to the Survey Wiki and all survey resources developed by our team.

Best Practice Data Collection Protocols n Knowledge aggregation process: DECIE has supervised more than 208 surveys since 2009, and

Analytics staff have personally been involved with more than 50 surveys at DIME. We will rely on

that experience to document agreed-upon best practice protocols and share with our network of

field coordinators for inputs and comments. The revised version will be a dynamic document, as it

will live on the Wiki, where all users can suggest additions or revisions (in a process moderated by

our team).

n Impact on quality of future surveys (our output): We will create guidelines for each key step of the survey

process, with associated checklists for field coordinators, covering topics such as questionnaire

design, piloting, ethical clearances, survey-firm procurement, high-frequency data checks, fieldwork

supervision, and more. This will provide a learning resource for anyone preparing to collect data, and

eliminate the need for researchers to train every new team member. Checklists will make sure that

even experienced staff do not forget key steps during the busy survey-preparation process.

Research Transparency n Knowledge aggregation process: Pilot a system for internal peer review of data-analysis coding.

Develop a series of trainings on doing reproducible research in collaboration with the Berkeley

Institute for Transparency in the Social Sciences (BITSS). Create standard guidelines for

anonymizing survey data, and set protocol for publishing data in the microdata catalog.

n Impact on quality of future surveys (our output): All peer-reviewed analytical outputs will also undergo

a code review in which all code is re-run to ensure that results replicate. All survey data will be

anonymized and submitted to the microdata catalogue in a timely fashion in accordance with the

newly-set protocols.

—continued


Software Codes and Guidelines n Knowledge aggregation process: Compile requests for coding assistance from DECIE’s network of 50+

field coordinators and research assistants, and organize into a searchable library. Requests currently

come to the data coordinator on the team, who is well placed to categorize common concerns and

develop code snippets with optimal solutions.

n Impact on quality of future surveys (our output): Improve coding from programming of the survey

instrument to data cleaning and analysis, by providing off-the-shelf solutions and a reference library

for programmers. Research assistants follow optimal coding conventions, and have easy-to-access

resources for help with common problems with working with survey data. Coding solutions are

standardized across the team, increasing consistency and transparency.

Gold Standard Survey Modules n Knowledge aggregation process: We will first review a subset of the 200+ existing questionnaires to

identify commonalities, focusing on questionnaires identified as best practice by the implementing

team. Then, we will review the survey design literature for existing research on best practices. We

will summarize findings in a two-page design note for each module, and prepare an example gold-

standard module, programmed to be used in CAPI surveys. Coding will also reflect best practice. The

modules and accompanying design notes will be available to all teams as off-the-shelf solutions,

which they can mix-and-match to create high-quality surveys.

n Impact on quality of future surveys (our output): Teams have access to carefully developed standardized

survey modules. This will improve the quality and consistency of the data, by increasing quality

of survey content (especially in contexts in which extensive piloting is not feasible) and reducing

programming errors. It will also increase the comparability of data across surveys, creating new

research opportunities.


8. Cases of Policy Influence

Introductioni2i works with clients to incorporate a dynamic

agenda that generates lessons during all stages

of the IE. This includes using existing evidence to

guide policy design in real time to incorporate evi-

dence from baseline analysis, adopting tested caus-

al mechanisms, and finally, to making decisions on

scale-up or down of interventions. This type of

influence or advocacy can be critical to delivering

an instrument of sustainable development into the

hands of the policymaker.

i2i has provided support in varying dimensions by:

(i) allowing for a more programmatic approach to

evidence-based policymaking; (ii) capacity building

around a broad set of stakeholders through poli-

cy-relevant research agendas; (iii) expanding the

reach of underdeveloped areas in impact evaluation

(IE) and evidence-based policy. These are all critical

elements to facilitate impact and influence policy

at different stages of a project and the policymak-

ing process.

In particular, i2i has funded a web-based monitor-

ing system (MyIE) that will complement current

advocacy and campaigning efforts by reporting on

periodic progress of IEs through the life of i2i as

more IEs are completed at different phases of their

lifecycle. These indicators showcase the diversity of

influence at all levels and well beyond the typical IE

results and impact.

TABLE 5: Selection of Policy Influence Indicators Collected in Mind MyIE System

Section 7: IE Influence on Program/Policy*

IE actions led to improved counterpart M&E *

IE data requirements led to improvements in monitoring and evaluation (M&E), data collection and/or reporting activities of the counterpart(s).

IE team provided training for data analysis **

IE team delivered skills training to local institutions and/or staff for general monitoring and other data analysis independently (through discussions, technical assistance, workshops, and other training channels).

IE team provided other types of training**

IE team delivered training to local institutions and/or staff on other topics than data analysis.

Rationalized policy design * IE improved design based on clear understanding of the underlying theory of change (causal links between the intervention components and the outcomes) and highlighted areas of uncertainty and critical assumptions.

IE baseline results discussed with clients **

If baseline results were discussed with client (if IE has baseline).

Baseline informed policy design/implementation **

IE baseline data was used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions.

IE final results were discussed with clients ***

IE analysis and results were discussed with the client to understand their policy relevance and application.

—continued


Below we present four case studies that exemplify

how i2i-funded IEs have influenced program design,

supporting regulation, and scale up.

Catalyzing Regulatory Reform in KenyaThe Kenya Patient Safety Impact Evaluation (KeP-

SIE) recently started implementation of the inter-

vention. But, there are already two significant con-

tributions to policy during the pre-implementation

phase, which includes design, baseline, and prepara-

tion for implementation: (1) an enhanced regulatory

framework for health inspections, and (2) strength-

ened institutional capacity through inspection pro-

tocols and an inspection-monitoring system.

KePSIE is a unique partnership between the Ken-

yan Government and the World Bank Group to de-

sign, implement, and evaluate an inspection system

for public and private health facilities in Kenya. This

system aims to improve patient-safety standards

(practices that reduce the probability of preventable

harm to patients and healthcare workers during the

process of healthcare, such as availability of hand

hygiene supplies and new syringes), which consti-

tute a public health problem in Kenya and across

the world.

Estimates suggest that globally, approximately

42.7 million adverse events result every year from

unsafe medical care in inpatient services. Most are

in low and middle-income countries.13 According to

the project’s baseline data, 97 percent of health

facilities in the three KePSIE study counties—Ka-

kamega, Kilifi, and Meru—are non-compliant with

minimum patient-safety standards. To assess

the extent to which governance and accountabili-

ty mechanisms can improve patient safety, KeP-

SIE experimentally allocates all private and public

health facilities in these three counties to one of

three arms: (1) high-intensity inspections with en-

forcement of warnings and sanctions for non-com-

pliant facilities; (2) high-intensity inspections

with enforcement of warnings and sanctions for

non-compliant facilities coupled with public disclo-

sure of inspection results; and (3) “business-as-usu-

al” low-probability inspections (the control group).

Enhanced Regulatory Framework: At the outset

of KePSIE, stakeholders recognized the progress

achieved with previous reforms but identified the

following challenges: (i) unclear and discretionary

rules of the game; (ii) lack of incentives to improve

patient safety at different levels of compliance with

the standards (for example, unclear (virtually inexis-

tent) sanctions and weak enforcement, except for

extreme cases of malpractice); and (iii) inadequate

13 Jha AK, Larizgoitia I, Audera-Lopez C, Prasopa-Plaizier N, Waters H, and Bates DW. The Global Burden of Unsafe Medical Care: Analytic Modelling of Observational Studies. BMJ. 2013; (10): 809-15.

TABLE 5: Selection of Policy Influence Indicators Collected in Mind MyIE System

Section 7: IE Influence on Program/Policy*

Adopted causal mechanism(s) based on IE results ***

IE evidence from experimental testing of alternative mechanisms was used by governments or other stakeholders to determine most effective program alternatives or to inform policy decisions.

IE results used to motivate scale-up/scale-down of policy ***

IE results reported success (or insufficient) impact of the intervention in achieving desired outcomes and were used by governments and/or other agencies/stakeholders to motivate scale-up (scale-down) of policy.

Presentations to (non-client) policymakers and academics of IE results ***

Details on presentations given to policymakers on the IE results.

* applicable after Concept Note review. ** applicable after baseline results are available and discussed. *** applicable after final results are available and discussed


government capacity to inspect and monitor a suf-

ficiently large number of facilities (around 4 percent

of facilities are inspected in a given year). As a re-

sponse to this assessment, KePSIE’s first output

was a new regulatory framework—gazetted under

Cap 242, Legal Notice No. 46 on March 21, 2016—

to conduct inspections in both private and public

facilities (only private facilities were inspected prior

to the reform).

This new framework includes the following ele-

ments: (A) a refined Joint Health-Inspection Check-

list (JHIC); (B) a scoring system that allows facili-

ties to be categorized according to the level of risk

presented to patients; and (C) scores that trigger

warnings and sanctions to be enforced according to

a facility’s level of risk.

Building on previous reform efforts that led to the

first JHIC in 2012, this constitutes one of the most

comprehensive efforts to monitor patient safety in

the region so far. To give some context, of 45 coun-

tries in the Africa region with de jure inspection

regimes, only five (South Africa, Mauritius, Namib-

ia, Equatorial New Guinea, and Seychelles) actual-

ly carry out any type of inspections, and that too

mostly for private health facilities (IFC, 2011).

Strengthened Institutional Capacity: At the out-

set, there was no job description for inspectors, no

training materials or protocols, no monitoring sys-

tems, and no institutional arrangement to link the

Ministry of Health, the regulatory bodies participat-

ing, the three county governments, and the inspec-

tors for an intervention such as the one we were

designing. The team developed all these materials,

standards, and protocols as the basis for the imple-

mentation and the monitoring function, reflected

in an operations manual and a training manual for

inspectors.

Currently, there is no administrative system to

check for inspections results in the country. All are

paper-based records. A system was put in place to

conduct inspections by tablets and manage prog-

ress and monitoring through a web-based monitor-

ing system that reports progress, performance, and

challenges in real time. This customized solution

then leads to the availability of timely and action-

able information to identify challenges in the im-

plementation and enhance accountability to make

mid-course corrections, without the intensive use of

resources, expertise, or equipment, commonly ab-

sent in poor-resource contexts.

Supporting an Adaptive Learning Agenda in Tanzania

As part of the broader engagement with DFID,

DIME supports the testing of an “adaptive learn-

ing” approach to DFID’s Payment by Results (PbR)

program, which started in 2015 and will run up to

2019. The £150 million PbR is DFID’s flagship pro-

gram in Tanzania, aiming to improve rural water

access. One of the key drivers of low water cover-

age in rural Tanzania is a disproportionate focus on

building new schemes, at the cost of maintaining

existing ones. Of the approximate 74,000 rural wa-

ter points in Tanzania, an estimated 46 percent are

non-functional at any point in time. To the extent

that those can be fixed at a reasonable cost, in-

creased efforts on maintenance could dramatically

increase coverage, improving welfare in various ar-

eas ranging from health to agricultural productivity.

PbR’s aim is to shift the focus and resources of gov-

ernment into maintenance, by providing local gov-

ernment authorities (LGAs) with bonus pay-outs to

reward increased functionality at the district level.

As part of this effort, DIME’s role is to provide DFID

with evidence on how to tackle the water mainte-

nance challenge.

In the spirit of adaptive learning, which places the

focus on the outcome rather than the particular in-

tervention modality, the evaluation has a wide re-

mit—it explores the general bottlenecks of that en-

vironment, going beyond estimating PbR’s impact


itself. So far, DIME has provided support in three

ways. First, the team worked with DFID to analyze

the program’s design (for example, incentive struc-

tures, funding flows) and incorporate available evi-

dence into programmatic decisions. Second, DIME

conducted formative research to generate a “thick

description” of the program’s context and challeng-

es. This included both quantitative and qualitative

research. Finally, DIME is currently designing a pro-

cess evaluation and an experimental evaluation to

tackle some of the challenges identified in the for-

mative research.

Although DIME’s research is at an early stage and

the experimental work has not started, the team’s

work has directly and indirectly influenced policy

design and implementation.

Understanding the bottlenecks to water main-tenance: The first year of the partnership (2015-

2016) was dedicated to building a “thick de-

scription”, to better understand the challenges

surrounding rural-water maintenance. DIME as-

sembled a multi-disciplinary team (including econ-

omists, engineers, and an anthropologist) and

conducted a combination of quantitative analy-

ses using secondary data and qualitative research

through an in-depth anthropological study of two

districts. On completion, DIME provided partners

with a set of five notes shedding light on various

drivers of water-point sustainability. Those drivers

included: specific characteristics of water points

(for example, age, extraction method, location); in-

stitutional characteristics (local government staff-

ing, capacity); and political cycles. Some of those

findings are briefly presented here.

Generating continuous feedback to support adaptive learning: A key characteristic of DIME’s

partnership with DFID is the “adaptive learning”

aspect of the PbR. The flexible operational model

means that findings can be incorporated directly

into DFID’s ongoing program, not only into future

ones. After formative research in Year 1, the focus

of Year 2 is to provide DFID with a process evalua-

tion, as the PbR starts disbursing its first payouts.

Project expectations will be compared with actual

implementation, possibly highlighting bottlenecks,

which will be analytically reviewed. The evidence will

be used to update the implementation strategy for

the rest of the program.

Using experimental research to calibrate PbR’s impact through a review of incentives and in-complete contracts: Combining field visits with the

learning generated from the formative research, the

evaluation team, together with DFID and govern-

ment counterparts, identified an important bottle-

neck to PbR’s impact. Tanzania’s Water Supply and

Irrigation Act states that minor repairs of water

infrastructure are the responsibility of communi-

ties, whereas major repairs are the responsibility of

LGAs. However, this distinction is necessarily am-

biguous, and not effectively enforced on the ground.

This creates an incomplete contract between com-

munities and local government, which typically

leads to shirking of responsibilities.

While the government is moving towards full de-

centralization, increasingly emphasizing all-encom-

passing roles for communities, these communities

still require substantial support from government.

Therefore, it is unclear whether LGAs will have the

incentive or institutional structure to translate PbR

payouts into improved functionality. It is suspected

that they may require a more direct approach to

overcoming the inherent problem of incomplete con-

tracts, and the associated vagueness in responsibili-

ties. The agreed impact evaluation will thus aim to

bridge this gap by strengthening the co-production

arrangements between communities and govern-

ment. In particular, the impact evaluation will roll

out a program that brings community water lead-

ers to the LGA offices on a regular basis to receive

training on maintenance-related issues and medi-

ation to resolve existing maintenance concerns.

Perhaps, more importantly, this will create an op-

portunity for government and communities to dis-


cuss maintenance challenges face-to-face, and how

they can collaborate to solve them. The objective

is to help LGAs develop a clearer strategy for how

PbR (and other government) funds could be used to

address sustainability challenges in their commu-

nities. Through this experiment, DIME is combining

state-of-the-art contracts theory and evaluation

methods to produce a tailored solution, which aims

to magnify the impact of a large program on rural

water access.

Using IE to Shift the Policymaking Culture in Nigeria

DIME has worked in the Nigerian health sector since

2007. Beginning with a single evaluation focused

on expanding malaria-related public-health services

using the private sector and community volunteers,

DIME developed collaborative relationships to gen-

erate and use IE evidence to inform policymaking.

This was possible through partnerships with the

Federal Ministry of Health and its agencies, includ-

ing the National Primary Health Care Development

Agency, the National Malaria Elimination Program,

the National Health Insurance Scheme, and the Na-

tional Agency for the Control of AIDS.

These relationships served as the foundation for

a series of IEs with the common objective: to pro-

mote and support innovative thinking and gener-

ate evidence to improve health outcomes in Nige-

ria. In working with our partners, DIME promoted

and implemented IE research to address multiple

constraints in the health sector. At the same time,

DIME used this process to promote a shift in the

policymaking culture towards evidence and re-

sults-informed decisions. With this, we positioned

ourselves as a knowledge broker and provider of

high-quality research services and policy advice.

In 2012, DIME entered into a formal partnership

with the Federal Ministry of Health and the Bill &

Melinda Gates Foundation to use IE in strategic

government programs aiming to reduce maternal

and child mortality. Through this partnership, DIME

conducted IEs on expanding the availability and ac-

cessibility of primary healthcare services, improving

the quality of these services, and motivating front-

line health workers. In a further partnership with the

Gates Foundation, DIME tested whether embedding

culturally relevant messaging in mass media could

alter norms and behaviors related to sexual and re-

productive health and gender-based violence, areas

that are notoriously difficult to change.

DIME IEs include a series of research and analyt-

ical products that provide input at all stages of

policy design and program management. Examples

of such inputs provided through our Nigeria health

work include:

Program design DIME research introduced variations in program design to learn which is most effective, answering questions such as “What is the added value of providing ongoing monitoring and coaching to public primary health facilities—in addition to information on baseline quality—in terms of improving the quality of care?”

Program delivery DIME supported the delivery of routine program elements, such as working to improve the timeliness of payments to frontline health workers (subject to budget constraints).

Documenting experience DIME helped to document the experience of interventions that did not make it to scale (“learning from failure”), such as a conditional cash transfers for maternal health and a community-monitoring intervention to reduce drug stock-outs at primary health facilities. This will help avoid the re-learning of the same lessons in the future.

—continued


This is a snapshot of the type of policy influence

achieved through a sector-wide approach to us-

ing IE to deliver better health outcomes. Achieving

such results across programs and agencies re-

quires staying power to progress beyond capacity

constraints, unpredictable institutional environ-

ments, and external shocks. Notably, this work

provides a bridge for policy continuity: while this

research was largely launched under President

Goodluck Jonathan, discussions on the implica-

tions for future policy are now being held with the

administration of current President Muhammadu

Buhari. This would not have been possible with-

out DIME’s extended engagement in the Nigerian

health sector, and the building over time of a net-

work of persons committed to changing the sta-

tus quo. It is only through long-term investment

that change takes effect. While others may enter

for the quick win, DIME and Nigeria have engaged

for the long haul.

How Partnerships Can Have Sector-Wide Influence in Rwanda DIME’s work with the Government of Rwanda be-

gan in 2011 under the Global Agriculture and Food

Security Program (GAFSP) and is an example of

how governments can take on a sector-wide ap-

proach to impact evaluation (IE). What started as

one GAFSP-financed evaluation, evolved into a large

portfolio of IEs in the agricultural sector, driven by

the keen interest of the Rwandan Ministry of Ag-

riculture and Animal Resources (MINAGRI) to sys-

tematically learn from robust evidence.

This program of IEs contributes to building science

of delivery in a number of areas: investment in

large infrastructure (terracing, irrigation, and feed-

er roads), rural finance, accountability in extension

service delivery, as well as understanding mecha-

nisms for operation and maintenance of rural roads

and irrigation projects. The scale and lifecycle varies

across each of the research areas, and has led to

studying several unique but interlinked questions in

a close partnership where research feeds into pro-

gram design and operations.

For long-term large-scale projects like the Land

Husbandry, Water Harvesting, and Hillside Irriga-

tion (LWH) program, the evaluation aims to for-

mally document the impact of the intervention in

project sites, using as a comparison group similar

pre-identified watersheds that will not receive LWH

project activities. In addition, there is an active col-

laboration between DIME and MINAGRI to strate-

gically test delivery mechanisms on a number of

sub-interventions that aim to directly affect the

results of the broader LWH program.

Capacity building DIME provided IE training to hundreds of policymakers and government technical staff and established the Nigeria Health IE Community of Practice, a forum for exchange of health research and impact evaluation-related capacity building (comprising government, civil society, and the private sector) to stimulate the exchange of ideas and cross-programmatic learning.

Policy debate and design DIME engaged high-level policymakers, including the Minister of State for Health, in a debate on the implications of recent IE findings and provided inputs into the draft National Health Policy (NHP).

Data generation and documentation

DIME IEs have collected a wealth of microdata in a country that, despite its size, has limited rigorous health research, that too in areas that are infamously data poor. This data serves as an input into DIME’s own analyses, but beyond these there is great potential for its use to understand and inform additional aspects of policy design.


One example of this included running a random-

ized control trial (RCT) with 80 self-help groups in

Karongi district in 2012-13 aimed to introduce and

rigorously test two new types of savings products:

a targeted savings account and a commitment

savings account. An important component of LWH

was transitioning from subsistence to commercial

agriculture. DIME worked with the rural-finance

specialist to create and rigorously test innovative

financial products designed to help farmers man-

age their finances to provide for their families and

have money available at the beginning of each sea-

son to purchase agricultural inputs.

The new savings products found promising results

and, on this basis, the LWH team decided to test

similar products at scale and worked with DIME to

implement an RCT with five agricultural cooperatives

in Rwamagana district in 2013-14. However, program

scale up did not translate into similar positive im-

pacts when rolled out beyond the careful manage-

ment of the project team. Based on the evidence of

the lack of impact in the larger program in Rwama-

gana, the LWH team decided not to extend the pro-

gram to other districts. This presents an important

lesson. While many IEs run proof-of-concept evalu-

ations to advise national scale up, testing the effec-

tiveness of a program on a larger scale in a more re-

al-world environment is critical to understanding the

true potential impact of program scale-up.

In addition to rural finance, testing the modalities to

improve extension services are central to LWH’s aims

of capacity-building and technology diffusion. In LWH

project areas, farmers purchase agricultural services

(inputs and extension) from One Acre Fund (OAF).

DIME worked with OAF and MINAGRI to design, intro-

duce, and test innovative farmer-feedback tools. The

government was interested in monitoring the exten-

sion services provided by OAF, and farmer satisfaction

with the service. In addition, the research team was

interested in whether feedback tools could actually

increase demand and improve low participation rates.

Together, the team set up a large field experiment, in

which two types of feedback tools were randomly as-

signed to groups of OAF clients. The team also tested

the cost-effectiveness of different feedback modal-

ities. The most cost-effective feedback mechanism

piloted (a hotline) was adopted and scaled up by OAF

throughout Rwanda the following season. In addition,

the satisfaction data drawn from the feedback tools

themselves helped convince MINAGRI to continue its

partnership with OAF and scale it up to new LWH sites.

Over the course of the partnership between DIME

and MINAGRI, there has been consistent efforts to

build local capacity surrounding the design and de-

ployment of impact evaluations. The DIME team

has led four technical workshops for officials within

the ministry, ranging across IE design and methods

to the use of statistical packages. In addition, par-

ticipating in LWH missions from very early on, the

research team was able to contribute to and inform

specific elements of program delivery, including mo-

tivating the government’s interest in piloting, test-

ing, and scaling up different complementary inter-

ventions—the rural finance experiment in Kayonza

and Rwamagana being prime examples.

Moving forward, through its partnership with local

counterparts on the Rural Feeder Roads project,

DIME is putting together a large-scale data system—

digitizing existing records and working closely with

government stakeholders to set up the infrastructure

to gather information in ways that will alter the way

in which the government seeks to pose policy ques-

tions and deliver relevant results. Finally, by testing

the science of delivery around the sustainable oper-

ation and maintenance of irrigation systems, DIME

and MINAGRI are displaying how the iterative pro-

cess of program implementation, evaluation, and pol-

icy retooling can be leveraged to deliver significantly

improved results. This presents a sector-wide ap-

proach to learning through individual project cycles

and linking the learning across projects to build com-

prehensive evidence-led sector strategies.


9. Communicating Lessons and Results

Strategy

The i2i dissemination strategy last year focused

on raising global, regional, and local country-level

communications coverage of i2i-related activities

among key audiences in the development commu-

nity, particularly country clients, governments, do-

nors and shareholders, researchers, and operational

staff at the World Bank Group and other IFIs.

In parallel, the strategy emphasized outreach via a

variety of existing and new technologies including

the web, social media channels, blogs and other av-

enues more easily accessible and where efficiency

and impact could be magnified.

Stocktaking 2016

Research economists continued to work with proj-

ect clients and government officials to document

evidence during several points of project lifecycles.

Outputs produced at this level include 20 baseline

reports and 16 IE reports. Baseline reports and IE

reports stimulate policy dialogue and support the

adoption of casual mechanisms based on results.

These can influence the scale-up (or scale-down) of

a policy at the national level.

A number of DIME workshops and smaller confer-

ences and events were held during the past year.

As projects and programs mature and produce

results, our outreach focus has shifted. Where, in

previous years, a lot of emphasis was on launch-

ing new programs, for example through our large-

scale global workshops, DIME has expanded this to

sharing evidence generated at different stages of

projects. This is done both by bringing together de-

velopment practitioners, academics, and donors in

conferences, as well as in country events with local

policymakers at different levels.

A number of DIME workshops and smaller confer-

ences and events were also held during the past

year. This included (1) a policymaker summit in

Nigeria summarizing and presenting several years’

work in the health sector of the country, (2) an im-

pact-evaluation workshop in Mexico on Entertain-

ment Education aligning researchers, practitioners,

and producers from entertainment hubs including

Hollywood, Nollywood, and Bollywood to design and

evaluate behavior-change campaigns under a new

DIME program, (3) an evidence day in Kenya bring-

ing together all IE work in the country, and final-

ly, (4) a number of workshops and conferences in

Washington, DC in various thematic areas including

child labor and agriculture. In all, 25 DIME seminars,

workshops, or trainings took place over the course

of the year.

The team also continued to garner wider policy

outreach through (a) policy briefs, (b) World Bank

working papers, and (c) journal submissions. This

past year (March 2016 to February 2017), seven

working papers were published under the catego-

ry of impact evaluation at the World Bank Group.

The team produced 26 policy briefs and seven blog

posts, providing an opportunity for researchers to

proffer advice or solutions stemming from research.


Below are several examples of how dissemination

takes place in practice at different stages of a proj-

ect or program.

Narrating Behavior ChangeThe Entertainment Education Program component

of DIME highlights how the World Bank can use its

comparative advantage in research and convening

power to bring a variety of stakeholders to intro-

duce and mainstream a development practice. Two

World Bank Flagship publications (the 2015 and

2016 World Development Reports) highlighted the

untapped potential of entertainment education and

mass media in development practice. In 2016, DIME

launched a research program on entertainment

education. The program is supported by leading

researchers, development partners—for example,

Gates, DFID, and IADB—and global media power-

houses—for example, MTV Staying Alive Founda-

tion, USC Hollywood Health and Society, UCLA

Global Media Center, Rede Globo, Brazil, and Third

Eye, India. It uses DIME’s existing partnerships with

traditional and non-traditional partners to expand

the research agenda in the entertainment hubs of

Brazil, Mexico, Nigeria, and India. The generated

evidence base should support the systematic and

effective scale up of edutainment in development,

from radio to TV to videogames to mobile apps. The

Bank’s External and Corporate Relations (ECR) unit

supported dissemination activities of the new DIME

program. We received media coverage from glob-

al media outlets such as TED talks, Bloomberg TV,

MTV, and The Hollywood Reporter, among others.

To this end the unit held an inaugural DIME work-

shop in Entertainment Education in Mexico City. The

new program brought together 22 project teams

from Latin America, Sub-Saharan Africa, and South

Asia with producers and researchers from leading

media organizations and universities to design the

next generation of entertainment research. Further

outreach ensued from an MTV Shuga launch with

the MTV Staying Alive Foundation in New York and

also through individual outreach efforts conducted

by the program leader at various venues including

USAID events, Morelia Film Festival, and the Rocke-

feller Foundation.

Educating the Masses in Financial Literacy in BrazilBetween 2010 and 2011, DIME supported the larg-

est experimental evaluation of a financial-literacy

program for high school students. The pilot program

was sponsored by the Brazilian Committee of Fi-

nancial Education (CONEF) and took place in six

Brazilian states and covered approximately 900

schools and 20,000 high-school students. The in-

tervention was intensive and generated positive

results on learning, attitudes, and behavioral out-

comes. The results of the IE were disseminated in

policy talks and academic seminars, and were cov-

ered extensively by the main newspapers and mag-

azines in the country.

This evaluation had important policy implications as

the Brazilian Ministry of Education decided to scale

up the program nationally. Based on this successful

experience, CONEF decided to test a new pilot tar-

geting primary-school students under the assump-

tion that impacts should be stronger, the earlier

students are introduced to financial concepts. The

The Entertainment Education Program component of DIME

highlights how the World Bank can use its comparative

advantage in research and convening power.


new pilot was implemented in 2015 in two states,

Santa Catarina in the south and Manaus in the

north. The pilot included 100 schools and covered

about 18,000 students aged six through 15.

This is the largest RCT of a pilot program targeting

this age group and the first conducted in a develop-

ing country to find positive effects of a financial ed-

ucation program on student learning and attitudes.

The results were disseminated in an event in May

2016 that brought together donors, academics, and

policymakers and were covered by two of the main

newspapers in the country.

Mozambique: Localizing Knowledge and Globalizing Outreach Promoting sustainable irrigation and drainage is es-

sential for smallholder farmers to be resilient to the

intensifying climate variations. DIME has developed

a long-term research agenda aimed at understand-

ing (1) how to best leverage irrigation investments

to increase resilience of small farmers and (2) how

to build local institutions to ensure sustainability of

these schemes. In Mozambique, the team developed

a series of studies aimed at shedding light on these

critical research questions.

The team is involved with several stakeholders in

the Mozambique irrigation sector and, since incep-

tion of the project, has sought to maximize learning

across projects and stakeholders. Extensive moni-

toring systems have been set up with government

counterparts and, last year, comprehensive agricul-

ture baseline surveys were collected for both proj-

ects. The baseline results provided useful insights

into the way the beneficiaries are selected as well

as potential ways to improve water efficiency by

altering watering practices.

In November 2016 DIME organized a brown bag-

lunch event in the World Bank country office, bring-

ing together representatives from the Ministry of

Agriculture and Food Security, including regional

representatives, Ministry of Land, Environment, and

Rural Development, the National Irrigation Insti-

tute, World Bank, AfDB Operations, and DFID. The

session covered baseline results data systems em-

ployed for each project, challenges, lessons learned,

and next steps.

The results were shared with other irrigation-re-

lated projects in the global agriculture portfolio at

the World Bank during a recent DIME Agriculture

conference in Washington, DC as well as at Johns

Hopkins University. A working paper and policy brief

results are currently being drafted.

Kenya Evidence DayThe World Bank’s active lending support to Ken-

ya amounts to around $6.8 billion, of which, around

$700 million (or 12 percent) is associated with im-

pact evaluations. Despite the fact that Kenya is

one of the largest producers of impact evaluations

in the world, evidence from the country often does

not reach intended audiences. To this end, the DIME

group brought its convening power and organized

a one-day event bringing together policymakers,

practitioners, and researchers for a high-level dis-

cussion on how evidence can be used to generate

relevant insights from conception to completion of

projects. The day was also utilized to showcase a

series of ongoing impact evaluations that provide

examples of how to do this in practice.

The workshop focused on how IE can and has been

used to improve government programs along two

dimensions: (1) strengthening accountability sys-

tems to improve service delivery; and (2) maximiz-

ing the impact of large-scale investments in public

infrastructure.

The event included a combination of (i) presenta-

tions of the latest evidence and learnings from six

ongoing impact evaluations in the country; and (ii)

panel discussions on what more can be done to im-

prove accountability systems and maximize pub-

lic-infrastructure returns. The event led to a stra-

tegic approach to combining fragmented research

and a set of comprehensive lessons to inform activ-

ities across operations, research, and policy.

Reconciliation in Sierra Leone

A recent paper on reconciliation in Sierra Leone ti-

tled “Reconciling after civil conflict increases social

capital but decreases individual well-being” was ini-

tially published in Science Magazine, a partnership

effort led by Innovations for Poverty Action. This

highlights how a topic of global interest can quickly

and easily find extensive dissemination coverage at

all levels of outreach.

The article created a media blitz and was men-

tioned significantly in a number of news outlets

around the world including in the Washington Post,

Le Monde, and Reuters. The paper also had heavy

social-media outreach and subsequent mentions by

policymakers, researchers, and journalists.

http://science.sciencemag.org/content/352/6287/787

Impact Evaluation to Development Impact - World...

Documents

Transcript of Impact Evaluation to Development Impact - World...