$

RESEARCH

The development

impact of impact

evaluation

Arianna Legovini

Development Impact Evaluation

World Bank, Tokyo, December 5, 2014

2

Get the delivery right

Subsidy for malaria treatment $350 million

Sealing packages saved 16,600 under five +

2,200 adult lives in 5 years

Problem 83% reduction in stockouts

3

Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M.

4

Get people to do it Source: Goldstein, Thirumurthy, and Zivin

HIV treatment $ 2.1 billion

Reminders increase

adherence to treatment by 35%

Problem IE scalable solution

5

6

does IE make a difference?

1. Impact of impact evaluation on World Bank project implementation

– Rigorous econometric analysis

2. Theory of how IE influences policy decisions

– Monitoring indicators

3. Institutional citations of impact evaluation

– Descriptive trends in Bank documents

4. Is the cost worth the benefit?

– Basic cost-benefit analysis

7

IMPACT EVALUATION THAT DELIVERS

Legovini, Piza, Di Maro (2014)

8

It does!

…when the IE starts

IE increases disbursements

IE helps projects stick to plans

9

Does IE affect project implementation? 1000 World Bank projects - 101 impact evaluations

(2005-2011)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

IE=1 & IE_quarter==0 IE=1 & IE_quarter==1 IE=0

Disbursements increase after IE starts

10

IE increases disbursements (proportion of total commitments)

statistically significant at 1% level

14.2

19.4

0

5

10

15

20

25

Cumulative quarterly disbursement without IE Cumulative quarterly disbursement with IE

without IE with IE

37%

11

Plan to actual disbursement gap: IE decreases delays in implementation

12

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

IE=1 & IE_quarter==1 IE=0

without IE

with IE

A THEORY OF CHANGE WHAT DO MONITORING INDICATORS SAY? LEGOVINI, BEDOYA AND DIME TEAM 2014

13

Why? Clear roadmap

Clearer roadmap

What

To whom When

Where

14

Why? Better capacity

Better capacity

Research team

Full time field coordinator

Training

More and better data

15

dime results indicators

88%

74%

75%

36%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Client participated in IE workshop(s)

Baseline results discussed with client

IE results discussed with client

Training provided for data analysis

Capacity Building

16

dime workshops increase participants’ knowledge

(pre/post test)

dime workshops increase quality of proposals

FCS window, July 2014 - double blind external evaluation

dime results indicators

84%

85%

48%

50%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

High-quality baseline survey

High-quality follow-up survey(s)

Improved administrative data

Availability of Data as a Public Good

Quality of Data

19

Why? More effort into implementation

Motivation to perform

Goal: this is what we are

trying to achieve

Control: this is how we want

to do it

…and BTW somebody

measures (and cares)

20

Why? Better decisions 21

dime results indicators

36%

23%

30%

0% 5% 10% 15% 20% 25% 30% 35% 40%

Baseline informed policydesign/implementation

Adopted delivery or behavioral mechanismbased on IE results

IE results were used to motivate scale-up/scale-down of policy at national level

Quality of Policy Decisions

22

DOES ANYONE CARE? TRENDS IN INSTITUTIONAL CITATIONS OF IE

23

Source: WB Archives, author’s calculations Note: 1. IE citations include “IE” and “Impact Evaluation” 2. The statistics include 158 types of documents at WB, such as Country Assistance Strategy Document, Project Appraisal Document, Implementation Status and Results Report and Working Paper.

IE Citations multiplied by 6

3,691

21,271

0

5,000

10,000

15,000

20,000

25,000

05 06 07 08 09 10 11 12

Tota

l IE

Cit

atio

ns

Fiscal Year

All Documents 2 per. Mov. Avg. (All Documents)

24

1,092

7,684

0

2,000

4,000

6,000

8,000

10,000

05 06 07 08 09 10 11 12

Tota

l IE

Cit

atio

ns

Fiscal Year

Source: WB Archives, author’s calculations Note: The statistics include several types of documents, such as Working Papers and other Research Documents.

Research documents

25

Operational documents

400

2,773

0

500

1,000

1,500

2,000

2,500

3,000

05 06 07 08 09 10 11 12

Tota

l IE

Cit

atio

ns

Fiscal Year

Project Appraisal Documents

25

181

0

20

40

60

80

100

120

140

160

180

200

05 06 07 08 09 10 11 12

Tota

l IE

Cit

atio

ns

Fiscal Year

Country Assistance Strategies

Source: WB Archives, author’s calculations 26

433

3,367

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

05 06 07 08 09 10 11 12

Tota

l IE

Cit

atio

ns

Fiscal Year

Source: WB Archives, author’s calculations Note: The statistics include 3 types of documents: Board Report, Implementation Status and Results Report, and Implementation Completion and Results Report.

Project completion documents

27

AT WHAT COST? SOME SIMPLE COST-BENEFIT OF IMPACT EVALUATION

28

Is the cost worth the benefit?

• Zambia best supply chain – IE solution: 83% reduction in stock out

21-25% reduction in malaria mortality

– Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2

– Scaled up: Benefit ($60.7M) / Cost ($12.4M) = 4.9

• Malawi commitment saving account – IE solution: 22% increase in farmer production value

– Pilot: Benefit ($0.7M) / Cost ($0.5M) = 1.4

– If scaled: Benefit ($15.6M) / Cost ($1.7M) = 9.2

29

What does dime do?

Transform development policy

30

1. Run experiments to inform decisions

2. Build agencies’ capacity to do this systematically

3. Draw lessons and share them face-to-face to global audiences

31

dime in action

Inform policy design

Guide mid-course

corrections

Inform adoption and

scale-up

Syst

em

atic

use

of

evid

ence

Train & apply

Learn by doing

Apply knowledge

Capacity

build

ing

IE Product

IE DESIGN

IE IMPLEMENTATION

IE DISSEMINATION

175 IEs in 47 Countries Distribution of dime ’s IEs by GP and CCSA

(number of IEs)

32

32

18

4

2

11

15

14

25

5

21

16

8

4

0 5 10 15 20 25 30 35

Agriculture

Education

Energy & Extractives

Environment & Natural Resources

Finance & Markets

Fragility & Conflict

Governance

Health, Nutition & Population

Social Protection & Labor

Social, Urban, Rural & Resilience

Trade & Competitiveness

Transport & ICT

Water

Policy questions in dime’s portfolio

(% of questions)

33

3%

7%

9%

12%

20%

21%

28%

0% 5% 10% 15% 20% 25% 30%

Top-downaccountability

Demand-sideaccountability

Behavioral biases

Incentives

Constraints

Delivery mechanisms

Packages

• IE has real impacts on developmental effectiveness and project delivery

• High return on investment

• Spillover effects

• i2i opportunity to increase scale and scope

• Open invitation to donors and governments

dime & i2i

34

partners

CRIME LAB

35

The strategic framework for learning and accountability

•Lesson 1: Base development policy decisions on evidence

•Lesson 2: Make learning part of the culture of development co-operation

•Lesson 3: Define a clear role for evaluation

Delivering evaluations effectively

•Lesson 4: Match ambitions with adequate resources

•Lesson 5: Strengthen program design and management systems

•Lesson 6: Ask the right questions and be realistic about expected results

•Lesson 7: Choose the right evaluation tools

•Lesson 8: Work together

•Lesson 9: Help strengthen partner country capacities and use them

Strengthening the culture of learning

•Lesson 10: Act on evaluation findings

•Lesson 11: Communicate evaluation results effectively

•Lesson 12: Evaluate the evaluators

Shared Objectives Evaluating Development Activities, 12 Lessons from the OECD DAC

(OECD 2013)

Human Development

• Education • Health • Social Protection

i2i

pillars and thematic areas

36

Shared Prosperity

• Finance, Private Sector and Jobs

• Agriculture • Infrastructure

Governance

• Public Sector Governance

• Justice • Local Development

Climate Change

• Energy • Environment • Natural Resource

Management • Agriculture • Transport

Fragility & Conflict

• Reintegration • Governance • Gender-based

Violence • Urban crime &

violence

Gender

• Human Capital • Economic

Opportunities • Voice/Agency

5

1 2 3

6 4

Total ODA, Fast Growing Sectors 2006-2012 (USD billion)

37

5.1 5.8

1.9

7.2

5.1

1.6

4.7

9.3 10.3

3.5

16.5

13.9

4.9

11.5

Pop. &Reproductive

Health

Water Supply &Sanitation

Conflict, Peace &Security

Transport &Storage

Energy Banking & FinancialServices

Agriculture,Forestry, Fishing

2006 2012

Source: www.oecd.org/dac/stats

38

Impact Evaluation 4 Peace: Evidence to lower violence and conflict

• In DRC: can short-term employment reduce youth participation in violent and armed activities?

Prioritized themes:

• Urban crime and violence

• Jobs and productive opportunities for youth at risk

• Sexual and gender-based violence

• Public sector /civil service reform

• In Mexico: can behavioral therapy, intensive mentoring and jobs reduce recidivism of youth incarcerated for serious crimes?

• In India: Which of three different strategies can help shift norms to prevent sexual and gender-based violence?

How do we supply reliable, clean energy to support sustainable growth?

Challenge Testing Solutions

Reduce transmission losses

Consume responsibly

Increase reliability

Slow down deforestation

Limit overgrazing and desertification

Mainstream climate smart technologies

Ensure sustainability of infrastructure

Institutions for O&M

Demand-side management

Infrastructure prioritization

Property rights/incentives/

governance

Capacity building and networking

Governance of irrigation

How do we align private, social and environmental objectives to sustainably manage natural resources?

Impact Evaluation in Energy and Environment

Challenge Testing Solutions

ieGovern: Governance & Justice Rigorous evidence on what works in

governance is in short supply… The World Bank has a comparative advantage

for leading IE research in the sector.

79

87

98

103

267

299

380

571

1,551

0 500 1,000 1,500 2,000

Transport & ICT

Governance

Environment & Natural Resources

Water

Urban, Rural, and Social Development

Finance & Markets / Trade & Comp.

Social Protection & Labor

Education

Health, Nutrition, & Population

Number of Impact Evaluations

Glo

bal

Pra

ctic

e

Governance IEs represent less than 3% of globally registered research studies

Source: 3ie Impact Evaluation registry.

Largest Public Sector Modernization lending portfolio among donors

Strong managerial support for IE

2015201 2014: Outreach and thematic expansion 2013: 10 IEs designed and launched in Africa Region

May 2014: 2nd Steering Group meeting

Chairs: Arianna Legovini (DIME) and Bill Dorotinsky

(GGP)

January 2015: Global ieGovern workshop

in Istanbul

Civil Service Reform Justice

Subnational PSM / Service Delivery

New Themes • Citizen

Engagement • Decentralization • Procurement • Public Financial

Management • Tax Admin.

Countries • Colombia • Ethiopia • India • Jamaica • Kazakhstan • Pakistan • Vietnam

- Design interventions to mainstream gender into operations - Run RCTs to measure results - Inform course corrections - Build knowledge across DIME’s knowledge areas.

Gender and Impact Evaluation

Financial Empowerment & Women

Gender Empowerment to Combat Domestic Violence

Addressing Human Capital Gender Gaps through the Lifecycle

Maternal & Child Health Initiative

Malaria Control Booster Project

our gender impact

• build women’s empowerment and agency – women can reduce informational constraints to adoption (Malawi and

Mozambique) – clarifying land rights can be especially productive for women farmers

(Benin) – gender-tailored saving products lift investment constraints (Rwanda)

• address human capital gender gaps through the lifecycle – reproductive health (Nigeria) – vocational training to ease the school-to-work transition (Malawi)

• combat domestic violence – Shifting social norms (India) – Address reproductive coercion through health programs (Mexico)

More cooperation in Operations

In 2007-2010, multi-bilateral ODA jumped from $9 billion to $16.7 billion

Limited Evaluation Resources

• The central evaluation unit of DAC member country has a budget of USD 2.4 million, or 0.16% of that country ODA.

• The evaluation budget of multilateral institutions is USD 10 million, a tiny fraction of lending.

Call for Collaboration

• The 2010 OECD report notes new demands on DAC evaluation units for rigorous impact evaluations, working in a more collaborative way with country partners, and synthesizing findings to report on development results

• i2i offers a framework for collaboration to exploit economies of scale in learning and knowledge sharing

44

Increasing Cooperation in Evaluation: small budgets, large task, economies of scale

Discussion

• In 2013, net official development assistance reached a record of US$138 billion.

• Which development interventions are effective and worth scaling up? Which promising mechanisms are worth testing across different sectors? Impact Evaluations provide guidance.

• The i2i platform can facilitate the rapid expansion of IEs, especially in under-evaluated sectors.

$

RESEARCH

Thank You!

facebook.com/IEKnow

#impacteval #ieknow_dime

blogs.worldbank.org/impactevaluations

microdata.worldbank.org/index.php/catalog/impact_evaluation