Www.3ieimpact.org Howard White What should be in a systematic review? Howard White, 3ie...

www.3ieimpact.orgHoward White

What should be in a systematic review?

Howard White, 3ie

International Initiative for Impact Evaluation

Campbell ColloquiumChicago May 2013


Effectiveness matters

30-50% reduction in mortality


Source: Gelb and Clark, ‘The Biometrics Revolution’, CGD, 2013


Major policy shifts:

•Private sector involvement•Poverty focus•Pro-environment

Total around US$3 billion

Just 7 per cent of a total of 120 RE and energy projects mentioned poverty in their objectives


Rural electrification design

• 82% build infrastructure and 85% have institutional development

• Tariff conditionality: 1/3 projects prior 1995, 14% thereafter

• One quarter address rehabilitation or system losses

• Few have explicitly pro-poor components (only two subsidize connection charge)


RET & Off grid electrification

• Rapid growth: only 2 off-grid projects before 1995 and 31 after

• Most (75%) include photovoltaic, usually solar home systems

• Others are micro-hydro (34%) and wind power (31%)


Focus of Farmer Field Schools


Causal chain evidence matrix

FFS 1

FFS 2

The rows are by intervention, not study

Links in the causal chain


Evidence matrix: targeting FFS

Design Outcomes Explan-atory factors

Target groups

Targeting mechanism

Who participates?

Who is excluded?

Drop-outs

FFS 1

FFS 2

Does targeting work?

Who is targeted?

How?

Who participates? Who doesn’t

(and why?)


Evidence matrix: vertical synthesis


Target groups

Tgtg mchnsm

Who participates?

Who is excluded?

Drop-outs

FFS 1

FFS 2

…

FFS n

Summary

Vertical synthesis


Vertical integration: target groups


Vertical synthesis: Targeting mechanisms

10

1

2

20

Categorical 45

Individual/household assessment 26

Self-selection 18



Narrative summaryTarget

groupsTgtg mchnsm

Who … Drop-outs

FFS 1

FFS 2

…

FFS n

Summary Horizontal synthesis

Evidence matrix: horizontal synthesis


Design Outcomes Narrative summary

Target groups

Tgtg mchnsm

Whoparticipates

Indonesia Existing groupAccess to irrigationLiterateHigh social standing

Chosen by community and extension workers

Better offBetter educatedLarger farms

Local selection succeeded in targeting better off, educated farmers

Horizontal synthesis 1


Horizontal synthesis 2

Design Outcomes Explanatory factors

Narrative summary

Target groups

Tgtg mchnsm

Whoparticipates

Liberia 50-50 men and women

Categorical (women)Community based

Mostly men Women•Lacked tools and land•Lack of time and no childcare •Need husband’s permission

Targeting failed to reach women as they did not have the requisites for participation, notably time, land and tools



Narrative summaryTarget

groupsTgtg mchnsm

Who … Drop-outs

FFS 1

FFS 2

…

FFS n

Summary

Overall synthesis

Evidence matrix: overall synthesis

Targeting has worked where it has targeted the farmers who are amongst the better off (as more educated, social leaders, more land, right crop etc.) but been less successful – though there are exceptions – when targeting the poor and women who don’t have these attributes



Narrative summary… …. … ….

FFS 1

FFS 2

FFS 3

Sub-group 1 summary

FFS 4

FFS 5

FFS 6

…

FFS n

Sub-group 2 summary

Sub-group synthesis

Evidence matrix: overall synthesis


So what goes in a systematic review?

• Design and importance: Define intervention, its dimensions and design (global portfolio review)

• Implementation experience

• Beneficiary knowledge, perceptions and participation (access/use)

• Outputs produced

• Intermediate and final outcomes (effectiveness)


Rural electrification: outputs

• Bangladesh: rural connections near 0 in early 80s to more than 4 million in 2002 of which Bank 600,000 (15%)

• Indonesia: coverage rose 33 to 85% from 1993 to 2003, 11 million new h/h, of which 10 million under First and Second Indonesia RE projects (1/2 paid

for by Bank)• Laos: 1993-2004 coverage from

14-45%


And so what about effectiveness?

Grid ERR=28% Off-gird ERR=21%


Conclusions

• Systematic reviewing is a method in which the approach should be systematic, but that need not be restricted to effectiveness

• By addressing a broader range of policy questions SRs will be of interest to a broader range of policy makers

• … And more interesting to read

Thank you. Go forth and influence policy

Www.3ieimpact.org Howard White What should be in a systematic review? Howard White, 3ie...

Documents

Transcript of Www.3ieimpact.org Howard White What should be in a systematic review? Howard White, 3ie...