Saving Moms and Babies;What Does the Impact Evaluation Evidence Show?

Jeffery C Tanner, Team Leaderjtanner@worldbank.org

1. Introduction to Systematic Reviews2. SR on Maternal & Child Mortality3. Results4. Knowledge Gaps5. Summing Up

Evidence-Based Decision-Making

the contribution of Systematic Reviews► “Sum up the best available research on a

specific question” (The Campbell Collaboration)

► Make strong claims on comprehensiveness of search

► Are a form of research• Unit of Analysis: Secondary observations

(Studies)• Follow basic steps of research process• Aim to minimize bias and error

3

• Sheer amount and flow of information/ research

• Variable quality of research outputs

• Need to ‘separate the wheat from the chaff’

• Problems of publication bias

• Limitations of single studies

Why do we need systematic reviews?

2. SR on Maternal & Child Mortality

MDGs 4 and 5 Continue to Lag

► MDG5: ¾ Reduction in Maternal Mortality by 2015• Main Indicator: Proportion of births attended by skilled health

personnel

► MDG4: 2/3 Reduction in Under-Five Mortality► Knowing what to do is no longer the problem; knowing

how to do it remains a challenge

MD

G 1

.a E

xtre

me

pove

rty

(%

popu

latio

n be

low

$1.

25 a

day

, 20

05 P

PP

)

MD

G 2

.1 P

rim

ary

com

plet

ion

rate

(%

rel

evan

t age

gro

up)

MD

G 3

.1 R

atio

of

girl

s to

boy

s in

pr

imar

y an

d se

cond

ary

educ

atio

n (%

)

MD

G 4

.a M

orta

lity

rate

, inf

ants

(p

er 1

,000

live

bir

ths)

MD

G 4

.a M

orta

lity

rate

, chi

ldre

n un

der

5 (p

er 1

,000

)

MD

G 5

.a M

ater

nal m

orta

lity

ratio

(m

odel

ed e

stim

ate,

per

100

,000

live

bi

rths

)

MD

G 7

.c I

mpr

oved

wat

er s

ourc

e (%

pop

ulat

ion

with

out a

cces

s)

MD

G 7

.c I

mpr

oved

san

itatio

n fa

cil

-iti

es (

% p

opup

latio

n w

ithou

t acc

ess)0

20406080

100 7894 96

80 8068

87 87100

87 96

51 5238

100

72

Corresponding target Latest available value

Objective of this Systematic Review► Scope: Reviews impact evaluations of interventions

to improve five MCH outcomes (SBA, MM, NM, IM, U5M) and those of SBA as an intervention from scalable programs in IDA/IBRD countries

► Outcome-oriented approach: Include full range of interventions

► Aims to answer the following questions:What interventions demonstrate reductions in

maternal and child mortality and increase skilled birth attendance?

What do we know about the effects of increasing skilled birth attendance?

What important knowledge gaps remain on interventions to reduce maternal and child mortality?

Frequency of Impact Evaluations by Outcome and Quality

► AAA-quality Impact Evaluations: Established Causality• Few, if any, remaining threats to internal validity.

► AA-quality Impact Evaluations: Likely Causality• Some identifying assumptions untested or unclear

► Consistency Analysis. Key Results driven by AAA-rated IES

► External Validity, Construct Validity also considered

Skilled birth attendant

Neonatal mortality

Infant mortality

Under-5 mortality

Maternal mortality

10

9

11

5

3

23

17

12

12

5

33

8

26

23

17

AAA rating AA ratingTotal

68 Impact Evaluations, 62 Studies

3. Results

Standardized Effect Size: SBA—Outcome

Results: Increasing Skilled Birth Attendance—Outcome

► Skilled Birth Attendance rates can be improved through • Conditional Cash Transfers and Vouchers• Interventions that bundle quality

improvements with increased accessibility

► Solely training health workforce or increasing awareness of safe motherhood was not observed to yield significant results on SBA rates.

► Where reported, effects are larger for more disadvantaged households

Results: Skilled Birth Attendance – Intervention

► No Robust Evidence that solely increasing proportion of births with SBA affects mortality• Only evaluated program is JSY in India: Null

results for NM– 2 IEs, AAA and AA quality, both high-powered– No effect even in areas with high (or low) quality of

health services

• Critical Knowledge gap: Need more IEs on this MDG indicator

► SBA “+” Can affect mortality and intermediate outcomes• PLUS=Provision & Utilization: quality of care, knowledge,

access• But evidence is mixed across outcomes, even within a given

study• Unclear what explains variation in results• Consistent, if thin, evidence on better U5M, Breastfeeding,

Family planning, Postnatal visits, Immunization, Anthropometric outcomes

Intermediate Outcomes of SBA as an Intervention► SBA PLUS = Provision & Utilization► Quality of care, knowledge, access► Consistent, if thin, evidence that SBA+

results in better • U5M• Breastfeeding• Family planning• Postnatal visits• Immunization• Anthropometric outcomes

Results: Maternal Mortality

► Few (8) studies exist, concentrated in SA (5)

► Most studies underpowered to detect effects in MM

► Interventions bundling components of both health care provision and utilization can reduce maternal mortality. • Specifically, bundling health worker training

and mothers’ knowledge and information (with and without insurance)

► More evaluations are needed• 3 Delay Model, especially transport and

referral systems• Family planning, universal health

Results: Neonatal Mortality

► Health: Knowledge & Information interventions in the sample which change home-based care practices at the community level reduced mortality

► Non-health: Interventions in non-health sectors associated with maternal education consistently lowered neonatal mortality

► More IEs are needed in• 3 Delay Model, esp. Transportation and

Referral Systems• Improvements in Quality and Availability of

Health Infrastructure for newborns

Infant Mortality

► Interventions in non-health sectors consistently reduced IM• Water and Sanitation• Energy• Education

► Governance interventions report significant effects in lowering infant mortality

► Training health workers to provide continuum of care services within communities can reduce IM

► Where reported, households from lower SES benefited more

Under-Five Mortality

► Interventions in non-health sectors consistently report large reductions in under-five mortality.

► Public Participation, Service Packages may reduce U5M

► Insecticide Treated Nets are only intervention targeting three main causes of mortality that has IE evidence on U5M

4. Knowledge Gaps

Gaps by Region

15 IEs3 SBA1 NM9 IM7 U5

9 IEs5 SBA3 NM3 U5

0 IEs

1 IE1 MM1 IM

28 IEs15 SBA5 MM18 NM6 IM4 U5

15 IEs10 SBA2 MM4 NM7 IM3 U5

Gaps by Severity

[98] [94] [93][84]

[49] [49]

[32]

[83] [81] [80]

[220]

[500]

[8][11] [10]

[14]

[32][34]

[14][17] [16]

[24]

[49]

[69]

[17] [20] [20][29]

[63]

[108]

0

10

20

30

40

50

60

70

80

90

100

0

5

10

15

20

25

30

35

40

ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA

Seve

rity I

ndex

(100

=wor

se in

dicato

r valu

e)

Impa

ct Ev

aluati

ons

Impact Evaluations Severity

Skilled birth attendant Maternal mortality Neonatal mortality Infant mortality Under 5 mortality

Other Gaps in Impact Evaluation Evidence

► By Outcome• Maternal Mortality: limited number of studies

but highly concentrated in South Asia (5 out of 8)

► By Intervention Type• 3 Delay Model (especially transport and

referral systems)• Governance• Health information systems, infrastructure,

financing• Income generating / Labor market

interventions• Transportation infrastructure

External Validity Implications: Beneficial Impacts are more likely in problematic areas

Skilled Birth Attendance

Infant Mortality

Neonatal Mortality

Under-Five Mortality

5. Summing Up

Key Messages

► There is no IE evidence that increasing skilled birth attendance alone reduces maternal or neonatal mortality: • Importance of EVIDENCE-BASED INDICATORS for post-MDGs

► Slow progress on MDGs 4 & 5, but evidence of effective interventions• SBA: vouchers, CCTs, bundled interventions• MM: SBA+ combining provision and utilization elements• NM: knowledge & information, maternal education • IM: Governance, Energy, WASH, Ed; training community

health workers• U5: Gov & Participation, WASH, Education; health Service

Packages, ITNs

► Countries & households with higher burdens may see larger results

► Important knowledge gaps remain• Intervention: including SBA, Nutrition, 3 Delays Model

(esp transportation)• Evaluation components (subgroup analysis)

25

The Systematic Review can be downloaded from

https://ieg.worldbankgroup.org/Data/reports/mch_eval.pdf

The Database of all IEs is at

https://ieg.worldbankgroup.org/Data/mch/mch_dataset.xlsx

Thank You!

General Systematic Review resources and international bodies

► Cochrane Collaboration; 1993; www.cochrane.org• Producing high quality information about the effectiveness of health

care (> 5000 published online – Cochrane library)

► Campbell Collaboration; 2000; www.campbellcollaboration.org• Producing systematic reviews of the effects of social interventions

(>200 published online – Campbell library)

► International Development Coordinating Group (IDCG); 2010 www.campbellcollaboration.org/international_development• Producing systematic reviews of high policy-relevance focusing on

social and economic development interventions in LMICs

► International Initiative for Impact Evaluation, 3ie; 2008 www.3ieimpact.org/en/evidence/systematic-reviews

► EPPI Centre - An Institute of Education centre focusing on systematic reviews in education, health and social policy

► Collaboration for Environmental Evidence producing systematic reviews for environmental management

Backup Slides

Overview

1. Introduction

2. Methods

3. Results

4. Knowledge Gaps

5. Summing Up

28

1. Introduction

MDGs 4 and 5 Continue to Lag

► MDG5: ¾ Reduction in Maternal Mortality by 2015• Main Indicator: Proportion of births attended by skilled health

personnel

► MDG4: 2/3 Reduction in Under-Five Mortality► Knowing what to do is no longer the problem; knowing

how to do it remains a challenge

MD

G 1

.a E

xtre

me

pove

rty

(%

popu

latio

n be

low

$1.

25 a

day

, 20

05 P

PP

)

MD

G 2

.1 P

rim

ary

com

plet

ion

rate

(%

rel

evan

t age

gro

up)

MD

G 3

.1 R

atio

of

girl

s to

boy

s in

pr

imar

y an

d se

cond

ary

educ

atio

n (%

)

MD

G 4

.a M

orta

lity

rate

, inf

ants

(p

er 1

,000

live

bir

ths)

MD

G 4

.a M

orta

lity

rate

, chi

ldre

n un

der

5 (p

er 1

,000

)

MD

G 5

.a M

ater

nal m

orta

lity

ratio

(m

odel

ed e

stim

ate,

per

100

,000

live

bi

rths

)

MD

G 7

.c I

mpr

oved

wat

er s

ourc

e (%

pop

ulat

ion

with

out a

cces

s)

MD

G 7

.c I

mpr

oved

san

itatio

n fa

cil

-iti

es (

% p

opup

latio

n w

ithou

t acc

ess)0

20406080

100 7894 96

80 8068

87 87100

87 96

51 5238

100

72

Corresponding target Latest available value

Role of IEG

► Independent Evaluation in the World Bank Group

► Impact Evaluations in the World Bank Group

► Why this Systematic Review• Are we doing the right things to achieve

MDGs?• Compare Causal evidence vs Bank Portfolio• Compare stock vs need of evidence, regionally

31

2. Methods

Search Process

► 3 Search Rounds• Electronic, “hand”

and snowball search strategies

► Review and coding into 300+ fields

► 7,000 62 studies► Quality ratings by

Internal Validity• Elements of

Construct and External Validity also considered

Challenges and Cautions for Systematic Reviews► Representativeness of Interventions—non-random selection

• Overrepresentation of easily evaluable interventions• Focus on “reduced form” studies excludes those with intermediate

outcomes• Lack of evidence does not imply no effect

► Representativeness of Impact Evaluations• Includes only existing studies• Publication bias (file drawer bias)

► Interpretation of Results• IEs measure partial equilibrium; general equilibrium may be

different• Null results must be interpreted carefully—we never “accept” zero• External validity—changes to time, place, or scale may affect

results

Standardized Effect Size: Maternal Mortality

Standardized Effect Size:Neonatal Mortality

Search Criteria

► Impact Evaluations• Experimental or Quasi-Experimental design• Counterfactual

► Completed 1995 – Present► Effectiveness / Policy / Field studies

• (Rather than bio-medical and efficacy trials)

► Low and Middle-Income Countries► Representative Sample of population of interest► Peer Review► Report impacts on at least 1 outcome of interest

• Skilled Birth Attendance, Maternal Mortality• Neonatal, Infant, Under-five Mortality

What of Nutrition?

► No studies on nutrition explicitly; some “bundling”• Mortality Outcomes• Effectiveness, not clinical/efficacy

► 18/93 studies mentioned “nutrition” or supplements (or variants)• 3 AAA—Intervention: Supplements to moms/kids

– 2 on Progresa CCT in Mexico—Impacts on IM; not significant for NM, SBA

• 7 AA, only 3 Interventions (4 Outcomes) – Bangladesh—Converted nutrition workers to Kangaroo Care; not

significant for NM, IM– Bangladesh—Family planning through Community Health Workers;

highly significant for U5– Vietnam—Provided training in child malnutrition; not significant for SBA

► 6/93 with Breastfeeding: 3 AAA (all India), 3AA (India, Pakistan, Bangladesh)• 5 Bundled Interventions—all reduced NM, none improved

SBA– CHW in India also reduced IM– Women’s Group in India not significant for MM

38

Standardized Effect Size:Infant Mortality

Standardized Effect Size:Under-Five Mortality

IEs of World Bank Funding and Projects: Mortality

► No significant results for maternal or neonatal mortality► Significant, but often small effects on infant mortality.

Larger for U5.

Income increasing; India; SME(N=429,445)

Monitoring/Evaluation/Accountability, Service management; Uganda; SME(N=50)

Planning/Policy, Delivery Modality; Brazil; SME(N=38,762)

Planning/Policy, Delivery Modality; Brazil; SME(N=38,762)

Water/Sanitation; Brazil; SME(N=3,568)

Delivery modality, Health workforce, Service package; Indonesia; SME(N=1,590)

Delivery modality, Health workforce, Service package; Indonesia; SME(N=1,590)

Health workforce, Knowledge/Information, Service Package; Indonesia; OR(N=80)

Health workforce, Knowledge/Information, Service Package; Indonesia; OR(N=63)

Health Infrastructure; Planning/Policy; Water/Sanitation; Bolivia; SME(N=8,009)

Income Increasing; India; SME(N=182,869 )

Income Increasing; India; SME(N=182,869 )

Planning/Policy; Delivery modality; Brazil; SME(N=3,336)

[50] Newman and others 2002; F(U5)

[57] Rocha and Soares 2010; F(U5)

[4] Baird abd others 2011; P(U5)

[12] Bjorkman and Svensson 2009; P(U5)

[43] Macinko and others 2007; F(IM)

[57] Rocha and Soares 2010; F(IM)

[58] Shrestha 2010; F(IM)

[4] Baird abd others 2011; P(IM)

[28] Gamper-Rabindran 2010; P(IM)

[58] Shrestha 2010; F(NM)

[47] Mazumdar and others 2011; F(NM)

[39] Lim and others 2010; F(NM)

[39] Lim and others 2010; F(MM)

-1 -.5 0 .5 1size effect

estimate (AAA) estimate(AA) 95% conf. int.

IEs of World Bank Funding & Projects:Skilled Birth Attendance (Outcome)

► SBA often significant, but small effect size► Of 15 IEs on World Bank, 11 from 3 countries ► SBA & IM IEs concentrated in regions with 2nd –lowest

burden

Health financing; Planning/Policy ; Indonesia; SME(N=12,000)

Health financing; Planning/Policy ; Indonesia; SME(N=12,000)

Income increasing; India; SME(N=429,445)

Monitoring/Evaluation/Accountability, Service management; Uganda; SME(N=50)

Health Financing; Rwanda; OR(N=2,108)

Health workforce, Delivery modality, Service Package; Indonesia; OR(N=6,730)

Health workforce, Knowledge/Information, Service Package; Indonesia; SME(N=52)

Health Infrastructure; Planning/Policy; Water/Sanitation; Bolivia; SME(N=8,009)

Income Increasing; India; SME(N=182,869 )

[7] Basinga and others 2011; F(SBA)

[26] Frankenberg and others 2009; F(SBA)

[39] Lim and others 2010; F(SBA)

[47] Mazumdar and others 2011; F(SBA)

[50] Newman and others 2002; F(SBA)

[55] Olken and others 2012a; F(SBA)

[55] Olken and others 2012b; F(SBA)

[4] Baird abd others 2011; P(SBA)

[12] Bjorkman and Svensson 2009; P(SBA)

-1 0 1 2size effect

estimate (AAA) estimate(AA) 95% conf. int.

Specific Knowledge Gaps

► SBA: no robust evidence that solely increasing proportion of births with SBA affects mortality

► Mortality: • Maternal Mortality: limited number of studies but

highly concentrated in South Asia (5 out of 8)• Child Mortality: needs attention to IE quality and

intervention details

► More high quality evaluations are needed• Family planning, universal health• Improvements in quality and availability of health

Infrastructure for newborns• “3 Delay” models (especially Transport and Referral

Systems)

1. Scoping: defining answerable question, methods set out in study protocol

2. Rigorous search to identify published and unpublished sources, in any language

3. Application of study inclusion criteria (PICOS)

4. Critical appraisal of study quality, to assess how reliable is the evidence

5. Data extraction and organisation

6. Synthesis of evidence (outcomes along causal chain)

7. Interpreting results (policy and practice, research recommendations)

8. Improving and updating reviews as new evidence emerges

What makes a systematic review ‘systematic’?