Using case studies to explore the generalizability of ‘complex’ development interventions

Michael Woolcock Development Research Group, World Bank

Kennedy School of Government, Harvard University mwoolcock@worldbank.org

Exploring Complexity in the Health Sector: A Panel on Case-Based Evaluation Approaches

John Snow, Inc, George Washington University School of Public Health,

in collaboration with the Institute of Medicine and the World Bank

December 4, 2014

[T]he bulk of the literature presently recommended for policy decisions… cannot be used to identify ‘what works here’. And this is not because it may fail to deliver in some particular cases [; it] is not because its advice fails to deliver what it can be expected to deliver… The failing is rather that it is not designed to deliver the bulk of the key facts required to conclude that it will work here.

Nancy Cartwright and Jeremy Hardie (2012) Evidence- Based Policy: A Practical Guide to Doing it Better (New York: Oxford University Press, p. 137)

What ‘key facts’ do we need? How might we acquire them?

10 mins, 3 points

• Using case studies to extrapolate from – Present to future (time) – Pilot to scale-up (size) – ‘There’ to ‘here’ (place)

• In social science, even ‘rigorously’ identified impact claims rarely provide warrant for such extrapolations – Especially for ‘complex’ interventions

• Extrapolation requires ‘key facts’ regarding – Context dynamics – Process mechanisms – Implementation capability – Trajectories of change Which careful analytic case studies can help to provide

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

“Same” impact claim, but entirely a function of when the assessment was done…

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

A

B

C

If an evaluation was done at ‘A’ or ‘B’, what claims about impact would be made?

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

A

B

C

?

D

t = 2

Extrapolation from ‘there’ to ‘here’: Using RCTs to test generalizability of RCTs

• Bold, Sandefur et al (2013)

– Take a project (contract teachers) with a positive impact from India, as determined by an RCT…

– …to Kenya; 192 schools randomly split into three groups to receive a contract teacher:

• a control group

• through an NGO (World Vision)

• through the MoE

– Result?

Implementation matters (a lot)

Bold et al (2013)

Getting (some of the) ‘key facts’: the role of case studies

• World Bank report on service delivery in Middle East North Africa (MENA) region, 2015 – Using HH surveys to map, explore subnational variation in

quality of health care (breadth)… – …complemented by case studies of ‘positive deviance’ to

explain how such outcomes occur (depth)

• Effectiveness of national community development program in Indonesia – (Barron, Diprose and Woolcock, 2011; Contesting

Development; Yale University Press) – Explaining forms, sources of impact heterogeneity over time

and place – Initial pilot results ‘modest’; but big success once scaled-up.

• How and why did this occur?

Primary source material

• Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) ‘Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development’, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641

• Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press

• Pritchett, Lant, Salimah Samji and Jeffrey Hammer (2012) ‘It’s All About MeE: Using Experiential Learning to Navigate the Design Space’ WIDER Working Paper No. 2012/104

• Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14

• Woolcock, Michael (2013) ‘Using Case Studies to Explore the External Validity of Complex Development Interventions’ Evaluation 19(3): 229-248