Ukes process tracing presentation
-
Upload
itad-ltd -
Category
Data & Analytics
-
view
192 -
download
1
Transcript of Ukes process tracing presentation
1
Process tracing for impact evaluationPotentials and Pitfalls
Melanie PuntonUK Evaluation Society Conference (12 May 2015)
2
Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation
Stedman-Bryce (2013) Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376
3
Process Tracing:
A qualitative method for assessing causal inference within a single case
Sits in the ‘evaluator’s toolbox’ alongside• Theory-based evaluation• Contribution analysis• Realist evaluation
4
Why process tracing?
0 Useful where pathways of change are uncertain 0Tells you WHY change happened0 Rigorous approach through ex-post design without a
control group
0Time intensive0 Requires the outcome to be known0 Uncharted territory
5
A brief history of process-tracing
0 A method stemming from the social and political sciences. Major contributors: Alexander George, Andrew Bennett, David Collier
0 Originally used to explain historical events, e.g. the cause of the Cuban Missile Crisis (Allison and Zelikow 1999)
0 A ‘contested method’0 Beach and Pederson’s book (2013): detailed theory and
step-by-step guidance0 Limited examples of its application in evaluation
NB: See Punton & Welle (2015) for references
6
Counterfactual framework
Generative framework
Experimental (RCTs)Quasi-experimental
Process tracingRealist evaluation
Intervention A
Outcome B????
Opening up the black box
7
Causal mechanisms
Intervention A
Outcome B????
Causal mechanism
Generative causal framework
8
Causal mechanisms
Part 3Part 2Part 1
Intervention A
Entity 1 Entity 2 Entity 3
Activity 1 Activity 2 Activity 3
Outcome B
9
The Universal Health Care Campaign in Ghana
0Collaborative advocacy effort by civil society to promote universal free access to health care
0Core funding from Oxfam until 20130Lobbying, mobilising, media and research0Effectiveness review in 2012-13, using elements of
process tracing
Stedman-Bryce (2013) Effectiveness Review: Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376
10
A (simple) example of a process tracing mechanism
Universal Health Care
campaign
Free universal
health care
Civil society
Conduct coordinated advocacy activities for free universal health care
Part 1
Public
Part 2
Become aware of the limitations of current health care financing
Public
Part 3
Demand free universal health care from government actors
Govern-ment
actors
Part 4
Increasingly support free universal health care, based on desire for public support
Govern-ment
Part 5
Amend policies and processes to move towards free universal health care
Activity
Entity
11
Differences between a mechanism and a theory of change…
0No leaps in logic between the parts0Every part should be necessary0Every part should be observable and empirically
measurable
12
Collecting evidence
• Examples of coordinated advocacy linked to campaign (e.g. events, platforms, meetings)
• Interview data suggesting that the campaign increased knowledge about and interest in healthcare reform among civil society actors
• Number of advocacy events held before the campaign started, compared to number held during campaign
Hypothesis: civil society conducts coordinated advocacy for free healthcare as a result of the Campaign
13
Collecting evidence
• Examples of coordinated advocacy taking place before the campaigns
• Interviews suggesting that the campaign made little difference to activities
Alternative hypothesis: The advocacy campaign made little difference to the coordination or activities of CSOs
14
“We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – The Hound of the Baskervilles
15
Assessing evidence: four tests• Analogous to a criminal trial• The evidence for each hypothesis
is weighed….• …in order to increase the
researcher’s confidence in the hypothesis.
• Bayesian probability logic followed to assess the strength of each part of the causal chain.
• Evidence examined using concepts of necessity and sufficiency.
16
Assessing evidence: four tests
Straw in the wind test
Evidence: hotel receipts, suggestive text messages- Neither necessary
nor sufficient
Hypothesis: John shot Mary
Hoop test
Evidence: John lacks an alibi- Necessary, but not
sufficient
Smoking gun test
Evidence: John was found holding a gun over Mary’s body- Sufficient, but not
necessary
Double decisive test
Evidence: A tamper-proof CCTV camera recorded the murder- Necessary and
sufficient
17
Assessing evidence: four testsU
niqu
enes
s of
evi
denc
e
Low (evidence is insuffi cient for h )
High (evidence is suffi cient for h )
Hig
h(e
vide
nce
is
nece
ssar
y fo
r h
)
Low
(evi
denc
e is
not
ne
cess
ary
for
h)
Hoop tests Double-decisive
Straw in the wind tests
Smoking gun tests
18
Assessing evidence: four testsStraw in the wind test
Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care
Evidence:1. Campaign members were given the opportunity to
discuss and debate a report on the complex National Health Insurance Scheme, in order to increase their understanding on the topic
2. Before the report was published, there was little coordinated advocacy for free universal healthcare
Increase the plausibility of the hypothesis but do not prove it or disprove alternative hypotheses. BUT together they provide stronger evidence than they do alone
19
Assessing evidence: four tests
Hoop test
Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care
Evidence: Until the campaign started, there was little notable coordinated advocacy on this issue.
Not enough on its own to prove the hypothesis, BUT evidence must pass the test in order to keep the hypothesis under consideration
20
Smoking gun test
Hypothesis: The National Health Insurance Authority in Ghana revised methodology for calculating membership because of pressure created by the campaign
Evidence: Ghana delegation at international meeting stated that the campaign’s report ‘prompted us to revise our figures’ (despite initially publically dismissing the report)
Evidence is sufficient to give high confidence that the hypothesis is true. BUT this type of evidence is hard to come by.
Assessing evidence: four tests
21
The end goal…
0Researcher can express level of confidence in each part of the mechanism
0Mechanism is only as strong as its weakest part0 If the researcher is confident in the whole
mechanism…0…the researcher can express confidence that the
mechanism holds in that particular case: that A caused B, and why.
22
Why process tracing?0 Useful where pathways of change are uncertain 0 Rigorous approach through ex-post design without a control
group0Tells you WHY change happened0Useful in a mixed methods design, or in combination with e.g.
Contribution Analysis?
0 Time intensive0 Requires the outcome to be known0Uncharted territory0 Risk of one weak link causing the whole mechanism to be
rejected
23
Questions?
Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation