Evaluating complex change across projects and contexts: Methodological lessons from a macro...
Transcript of Evaluating complex change across projects and contexts: Methodological lessons from a macro...
P r e s e n t a t i o n a t U K E S a n d C D I , 2 7 a n d 9 A p r i l 2 0 1 6
J e r e m y H o l l a n d a n d F l o r i a n S c h a t z
Evaluating complex change across projects and contexts: Methodological lessons from a macro evaluation of DFID's social accountability portfolio
Background
Part of a wider macro evaluation of DFID’s Policy Frame for Empowerment and Accountability (E&A)
Objectives:– To understand what works, for whom, in what contexts and why, by
conducting cross-case analysis of DFID’s E&A project portfolio since 2011; and
– To generate new evidence that informs policy and practice in DFID and other development organisations.
DFID’s E&A portfolio: 361 diverse projects (different focus areas, different countries and regions, different budgets and timeframes, different modalities and approaches, etc.)
Methodology
Project selection
Qualitative Comparative
Analysis (QCA)
Narrative Analysis
Theory development
QCA and Narrative Analysis
QCA (50 cases) Social science research method that applies a systematic comparison to
case study research Helps to identify the determinants of outcomes by looking at the similarities
and differences of cases in terms of the causal factors and outcomes obtained (Cress and Snow 2000)
Situated between qualitative and quantitative research approaches
Narrative analysis (13 case studies) In-depth qualitative comparative analysis to interpret and explain the
patterns identified through QCA Focus on identifying explanatory models that explain differences between
True Positive cases and False Positive cases (i.e. cases with the same configurations of conditions but a different outcome)
Based on secondary (project documentation) evidence supplemented by primary (Key Informant Interviews) evidence
5
1. Database of projects meeting
inclusion/ exclusion
criteria (produced 180 SAcc projects)
2. Screen for quality of outcome
contribution analysis (produced 50 SAcc
projects)
5. Identifying and coding project
‘conditions’
QCA contexts, mechanisms
and outcomes (CMOs)
6. Extracting data and scoring conditions
QCA binary scoring
4. Literature review and
DFID consultations
7. Developing
testable hypotheses
Hypotheses expressed as
CMO configurations
8. QCA of hypothesis
configurations
9. Selecting hypotheses and project cases for
narrative analysis
10. Narrative analysis of 13
projects
Analysis supplemented
with key informant interviews
Sample included ‘true positive’ and
false positive’ cases selected via Hamming distance
measure
3. Representativeness
analysis
Population of 2379
DFID projects
screened
10 Step Approach
Challenges and Lessons
1. The data quality challenge
…using QCA:• Despite selecting the projects with the best available data and filling data
gaps through Key Informant Interviews, we had 104 out of 1200 data points missing
• This required the manual construction of different sub-datasets for each hypothesis and limited our ability to perform more inductive analysis
…using narrative analysis:• The quality of data varied considerably in terms of coverage and
analytical depth• Key Informant Interviews were effective in deepening our understanding
but a tight timeline prevented us from reaching more than 20 stakeholders -> mixed evidence base
Challenges and Lessons
2. The challenge of unpacking ‘context’
…using QCA:• Despite including context conditions in our hypotheses, we were not able
to generate interesting or strong associations involving project contexts• Our context conditions were based on nationally comparable global
indices and possibly too broad
…using narrative analysis:• At the level of individual case studies, context factors were important but
so specific that it was difficult to generalise across cases while also retaining ‘granularity’
• This limited our ability to generate evidence on ‘what works in what contexts’
Challenges and Lessons
3. The challenge of sequencing and iteration
Combining QCA with narrative analysis required sequencing each evaluation step carefully, which resulted in a long timeline:– Finalising hypotheses before data extraction/coding– Finalising QCA before narrative analysis
But subsequent steps threw up additional factors, hypotheses and data points that would require revising preceding steps-> Iteration, for which there was insufficient time
Challenges and Lessons
4. The utility of mixing methods
Combining QCA with narrative analysis proved useful. The narrative analysis helped interpret and explain QCA findings, while QCA provided numerical evidence based on a relatively large number of cases to back up our findings.
The next slide presents an example finding from the macro evaluation:
Hypothesis QCA finding Narrative analysis findingHigher-level (at-scale) service delivery (O2) is achieved only when SAcc mechanisms include support for feeding evidence and learning into higher-level discussions (M7) and higher- level legislative and policy change (M1).
Hypothesis rejected.
The combination of feeding evidence upwards (M1) and directly supporting policy change (M7) is neither necessary nor sufficient for improved service delivery at scale. However, it improves the likelihood of success. In the cluster of (24) cases where both these mechanisms were present, there was evidence of improved higher-level service delivery in 7 (or 29%) of cases. In the cluster of (5) cases where both of these mechanisms are absent there is no case of improved higher-level service delivery. With coverage of 53% and consistency of 29%, the combination of M1 and M7 shows the strongest association with the outcome.
No single condition was necessary, sufficient, or strongly associated with the outcome.
The narrative analysis confirmed that linking SAcc to higher-level policy advocacy (M1) through upward feeding evidence (M7) could help improve service delivery at scale. The contribution of these mechanisms to improved outcomes was explained by the following factors:• SAcc processes worked
better when embedded in policy or programme frameworks
• Upward feeding evidence was effective when channelled directly into these policy and programme processes
• Upward feeding evidence was effective when combined with citizen participation key decision making processes.
Thank you
For more information:http://www.itad.com/knowledge-and-resources/dfids-macro-evaluations/