Www.3ieimpact.org Philip Davies Making Evidence Accessible and Relevant for Policy and Practice...
-
Upload
christine-goodwin -
Category
Documents
-
view
217 -
download
2
Transcript of Www.3ieimpact.org Philip Davies Making Evidence Accessible and Relevant for Policy and Practice...
www.3ieimpact.orgPhilip Davies
Making Evidence Accessible and Relevant for Policy and Practice
Philip Davies
International Initiative for Impact Evaluation [3ie]
Africa Evidence Network Regional MeetingJohannesburg, South Africa, 3rd June 2015
www.3ieimpact.orgPhilip Davies
• Helping people make better decisions and achieve better outcomes, by using the best available evidence from research and other sources
• Knowing what are effective interventions (“what works?”)
• In achieving which outcomes?
• For which groups of people?
• Under what conditions?
• Over what time span?
• At what costs?, plus
• Integrating research with decision makers’ knowledge, skills, experience, expertise and judgement
What is Evidence-Based Policy
www.3ieimpact.orgPhilip Davies
Evidence
Experience & Expertise
Judgement
Resources
Values, Beliefs and
Ideology
Habits & Bureaucratic
Culture
Lobbyists & Pressure Groups
Pragmatics & Contingencies
Factors Other Than Evidence
www.3ieimpact.orgPhilip Davies
Understanding
the Problem(Conceptualisation
) Developing
Solutions
(Policy Develop
ment)Putting Solution
s Into Effect
(Implementation)
Monitoring and
Evaluation
(M&E)
The ‘Classic’ Policy Cycle The ‘ROAMEF’ Policy Cycle
Evidence is required across the entire policy cycle
www.3ieimpact.orgPhilip Davies
The Kind Of Evidence Decision-Makers Look For
• Identifying the nature, size and dynamics of the problem
• Specifying the desired objectives• Identifying viable policy options• Identifying how the policy is supposed to work• Identifying the likely and achieved outcomes/impacts• Identifying the social distribution of
outcomes/impacts• Understanding people’s attitudes, experiences,
behaviour• Valuing the impacts (cost-benefit/cost-effectiveness)• Identifying effective implementation and delivery
www.3ieimpact.orgPhilip Davies
Types of Evidence for Policy Making
www.3ieimpact.orgPhilip Davies
Theory of Change/Logic Model/Programme Theory
• How is a policy/programme supposed to work?
• What activities, mechanisms, people, outputs have to be in place?
• And in what sequence – what is the causal chain?
• What resources are required – and are available?
• What data are required – and are available?
• Is the policy/programme feasible/achievable?
www.3ieimpact.orgPhilip Davies
Constituent Features of a Theory of Change
Assumptions?
www.3ieimpact.orgPhilip Davies
Constituent Features of a Theory of ChangeData Required
•Surveys, statistics, demographic data
•Qualitative data•Costs/benefits data•Systematic review data•Documentary analysis
• Performance data• Historical data• Diversity data• Qualitative data• Effectiveness data
• Stakeholder data• Qualitative data• Public opinion data• Effectiveness data
• Performance data• Effectiveness data• Stakeholder data• Qualitative data• Costs/benefits data
• Administrative data• Performance data• Costs/benefits data
• Administrative data• Performance data• Qualitative data
• Counterfactual data• Administrative data• Survey data, statistics• Cost/benefit data
www.3ieimpact.orgPhilip Davies
Types of Systematic Review
• Statistical Meta-Analyses
• Narrative Systematic Reviews
• Qualitative Systematic Reviews
• Rapid Evidence Assessments
• Evidence Maps and Gap Maps
www.3ieimpact.orgPhilip Davies
Single studies can:
Misrepresent the balance of research evidence
Illuminate only one part of a policy issue
Be sample-specific, time-specific, context-specific
Often be of poor quality
Why Do We Need Systematic Reviews?
Consequently, give a biased view of the overall evidence
www.3ieimpact.orgPhilip Davies
• Systematic searching for studies
• Systematic critical appraisal of identified studies – separating the wheat from the chaff
• Systematic and transparent inclusion/exclusion of studies for final review
• Systematic and transparent extraction of data
• Systematic statistical testing and analysis
• Systematic reporting of findings
What Makes a Review Systematic?
www.3ieimpact.orgPhilip Davies
Statistical Meta-Analytical Reviews
Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism
www.3ieimpact.orgPhilip Davies
• Synthesise qualitative and ethnographic evidence
• In-depth interviews, focus groups, observational studies, documentary analysis, case studies
• Seek common themes, concepts and principles across different studies
• Detailed attention to context/contextual specificity
• And stakeholders’ views
• Do not seek generalisations
Qualitative Systematic Reviews
Photo © Albert Gonzalez Farran - UNAMID
www.3ieimpact.orgPhilip Davies
• Scaled down systematic reviews of existing evidence
• Timed to meet the needs of policy makers/practitioners (1-3 months)
• Strategically using the ‘three arms’ of systematic searching, but less exhaustively
Rapid Evidence Assessments – What Are They?
• Critical appraisal of identified studies is included
• Summary of findings, with caveats and qualifications
Photo © Panos East Africa
www.3ieimpact.orgPhilip Davies
Rapid Evidence Assessments – How Scaled Down?
www.3ieimpact.orgPhilip Davies
3ie Evidence Gap Maps
• Maps of the existing evidence base on a policy issue, topic or sector such as maternal health, HIV/AIDS, agriculture, extreme poverty
• Structured around a framework of interventions and outcomes (intermediate and final)
• A ways of identifying where there is evidence, and where there is not
• An indication of the quality of this evidence
• Links to user-friendly summaries in the 3ie database of systematic reviews.
www.3ieimpact.orgPhilip Davies
Rapid Evidence Assessments - LimitationsEvidence Gap Maps
www.3ieimpact.orgPhilip Davies
www.3ieimpact.orgPhilip Davies
www.3ieimpact.orgPhilip Davies
Some Key Sources of Sythesised Evidence
• 3ie Impact Evaluations Database(http://www.3ieimpact.org/en/evidence/impact-evaluations/)
• 3ie Systematic Reviews Database • (http://www.3ieimpact.org/en/evidence/systematic-reviews/)
(http://www.3ieimpact.org/en/evidence/systematic-reviews/)• Best Evidence Encyclopedia (http://www.bestevidence.org/)• Cochrane Collaboration (www.cochrane.org)• Campbell Collaboration (www.campbellcollaboration.org)• Collaboration for Environmental Evaluation (http://www.environmentalevidence.org/)• National Institute for Health and Clinical Excellence (http: www.nice.org.uk/)• NHS Evidence (http://www.evidence.nhs.uk/)• National Guidelines Clearinghouse (USA) (www.guidelines.gov)• Prospero: International Prospective Register of Systematic Reviews (http://www.crd.york.ac.uk/prospero/• Social Care Institute for Excellence (http://www.scie.org.uk/)• Social Programs That Work (http://evidencebasedprograms.org/)
www.3ieimpact.orgPhilip Davies
UK Policymakers’ Views of Evidence
• Focus on the ‘end product’, rather than how the information was either collected or analysed
• Use of ‘anecdotal’ evidence (“tells a story”)• Drawing on such things as ‘real life stories’,
‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence
But:
• “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House”
• And: DfID Evidence into Action Team
• And: BCURE Programme + DPME (South Africa)
www.3ieimpact.orgPhilip Davies
Sharks
Where Do UK Civil Servants Go For Evidence?
PlanktonAcademic/Evaluation Research?
www.3ieimpact.orgPhilip Davies
UK Policymakers’ Views of Research Evidence
• Too Long • Verbose• Too Detailed • Too Dense• Impenetrable• Too Much Jargon• Too Methodological• Untimely• Irrelevant for policy
Source: Campbell, S., et al; 2007, Analysis for Policy
www.3ieimpact.orgPhilip Davies
• Establish what research says and does not say
• Establish the policy messages and policy implications
• Use a 1:3:25 format
• Very little mention of methodology
• Be clear - plain English summary
• Be persistent and opportunistic
Improving Communication of Evidence
www.3ieimpact.orgPhilip Davies
Thank you
Philip Davies
Email: [email protected]
+44 (0)207 958 8350
Visit www.3ieimpact.org