Can increased understanding of policymaking enhance the ... · 09/04/2018 Myriam Van Parijs - WUR...

Empathic evaluationCan increased understanding of policymaking

enhance the use of evaluation evidence?

Myriam Van Parijs - Evaluation Office Enabel

Enabel – Belgian Development Agency

09/04/2018 Myriam Van Parijs - WUR Conference 2018

EVAL Office strategy

Evaluation evidence

(systems -development

strategies)

Portfolio M&E+L systems

Evaluation culture

EVALUATION Office

Knowledge brokering - User oriented

Evidence informed strategies & policiesAffecting views on how change occurs

Evaluative mindset

High quality

M&E practices


INTRODUCTIONWhy have an interest in policymaking?


Evidence matters in making effective policies

assuming use of evidence

Evaluation phase : communicate and disseminate


Evidence matters in making effective policies…

• Supply - disillusionment with limited uptake of evidence

• Demand - policymakers are craving for wisdom, while drowning in information

• Evidence supporting policymaking –little proof of this link in real-world practice


Evidence matters in making effective policies –the GAP narrative


Evidence matters in making effective policies

serviceempathy?

policymaking needsUse of evidence by policymakers

EVIDENCE ISPRODUCED

EVIDENCE IS ACCESSIBLE

POLICY MAKERS USE EVIDENCE

SUPPLY SIDE DEMAND SIDE

GAP?


Empathic evaluation - Content

• Shift attention from ‘use of evidence’ to :– the imperatives of the policymaking systems, and the use of evidence

herein– improving the contribution that evidence makes to policies

• Can evaluation processes and evaluation systems become more empathic to the needs of policymaking?– policymaking– use of evidence


ON POLICYMAKING

What do policymakers actually do when they make policies?


What is policymaking?

Lack of definitions & understanding of ‘real-world policymaking’ is part of the problem

Source : ODI


Evidence in policymaking : a new call?

• 1997 : a call for ‘what works’ – Labour Government UK ‘Nobody rational could possibly want a government based on any other type of policymaking.’

• 3 drivers of the modern evidence based policy movement :– enlightenment– role of evidence in the radical improvement of health outcomes– New Public Management

• What works? – challenged by wicked problems – policymaking who gets what, when and how


Two policymaking paradigms on use of evidence

Rational - technocratic

Politicised-relational


Two policymaking paradigms From rational and technocratic paradigm

Use of evidence inpolicymaking

Rational use of evidence - take decisions.

Policy process Several separate stages in a process.Policy issues Are seen as ‘problems’, outside the

political sphere.

No distinction in nature of issues.Management of policypractice

Rational, related to the different stages.

Relation betweenevidence and policymaker

Focuses on the policy maker as ‘individual’.

Improve use ofevidence in policypractice

Bridge the ‘gap’ between policy maker andevidence (providers).

Focus on production of evidence (supply-side).


Two policymaking paradigms

From rational and technocratic paradigm To politicised and relational paradigmUse of evidence inpolicymaking


Ratio.

Political use, negotiation over evidence.

Based on values, interests & beliefs.Policy process Several separate stages in a process. Incrementalist, messy and complex

process, policy window.Policy issues Are seen as ‘problems’, outside the

political sphere.

No distinction in nature of issues.

Are considered as ‘problematisations’,constructed within the political sphere,subjectivity. Values, interests & beliefs.

Wicked problems and distinction betweensimple – complicated – complex issues.

Management of policypractice

Rational, related to the different stages. Intractable, muddle through.

Relation betweenevidence and policymaker

Focuses on the policy maker as‘individual’.

Focuses on individual level, increasingattention for organisational andinstitutional aspects (networks).

Improve use ofevidence in policypractice

Bridge the ‘gap’ between policy makerand evidence (providers). Focus onproduction of evidence (supply-side).

Interaction between policy maker andevidence providers. Focus on demand side(understand and enhance access and useevidence).


source : Van Parijs M., 2017

Politicised-relational paradigms

Key concepts• Policy arena• Values, interests, beliefs :

framing of policy issues –policy options

• Interactions

Policymaking

Model of Kingdon


Policymaking : evidence-informed or evidence-based?

• Use of evidence is not self-evident – results from trade-offs between competing values, beliefs, interests

• Politicised process, part of democratic debate – who gets what, when and how :

– debate needs to remain democratically representative of the public’s values - cannot remain on technical level or in a closed network,

– network of actors– policy based evidence making

• Need for trust and interactions to interpret the value of evidence – policymaking = meaning-making

process– policy-decisions : context

dependent– evaluation : from accountability

to learning– conceptual use of evidence in

evaluation– timing is seldom synchronised


Evidence-informed policymaking – an art?

On government based practices related to policymaking:What needs to be more wisely understood is that there is also an art to policymaking. The art consists in :• knowing when and how best to deploy evidence• understanding its limits as a means to steer

policy formation. • accommodating the political and subjective

modifications which is the inevitable consequence of creating policy in a democracy.

Frustrating though it can be, it is necessary to be both scientific and artful to make an impact in the world (Simmons, 2015). • Textbook policymaking versus ‘real-

world’ policymaking : tackle ‘machine’ approach evidence-informedpolicymaking


Source : ODI

Bridging the gap or connecting two worlds?

EVIDENCE ISPRODUCED

EVIDENCE IS ACCESSIBLE

POLICY MAKERS USE

EVIDENCE

SUPPLY SIDE DEMAND SIDE

GAP

Evaluation

Policymaking


ON EVIDENCE – TOWARDS IMPROVED USE OF EVIDENCE


What counts as evidence?

What is or what counts as evidence?• Evidence can be independently

observed and verified, and there is a broad consensus as to its contents and even interpretation; systematic investigation towards increasing the sum of knowledge.

• ‘Multiple evidences’ on a broad continuum - policy is to be informed by a wide breadth of evidence.

• Evidence is influenced by many aspects (experience, expertise & judgment, values, resources, culture, pressure groups, …).

Challenge?• “we are still far from a consensus –

intellectually or politically – regarding what should count as evidence, how it should be produced and validated, ad how it should be used to influence policymaking” (Heinrich, cited in Head, 2015)

• Lack of clarity hampers research on use of evidence

• What counts as evidence ? Differs : type of question asked, who asks, purpose of the use of evidence, evolution in understanding evidence over time, research culture.


Multiple evidences : ways of knowing, hierarchies, and horses for courses

• Hierarchies of evidence :

• More rigorous and systematic issues of evidence will improve efficiency & effectiveness of public policies // medicine

• Exclusive focus on methodological rigor <–> other determining factors, such as contextual factors

• Critics on hierarchies

• Focus on methods loss of other useful evidence, and thus for quality of policy decisions.

• Hierarchies ignore why & how evidence will be used, context-influence, social desirability of what is measured, and generalisability of effects – for who.

• Evidence hierarchies can’t answer what his good evidence


Horses for courses• Different ways of knowing – different types of policy issues and policy questions• Evidence matrices – appropriate evidence according to question asked

Multiple evidences : ways of knowing, hierarchies, and horses for courses


Understanding evidence in policymaking context

• Evidence : – no conclusive proof, objectivity, ‘truth’, wisdom– timing, context, sources, perceptions– good evidence may be ignored, bad evidence may be used misleadingly

• Quality of the evidence process – refer to context, politics, purpose and use– Beyond common ‘quality criteria’ (credibility, accuracy, relevance) – Usefulness of evidence for policymakers : persuasive and actionable evidence– Social desirability of evidence – Accept nuanced generalisability – what works here, will it work there?

Some interventions worked for some people in some situations

• Strategies for evidence use - focused on push & pull efforts to get evidence in policy arena :– Push efforts : improving dissemination of evidence (briefs, systematic review – Pull efforts : capacities of policymakers to use evidence


Understanding evidence in policymaking context

• Knowledge management : – Focuses on transmission, mobilisation, brokering– Mostly fails to recognise that decision to promote some pieces of evidence

over others, is a political choice (Parkhurst, 2016)– Assuming that evidence use is inherently an uncontested ‘good thing’ with

‘politics seen to be a ‘negative barrier’ to research utilisation, is a barrier to overcome (Parkhurst, 2016)

• Politics of evidence :– Rational paradigm : problem is the politicisation of science– Politicised paradigm : problem is the de-politicisation of politics– Both ‘problems’ lead to biases.


Technical bias• Pieces of evidence are ‘scientifically

robust’ or ‘flawed’• Faithful interpretation according to

methods used & results obtained : invalid conclusions from valid evidence (absence of evidence seen as evidence of absence)

• Strategic cherry picking of evidence

undermines scientific consensus, increases uncertainty, perpetuates disagreements over what the evidence actually says or means

Issue bias• Situation where choice of evidence

can bias policy decision : not the choice creates the bias, but the obfuscation or the imposition of values in evidence.

• Decision to include one or another body of evidence, or to create evidence : prioritises some concerns over others.

can lead to selection or creation of evidence according to ones goals and values, and exclusion of others.

Towards improved use of evidence : address politics of evidence (Parkhurst, 2016)


Evidence-use according to the two paradigmsFrom rational and technocratic paradigm To politicised and relational paradigm

Use of evidence inpolicymaking


What works?

Political use of evidence, negotiation overevidence.

What works, for who, said by whom, onwhat basis of evidence, selected accordingto which values & interests?

Type of evidence Hierarchy of evidence on top of ‘robust’evidence.

Best evidence.

Evidence comes without hierarchy anddepends on relation to the issue.

Appropriate evidence – horses for courses.Politics of evidence ‘de-politicise’ the use of evidence –

develop evidence free from politicalinfluence – use evidence ‘properly’

Recognise the influence of politics inevidence use

Quality of evidence Relate to ‘what works’ :quality/accuracy/objectivity, credibilityand generalisability, relevance

+ processes of using information +usefulness for policymaking (persuasiveand actionable) +


source : Van Parijs M., 2017

Towards improved use of evidence : appropriate evidence (Parkhurst, 2016)

1. 2. 3.


source : Parkhurst, 2016

Good evidence ? A matter of appropriate evidence (Parkhurst, 2016)

Evidence addressing policy concerns at hand

Evidence applicable to local context

Evidence constructed in ways useful to address policy

concern

Appropriate evidence for policy making


source : Parkhurst, 2016

Bridging the gap?

• Two different worlds – two different modus operandi between evaluation and policymaking :

– Timescales : fast vs slow– Perspective : present + forward/prospective – retrospective– More incentives not to use evaluation evidence, than to use : politics - blaming

• No ‘maggic bullet’ : evidence is mostly imprecise, inconclusive, complex, whereas policymakers can only use evidence if it is precise, provides clear guidance, and is formulated in sufficiently simple terms that it can be applied (Nutley et al.)

• Top 5 Barriers & Facilitators of evidence use :– Barriers : availability & access, clarity, timing/opportunity, policymaker research skills,

costs– Facilitators : availability & access, collaboration, clarity/relevance/reliability ,

relationships with policymakers, relationships with researchers


Providing new evidence as necessary

Framing the issuePolicy makers and evidence providers

jointly scope the questions

Re-framing the issue: policy makers

and evidenceproviders jointly

interpret the evidence

Assembling existingevidence

Multi-stakeholder

evidence-policyinterface

Policy issues

Policy optionsA longer time frame allows evidence from primary

research to complement existing knowledge

In a shorter time frame, research reviews, synthesesand evidence networks will be more cost-effective

From a bridging a gap to an interaction model

Source : by Shaxson, 2016 – adapted from Defra, 2006

WHAT CAN WE CONCLUDE?


What does this mean for bringing two realms closer?

1. The dominant model of policymaking does not fit ‘real-world’ policymaking. This fields is complex, and needs craftsmanship to interact in an agile way. Use a map instead of a GPS.

2. Public policy issues are mostly wicked problems. They are not ‘solved’, only ‘re-solved’.

3. Using evidence : from individual approach, to organisational and institutional approach

4. Evaluation evidence is valuable towards policymakers, but remain realistic on ‘uptake’ – and the potential of ‘instrumental use’ of evaluation and re-value the conceptual use/enlightenment function of evaluation.

5. Evaluation evidence is also political. Recognise politics of evidence – both technical bias and issue bias.

6. Focus on appropriate evidence and appropriate use of evidence, rather than on more evidence & increase use of evidence.


Evaluation empathic towards policymaking?1. Interaction is key! Only when the condition is met that policy makers and scientists

can negotiate the available evidence and interpret the value of this evidence for policymaking, can knowledge effectively contribute to policy processes. (Simmons, 2015).

2. Finding a ‘magic bullet’/nudge to enhance uptake of evidence might be an illusion. It may not have solved any problems, but it [may have] led to a better class of problems (Rivlin, 1971)

3. Re-recognise the value of ‘conceptual use’ of evaluation evidence – instrumental.4. Evaluation processes & principles are rooted in accountability – suited for learning?

(independence vs interaction, relativity of objectivity).5. Consider more evaluation as a ‘service’ – understand the realm of the policymaker

and accept ‘politics of evidence’.6. Improving use of evidence in policymaking is a dynamic debate in turbulent times

with both a stronger demand for evidence, and increasing contestation –‘alternative facts’ & evaluation community shouldn’t stay out.

7. Understanding use of evidence will evolve through interdisciplinary understandings (policymaking, evaluation research, cognitive sciences).


THANK YOU!

[email protected]


Can increased understanding of policymaking enhance the ... · 09/04/2018 Myriam Van Parijs - WUR...

Documents

Transcript of Can increased understanding of policymaking enhance the ... · 09/04/2018 Myriam Van Parijs - WUR...