Brian Head

brian.head@uq.edu.au

School of Political Science

University of Queensland

Transforming the field: use of research evidence in policy and practice

Nuffield Foundation, 28 Bedford Square, London

24-25 September 2018

1

2

Science or objective evidence is supposed to solve problems

Kirkaldy Testing Museum, London, SE1

Do we now live in a world of “alternative facts” and self-enclosed

“echo chambers” that confirm our prejudices rather than challenge

the veracity of our thinking?

• IF we agree that using best-available evidence is necessary

for better problem-solving and service delivery, how can we

harness ‘research expertise’ (internal and external to govt)

and show how it contributes to better outcomes?

• Knowledge-based systems require strengthening. They

require investment and long-term commitment to:

• 1. good data gathering

• 2. good analytical skills

• 3. an open (transparent) political culture

• Evidence is plural – in its content, its modes of production,

forms of circulation, and types of use.

• Evidence in public policy assists in various task-domains: Regulation and compliance oversight; Policy development;

Service quality; Administrative processes; Performance

management; and Stakeholder engagement.

4

CRITICS of ‘evidence-based’ aspirations tend to emphasise

three political concerns and three science-related concerns:

POLITICAL CONCERNS:

• 1. Actual use of evidence in practical policy-making is often

highly selective (ad hoc and ‘cherry-picking’).

• Sometimes explicitly politicised (“policy-driven evidence”);

• 2. Political opportunism, reinforced by media distractions,

promote short-termism; this can undermine use of solid

evidence for long-term policy.

• 3. Concern about technocracy: if technical experts

dominate policy advisory processes, they may displace

stakeholder consultation and the democratic voices of

community and lay groups.

SCIENCE and KNOWLEDGE CONCERNS

• 1. Concerns about privacy, ethics and confidentiality in

a data-rich world.

• 2. Concerns about better integrating ‘citizen science’

and stakeholder knowledge/experience.

• 3. Pragmatic concern that the social world is inherently

complex, changing, uncertain, and interactive;

systems thinking supports the importance of building

shared understandings and capacities;

• policy solutions and effective programs are a product of

negotiation and interpretation, and cannot be deduced

from experimental science;

• strategic direction-setting needs to be adaptive and draw

on many sources of knowledge.

• In summary, we have seen that much depends on how

broadly (or narrowly) we conceive of 'research evidence’,

its value, and its reliability; hence, there are various

modalities of its 'uses’ for policy and practice.

• Generalisation is problematic, given there are so many

different contexts and situations for knowledge use.

Diversity is a crucial dimension in relation to:

• The types or forms of evidence

• Their mobilisation and linkage to political ideas and social

values

• The roles and perspectives of knowledge users and funders

• The power of organisational and occupational cultures

• The role of leaders and brokers.

We have known all this through the early work of Weiss,

Schorr, Nutley & Davies, Pawson & Ferlie, and many others.7

• Arguably, in most cases where key evidence tends to be

very specific and technical, it is more likely to be

amenable to ‘high fidelity' approaches, where it is

appropriate to focus on 'measuring' the degree of

uptake or impact.

• Legitimacy is generally high, at least in countries with long-

established trust in the science validation process.

• In other cases [i.e., where very detailed technical data

are missing or constitute only a small part of the

knowledge ecology], the main focus is likely to be on

creating shared understandings, purposes and

capabilities to better manage the challenges, while also

taking account of best-available evidence/ judgement.

• Legitimacy has to be created; the policy studies literature

draws attention to agenda-setting and plausible narratives.

8

1. Rigorous trials are undertaken to test and improve the

reliability/efficacy of technical evidence about therapeutic

drugs or clinical procedures; when robust evidence about

efficacy and safety is available, there is pressure to increase

its uptake into clinical practices.

2. Academic research teams may work on socially important

topics (often interacting with non-university stakeholders); they

are then under pressure to 'sell' the relevance of their work

and demonstrate the utility and 'impact' of their work, e.g. the

REF assessment.

Measuring ‘relevance’ to third parties is inherently perceptual;

Measuring contribution to improved social outcomes is

worthwhile - the ODI model (5 levels) is one of several good

frameworks.

9

AwarenessAttitudes,

perceptions, ideas

Knowledge &

understandingPractice & policy

change

CONCEPTUAL USE INSTRUMENTAL USE

Source: C.H. Weiss; S.Nutley et al

Forms of influence and impact

3. Professional evaluation studies (= contract research

projects) are commissioned by governmental /NGO

organisations; these generate attempts to identify useful

“lessons” for improvement.

Research ‘utilisation’ concerns in the 1970s-1990s were

centred on the patchy uptake of evaluations.

4. Local communities facing complex problems may seek

to deliberate on priority issues and develop ‘placed-based’

collaborative approaches to community development and

collective problem-solving;

this may entail co-design of interventions; iterative

review/adjustment; and real-time monitoring of processes

and outcomes.

11

“A place-based approach is a collaborative, flexible

response to local disadvantage in which the community is

the primary driver of change, and where data and

evidence are used to guide practice and innovation.

“The goal is long term, large scale change, and the

development of the approach is shared between the

community, governments and other stakeholders, and

recognises the value of local knowledge and the primacy

of social relationships”.

[Aust Dept of Social Services, 2017]

12

13

Community capacity to address disadvantage

Complexity/high capacity(Complex disadvantage and high community capacity or readiness)

Example response:Collective Impact type approach

Complexity/low capacity(Complex disadvantage and low community capacity or readiness)

Example response:Community capacity building followed by Collective Impact type approach

Co

mp

lex

dis

ad

van

tage

HighLow

Complicated/high capacity(Complicated disadvantage and high community capacity or readiness)

Example response:Service coordination

Complicated/low capacity(Complicated disadvantage and low community capacity or readiness)

Example response:Community capacity building and service coordination

Source: Adapted from the NZ Productivity Commission report on More effective social services, 2015

Co

mp

lica

ted

soci

al p

robl

ems

1,410

people

63 service

providers

$58.7m

per year

206

services

$41,600

per person

per year

most

programs

meeting

requirements

Roebourne

WESTERN

AUSTRALIA

Source: DSS

15

Key lessons for Evaluation of complex community interventions

1. Place-based approaches are ‘big picture’ and dynamic in nature

2. Evaluators should gain an understanding of communities within a place

3. Capturing and sharing lessons learnt in real time should be a key objective of place-based evaluations

4. Key questions are what, for whom, how, when, in what context

5. Establishing a baseline and/or counterfactual is important

6. Innovative evaluative methods should be considered

7. Effective evaluation governance is critical for outcomes

Source: T Reddel 2017. Emerging patterns in placed-based approaches. DSS Canberra

16

attachments

17

Mismatch between Complex Interventions and "Gold Standard"

Evaluations [SCHORR, 2009]

ATTRIBUTES OF EFFECTIVE COMPLEX

INTERVENTIONS

ATTRIBUTES ASSOCIATED WITH "GOLD

STANDARD” EVALUATIONS

Significant front-line flexibility exists within

established quality standards

Intervention is standardized, discretion

minimized

Intervention evolves in response to experience

and changing conditions

Intervention remains constant over time

Intervention/program design reflects local

strengths, needs, preferences

Intervention is centrally designed and

uniform across sites

Intake/recruitment into program falls under

local control, within broad parameters

Intake is centrally designed to permit

random assignment

Multiple components respond to children in

family, peer, and neighborhood contexts

Single-factor, single-sector interventions

Interactive components take into account the

interrelationships among health, social,

educational needs

Components are clearly separable

Training and setting emphasize continuing,

respectful relationships and other hard-to-

measure attributes

Focus on readily measured inputs

Implementers believe in the intervention and

go beyond their job description to respond to

clients

Implementation is value-free

18

Political

judgement: diffuse, fluid, often

populist and

adversarial

Professional

practices: Organisational

knowledge,

implementation,

practical experience

Scientific

research: systematic

approaches,

quantitative and

qualitative,

experimental and

action-oriented

POLICY PROBLEM

shaped by media & advocacy

groups

Inform and influence POLICY

RESPONSE

Each element is vital, but is only part of the problem-solving story

19

INTERNAL

(this state department)

OTHER governmental

agencies

EXTERNAL

Stakeholders

EXTERNAL

(Others)

Work team and

colleagues

Other QLD state

agencies

INDUSTRY associations GOOGLE search

Internet sources

Departmental data files

and reviews

Interstate agencies or

NATIONAL GOVT

COMMERCIAL and

COMMUNITY

lobbyists

MEDIA commentators

Senior mngt priorities LOCAL govt bodies Clients and citizens RESEARCH

publications

Ministerial &

legislative priorities

Statutory & regulatory

bodies

Service-provider

organisations

(contracted to

government)

EVALUATION &

Review reports

Diverse sources of information identified as available for a Queensland govt dept