Kittens are Evil

Heresies in Public Policy #KittensAreEvil

Language

Outcomes-Based Performance Management“Outcomes Based Evaluation”“Outcomes/Results Based Accountability”TM

“Results Based Management”“Payment by Results”

= measure performance by the impact a person/team/organisation/project has in the

world#KittensAreEvil

Research Findings

• Measurement Problem: Outcomes don’t measure impact in people’s lives

• Attribution Problem: Outcomes aren’t delivered by an organisation

• OBPM distorts organisations’ priorities

• OBPM undermines good frontline practice

#KittensAreEvil

Measurement Problem:Outcomes Don’t Measure Impact

#KittensAreEvil

“One clear and compelling answer to the question of "Why measure outcomes?" is: To see if programs really make a difference in the lives of people.” United Way of America

#KittensAreEvil

The outcomes bible…

“Outcomes Based Evaluation”,

by Robert Schalock, 1995

#KittensAreEvil

How to measure an outcome

• Delivery group/control group

• In-depth qualitative research

• Large scale quantitative research – designed by participants

#KittensAreEvil

How to measure an outcome

Minimum post-programme research time?

18 months

#KittensAreEvil

What does get measured?

Netten, A., Beadle-Brown, J., Caiels, J., Forder, J., Malley, J., Smith, N., Trukeschitz, B., Towers, A., Welch, E. and Windle, K. (2011) Adult Social Care Outcomes Toolkit v2.1: Main guidance, PSSRU Discussion Paper 2716/3, Personal Social Services Research Unit, University of Kent, Canterbury

What does get measured?

Netten, A., Beadle-Brown, J., Caiels, J., Forder, J., Malley, J., Smith, N., Trukeschitz, B., Towers, A., Welch, E. and Windle, K. (2011) Adult Social Care Outcomes Toolkit v2.1: Main guidance, PSSRU Discussion Paper 2716/3, Personal Social Services Research Unit, University of Kent, Canterbury

ASCOT

Control group? - No

Qualitative techniques? - No

Quantitative techniques? - Yes

designed by service users? - No

18 month follow up? – Maybe

#KittensAreEvil

Focusing Illusion

“Nothing in life is quite as important as it seems to be while you’re thinking about it”Daniel Kahneman, Professor of Psychology and Public Affairs, Princeton University

#KittensAreEvil

What does get measured?

Quality Dimension

Accountability Dimension

#KittensAreEvil

#KittensAreEvil

#KittensAreEvil

The attribution problem:

Outcomes aren’t delivered by organisations

What is an outcome?

Programme Logic Model

Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and

Program Planning, 2003

Programme Logic Model

Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and

Program Planning, 2003

Programme Logic Model

Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and

Program Planning, 2003

What else is missing?

Program participants?

?

?

“Outcomes are by definition results over which organizations do not have complete control”

John Mayne, “Challenges and Lessons in Implementing Results-Based Management”,

Evaluation, 2007

Theoretical problems:

• Measurement problem: Outcomes don’t measure impact in people’s lives

• Attribution problem: Outcomes aren’t delivered by an organisation

The uncertainty principle in action?

What happens when people implement OBPM?

What’s the evidence?

OBPM creates:

•“Goal displacement”

•“Creaming”

•“Making the numbers”

Burt Perrin “Effective Use and Misuse of Performance Measurement”, American Journal of Evaluation, 1998

Targets for results “frequently distort the direction of programs, diverting attention away from, rather than towards, what the program should be doing.”

Burt Perrin, 1998

“Unintended consequences”:

•focusing on those who are easiest to help

•“difficult” clients are skipped in favor of the “easy” onesS van Thiel and F. L. Leeuw “The Performance Paradox in the Public Sector”, Public Performance and Management Review, 2002

“Ossification, a lack of innovation, tunnel vision and suboptimization”S van Thiel and F. L. Leeuw “The Performance Paradox in the Public Sector”, Public Performance and Management Review, 2002

“Target based performance management always creates ‘gaming’” (my emphasis)

Bevan, G. and Hood, C. “What’s measured is what matters: targets and gaming in the English public health care system”, Public Administration, 2006

Triage “parks” disabled people on Work ProgrammeIndependent, Monday 28th January, 2013

“Work advisers 'pushing jobless into self-employment”BBC, 3rd February, 2013

“Private health contractor's staff told to cut 999 calls to meet targets”Guardian, Wednesday 23 January 2013

“NHS targets 'may have led to 1,200 deaths' in Mid-Staffordshire”Daily Telegraph, March 2009

“A4e employee forged signatures to boost job placement numbers”The Guardian, 6th March, 2012

“Serco gave NHS false data about its GP service 252 times”Guardian, Thursday 20 September 2012

Implementing outcomes approaches

“Always results in gaming”:

• Creaming/cherry picking (helping the easiest to help)

• Targeting resources to produce data (teaching to the test)

• Reclassifying results (pretending)

• Making things up

Impact onfrontline practice

Outcomes Based Accountability

• Makes “it more difficult to engage with and build relationships with homeless and at risk young people”

• Has significant impacts on the daily practice of workers

• Reduces the time available to create a sense of belonging

• Reduces the time to “develop young people’s life skills”

Lynn Keevers (et al) “Made to Measure: Taming Practices with Results-based Accountability”, Organization Studies, 2012

What Social Workers do…

• 86 per cent of time is system driven - filling in forms for accountability and discussing them with colleagues.

• The 14 per cent of time spent face to

face with a family member is not developmental.

Hilary Cottom, Relational Welfare, 2011

Frontline practice

= Reversal of relationship between worker/client

From: how can I help you achieve your goals?

To: how can you help me achieve my targets?

Models of

Outcomes Based Performance Management

Payment by Results:Managing for Genuine Impact?

1. Accept you can’t do it

2. Define an outcome you can live with

3. Set targets for performance

5. Providers deliver activity

6. Gather outcomes data

7. Realise limits of influence

8. Game:

•Cherry-pick•Teach to the test•Reclassify•Change practice to focus on data•Make up data

4. Providers plan activity to meet outcome targets

OBPM: Managing Performance for Genuine Attributability

1. Define performance you are responsible for

2. Set targets for performance

4. Deliver activity

5. Gather performance data

6. Act so as to improve performancedata

3. Providers plan activity to meet performance targets

Summary

• Outcomes don’t measure impact in people’s lives

• Outcomes aren’t delivered by an organisation

• OBPM distorts organisations’ priorities

• OBPM undermines good frontline practice

If not outcomes, then what?

• Bottom up is key – start from actual people’s needs

• Deal with complexity: Put human judgement on the frontline

• Trust & Transparency

• Use social theory: measure change in social context and identity

• Something about identity

Thanks for listening

Toby Lowe

E: tobylowe@hotmail.co.uk

Twitter: @tobyjlowe