Measurement for improvement

© NHS Improving Quality 2014© NHS Improving Quality 2014

Measurement for Improvement

Patient Safety Team

© NHS Improving Quality 2014

• What are your expectations of the next three hours?


Aims and objectives

Aims• To have a greater knowledge and practical

understanding of Data Collection and Analysis

Objectives• Understand how to do ‘real’ analysis• Use data to improve decision-making and identify

real improvements

3


Introduction

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Distributions – “How long do cars last?”

68.26%

95.44%

99.73%

4


“Measurement is for improvement not judgement.”

D. Berwick

Measurement for improvement5


“You can’t fatten a cow by weighing it” (Palestinian proverb)

Improvement is not about measurement, but……..

How do we know if a change is an improvement?

“If you can’t measure it, you can’t improve it”

Measurement for improvement6


Model for improvement

A PDS

A PDS

7


Measurement throughout the project cycle

Project Identification

Getting a baseline

Did project make a

difference

Will project sustain

Evaluating worth of the

project

A PDS

A PDS

A PDS

A PDS

A PDS

8


7 Repeat steps 4-6

Seven steps to measurement

1 Decide Aim

2 Choose Measures

3 Define Measures

4 Collect Data

5 Analyse and Present

6 Review Measures

9


Step 2 – Choose Measures

10


Step 2 – What to measure?

Patient experience and outcomes

Patient satisfaction

survey

% patients complication

free in recoveryPain score Average time

patient starved

Safety and reliability Clinical incidents ReadmissionsExceptions from

time out checklist

% correct equipment to

hand

Efficiency and value Turnaround time % theatre utilisation Cancellations

Delays(Late starts &

finishes)

Leadership and high performing teams Staff survey Training and

development Staff turnover Staff absence

11


Driver DiagramsSchematic view of a system on the left we depict outcomeAs we move right we drill down into the network of causes that drive the outcome, from ‘primary’ to ‘secondary’ drivers.

Aim: An improved system

PrimaryDriver

PrimaryDriver

Secondary

Driver

Secondary

Driver

Secondary

Driver

Secondary

Driver

Secondary

Driver

12


Driver Diagrams

On the right we depict ideas for system changes that might ultimately impact the outcome. Diagram represents our theory about how to modify the system to change the outcome.

Aim: An improved system

PrimaryDriver

PrimaryDriver

Change

Change

Change

Secondary

Driver

Secondary

Driver

Secondary

Driver

Secondary

Driver

Secondary

Driver

13


Aim: 2 stones lighter!

Energy Out

Energy In

Walk daily commute

Stairs not lift

Exercise

Reduce alcohol intake

Eat Less

Driver Diagrams – weight loss example

14

Pedometer

Gym work out 3 days

Squash weekends

No pub weekdays

Take packed lunch

Low fat meals


Aim

Defect FreeSurgery

Avoid Mistakes

AvoidComplications

AvoidDelays

Driver Intervention

Conduct team brief

Conduct team debrief

Conduct time out

Produce accurate lists

Implement SSI bundle

Implement VTE bundle

Have correct kit to hand

Ensure staff adequately trained

O1

O2

O3

O1 Overall glitch countO2 Never eventsO3 Number of surgical site infections

P1 % lists with Team BriefP2 % lists with Team DebriefP3 % compliance with SSI bundle

P2

P1

P3

Driver Diagramsclinical example

15


Exercise: Create your driver diagram

Thinking about your project• Start to create a driver diagram and choose measures

• Complete the driver diagram to link Aims with Measures

You have 20 minutes

16


Step 2 – Choose Measures

An important note:

As well as clinical and quality measures – you may need to consider what financial measures are required for your project? You may need initial and ongoing funding?

Your success in gaining access to funds will be helped if you have completed a project financial justification or return on project investment analysis

Covered in more detail later in the presentation …

17

•Measurement Definitions

18


Step 3 – Define Measures

• An operational definition is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently

• Are we measuring the same thing?

19


Exercise

• M&M Activity

20


RepeatabilityCan you, who created the definition, understand it and repeat it? Also known as test-retest error, used as an estimate of short-term variation

ReproducibilityAfter repeatability, try seeing if

the definition that you have created can be reproduced by other people?

Advice on creating measurement definitions

21


•The Measures Checklist

•Why important?•Who owns?•Definitions?•Goals?

22


Step 4 – Collect DataPractical considerations:5 W’s and 1H

23


Step 4 – Collect DataDecisions, decisions… • What - All patients or a sample?• Who – took the data? (what role?)• When – When was the data taken - real time or

retrospective?• Where is the data from?• How – was the data taken - hospital system or

audit?

• Turn the data into a different unit (hours into days)

24


Step 5 – Analyse and Present

We will now focus in more detail on methods of presenting and analysing our chosen measures….

25

•Understanding •and dealing with

• variation in analysis

26


The Airplane GameUsing the paper provided – make a paper planeYou have 5 minutes

• When instructed – throw your planes!

• What happened?

• Why are they not flying the same distance?

27

Exercise – The plane game


Fishbone diagram

Aeroplanes fly different

distances

Problem

Equipment People Procedures

Measurement (+Diagnostic)

Communication Materials

Causes

Types of paper e.g.

card, tracing paper,

No clear instructions provided

Some tables had scissors, rulers to help

Skills / ideas

Throwing styles

28


“We live in a world filled with variation –and yet there is very little recognition or understanding of variation”

William Scherkenbach

“Data should always be presented in such a way that preserves the evidence in the data…”

Walter Shewhart

Variation

29

What do people understand by the word variation?


What does this data tell us?

Patients treated in April

600

550

610

540

560

570

580

590

2008 2009

30



Patients treated

650

600

550

500

450

400

350

300April 2008 April 2009

31



This Month Last Month

Given two different numbers, one will always be bigger than the other!

Som

ethi

ng I

mpo

rtan

t

What action is appropriate?

32


CommonCause

Stable in time and therefore relatively predictable

Paper selection

Persons technique

Design of the plane

Special Cause

Irregular in time and therefore unpredictable

Water spill

Fire

Can we classify variation?

33


It is important to distinguish between the two types of variation because they require different approaches to deal with them

Can we classify variation?

“There are different improvement strategies depending of which type of variation is present (common cause or special cause), so it is important for a team to know the difference.”

M.L. George

34


Step 5 – Analyse and present

Presentation

Dot plotIndividual value Plot

Interval Plot

Bar Chart

RunChart

SPC

Pareto

Scatter Plot

Tally Chart

Pie Chart

Radar Chart

Area Chart

35

SPC

Run Char

t


Plotting the dots - example Run ChartNumber of calls to outreach team (weekly)November 2007 to June 2008

0

No

of C

alls

180

160

140

120

100

80

60

40

20

1st Nov 15th Nov 29th Nov 13th Dec 27th Dec 10th Jan 24th Jan 7th Feb 21st Feb 6th Mar 20th Mar 3rd Apr 17th Apr 1st May

Week

Calls per week Median

36


The Myth of Trends

Upward trend ? Downward trend ?

Downturn ?Setback ?

Turnaround ?Rebound?

Static ?Flatline ?

37


Time

Downward trend

Time

Upward trend

Looking for a trend

7 points all in upward direction

7 points all in downward direction

38


Looking for a trend

7 points above centre line 7 points below centre line

Time

Below centre

Time

Above centre

39


Exercise 7Creating your own chart

• Using graph paper, ruler and a pencil:

1. Draw and label the axis2. Plot the dots (daily or weekly data is the best)3. Work out the median and plot it4. Add a title (with dates)5. Add a legend6. Analyse it!

• You have 20 minutes

40


100908070605040302010

%

% Compliance with hand hygiene (weekly) April – Sept 2008

Week1 Apr 15 298 22 6 May13 20 273 Jun10 17 241 Jul 8 15 22 295 Aug12 19 262 Sep9 16

xx

x

x

x

x

x

x

xx

x

x

xx

x

xx

x

xx

x

xx

xx

% Compliance with hand hygiene Medianx

Your chart… should look like this?

41


Discussion …

• How could you apply these charts to your projects?

42

•What is SPC?

44


Statistical Process Control(SPC) charts…

• …use the pattern of events in the past to predict with some degree of certainty where future events should fall

• …distinguish between the natural/common cause • variation and special cause variation• …enable you to look for problems when they are there, not

when they are not• …can motivate staff to improve practice thereby reducing

adverse events and minimising variation

45


Statistical Process Control(SPC) Charts:

No

of A

dmis

sion

s

0

200

150

100

50

250

Performance Report – Number of Admissions

Week

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

No Admissions Median Lower Limit (66.5) Upper Limit (222.4)

46


Two ways to improve a process

If uncontrolled (special) variation

Process is unstable and unpredictable

Variation caused by factors outside process

External cause should be identified and sorted

If controlled (common) variation

Process is stable and predictable

Variation is inherent to the process

Therefore the process must be changed

47


The improvement process

Pat

ient

Wai

ting

Tim

e

0

200

150

100

50

250

Performance Report

Week

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Act Plan

DoStudy

Special causes present -

unpredictable

Process predictable

(within control limits)

Process improvement

48


Three dangers to beware of …

1. Reacting to special cause variation by changing the process

2. Ignoring special cause variation by assuming “it’s part of the process”

3. Comparing more than one process

49

•Interpreting •Charts

50


LCL

UCL

MEAN

Point above UCL

Special causes

XX

X

X

X

X

X

X

X

X

Point below LCL

X

X

X

XX

X

X

X

X

X

Rule 1 - Any point outside one of the control limits

51


LCL

UCL

MEAN

7 Points above centre line

Special causes

Rule 2 - A run of seven points all above or all below the centre line, or all increasing or all decreasing

XX

X

X

XX

X

XXX X

X

X

XX

X

X

XX

X

7 Points below centre line

52


LCL

UCL

MEAN

7 Points upward direction

Special causes

Rule 2 - A run of seven points all above or all below the centre line, or all increasing or all decreasing

7 Points downward directionX

X

XX

X X

X

X

X

X XX

X

XX

X

X

X

53


LCL

UCL

MEAN

Cyclic pattern

Special causes

Rule 3 - Any “unusual” pattern or trends within the control limits

Trend pattern

XX

X

X

X

XXX

X

X

X

X

X

X

X

X

X

X

X

X

XX

XX

XX

X

X

X

X

X

X

X

XX

X

X

54


LCL

UCL

MEAN

Less than 2/3 of all the points fall in this zone

Special causes

Rule 4

More than 2/3 of all the points fall in this zone

X

X

X XX

X

X

X

X

XX

XX

X

X

X

X

XX

X

X

XX

X

X

X

XX

X

XX

X

X

XX

XX

X

55


Process out of control

• These rules are important!• They tell us if the process is stable or unstable• They tell us if common or special cause • variation is present

Remember the rules!• Outside control limits• Run of 7 or more consecutive points• Patterns• Rule of thirds

56


Exercise 8Interpreting SPC charts

•Apply the SPC rules to the charts in the handout

• Are there rules present?• Is the chart in control?

57


Step 6 – Review Measures

“It is a waste of time collecting and analysing your data if you don't take action on the results

“

58

Measurement for improvement

Healthcare

Transcript of Measurement for improvement