Are We Making a Difference? Using Measurement to Guide Improvement

Andrew Wray

Quality Forum 2017

Presenter Disclosure

Presenter: Andrew Wray

Relationships with commercial interests: – Nothing to disclose

Performance Measurement in Health Care

• Accountability: for judgement, to reassure the public, to decide how much people are paid, to choose between two options.

• Research: to develop new knowledge, often generalizable

• Improvement: to guide our efforts to change, to monitor our progress, to understand the problem.

Reference: The Improvement Guide, 2nd ed. Langley,

Moen, Nolan, Nolan, Norman & Provost, p. 24

Purpose of Measurement for Improvement

Data can tell us:

- How we are progressing over time

- Where we can get some change ideas

What to measure?

• Each project will need to identify a couple of key measures related to the work

• Will be a mix of process, outcome and balancing measures

• Needs to be meaningful for the team, and reflective of the processes you are improving

We’ll also have measures as part of our PDSA cycles – more later.

Guideline: between 3-8 measures per project

• at least one outcome measure

• at least one process measure

A Few key indicators to track project progress

Family of Measures

Example: Improving Diabetes Care

• Outcome: – HbA1C meeting clinical target

• Process:

– % who have attended structured education session – % screened for retinopathy in the last year – % reporting sufficient social and spiritual supports

• Balancing:

– Staff satisfaction

• For your work, what sorts of indicators are important to track?

Operational Definitions

• Full description of a measure: – What is the measure

– Inclusion/exclusion criteria

– Calculation

– Sample size

– Sampling strategy

– Subgroup frequency

– Data collection strategy

– By whom

– Etc.

Measurement Plan Worksheet

Measure Operational

Definition Outcome, Process or Balancing

Data Collection Strategy

Frequency of Data Collection

How will measure be displayed

Baseline result Target result

Operational Definitions • Option 1

– Number of people who fell

– All patients over 5yrs

– Number of patients with a fall/total number of patients

– Complete enumeration

– Monthly – Data source: DAD

• Option 3

– Number of falls with an injury

– All patients

– Count of all falls in the facility that caused harm

– Sample of 50 charts

– Audit by quality department staff

– Quarterly

• Option 2

– Number of falls

– Patients 70+

– Count of all falls in the facility

– Complete enumeration

– Reports to PSLS

– Monthly

Sampling?

• Complete enumeration is best

• Sampling saves resources – usually recommended – use judgment samples

DATA COLLECTION

• Start right away

• Small, frequent measures

• Integrate into workload

• Timely

• For the indicators you started thinking about, use the measurement plan worksheet to start working on an operational definition.

• So we’re collecting data…

what do we do with it?

• So we’re collecting data…

what do we do with it?

Share it with your team!

The Run Chart: Tracking progress over time

Data displayed in time order

Data is collected and displayed weekly or monthly.

Pre and Post Change Bar Chart – What is the Interpretation?

Scenario 1 . Data displayed in a run chart over time.

0

2

4

6

8

10

change made between week 7 and 8

Scenario 1 . Pre-post data.

Let’s Look at the Data in a Run Chart: Scenario 1

0

1

2

3

4

5

6

7

8

9

10

Scenario 2 . Pre-post data.

Scenario 2 . Data displayed in a run chart over time.

Scenario 2

change made between week 7 and 8

Scenario 3. Pre-post data.

Scenario 3. Data displayed in a run chart over time.

change made between week 7 and 8

Scenario 3

What if we use a t-test? Average Before

Change =70.0

Average After Change =30.1

0

10

20

30

40

50

60

70

80

The t-test shows a significant difference:

t(22)=7.6, p<.001

Adapted from Perla R.J., Provost L.P., & Murray S.K.

(2011). The run chart: a simple analytical tool for

learning from variation in healthcare processes. BMJ

Quality & Safety, 20(1):46-51.

0

10

20

30

40

50

60

70

80

90

Week1

Week2

Week3

Week4

Week5

Week6

Week7

Week8

Week9

Week10

Week11

Week12

Week13

Week14

Week15

Week16

Week17

Week18

Week19

Week20

Week21

Week22

Week23

Week24

pre-change

post-change

Average Before Change

=70.0

Average After

Change =30.1

0

10

20

30

40

50

60

70

80

t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K.

(2011). The run chart: a simple analytical tool for

learning from variation in healthcare processes. BMJ

Quality & Safety, 20(1):46-51.

Data in real-time

• Data in real time allows us to track progress of the work

• Tells us if we getting better or worse

• Tells us if the changes we are testing are working

– The longer the lag, the slower the learning cycle

Visual analysis of run charts

No improvement. Random fluctuation.

Improvement. Trend going up.

Run charts: Evidence of non-random patterns

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Rule 1. Shift

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Rule 2. Trend

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Rule 3. Runs

Data line crosses once Too few runs: total 2 runs

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Rule 4. Astronomical Point

“Measurement should be used to speed things up, not to slow them down”

• - IHI Breakthrough Series Guide

Reference: The Improvement Guide, 2nd ed. Langley,

Moen, Nolan, Nolan, Norman & Provost, p. 24

What about PDSA cycles?

• We’ll have a family of measures to track over the duration of the project but we will also need other measurement

– We’ll want to use measurement to learn what is working when we test changes.

PDSA measurement

• Change idea: phone call reminders

• P: try 5 phone calls

• D: first 5 patients on Monday

• S: Number reached, length of call, did they attend

• A: What next?

Generate light, not heat, with data

• Reinertsen JL, Gosfield AG, Rupp W, Whittington JW. Engaging Physicians in a Shared Quality Agenda. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2007