Building a Measurement Plan for QI Projects

Quality Forum 2014

Melanie Rathgeber

Merge Consulting

Melanie@merge.ca

www.mergeconsulting.ca

Geoff Schierbeck

BC Patient Safety & Quality Council

gschierbeck@bcpsqc.ca

www.bcpsqc.ca

What is your experience with Measures and Data in Healthcare?

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

Traditional Problems with Data

1. The data are wrong.

2. The data are too old.

3. Don’t know what is might not be statistically significant?

4. We need to focus on this “outlier” or “trend”.

The data are wrong. Translation: stakeholders don’t agree with the way the data was collected in the first place.

The data are too old. Translation: Things have changed now; things are better now. These results don’t reflect our current environment.

These results might not be statistically significant. Translation: It is important to know what this data means before we jump to conclusions.

We need to focus on this “outlier” or “trend”.

Translation: There are results that jump out at us and we naturally feel compelled to fix them.

http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx

Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed). Provost L, Murray S (2011) The Health Care Data Guide. Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47 Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51. Perla R, Provost L, (2012). Judgment sampling: A health care improvement perspective., Q Manage Health Care 21;3: 169-175

Resources

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Voting

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

QI projects

“Systematic, data guided, activities designed to bring about immediate improvement in a health care setting”

Lynn et al. The ethics of using Quality Improvement methods in Health Care, Annals of

Internal Medicine. 2007; 146: 666-73

Purpose of Data in QI Projects

Need to know: - where we started (baseline) - how we change over time (e.g. each week) - when we have reached our target

- Not for Accountability (doesn’t go on dashboards or to external

agencies) - Not for Research

Source: Langley et al. (2009). The Improvement Guide. 2nd edition

Lean/Six Sigma

Define Measure Analyze Improve Control

What are we trying to accomplish?

How will we know that a change is an improvement?

What changes can we make that will result in

improvement?

Act Plan Study Do

Model for Improvement

What are we trying to accomplish?

How will we know that a change is an improvement?

What changes can we make that will result in

improvement?

Act Plan Study Do

Aim Statement

A measurement plan starts with an Aim statement

An Aim statements specifies What will improve? When will it improve? How much will it improve? For whom will it improve?

Example: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2014.

Dissecting the Aim statement

“Some is not a number; soon is not a time” Donald Berwick, Former CEO of IHI

What will improve? Percent of patients seeing their own GP When will it improve? By May 2013 How much will it improve? From 15% to 90% For whom will it improve? Diabetes patients at Canada Way Clinic

Examples – share with partner

What will improve? When will it improve? How much will it improve? (numerical goal) For whom will it improve?

What are we trying to accomplish?

How will we know that a change is an improvement?

What changes can we make that will result in

improvement?

Act Plan Study Do

Family of Measures

Family of Measures Outcome measures Based on your Aim statement What is ultimately better? Voice of the patient/customer

Process measures What are you changing – is it really happening? Voice of the system – what is being done differently? Change more quickly than outcomes

Balancing measures What unintended consequences might occur?

Example Aim Statement: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013. Outcome Measure: Percent of diabetes patients seen by their own GP Process Measure(s): Balancing Measure(s):

What are we trying to accomplish?

How will we know that a change is an improvement?

What changes can we make that will result in

improvement?

Act Plan Study Do

Identifying, testing, and

implementing changes

Process measures are based on the changes you plan to make:

- Changes that are tested and ready to implement:

- Example: - Patients are booked for a follow-up before they leave the office

Process Measure(s)

1. Percent of diabetes patients that leave the clinic with their next appointment booked

Example: Balancing Measure

Staff are concerned that other patients will have to wait longer for an appointment, if they are not able to be seen on Wednesday morning and Friday afternoon

Example: Balancing Measure

1. Average wait time for non-diabetes patients between calling for an appointment and being seen

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Geoff’s project: Listen for the following:

- What were the outcome/process/balancing measures?

- How were they chosen?

- How was the data useful in driving improvement?

- What was the data showing us?

Ophthalmology Turn Over Times

Where do I start? • I have a hunch

• I need to determine a target

• How am I going to get the information • What do I actually want to accomplish?

I really needed to develop my AIM

• What was I going to DO • by WHEN • by HOW MUCH

Aim Statement • By September the ophthalmology clinic turn over time will

be decreased to 3 minutes. (feet out to feet in)

• Change idea: each person is responsible to perform certain tasks to avoid the questions of who has done what. Maybe need to standardize the process.

• Hunch “that there is no consistency in tasks”

• I needed to gather the data to see:

• What the actual turn over time was • What is the optimal turn over time

Collecting Data

0

10

20

30

40

50

60

70

80

90

100

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

Participation in Checklist

Where did the project start showing improvement?

0

10

20

30

40

50

60

70

80

90

100

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

Participation in Checklist

Where did the project start showing improvement?

0

10

20

30

40

50

60

70

80

90

100

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

Participation in Checklist

Time in min

Turn Over Times

0

2

4

6

8

10

12

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

Goal

Time in min

Turn Over Times

0

2

4

6

8

10

12

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

Checklist vs Turn Over Times

0

10

20

30

40

50

60

70

80

90

100

Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb

- What were the outcome/process/balancing measures?

- How were they chosen?

- How was the data useful in driving improvement?

- What was the data showing us?

Partner discussion …

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Voting

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

Run Charts

A Simple Run Chart: Building Block of Measurement for Improvement

49

Data displayed in time order

Data is collected and displayed weekly or monthly.

A Simple Run Chart

50

Data displayed in time order. Time is along X axis

Data is collected and displayed weekly or monthly.

Centre line = median of the data points or = baseline value -Result along Y axis

-One “dot” = one sample of data

-Sample size = each “dot” should have the same n

0%

20%

40%

60%

80%

100%

April May June July Aug Sept Oct Nov Dec Jan

Percent of Patients with Diabetes with Self-Management Goals

Documented

The Value of Data in Real Time

51

Run Charts can tell us what is happening in real time.

The Value of Data Over Time

52

Run Charts detect true patterns and trends over time. Not just what is happening before and after a change.

Pre and Post Change Bar Chart – What is the Interpretation?

54

Scenario 1 . Data displayed in a run chart over time. 0

2

4

6

8

10

change made between week 7 and 8

Scenario 1 . Pre-post data.

Let’s Look at the Data in a Run Chart: Scenario 1

55

0123456789

10

Wee

k 1

Wee

k 2

Wee

k 3

Wee

k 4

Wee

k 5

Wee

k 6

Wee

k 7

Wee

k 8

Wee

k 9

Wee

k…

Wee

k…

Wee

k…

Wee

k…

Wee

k…

Scenario 2 . Pre-post data.

Scenario 2 . Data displayed in a run chart over time.

Scenario 2

change made between week 7 and 8

56

Scenario 3. Pre-post data.

Scenario 3. Data displayed in a run chart over time.

change made between week 7 and 8

Scenario 3

57

Scenario 4: What if we use a t-test? Average Before

Change =70.0

Average After Change =30.1

0

10

20

30

40

50

60

70

80

The t-test shows a significant difference: t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray

S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51.

0

10

20

30

40

50

60

70

80

90

pre-change

post-change

Average Before Change

=70.0

Average After

Change =30.1

0

10

20

30

40

50

60

70

80

t(22)=7.6, p<.001 change is not sustained

Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51.

Making a Run Chart:

- time along the bottom (X axis)

- results along the side (Y axis)

- a centre line which is the median of all data points on the chart

- each dot on a run chart is the result for:

one case or one day or one week or one month

Option 1: Making a Run Chart In Excel

Run Chart Excel Template http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx note: you need to complete a one time free registration

62

Option 2: Run Chart Template from IHI

- There are simple rules, based on probability, that are used

to determine evidence of improvement in our projects

- Interpretation: the rules tell us if there is a non-random pattern in our data.

- If we have implemented a change, and we see a non-random pattern (going in the right direction), it is evidence of improvement

Analyzing Run Charts

A Shift: 6 or more

An astronomical data point

Too many or too few runs

A Trend 5 or more

Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.

Perla, Provost, Murray (2011)

Shifts and Trends:

A Shift: 6 or more

consecutive points

all on one side of

the median

A Trend: 5 or more

consecutive points all

increasing or all

decreasing

Perla, Provost, Murray (2011)

0

10

20

30

40

50

A Trend: Five or More Consecutive Points All Going in the Same Direction

Example: The Trend Rule

0

10

20

30

40

50

60

Jan-10 Mar-10May-10 Jul-10 Sep-10 Nov-10 Jan-11 Mar-11May-11 Jul-11 Sep-11 Nov-1

Number of Falls

Jan-10

Feb-10

Mar-10

Apr-10

May-10

Jun-10

Jul-10

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-11

Feb-11

Mar-11

Apr-11

May-11

Jun-11

Jul-11

Aug-11

Sep-11

Oct-11

Nov-11

Number of falls 21 23 24 21 22 24 35 37 32 33 34 25 24 17 20 17 19 21 20 15 14 19 16

Baseline median 24 24 24 24 24 24 24 24 24 24 24

Baseline median extended 24 24 24 24 24 24 24 24 24 24 24 24 24

Number of Falls in Facility X Jan 2010 - Nov 2011

Example: The Shift Rule

Plain Language Interpretation?

0

10

20

30

40

50

60

Jan-10 Apr-10 Jul-10 Oct-10 Jan-11 Apr-11 Jul-11 Oct-11

Number of Falls

Jan-10

Feb-10

Mar-10

Apr-10

May-10

Jun-10

Jul-10

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-11

Feb-11

Mar-11

Apr-11

May-11

Jun-11

Jul-11

Aug-11

Sep-11

Oct-11

Nov-11

Number of falls 21 23 24 21 22 24 35 37 32 33 34 25 24 17 20 17 19 21 20 15 14 19 16

Baseline median 24 24 24 24 24 24 24 24 24 24 24

Baseline median extended 24 24 24 24 24 24 24 24 24 24 24 24 24

Number of Falls in Facility X Jan 2010 - Nov 2011

There is evidence of improvement – the chance we would see a “shift” like this in data if there wasn’t a real change in what we were doing is less than 5%

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Voting

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

Small samples per day or week are okay. Sample size builds over time How much data to satisfy team that it is representative? Simple strategies: - every 5th patient - all patients on Thursday morning If you are reporting externally, or if you want to publish results of QI – may need different strategy See papers by Perla, Provost, and Murray

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

Operational Definitions Deciding on an operational definition should be done with your QI team What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..

Operational Definition Example

Basic definition: Patient satisfaction ratings from patient survey

Operational Definition Example

Basic definition: Patient satisfaction ratings from patient survey Operational definition: Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey

“The Data Are Wrong” Not a matter of right versus wrong What is your operational definition? Involve others from the start in this decision.

Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips

Data Display Principles

0%

5%

10%

15%

20%

April May June July Aug Sept Oct Nov Dec Jan

Percent of Forms That Were Illegible

0%

5%

10%

15%

20%

April May June July Aug Sept Oct Nov Dec Jan

Percent of Forms That Were Illegible

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

April May June July Aug Sept Oct Nov Dec Jan

Percent of Forms That Were Illegible

0%

5%

10%

15%

20%

April May June July Aug Sept Oct Nov Dec Jan

Percent of Forms That Were Illegible

*hypothetical data – illustrative purposes only

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Large Teaching Hospitals

Large Community Hosptials

Medium Community Hospitals

Small Community Hospitals

SMALL MULTIPLES – all info on one page

0

10

20

30

40

1/1

/09

2/1

/09

3/1

/09

4/1

/09

5/1

/09

6/1

/09

7/1

/09

8/1

/09

9/1

/09

10

/1/0

9

11

/1/0

9

12

/1/0

9

1/1

/10

2/1

/10

ProvincialReadmission Rate

010203040

Large Teaching Hospitals

010203040

Large Community Hospitals

010203040

Medium Community Hospitals

010203040 Small Community

Hospitals

*hypothetical data – illustrative purposes only

QI Responses to Traditional Problems with Data

1. The data are wrong.

2. The data are too old.

3. Don’t know what is might not be statistically significant?

4. We need to focus on this “outlier” or “trend”.

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Voting