C6 Melanie Rathgeber and Heidi Johns - Building a Measurement Plan for QI Projects
Transcript of C6 Melanie Rathgeber and Heidi Johns - Building a Measurement Plan for QI Projects
Building a Measurement Plan for QI Projects
Quality Forum 2013
Melanie Rathgeber
MERGE Consulting
Heidi Johns
BC Patient Safety & Quality Council
This course if for you if ….
Objectives
This course is designed to demonstrate:
1. Guidelines for choosing project indicators
2. Tips for collecting data
3. The use of run charts to display data
http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx
Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed).
Provost L, Murray S (2011) The Health Care Data Guide.
Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47
Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51.
Perla R, Provost L, (2012). Judgment sampling: A health care improvement perspective., Q Manage Health Care 21;3: 169-175
Resources
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?
1 4Not at all confident Extremely confident
QI projects
“Systematic, data guided, activities designed to bring about immediate improvement in a health care setting”
Lynn et al. The ethics of using Quality Improvement methods in Health Care, Annals of Internal Medicine. 2007; 146: 666-73
QI projects
Trying to improve something, by changing or introducing a process
- Need to measure something- Need to measure the new process
Purpose of Data in QI Projects
Need to know:- where we started (baseline)- how we change over time (e.g. each week)- when we have reached our target
- Not for judgment (doesn’t go on dashboards or to external agencies)
- Not for research
Run Charts
Measurement Worksheet
Measure Operational Definition
Outcome, Process or Balancing
Data Collection Strategy
Frequency of Data Collection
How will measure be displayed
Baseline result
Target result
Source: Langley et al. (2009). The Improvement Guide. 2nd edition
Lean/Six Sigma
Define
Measure
Analyze
Improve
Control
What are we trying to accomplish?
How will we know that a change is animprovement?
What changes can we make that will result in improvement?
Act Plan
Study Do
Model for Improvement
What are we trying to accomplish?
How will we know that a change is animprovement?
What changes can we make that will result in improvement?
Act Plan
Study Do
Aim Statement
A measurement plan starts with an Aim statement
An Aim statements specifies
What will improve?When will it improve?How much will it improve?For whom will it improve?
Example: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013.
Dissecting the Aim statement
“Some is not a number; soon is not a time” Donald Berwick, Former CEO of IHI
What will improve? Percent of patients seeing their own GP
When will it improve? By May 2013
How much will it improve? From 15% to 90%
For whom will it improve? Diabetes patients at Canada Way Clinic
Examples
What will improve?
When will it improve?
How much will it improve? (numerical goal)
For whom will it improve?
What are we trying to accomplish?
How will we know that a change is animprovement?
What changes can we make that will result in improvement?
Act Plan
Study Do
Family of Measures
Family of Measures
Outcome measures Based on your Aim statement What are we trying to accomplish? What is ultimately better? Voice of the patient/customer
Process measures What are you changing – is it really happening? Voice of the system – what is being done differently? Change more quickly than outcomes
Balancing measures What unintended consequences might occur?
Example
Aim Statement: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013.
Outcome Measure:
Percent of diabetes patients seen by their own GP
Process Measure(s): Go back to Model – Question 3
Balancing Measure(s):
What are we trying to accomplish?
How will we know that a change is animprovement?
What changes can we make that will result in improvement?
Act Plan
Study Do
Identifying, testing, and
implementing changes
Example: What changes can we make?
Changes tested and ready to implement:
- Patients are booked for a follow-up before they leave the office- GP’s set aside Wed morning and Friday afternoon for diabetes
group visits
Process Measure(s)
1. Percent of diabetes patients that leave the clinic with their next appointment booked
2. Number of non-diabetes patients each week who were ‘fit in’ on Wed morning or Friday afternoon
Example: Unintended Consequences
Staff are concerned that other patients will have to wait longer for an appointment, if they are not able to be seen on Wednesday morning and Friday afternoon
Balancing Measure
1. Average wait time for non-diabetes patients between calling for an appointment and being seen
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?
1 4Not at all confident Extremely confident
Family of Measures in Action – An Improvement Project
- What were the outcome/process/balancing measures?
- How were they chosen?
- How was the data useful in driving improvement?
- What was the data showing us?
Where do I start?• I have a hunch
• I need to determine
a target
• How am I going to get
the information
• What do I actually want to
accomplish?
I really needed to develop my AIM
• What was I going to DO • by WHEN • by HOW MUCH
• By September 2011 the wait time between referral and being seen will decrease from X to X
• Change idea: look at referrals coming in to see what the problem is. Maybe need to standardize the process. Hunch that process wasn’t consistent.
• I needed to gather the data to see:• What the actual wait time was• What the referrals looked like
Gathering the data
• Here is what I did………………….
April May June July Aug0%
5%
10%
15%
20%
25%
30%
9%
12%11%
10%
16%
Percent of Forms That Were Illegible
April May June July Aug0
5
10
15
20
25
30
35
40
24
15
17
12
19
Number of Different Forms Seen
Tracking Key Process Measure over Time
Percent of referral forms fully complete
3235
2926 24
80 80
90
0
10
20
30
40
50
60
70
80
90
100
April '11 May June July August Jan '12 Feb April
* Calculations to be confirmed
Starting to Track Time Between Referral and Date Patient Seen for First Appointment
* Calculations to be confirmed
Patient1 Patient2 Patient3 Patient4 Patient5 Patient60
5
10
15
20
25
30
35
40
45
29
1815
30
15 15
Days Between Referral and Appointment Date
Day
s
Family of Measures in Action – An Improvement Project
- What were the outcome/process/balancing measures?
- How were they chosen?
- How was the data useful in driving improvement?
- What was the data showing us?
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?
1 4Not at all confident Extremely confident
Making a Run Chart:
- time along the bottom (X axis)
- results along the side (Y axis)
- a centre line which is the median of all data points on the chart
- each dot on a run chart is the result for:
one case or
one day or
one week or
one month
Making a Run Chart In Excel
Monday Tuesday WednesdayThursday Friday Saturday Sunday Monday Tuesday Wednesday0%
20%
40%
60%
80%
100%
Result
Collecting the data
“Measurement should be used to speed things up, not to slow them down”
- IHI Breakthrough Series Guide
Some tips for getting started
1. Get Started Right Away – Real Time Data on a Run Chart
2. Sampling – Small samples are okay. Sample size increases over time.
3. Seek Usefulness, Not Perfection - Discuss an Operational Definition With Your Team
1. Get Started Right Away – Real Time Data on a Run Chart
Pre = 8 days wait timePost = 3 days wait time
Why not pre and post?
Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss
Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss
Wee
k 1
Wee
k 2
Wee
k 3
Wee
k 4
Wee
k 5
Wee
k 6
Wee
k 7
Wee
k 8
Wee
k 9
Wee
k 10
Wee
k 11
Wee
k 12
Wee
k 13
Wee
k 14
0
1
2
3
4
5
6
7
8
9
10
Scenario 1 when we measure the same thing over time.
Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss
Wee
k 1
Wee
k 2
Wee
k 3
Wee
k 4
Wee
k 5
Wee
k 6
Wee
k 7
Wee
k 8
Wee
k 9
Wee
k 10
Wee
k 11
Wee
k 12
Wee
k 13
Wee
k 14
0
1
2
3
4
5
6
7
8
9
10
change made between week 7 and 8
Scenario 2 when we measure the same thing over time.
Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss
Wee
k 1
Wee
k 2
Wee
k 3
Wee
k 4
Wee
k 5
Wee
k 6
Wee
k 7
Wee
k 8
Wee
k 9
Wee
k 10
Wee
k 11
Wee
k 12
Wee
k 13
Wee
k 14
0
1
2
3
4
5
6
7
8
9
10
change made between week 7 and 8
Scenario 3 when we measure the same thing over time.
Showing improvement:
No improvement. Random fluctuation.
Improvement. Trend going up.
- There are simple rules, based on probability, that are used to determine evidence of improvement in our projects
- Interpretation: the rules tell us if there is a non-random pattern in our data.
- If we have implemented a change, and we see a non-random pattern (going in the right direction), it is evidence of improvement
Analyzing Run Charts
A Shift: 6 or more
An astronomical data point
Too many or too few runs
A Trend5 or more
Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.
The Health Care Data Guide, p 78
Some tips for getting started
2. Sampling – Small samples are okay. Sample size increases over time.
Small samples per day or week are okay
Sample size builds over time
How much data to satisfy team that it is representative?
Simple strategies:
- every 5th patient
- all patients on Thursday morning
If you are reporting externally, or if you want to publish results of QI – may need different strategy
See papers by Perla, Provost, and Murray
Some tips for getting started
3. Seek Usefulness, Not Perfection - Discuss an Operational Definition With Your Team
Operational Definitions
Deciding on an operational definition should be done with your QI team
What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..
Operational Definition Example
Basic definition:
Patient satisfaction ratings from patient survey
Operational Definition Example
Basic definition:
Patient satisfaction ratings from patient survey
Operational definition:
Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey
Operational Definition Example
Basic definition:
Patient satisfaction ratings from patient survey
Operational definition:
Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey
“The Data Are Wrong”
“The Data Are Wrong”
Not a matter of right versus wrong
What is your operational definition?
Involve others from the start in this decision.
Are we good at using data to drive decisions? Do we have a plan of action?
• What if we are not seeing evidence of improvement?
• What if we see improvement but not on target?
• How long do we collect the data?
Data Display Principles
April May June July Aug Sept Oct Nov Dec Jan0%
5%
10%
15%
20%Percent of Forms That Were Illegible
April May June July Aug Sept Oct Nov Dec Jan0%
5%
10%
15%
20%Percent of Forms That Were Illegible
April May June July Aug Sept Oct Nov Dec Jan0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Percent of Forms That Were Illegible
April May June July Aug Sept Oct Nov Dec Jan0%
5%
10%
15%
20%Percent of Forms That Were Illegible
*hypothetical data – illustrative purposes only
1 2 3 4 5 6 7 8 9 10 11 12 13 140
5
10
15
20
25
30
35
40
Large Teaching HospitalsLarge Community HosptialsMedium Community HospitalsSmall Community Hospitals
SMALL MULTIPLES – all info on one page
0
10
20
30
40
1/1/
09
2/1/
09
3/1/
09
4/1/
09
5/1/
09
6/1/
09
7/1/
09
8/1/
09
9/1/
09
10/1
/09
11/1
/09
12/1
/09
1/1/
10
2/1/
10
ProvincialReadmission Rate
0
10
20
30
40Large Teaching
Hospitals
010203040
Large Community Hospitals
010203040
Medium Community Hospitals
010203040 Small Community
Hospitals
*hypothetical data – illustrative purposes only
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?
1 4Not at all confident Extremely confident