Measurement Basics
-
Upload
bcpsqc -
Category
Health & Medicine
-
view
476 -
download
0
description
Transcript of Measurement Basics
Building a Measurement Plan for QI Projects
Quality Forum 2014
Melanie Rathgeber
Merge Consulting
www.mergeconsulting.ca
Geoff Schierbeck
BC Patient Safety & Quality Council
www.bcpsqc.ca
What is your experience with Measures and Data in Healthcare?
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
Traditional Problems with Data
1. The data are wrong.
2. The data are too old.
3. Don’t know what is might not be statistically significant?
4. We need to focus on this “outlier” or “trend”.
The data are wrong. Translation: stakeholders don’t agree with the way the data was collected in the first place.
The data are too old. Translation: Things have changed now; things are better now. These results don’t reflect our current environment.
These results might not be statistically significant. Translation: It is important to know what this data means before we jump to conclusions.
We need to focus on this “outlier” or “trend”.
Translation: There are results that jump out at us and we naturally feel compelled to fix them.
http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx
Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed). Provost L, Murray S (2011) The Health Care Data Guide. Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47 Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51. Perla R, Provost L, (2012). Judgment sampling: A health care improvement perspective., Q Manage Health Care 21;3: 169-175
Resources
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
Voting
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
QI projects
“Systematic, data guided, activities designed to bring about immediate improvement in a health care setting”
Lynn et al. The ethics of using Quality Improvement methods in Health Care, Annals of
Internal Medicine. 2007; 146: 666-73
Purpose of Data in QI Projects
Need to know: - where we started (baseline) - how we change over time (e.g. each week) - when we have reached our target
- Not for Accountability (doesn’t go on dashboards or to external
agencies) - Not for Research
Source: Langley et al. (2009). The Improvement Guide. 2nd edition
Lean/Six Sigma
Define Measure Analyze Improve Control
What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in
improvement?
Act Plan Study Do
Model for Improvement
What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in
improvement?
Act Plan Study Do
Aim Statement
A measurement plan starts with an Aim statement
An Aim statements specifies What will improve? When will it improve? How much will it improve? For whom will it improve?
Example: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2014.
Dissecting the Aim statement
“Some is not a number; soon is not a time” Donald Berwick, Former CEO of IHI
What will improve? Percent of patients seeing their own GP When will it improve? By May 2013 How much will it improve? From 15% to 90% For whom will it improve? Diabetes patients at Canada Way Clinic
Examples – share with partner
What will improve? When will it improve? How much will it improve? (numerical goal) For whom will it improve?
What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in
improvement?
Act Plan Study Do
Family of Measures
Family of Measures Outcome measures Based on your Aim statement What is ultimately better? Voice of the patient/customer
Process measures What are you changing – is it really happening? Voice of the system – what is being done differently? Change more quickly than outcomes
Balancing measures What unintended consequences might occur?
Example Aim Statement: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013. Outcome Measure: Percent of diabetes patients seen by their own GP Process Measure(s): Balancing Measure(s):
What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in
improvement?
Act Plan Study Do
Identifying, testing, and
implementing changes
Process measures are based on the changes you plan to make:
- Changes that are tested and ready to implement:
- Example: - Patients are booked for a follow-up before they leave the office
Process Measure(s)
1. Percent of diabetes patients that leave the clinic with their next appointment booked
Example: Balancing Measure
Staff are concerned that other patients will have to wait longer for an appointment, if they are not able to be seen on Wednesday morning and Friday afternoon
Example: Balancing Measure
1. Average wait time for non-diabetes patients between calling for an appointment and being seen
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
Geoff’s project: Listen for the following:
- What were the outcome/process/balancing measures?
- How were they chosen?
- How was the data useful in driving improvement?
- What was the data showing us?
Ophthalmology Turn Over Times
Where do I start? • I have a hunch
• I need to determine a target
• How am I going to get the information • What do I actually want to accomplish?
I really needed to develop my AIM
• What was I going to DO • by WHEN • by HOW MUCH
Aim Statement • By September the ophthalmology clinic turn over time will
be decreased to 3 minutes. (feet out to feet in)
• Change idea: each person is responsible to perform certain tasks to avoid the questions of who has done what. Maybe need to standardize the process.
• Hunch “that there is no consistency in tasks”
• I needed to gather the data to see:
• What the actual turn over time was • What is the optimal turn over time
Collecting Data
0
10
20
30
40
50
60
70
80
90
100
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
Participation in Checklist
Where did the project start showing improvement?
0
10
20
30
40
50
60
70
80
90
100
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
Participation in Checklist
Where did the project start showing improvement?
0
10
20
30
40
50
60
70
80
90
100
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
Participation in Checklist
Time in min
Turn Over Times
0
2
4
6
8
10
12
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
Goal
Time in min
Turn Over Times
0
2
4
6
8
10
12
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
Checklist vs Turn Over Times
0
10
20
30
40
50
60
70
80
90
100
Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
- What were the outcome/process/balancing measures?
- How were they chosen?
- How was the data useful in driving improvement?
- What was the data showing us?
Partner discussion …
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
Voting
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
Run Charts
A Simple Run Chart: Building Block of Measurement for Improvement
49
Data displayed in time order
Data is collected and displayed weekly or monthly.
A Simple Run Chart
50
Data displayed in time order. Time is along X axis
Data is collected and displayed weekly or monthly.
Centre line = median of the data points or = baseline value -Result along Y axis
-One “dot” = one sample of data
-Sample size = each “dot” should have the same n
0%
20%
40%
60%
80%
100%
April May June July Aug Sept Oct Nov Dec Jan
Percent of Patients with Diabetes with Self-Management Goals
Documented
The Value of Data in Real Time
51
Run Charts can tell us what is happening in real time.
The Value of Data Over Time
52
Run Charts detect true patterns and trends over time. Not just what is happening before and after a change.
Pre and Post Change Bar Chart – What is the Interpretation?
54
Scenario 1 . Data displayed in a run chart over time. 0
2
4
6
8
10
change made between week 7 and 8
Scenario 1 . Pre-post data.
Let’s Look at the Data in a Run Chart: Scenario 1
55
0123456789
10
Wee
k 1
Wee
k 2
Wee
k 3
Wee
k 4
Wee
k 5
Wee
k 6
Wee
k 7
Wee
k 8
Wee
k 9
Wee
k…
Wee
k…
Wee
k…
Wee
k…
Wee
k…
Scenario 2 . Pre-post data.
Scenario 2 . Data displayed in a run chart over time.
Scenario 2
change made between week 7 and 8
56
Scenario 3. Pre-post data.
Scenario 3. Data displayed in a run chart over time.
change made between week 7 and 8
Scenario 3
57
Scenario 4: What if we use a t-test? Average Before
Change =70.0
Average After Change =30.1
0
10
20
30
40
50
60
70
80
The t-test shows a significant difference: t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray
S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51.
0
10
20
30
40
50
60
70
80
90
pre-change
post-change
Average Before Change
=70.0
Average After
Change =30.1
0
10
20
30
40
50
60
70
80
t(22)=7.6, p<.001 change is not sustained
Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51.
Making a Run Chart:
- time along the bottom (X axis)
- results along the side (Y axis)
- a centre line which is the median of all data points on the chart
- each dot on a run chart is the result for:
one case or one day or one week or one month
Option 1: Making a Run Chart In Excel
Run Chart Excel Template http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx note: you need to complete a one time free registration
62
Option 2: Run Chart Template from IHI
- There are simple rules, based on probability, that are used
to determine evidence of improvement in our projects
- Interpretation: the rules tell us if there is a non-random pattern in our data.
- If we have implemented a change, and we see a non-random pattern (going in the right direction), it is evidence of improvement
Analyzing Run Charts
A Shift: 6 or more
An astronomical data point
Too many or too few runs
A Trend 5 or more
Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.
Perla, Provost, Murray (2011)
Shifts and Trends:
A Shift: 6 or more
consecutive points
all on one side of
the median
A Trend: 5 or more
consecutive points all
increasing or all
decreasing
Perla, Provost, Murray (2011)
0
10
20
30
40
50
A Trend: Five or More Consecutive Points All Going in the Same Direction
Example: The Trend Rule
0
10
20
30
40
50
60
Jan-10 Mar-10May-10 Jul-10 Sep-10 Nov-10 Jan-11 Mar-11May-11 Jul-11 Sep-11 Nov-1
Number of Falls
Jan-10
Feb-10
Mar-10
Apr-10
May-10
Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-11
Feb-11
Mar-11
Apr-11
May-11
Jun-11
Jul-11
Aug-11
Sep-11
Oct-11
Nov-11
Number of falls 21 23 24 21 22 24 35 37 32 33 34 25 24 17 20 17 19 21 20 15 14 19 16
Baseline median 24 24 24 24 24 24 24 24 24 24 24
Baseline median extended 24 24 24 24 24 24 24 24 24 24 24 24 24
Number of Falls in Facility X Jan 2010 - Nov 2011
Example: The Shift Rule
Plain Language Interpretation?
0
10
20
30
40
50
60
Jan-10 Apr-10 Jul-10 Oct-10 Jan-11 Apr-11 Jul-11 Oct-11
Number of Falls
Jan-10
Feb-10
Mar-10
Apr-10
May-10
Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-11
Feb-11
Mar-11
Apr-11
May-11
Jun-11
Jul-11
Aug-11
Sep-11
Oct-11
Nov-11
Number of falls 21 23 24 21 22 24 35 37 32 33 34 25 24 17 20 17 19 21 20 15 14 19 16
Baseline median 24 24 24 24 24 24 24 24 24 24 24
Baseline median extended 24 24 24 24 24 24 24 24 24 24 24 24 24
Number of Falls in Facility X Jan 2010 - Nov 2011
There is evidence of improvement – the chance we would see a “shift” like this in data if there wasn’t a real change in what we were doing is less than 5%
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
Voting
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
Small samples per day or week are okay. Sample size builds over time How much data to satisfy team that it is representative? Simple strategies: - every 5th patient - all patients on Thursday morning If you are reporting externally, or if you want to publish results of QI – may need different strategy See papers by Perla, Provost, and Murray
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
Operational Definitions Deciding on an operational definition should be done with your QI team What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..
Operational Definition Example
Basic definition: Patient satisfaction ratings from patient survey
Operational Definition Example
Basic definition: Patient satisfaction ratings from patient survey Operational definition: Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey
“The Data Are Wrong” Not a matter of right versus wrong What is your operational definition? Involve others from the start in this decision.
Objectives This course is designed to demonstrate: 1. Guidelines for choosing indicators 2. The use of run charts to display data 3. Importance of sampling 4. Importance of operational definitions 5. Data Display tips
Data Display Principles
0%
5%
10%
15%
20%
April May June July Aug Sept Oct Nov Dec Jan
Percent of Forms That Were Illegible
0%
5%
10%
15%
20%
April May June July Aug Sept Oct Nov Dec Jan
Percent of Forms That Were Illegible
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
April May June July Aug Sept Oct Nov Dec Jan
Percent of Forms That Were Illegible
0%
5%
10%
15%
20%
April May June July Aug Sept Oct Nov Dec Jan
Percent of Forms That Were Illegible
*hypothetical data – illustrative purposes only
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Large Teaching Hospitals
Large Community Hosptials
Medium Community Hospitals
Small Community Hospitals
SMALL MULTIPLES – all info on one page
0
10
20
30
40
1/1
/09
2/1
/09
3/1
/09
4/1
/09
5/1
/09
6/1
/09
7/1
/09
8/1
/09
9/1
/09
10
/1/0
9
11
/1/0
9
12
/1/0
9
1/1
/10
2/1
/10
ProvincialReadmission Rate
010203040
Large Teaching Hospitals
010203040
Large Community Hospitals
010203040
Medium Community Hospitals
010203040 Small Community
Hospitals
*hypothetical data – illustrative purposes only
QI Responses to Traditional Problems with Data
1. The data are wrong.
2. The data are too old.
3. Don’t know what is might not be statistically significant?
4. We need to focus on this “outlier” or “trend”.
On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident
Voting