M1 Presentation Identifying Improvement Measures -...
Transcript of M1 Presentation Identifying Improvement Measures -...
11/26/2014
1
Identifying and Defining Improvement Measures
M1December 8, 2014
Following the CAUTI Case…
1. Baselines, Gaps, Aims, OutcomesWhere are we now, and what are we trying to accomplish?
2. Building a Theory of Improvement (Driver Diagram)
What should we measure and why?
3. Mapping the measures (Measure Tree)How will we calculate the measures?
4. Defining the MeasuresAttributes of Useful Improvement Measures
5. Collecting Data and Testing Changes
P2
11/26/2014
2
What Are We Trying to Accomplish?
Dimensions of Quality
Baselines, gaps, outcomes
Aim Statement
Three Types of Measures
Outcome Measures� Point to qualities that stakeholders value.
� Is this system meeting the needs of those who care about its operation?
� Is our improvement work making a meaningful impact?
Process Measures� Voice of the process.
� Are the parts/steps in the system performing as planned? Are processes reliable? Efficient? Patient-Centered?
� Are we on track to improve?
Balancing Measures� Are we producing perverse unintended consequences in our
efforts to improve? What other factors may be affecting results?
11/26/2014
3
Balancing Outcomes: IHI Triple AimP5
IOM Report: Dimensions of Care Quality
Safe - as safe in healthcare as in our homes
Effective - matching care to science; avoiding overuse of ineffective care and underuse of effective care
Patient-centered - honoring the individual and respecting choice
Timely - less waiting for both patients and those who give care
Efficient - reducing waste
Equitable - closing racial and ethnic disparities in access and health status
Institute Of Medicine (2001). Crossing the quality chasm : a new health system for the 21st century. Washington, D.C., National Academy Press.
11/26/2014
4
IHI Triple Aim: Examples of Measures
Systems of Care P8
D: The environment (policy,payment, accreditation, etc.)
C: Organizations that supportmicrosystems
B: Microsystems
A: Lives of Patients
11/26/2014
5
Case Background: Reducing CAUTIs
A medium sized acute care hospital has noticed that there has been an increasing occurrence of catheter associated urinary tract infections (CAUTIs) over the past year. Not only has the occurrence of CAUTIs been gradually going up but also the severity of the infections has been increasing.
Indwelling urinary catheters are commonly used medical devices within acute and non-acute settings. But their use does increase the risk of CAUTIs by:
• Enabling organisms to gain entry to the bladder via external surface or opened connections
• Reducing the body's defense of flushing out organisms during urination
• Facilitating biofilm formation
Reducing CAUTIs would contribute to:
• Improving the patient experience
• Reducing the cost of antibiotic prescribing
• Reducing inpatient length of stay
• Reducing readmissions
• Improving patient outcomes
Baseline Data – Key Outcome P10
11/26/2014
6
Aim P11
AIM: Reduce CAUTI infections in all units
below 1.6 (10th percentile) within 12
months and to zero within 24 months.
Exercise
1. Case Discussion
� Why are catheter-associated infections measured as
‘Number of CAUTIs per 1000 Foley catheter days?’
� What is the evidence that the rate of infections has
actually been increasing?
2. Own Project: Reflect and discuss in pairs
� What are you trying to accomplish (your aim?)
� What is the outcome measure that best captures the
aim of your project?
� What is the baseline level of performance on the
outcome? How much does the outcome need to
improve?
3. Share with the group
P12
11/26/2014
7
Building a Theory of Improvement
Driver diagrams
Prioritization
Linking drivers and measures
Theory Drives Improvement
“Without theory, there are no questions; without questions, there is no learning.”
P14
11/26/2014
8
A Theory of How to Improve a System
CauseEffectDrives
P15
Version:
11/26/2014Theory for CAUTI Reduction
Reduce catheter associated urinary tract infections by 50% in one year
P1 Leadership and aligned policy for catheter use
S1 Clear policies for infection control
Outcomes Primary Drivers Secondary Drivers Changes / Interventions
P2 Eliminate unnecessary catheter insertions
P3 Reliable compliance with catheter insertion protocol
P4 Reliable compliance with catheter maintenance protocol
S2 Transparent reporting of process failures
S3 Staff training, with feedback on observed protocol compliance
S4 Insert catheters only for appropriate indications
S6 Minimize use of catheters for patients at risk for infections
S8 Insertion only by trained staff
S9 Standard insertion procedure
S10 Daily assessment of need, removal at earliest opportunity
S5 Consider alternative methods
S11 Standard cleaning and maintenance procedure
Maintenance Bundle:A Tamper seal intactB Secured in placeC Hand hygieneD Meatal hygieneE Disposal & clean containerF Maintain unobstructed flow
Insertion Bundle:A Hand HygieneB Sterile gloves, materialsC Aseptic insertionD Unobstructed flow
S7 Remove when no longer required
11/26/2014
9
Version:
11/26/2014Needed Measures for CAUTI Reduction
Reduce catheter associated urinary tract infections by 50% in one year
P1 Leadership and aligned policy for catheter use
S1 Clear policies for infection control
Outcomes Primary Drivers Secondary Drivers Changes / Interventions
P2 Eliminate unnecessary catheter insertions
P3 Reliable compliance with catheter insertion protocol
P4 Reliable compliance with catheter maintenance protocol
S2 Transparent reporting of process failures
S3 Staff training, with feedback on observed protocol compliance
S4 Insert catheters only for appropriate indications
S6 Minimize use of catheters for patients at risk for infections
S8 Insertion only by trained staff
S9 Standard insertion procedure
S10 Daily assessment of need, removal at earliest opportunity
S5 Consider alternative methods
S11 Standard cleaning and maintenance procedure
Maintenance Bundle:A Tamper seal intactB Secured in placeC Hand hygieneD Meatal hygieneE Disposal & clean containerF Maintain unobstructed flow
Insertion Bundle:A Hand HygieneB Sterile gloves, materialsC Aseptic insertionD Unobstructed flow
S7 Remove when no longer required
A fundamental assumption of
clinical QI:
Reliable execution of key clinical
driver processes improves
outcomes measured at the
population level
Measuring Improvement
Measures let us
• Monitor progress in improving
the system
• Identify effective changes
11/26/2014
10
Prioritizing Drivers
Limitations of resources, attention or will usually mean we cannot work on (or measure!) everything.
Priorities:
Where is the ‘Bang for Buck?’ Which drivers do we believe will deliver the biggest impact?
Which ones will be easiest to work on? Most difficult? Are some ‘beyond our control’?
What is our current level of performance on these drivers?
P20
CAUTI Driver Rankings
Difficulty
Impact
HIGH
LOW
HIGHLOW
P21
11/26/2014
11
CAUTI Priority Measure Concepts
S4: Insert catheters only for appropriate indications.
The most effective way to eliminate the possibility of a CAUTI is to eliminate an unneeded catheter.
S7: Remove when no longer required.
Since the risk of infection is roughly proportional to the time the catheter is in place, removing catheters as soon as possible will reduce the risk.
S9: Standard insertion procedure.
If trained staff follow strict protocols for aseptic insertion of catheters, the risk of bacterial infection will be minimized.
S11: Standard cleaning and maintenance procedure.
Similarly, careful adherence to the components of the maintenance bundle will reduce risk.
P22
Exercise
Use the Driver Diagram Rubric to guide the following:
Case Discussion
� Do you have questions or issues about the CAUTI
driver diagram?
� Discuss and resolve. If you get ‘stuck’, raise the
question to the group.
Own Project Discussion
� Review (or create) the driver diagram for your project
� Discuss in pairs (or to table)
� Be prepared to share with the group
P23
11/26/2014
12
Driver Diagram Rubric
1. Does the aim of the diagram focus on OUTCOMES?
2. Do the driver labels refer to the improvements needed to accomplish the aim?
3. Are all of the secondary drivers necessary for achieving the aim?
4. Are the secondary drivers sufficient to achieve the aim?
5. Do the drivers consider needed process, leadership, cultural, and structural changes?
6. (Optional) Does the diagram include change concepts or specific change ideas that might be tested as part of an improvement initiative?
Testing the diagram: Show the driver diagram to a knowledgeable person who is naïve to the system you want to improve. Ask them to explain what you are trying to accomplish and how. Identify areas of confusion, and consider revising your diagram.
P24
Mapping the Measures
Measure Tree Diagram
11/26/2014
13
Version:
11/26/2014Measures for CAUTI Reduction
Reduce catheter associated urinary tract infections by 50% in one year
P1 Leadership and aligned policy for catheter use
S1 Clear policies for infection control
Outcomes Primary Drivers Secondary Drivers Changes / Interventions
P2 Eliminate unnecessary catheter insertions
P3 Reliable compliance with catheter insertion protocol
P4 Reliable compliance with catheter maintenance protocol
S2 Transparent reporting of process failures
S3 Staff training, with feedback on observed protocol compliance
S4 Insert catheters only for appropriate indications
S6 Minimize use of catheters for patients at risk for infections
S8 Insertion only by trained staff
S9 Standard insertion procedure
S10 Daily assessment of need, removal at earliest opportunity
S5 Consider alternative methods
S11 Standard cleaning and maintenance procedure
Maintenance Bundle:A Tamper seal intactB Secured in placeC Hand hygieneD Meatal hygieneE Disposal & clean containerF Maintain unobstructed flow
Insertion Bundle:A Hand HygieneB Sterile gloves, materialsC Aseptic insertionD Unobstructed flow
S7 Remove when no longer required
M1
M5
M2
M6
M3
M4
CAUTI Measures P27
11/26/2014
14
CAUTI Reduction Measures
D1 Count of patients with
catheters in situ in measurement
month
M1 Percent of patients
with appropriate
catheter placements
N1 Count of patients
meeting critiera for
catheter insertion
D2 Count of catheters inserted in
measurement month
M2 Average catheter
duration
N3, M3 Count of CAUTIs
in measurement month
N4 Count of catheter
insertions with all
insertion bundle
elements in compliance
D3, N2 Sum of
days with
catheters in situ
M4 CAUTIs per 1000
patient days
N5 Count of catheters
with all maintenance
bundle elements in
compliance
M5 Percent of catheter
insertions with all
insertion bundle
elements in compliance
M6 Percent of catheter
insertions with all
maintenance bundle
elements in compliance
Denominators Numerators Measures
M4 (alternate) Catheter
days between CAUTI
events
P28
Exercise
Case Discussion
� Do you have questions or issues about the CAUTI
measure tree?
� Discuss and resolve. If you get ‘stuck’, raise the
question to the group.
Own Project Discussion
� Based on your own driver diagram, identify the
outcome and key process measures you will need (1
outcome, no more than 4 process)
� Create a measure tree that shows the numerators
and denominators for your measures.
� Be sure your process measures are linked to drivers.
P29
11/26/2014
15
Attributes of Useful Improvement Measures
Responsive
Valid
Comprehensible
Timely
Feasible
Relevant
Attributes of Useful Improvement Measures
Responsive The measure is sensitive to changes in the system state.
When the system improves, the measure says so.
Valid The measure aligns with the theory of changes (driver
diagram). Improvement in the measure means improvement
in the system.
Comprehensible The intended audience understands the meaning of the
measure for system improvement.
Timely The data are available soon enough to inform improvement
decisions (project planning, PDSA testing).
Feasible The data can be collected with minimum effort and cost, and
in a timely fashion.
Relevant The measure supports problem identification and testing at
the appropriate level of management.
Consistent The measure has a clear definition: it yields consistent
results when applied in different places and at different
times.
Ownership Someone is explicitly assigned to monitor the measure on a
regular basis, detect problems, and initiate change.
11/26/2014
16
Why Time Is Important for Measuring Improvement
“Improvement is temporal!” – Lloyd Provost
Displaying data over time (using run charts or control charts) allows us to make informed predictions, and thus manage effectively
Did We Improve?
Did we improve?
What will happen next?
Should we do something?
Source: R. Lloyd
Percent of ER patients with Chest Pain Seen by a
Cardiologist within 10 min
P35
11/26/2014
17
Version:
11/26/2014Validity: Aligned with Improvement Theory
Reduce catheter associated urinary tract infections by 50% in one year
P1 Leadership and aligned policy for catheter use
S1 Clear policies for infection control
Outcomes Primary Drivers Secondary Drivers Changes / Interventions
P2 Eliminate unnecessary catheter insertions
P3 Reliable compliance with catheter insertion protocol
P4 Reliable compliance with catheter maintenance protocol
S2 Transparent reporting of process failures
S3 Staff training, with feedback on observed protocol compliance
S4 Insert catheters only for appropriate indications
S6 Minimize use of catheters for patients at risk for infections
S8 Insertion only by trained staff
S9 Standard insertion procedure
S10 Daily assessment of need, removal at earliest opportunity
S5 Consider alternative methods
S11 Standard cleaning and maintenance procedure
Maintenance Bundle:A Tamper seal intactB Secured in placeC Hand hygieneD Meatal hygieneE Disposal & clean containerF Maintain unobstructed flow
Insertion Bundle:A Hand HygieneB Sterile gloves, materialsC Aseptic insertionD Unobstructed flow
S7 Remove when no longer required
M1
M5
M2
M6
M3
M4
Validity: Alignment with Improvement Work
• Improvement in a pilot population (1 practice, 1 unit, etc.) will not be
evident in measures based on the total population (city, hospital system)
Total
Population Measurement
SamplePilot Unit
P39
11/26/2014
18
Validity: Alignment with Improvement Work
Total
Population Target
population
• To track improvement, we must measure in the same target population
where we are working to improve.
P40
Comprehensible?
Percentage of patients discharged in the measurement month that suffered a CAUTI
Number of CAUTIs per 1000 Foley catheter days during measurement month
Number of CAUTIs per 1000 inpatient days during the measurement month
Count of CAUTIs in the measurement month
Number of catheter days since the last CAUTI event
11/26/2014
19
Importance of Timely Data
Consistency: Operational Definition
A procedural description of what to measure and the steps to follow to measure it consistently…
� Gives communicable meaning to a concept
� Tells what you need to count or measure, and how to
do it
� Specifies measurement methods and equipment
� Provides guidance on sampling
� Identifies detailed criteria for inclusion and exclusion
… is the basis for reliable measurement
Source: R. Lloyd
11/26/2014
20
Operational Definition Example
Measure: Percentage of patients undergoing hip and knee replacement surgery during the measurement period who have had preoperative nasal swabs to screen for Staphylococcus aureus carriage
Goal: 95%Measurement Period Length: MonthlyNumerator Definition: Number of patients undergoing hip or knee replacement surgery who have had a nasal swab specimen processed to screen for Staphylococcus aureus carriage prior to surgeryDenominator Definition: Number of patients undergoing elective hip or knee replacement surgeryNumerator and Denominator Exclusions:• Patients who are less than 18 years of age • Patients who had a principal or admission diagnosis suggestive of preoperative
infectious diseases• Patients with physician-documented infection prior to surgical procedures• Patients undergoing non-elective hip or knee replacement surgeryDefinition of Terms:Hip or knee replacement surgery includes operations involving placement of a nonhuman-derived device into the hip or knee joint space. ICD-9 Codes include 00.70-00.73, 00.85-00.87, 81.51-81.53, 00.80-00.84, 81.54, and 81.55.Calculate as: (numerator/denominator*100)
Responsive
Percent of catheters removed during the measurement
month within 2 days of insertion
Average catheter duration by month
• Which measure better reflects the
improvement work of teams trying to reduce
unnecessary catheter placement?
• Which measure better reflects protocol
compliance?`
11/26/2014
21
January
April
June
August
September
Average Time versus Percent Conforming*
Specification = 30 min or less
Which measure is most useful to an improvement team?� % of cases with Abx within 30 min
� Average time to Abx admin
*Simulated data via @Risk
Population MeasuresThroughput =
visits…
…with reliable
care process…
Population: who’s health are we responsible for?
…have an
incremental
impact on
population.
DM pts with LDL<100
Active DM pts in practice panel
P50
11/26/2014
22
Typical Population Questions
• What is the current state of the population for whom we are responsible (even those we haven’t seen for awhile?) re: Health status? Pt. Experience? Cost of care?
• How do our population’s risk factors and outcomes compare with those of other provider organizations?
• How should we plan for the long term?
• What has the impact of our improvement work been on the population? Are there other factors effecting changes in outcomes?
Outpatient ‘Look-Back’ Measures
Percent of population with current self-management plan as of
most recent visit within the past 12 months.
Each measurement contains mostly the same patients as the previous month.These measures are slow to show improvement, but reflect the state of care for the population!
12 months
Current test
No current test
11/26/2014
23
“Current Care” Measures
Percent of patients seen last month who lacked an up-to-date
A1C and who got the test during the visit or were referred.
Each subgroup contains different patients & represents current workThese measures are great for tracking process changes!
Current test
No current test
1 mo
Exercise
Case Discussion
� The worksheet titled ‘Attributes of Useful
Improvement Measures’ shows alternative measures
that might be used to capture the key measure
concepts in the case.
� Compare the alternative measures with respect to the
attributes of useful improvement measures
Own Project Discussion
� Refer to your proposed project measures
� How do your measures relate to the attributes of
useful improvement measures?
� Discuss at your table
Share your insights
P59
11/26/2014
24
P60
Data Collection
Data for PDSA testing
Concurrent data collection
Segmentation
Sampling
11/26/2014
25
Project Data Collection
Existing EMR system� PRO: data collected as component of routine care
� CON: needed process measures may not be included; data may lag by weeks or months; process failures lack context; usually requires custom reports
Paper chart review� PRO: notes may provide useful context; may be necessary
if no electronic system
� CON: labor intensive (but sampling helps); data may lag by days or weeks
Concurrent log or registry� PRO: ad hoc data can target PDSAs, project measures; no
lag; context available;
� CON: extra work for caregivers; special data process necessary
Measuring Reliability
Reliability =
Number of Actions That Achieve The Intended Result
Total Number of Actions Taken or Intended
= Percent ‘Conforming’
11/26/2014
26
Proposal that
patient requires
urinary catheter
Check pt. for past
problems,
allergies, etc.
Explain procedure
to pt. and/or
caregivers
Decontaminate
hands
Clean and prepare
the work area,
assemble materials
Prepare patient
Put on personal
protection equipment
(PPE) and sterile gloves
Is the
patient
male?
Follow male
procedures for
urinary catheter
insertion
Follow female
procedures for
urinary catheter
insertion
Record patient experience, document
technical specifications and time of
completion into the chart
Dispose of equipment and materials in
designated bag. Remove PPE and wash hands
No
Yes
A B
Alternative
methods
available?
Indications
are appro-
priate?
Yes No
No YesStop
% of cases with appropriate indication
% of females with proper insertion procedure% of males with
proper insertion procedure
% of cases with proper hand hygiene
Measuring Process: Total Joint Arthroplasty
Aim: Pre-screen all total hip or knee replacement patients for nasal Staph; those who test positive will complete a course of mupirocin.
Population: All patients undergoing TKA or HKA in our hospital (with exclusions)
Process: Screening and decolonization
Measurement interval: monthly
Process reliability measure:Percent of patients who screened positive for SA who report they had completed a course of mupirocin prior to day or surgery.
11/26/2014
27
Schedule
procedureS
ch
ed
ulin
gL
ab
Ho
sp
ita
l /S
urg
eo
n
TKA or THA?
Insert lab
request for SA
culture
Inform patient
of SA screening
Pt presents for
nasal swab
Positive for
SA?Process
specimen
Results to
surgeon &
hospital
Document in
record
Confirm Rx
complete
Surgery
1-4 weeks pre-procedure 2-3 weeks pre-procedure Day of surgery
Staph aureus (SA)Screening and Decolonization Process Example
Yes
Yes
No
No
Prescribe 5 day
mupirocin
Contact patient
% of no-shows for swab
Suggested
measures
% of positive results not acted on
KEY RELIABILITY MEASURE
% of colonized patients with completed Rx
Notify hospital
% of cases with missing lab order
Time to receive lab results
Time to notify patient
Individual Patient Data to Assess Process Improvement
0
20
40
60
80
100
120
140
160
9/1
6/1
0
10/1
/10
10/1
1/1
0
10/3
1/1
0
11/4
/10
12/2
/10
12/8
/10
12/1
9/1
0
12/2
7/1
0
1/1
7/1
1
2/2
/11
2/1
0/1
1
2/1
0/1
1
2/1
5/1
1
2/2
5/1
1
2/2
8/1
1
3/2
/11
3/5
/11
3/2
4/1
1
3/2
8/1
1
4/2
8/1
1
5/8
/11
5/1
3/1
1
5/1
7/1
1
5/2
3/1
1
5/2
3/1
1
6/1
5/1
1
6/2
5/1
1
6/2
7/1
1
7/3
/11
7/8
/11
7/1
0/1
1
7/1
8/1
1
7/2
8/1
1
7/3
0/1
1
8/4
/11
8/1
0/1
1
8/1
8/1
1
8/2
5/1
1
8/2
8/1
1
8/3
1/1
1
9/5
/11
9/8
/11
9/2
1/1
1
9/2
5/1
1
9/2
7/1
1
10/2
/11
10/5
/11
10/1
8/1
1
10/2
1/1
1
10/2
5/1
1
10/3
1/1
1
11/1
0/1
1
11/1
6/1
1
11/2
6/1
1
12/5
/11
12/1
4/1
1
1/2
/12
1/5
/12
1/2
8/1
2
1/3
0/1
2
2/8
/12
2/1
4/1
2
2/2
0/1
2
Time(min)
Individual Patients Over Time
Time to Antibiotic Medication Sept 2010 - Feb 2012
Updated: March 7, 2012
Shift = 6; Trend = 5
Mean = 46 Mean = 38
Mean = 35
Mean = 27.3
Mean = 29.9 Guidelines
#1 Guidelines
#2ED Comm
Stickies/ QI ED Comm
Excel Dose Calc
VOE Guidelines
Goal:
Source: James Moses, BMC
11/26/2014
28
Sampling
… when you can’t measure
the entire population, you
can estimate its
characteristics by sampling
• Systematic sampling
• Random sampling
• Stratified sampling
• Convenience sampling
• Judgment sampling
Sampling Methods
Source: R. Lloyd
Convenience Sample
“Gosh I’m in a hurry. Why don’t I just review these?”
11/26/2014
29
Simple
Random
Sample
Every element has an equal chance of being selected
Sampling Methods
Systematic
Sample
…then select every nth element
First element selected at random…
Possible bias if there are patterns in the sequence of elements
You might survey every 10th patient who arrives at a clinic
beginning at a randomly selected time
Sampling Methods
11/26/2014
30
Sampling Methods
Judgment Sample
Especially for PDSA testing,
someone expert with the process
selects items to measure:
• To include a range of conditions
• Selection criteria may change as
understanding increases
• Successive small samples
instead of one large sample
What Sample Size?
To be useful, samples should be large enough to reveal improvement shifts and trends.
This also depends on magnitude of the change, and the inherent variability of the measure.
30 is a good rule of thumb for current care measures
You can approach this issue empirically
Don’t sample unless you need to
Small samples ok for PDSA testing
11/26/2014
31
Tracking Change– Segment by Segment
Segment 1 - Pilot Segment 2
Jan 10 Mar 11
Segment 3
Segmentation
By
Organization
Site
Provider
Region
Diagnosis
Patient process ‘trajectory’
© R. Scoville • 76
11/26/2014
32
Exercise
How did the CAUTI team approach their data collection task?
Own Project
� What data are available to support your improvement
measurement plan? Is it possible to gather concurrent
data?
� What are some of the change ideas that you will test?
What data will be needed to assess their impact?
How will you gather those data?