Building a Measurement System That...

1

Building a Measurement System That Works

Session B713 April 20161330 – 1500

Robert Lloyd, Ph.D.Vice President, Institute for Healthcare Improvement

Gary Sutton, BSc (Honors) Statistician

International Forum on Quality and Safety in Healthcare

Gothenburg Sweden

The presenters have

nothing to declare

IHI Faculty(bio sketches can be found at the end of this presentation)

Gary Sutton, BSc (Honors) Scottish Government, Statistician

Improvement Advisor; Scotland

[email protected]

@scotgov_ia

Robert Lloyd, PhDVice President, Institute for Healthcare Improvement

[email protected]

@rlloyd66

2

and the

3

1. What is your current level of knowledge about quality measurement?

2. What is your motivation for measuring?

3. Do you know the milestones in the Quality Measurement Journey (QMJ)?

4. Do you understand variation conceptually?

5. Do you understand variation statistically?

6. How well do you link measurement to improvement

Our Objectives for today…To answer Six Key Questions on Measurement

Question #1What is your current level of knowledge about

quality measurement?

This self-assessment is designed to help quality facilitators and improvement team

members gain a better understanding of where they personally stand with respect to

the milestones in the Quality Measurement Journey (QMJ). What would your

reaction be if you had to explain why is it preferable to plot data over time rather than

using aggregated statistics and tests of significance? Can you construct a run chart

or help a team decide which measure is more appropriate for their project?

You may not be asked to do all of the things listed below today or even next week.

But if you are facilitating a QI team or expect to be able to demonstrate

improvement, sooner or later these questions will be posed. How will you deal with

them?

The place to start is to be honest with yourself and see how much you know about

concepts and methods related to the QMJ. Once you have had this period of self-

reflection, you will be ready to develop a learning plan for yourself and those on your

improvement team.

Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using

Indicators. Jones & Bartlett Publishers, 2004: 301-304.

3

ExerciseMeasurement Self-Assessment

Select the one response which best captures your opinion:

1. I'd definitely have to call in an outside expert to explain and apply

this topic.

2. I’ve heard of this topic but I would not feel comfortable applying it to

a team’s work.

3. I am familiar with this topic but would have to study it further before I

felt comfortable explaining it to a team.

4. I have knowledge about this topic and feel confident that I could help

a team apply it to their improvement efforts but I would not want to

stand up and teach this to a large group.

5. I consider myself an expert in this area and could apply easily to a

team’s work as well teach this topic to large groups.

Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using

Indicators. Jones & Bartlett Publishers, 2004: 301-304.

Exercise: Measurement Self-AssessmentSource: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators.

Jones & Bartlett Publishers, 2004: 301-304.

Measurement Topic or SkillResponse Scale

1 2 3 4 5

Help people in my organization determine why they are measuring (improvement, judgment or research)

Move teams from concepts to specific quantifiable measures

Building clear and unambiguous operational definitions for our measures

Develop data collection plans (including stratification and sampling strategies)

Explain why plotting data over time (dynamic display) is preferable to using aggregated data and summary

statistics (static display)

Explain the differences between random and non-random variation

Construct run charts (including locating the median)

Explain the reasoning behind the run chart rules

Interpret run charts by applying the run chart rules

Explain the statistical theory behind Shewhart control charts (e.g., sigma limits, zones, special cause rules)

Describe the basic 7 Shewhart charts and when to use each one

Help teams select the most appropriate Shewhart chart for their measures

Describe the rules for special cause variation on a Shewhart chart

Help teams link measurement to their improvement efforts

4

QualityBetter

Old WayQuality Assurance

(Data for judgment))

QualityBetter Worse

New Way(Quality Improvement)

Action taken on all occurrences

Reject

defectives

Question #2What is your motivation for measuring?

Source: Robert Lloyd, Ph.D., 2009.

Requirement,Specification or Target

No action

taken

here

Worse

Source: Provost, Murray & Britto (2010)

Example of Data for Judgement

5

Slide #9

Slide #9

How Is Error Rate Doing?


Slide #10

Slide #10

How is Perfect Care Doing?


6

20-20 Hindsight

“Managing a process on the basis of monthly (or quarterly) averages is like trying to drive

a car by looking in the rear view mirror.”

D. Wheeler Understanding Variation, 1993.

If you are serious about your quality improvement efforts, you should be collecting and analyzed data as close to the production of work as possible.

• What would it take to collect data on individual patients waiting to see the doctor?

• To track the number of patients being assessed for pressure ulcers each day?

• The percent of “did not attend” appoints for each week?

• Most measures can be collected more frequently than monthly!

Question #3Do you know the milestones in the Quality

Measurement Journey (QMJ)?

7

13

AIM (How good? By when?)

Concept

Measure

Operational Definitions

Data Collection Plan

Data Collection

Analysis ACTION

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.

Milestones in theQuality Measurement Journey

© 2015 Institute for Healthcare Improvement/R. Lloyd

AIM – reduce patient falls by 37% by the end of the year

Concept – reduce patient falls

Measures – Inpatient falls rate (falls per 1000 patient days)

Operational Definitions - # falls/inpatient days

Data Collection Plan – monthly; no sampling; all IP units

Data Collection – unit submits data to Quality

Improvement Dept. for analysis

Analysis – control chart ACTION


8

15


Concept

Measure



Data Collection

Analysis ACTION



Moving from a Concept to Measure

“Hmmmm…how do I move from a concept

to an actual measure?

Every concept can have MANY measures.

Which one is most appropriate?

9

Concept Potential MeasuresHand Hygiene Ounces of hand gel used each day

Ounces of gel used per staffPercent of staff washing their hands

(before & after visiting a patient)Percent of inpatients with C.Diff

Patient Falls Percent of patients who fell Fall rate per 1000 patient daysNumber of fallsDays between a fall

Employee Evaluations Percent of evaluations completed on time

Number of evaluations completed

Variance from completion due date

Every concept can have many measuresSource: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

© 2015 Institute for Healthcare Improvement/R. Lloyd18

Three Types of Measures� Outcome Measures

� Point to qualities that stakeholders value (voice of the customer)

� Is this system meeting the needs of those who care about its operation?

� Is our improvement work making a meaningful impact?

� Process Measures� Voice of the process.

� Are the parts/steps in the system performing as planned? Are processes reliable? Efficient? Patient-Centered?

� Are we on track to influence the Outcome measure(s)?

� Balancing Measures� Are we producing unintended consequences in our efforts to improve?

� What other factors may be affecting results?

� Looking at a system from different directions/dimensions.

� What happened to the system as we improved the outcome and process measures?

10

Provost, L.P. & Murray, S.K. (2011). The health care data guide: Learning from data for improvement. San Francisco: Jossey-Bass.

Types of Measures Description

The Surgical Sight Infection Family pf Measures

Outcome The voice of the customer or

patient. How is the system

performing? What is the result?

Surgical Sight Infection Rate

Process The voice of the workings of the

process. Are the parts or steps in

the system performing as planned.

Percentage of appropriate

prophylactic antibiotic selection.

Percentage of on time administration

of prophylactic antibiotics.

Percentage of a safety climate score

great than 4.

Balancing Looking at a system from different

directions or dimensions. What

happened to the system as we

improved the outcome and

improvement measures?

Patient satisfaction

Cost per case

Expectations for Improvement

% Compliance with process

% Compliance with process

Outcome measure - % reduction in?

When will my data start to move?

• Process measures will start to move first

• Outcome measures will most likely lag behind process measures

• Balancing measures – just monitoring – not looking for

movement (pay attention if there is movement)

11

Balancing Measures:Looking at the System from Different Dimensions

� Outcome (quality, time)

� Transaction (volume, no. of patients)

� Productivity (cycle time, efficiency, utilisation, flow,

capacity, demand)

� Financial (charges, staff hours, materials)

� Appropriateness (validity, usefulness)

� Patient satisfaction (surveys, customer complaints)

� Staff satisfaction

Balancing measures help keep you from sub-optimizing the system!

22Balancing Measures also help you be aware of Unintended Consequences

12

© 2015 R. C. Lloyd & Associates

1. A starting point for any QI project is to move from concepts to

measures that appropriately capture the concepts of interest.

2. Use the Organizing Your Measures Worksheet on the next

page to start this part of your journey.

3. List the concepts of interest in the far left column. Then identify

potential measures for these concepts in the second column.

Remember that a single concept might have more than one

potential measure.

4. Finally, indicate whether each potential measure is an

Outcome, Process or Balancing measure.

ExerciseOrganizing your Measures

© 2015 R. C. Lloyd & Associates

Concept Potential Measure(s) Outcome Process Balancing

Organizing Your Measures Worksheet

Topic for Improvement:

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

13

Concept Potential Measure(s) Outcome Process Balancing

Patient Harm Inpatient falls rate

Patient Harm Number of falls

Compliance Percent of inpatients assessed for falls

Staff Education Percent of staff fully trained in falls assessment protocol

Assessment Time

The additional time it takes to conduct a proper falls assessment

ExampleOrganizing Your Measures Worksheet


Topic for Improvement: Inpatient Falls

Conclusions:Moving from a Concept to a Measure

1. Moving from concept to measures requires focused work

to create agreement about adjectives such as recovery,

major, timely, complete, accurate or excellent.

2. A concept may need more than one measure and,

therefore, the development of more than one operational

definition.

3. The transition from concept to measure doesn’t just

happen, it requires both technical and clinical decision-

making to be blended with pragmatism and acceptance of

the imperfections of the measures.

14

27


Concept

Measure



Data Collection

Analysis ACTION



28An Operational Definition...

… is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently.

� It gives communicable meaning

to a concept

� Is clear and unambiguous

� Specifies measurement

methods and equipment

� Identifies detailed criteria for

inclusion and exclusion.

� Provides guidance on sampling


15

© 2015 Institute for Healthcare Improvement/R. Lloyd29

From a food science perspective, it is difficult to define a food product that is 'natural' because

the food has probably been processed and is no longer the product of the earth. That said,

FDA has not developed a definition for use of the term natural or its derivatives. However, the

agency has not objected to the use of the term if the food does not contain added color,

artificial flavors, or synthetic substances.

What is the definition of 'natural' on the label of food?

September 23, 1999An expensive operational definition

problem!

NASA lost a $125 million Mars orbiter because one engineering team used metric units (newton-seconds) to guide the spacecraft while the builder (Lockheed Martin) used pounds-second to calibrate the maneuvering operations of the craft.

Information failed to transfer between the Mars Climate Orbiter spacecraft team at Lockheed Martin in Colorado and the mission navigation team in California. The confusion caused the orbiter to encounter Mars on a trajectory that brought it too close to the planet, causing it to pass through the upper atmosphere and disintegrate.

16

How do you define the following healthcare concepts?

• Medication error• Co-morbid conditions• A healthy life style• Cancer waiting times• Health inequalities• Asthma admissions• Childhood obesity• Patient education• Health and wellbeing• Adding life to years and years to life• Children's palliative care • Safe services• Smoking cessation• Urgent care• Complete history & physical• Surgical Screening

• Delayed discharges

• End of life care

• Falls (with/without injuries)

• Childhood immunizations

• Complete maternity service

• Patient engagement

• Moving services closer to home

• A well-baby visit

• Ambulatory care

• Access to health in deprived areas

• Diagnostics in the community

• Productive community services

• Vascular inequalities

• Breakthrough priorities

ExampleMedication Error Operational Definition

Measure Name: Percent of medication errors

Numerator: Number of outpatient medication orders with one or more errors. An error is defined as: wrong med, wrong dose, wrong route or wrong patient.

Denominator: Number of outpatient medication orders received by the family practice clinic pharmacy.

Data Collection:

• This measure applies to all patients seen at the clinic

• The data will be stratified by type of order (new versus refill) and patient age

• The data will be tracked daily and grouped by week

• The data will be pulled from the pharmacy computer and the CPOE systems

• Initially all medication orders will be reviewed. A stratified proportional random sample will be considered once the variation in the process is fully understood and the volume of orders is analyzed.

17

• Select one measure for your project (outcome or process) and write a clear and specific Operational Definition.

• Is the measure clearly defined? If you gave the definition of your measure to another person would they know precisely what you are attempting to measure?

• Are you clear about the name of the measure, what is to be included and the measurement steps required to obtain data?

• Use the Operational Definition Worksheet to record your responses.

ExerciseBuilding Your Operational Definitions

34

Measure Name: ________________________________________(Remember this should be specific and quantifiable, e.g., the time it takes to…,the number

of…, the percent of… or the rate of…)

Operational DefinitionDefine the specific components of this measure. Specify the numerator and denominator if

it is a percent or a rate. If it is an average, identify the calculation for deriving the average.

Include any special equipment needed to capture the data. If it is a score (such as a patient

satisfaction score) describe how the score is derived. When a measure reflects concepts

such as accuracy, complete, timely, or an error, describe the criteria to be used to determine

“accuracy.”

Operational Definition Worksheet

See Appendix D for a detailed Operational Definition worksheet

18

� Operational definitions are not universal truths!

� Operational definitions require agreement on terms, measurement methods and decision criteria.

� Operational definitions need to be reviewed periodically to make sure everyone is still using the same definitions and that the conditions surrounding each measure have not changed.

Conclusions:Developing Operational Definitions

36


Concept

Measure



Data Collection

Analysis ACTION



19

Stratification

• Separation & classification of data according to predetermined categories

• Designed to discover patterns in the data

• For example, are there differences by shift, time of day, day of week, severity of patients, age, gender or type of procedure?

• Consider stratification BEFORE you collect the data

Common Stratification Levels

� Day of week

� Shift

� Severity of patients

� Gender

� Type of procedure

� Unit

� Age

What stratification

levels are appropriate

for your data?

20

Sampling

When you can’t capture data on the

entire population (an enumeration), you can estimate its characteristics

by sampling.

© Richard Scoville & I.H.I.

© 2015 Institute for Healthcare Improvement/R. Lloyd

Options for Sampling

Non-probability Sampling Methods

• Convenience sampling

• Quota sampling

• Judgment sampling

Probability Sampling Methods

• Simple random sampling

• Stratified random sampling

• Stratified proportional random sampling

• Systematic sampling

• Cluster sampling


21

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Sample Size = 1 per week

-10

10

30

50

70

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


-10

10

30

50

70

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


-10

10

30

50

70

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


-10

10

30

50

70

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


Sample Size will affect your ability to

detect a change

Provost. L. & Murray. (2008, November) The Date Guide: Learning from Data to Improve Health Care. Austin, TX: API.

Tips for building an effective measurement system

� Seek useful measures not perfection

� Think about stratification

� Use sampling (when appropriate)

� Integrate measurement into daily routine

� Collect qualitative and quantitative data

� Plot data over time

22

Integrate measures into daily routineand make it easy:

(example of attendance at a restaurant)

What measure could you collect in this way?

• Number of falls

• Number of pressure ulcers

• Number of cancelled appointments

• Number of patients coming to clinic

• Number of medication errors

• Number of staff evaluations not completed on time

©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd

1. Sampling should produce representative and workable numbers for the unit of

interest.

2. Customers providing feedback about their service or the care they receive can

be very susceptible to sampling bias.

3. Sampling bias can be introduced if you always use the same place or time to

gather data and this location or time period (e.g., Tuesday afternoon in the

clinic at 1330) is not representative of the whole. This is a major problem when

single point in time audits are relied on as the sampling method.

4. When conducting surveys, recall bias occurs if the questions rely on the

individual’s to “think back and recall how they felt about their hospital stay.”

5. The worst case scenario occurs when you have no idea where the sample

came from or how representative it is of the population or organisation overall.

6. Clear guidance on data collection methods, in particular sampling and

stratification, is a critical for any successful QI project.

Conclusions:Data Collection

23

45

“If I had to reduce my message for

management to just a few words, I’d say it all had to do with reducing variation.”

W. Edwards Deming

Question #4Do you understand variation conceptually?

46

The Problem!

Aggregated data presented in tabular formats or with summary statistics, will

not help you measure the impact of process improvement efforts.

Aggregated data and summary statistics can only lead to judgment, not

to improvement.

24

47

Average Percent of Patients who FallBefore and After the Implementation of a New Protocol

Pe

rce

nt

of

Pa

tie

nts

w

ho

Fa

ll

Time 1 Time 2

3.8

5.2

5.0%

4.0%

WOW!

A “significant drop”

from 5% to 4%

Conclusion -The protocol was a success! A 20% drop in the average mortality!

48

24 Months

1.0

9.0

Now what do you conclude about the impact of the protocol?

5.0

UCL= 6.0

LCL = 2.0

CL = 4.0

Protocol implemented here

Average Percent of Patients who FallBefore and After the Implementation of a New Protocol

Pe

rce

nt

of

Pa

tie

nts

w

ho

Fa

ll

25

49

The average of a set of numbers can be created by many different distributions

Average

Measu

re

Time

Patients do NOT care about the average!

50

If you don’t understand the variation that lives in your data, you will be tempted to ...

• Deny the data (It doesn’t fit my view of reality!)

• See trends where there are no trends

• Try to explain natural variation as special events

• Blame and give credit to people for things over which they have no control

• Distort the process that produced the data

• Kill the messenger!

26

“A phenomenon will be said to be

controlled when, through the use of

past experience, we can predict, at least

within limits, how the phenomenon may be expected to vary in

the future”W. Shewhart. Economic Control of

Quality of Manufactured Product, 1931

Dr. Walter A Shewhart

“What is the variation in one system over time?” Walter A. Shewhart - early 1920’s, Bell Laboratories

52

time

UCL

Every process displays variation:• Controlled variation

stable, consistent pattern of variation

“chance”, constant causes

• Special cause variation“assignable”

pattern changes over time

LCL

Static View

Sta

tic V

iew

Dynamic View

27

53

Common Cause Variation• Is inherent in the design of the

process

• Is due to regular, natural or ordinary causes

• Affects all the outcomes of a process

• Results in a “stable” process that is predictable

• Also known as random or unassignable variation

Special Cause Variation• Is due to irregular or unnatural

causes that are not inherent in the

design of the process

• Affect some, but not necessarily all aspects of the process

• Results in an “unstable” process

that is not predictable

• Also known as non-random or

assignable variation

Types of Variation

Point …Variation exists!

Common Cause does not mean “Good Variation.” It only

means that the process is stable and predictable. For

example, if a patient’s systolic blood pressure averaged around

165 and was usually between 160 and 170 mmHg, this might be

stable and predictable but completely unacceptable.

Similarly, Special Cause variation should not be viewed as “Bad

Variation.” You could have a special cause that represents a

very good result (e.g., a downward trend in the turnaround time

for a particular med), which you would want to sustain. Special

cause variation merely means that the process is unstable and

unpredictable.

28

55

2 Questions …

1. Is the process stable? If so, it is predictable.

2. Is the process capable?

• The chart will tell you if the process is stable and predictable.

• You have to decide if the output of the process is capable of meeting

the target or goal you have set!

• Finally, note that you should only try to make improvements to

processes that exhibit common cause variation. Why? Because

they are stable and predictable. You have no idea where processes

with special causes will go next!

Attributes of a Leader WhoUnderstands Variation

� Leaders understand the different ways that variation is viewed.

� They explain changes in terms of common causes and special causes.

� They use graphical methods to learn from data and expect others to consider variation in their decisions and actions.

� They understand the concept of stable and unstable processes and the potential losses due to tampering.

� Capability of a process or system is understood before changes are attempted.

29


• Select several measures your organization tracks on a regular basis.

• Do you and the leaders of your organization evaluate these measures according the criteria for common and special causes of variation?

• If not, what criteria do you use to determine if data are improving or getting worse?

12/9

5

2/9

6

4/9

6

6/9

6

8/9

6

10/9

6

12/9

6

2/9

7

4/9

7

6/9

7

8/9

7

10/9

7

12/9

7

2/9

8

4/9

8

6/9

8

8/9

8

10/9

8

12/9

8

2/9

9

4/9

9

6/9

9

month

Perc

ent

C-s

ections

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

UCL=27.7018

CL=18.0246

LCL=8.3473

Percent of Cesarean Sections Performed Dec 95 - Jun 99

Week

Num

ber

of

Medic

ations E

rrors

per

1000 P

atient

Days

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

UCL=13.39461

CL=4.42048

LCL=0.00000

Medication Error Rate

DialogueCommon and Special Causes of Variation


1. The same data can show different patterns of variation

dependent on how much of it you present and how you

statistically analyse and display the data.

2. Data presented over time (i.e., plotting the data by day,

week or month) is the only way you will ever be able to

improve any aspect of quality or safety!

3. Avoid using aggregated data and enumerative statistics if

you are serious about improving quality and safety!

4. A leaders job is to understand patterns of variation and ask

why!

ConclusionsUnderstanding Variation

30

Question #5Quality Measurement Journey: Understanding Variation Statistically

60

How do we analyze variation for quality improvement?

Run and Control Charts are the best

tools to determine:

1. The variation that lives in the process

2. if our improvement strategies have had

the desired effect.

31

Process Improvement: Isolated Femur Fractures

0

200

400

600

800

1000

1200

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64Sequential Patients

Min

ute

s E

D t

o O

R p

er

Patient

Holding the Gain: Isolated Femur Fractures

0

200

400

600

800

1000

1200


Min

ute

s E

D t

o O

R p

er

Patient

3. Determine if we are holding the gains

Current Process Performance: Isolated Femur Fractures

0

200

400

600

800

1000

1200


Min

ute

s E

D t

o O

R p

er

Patient

Three Uses of SPC Charts

2. Determine if a change is an improvement

1. Make process performance visible

62

How many data points do I need?

Ideally you should have between

10 – 15 data points before constructing

a run chart

10 – 15 patients

10 – 15 days

10 – 15 weeks

10 – 15 months

10 – 15 quarters (not useful for QI projects)

• If you are just starting to

measure, plot the dots and

make a line graph.

• Once you have 8-10 data

points make a run chart.

32

Me

as

ure

Time

X (CL)~

The centerline (CL) on a

Run Chart is the Median

Elements of a Run Chart

Four run rules are used to determine if non-random variation is present

How do we analyze a Run Chart

“How will I know what the Run Chart is trying

to tell me?”

It is actually quite easy:

1. Determine the number of runs.

2. Then apply the 4 basic run chart

rules decide if your data reflect

random or non-random variation.

33

What is a Run?• One or more consecutive data points

on the same side of the Median

• Do not include data points that fall on

the Median

First, you need to determine the Number of Runs

Me

as

ure

Time

X (CL)~

How many Runs on this chart?

Points on the Median (don’t count these when counting the number of runs)

Run = a series of consecutive data points above or below the median.

Ignore points on the median.

34

Me

as

ure

Time

X (CL)~

Draw circles around the individual runs

Did you identify 7 runs?

Now apply the Run Chart Rules to Identify any non-random patterns in the data

• Rule #1: A shift in the process, or too many data points in a run (6 or more consecutive points above or below the median)

• Rule #2: A trend (5 or more consecutive points all increasing or decreasing)

• Rule #3: Too many or too few runs (use a table to determine this one)

• Rule #4: An “astronomical” data point

68

35

Rule 1 – A Shift

A shift in the process is six or more consecutive points either all above or all below the median. Values that fall on the median do not add to nor break a shift. Skip values that fall on the median and continue counting.

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Me

as

ure

or

Ch

ara

cte

ris

tic

Rule 1

Rule 2 – A Trend

Five or more consecutive points all going up or all going down. If

the value of two or more successive points is the same, ignore one of the

points when counting. Like values do not make or break a trend.

Rule 2

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Measure

or

Chara

cte

ristic

Median 11

36

Rule 3 – Runs Test(Too few or too many runs)

Rule 3

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Me

as

ure

or

Ch

ara

ce

ris

tic

Median

11.4

Data line crosses only onceToo few runs: total of only 2 runs

Rule #3: Requires a reference table

72

Use this table by first calculating the number of "useful observations" in your data set. This is done by subtracting the number of data points on the median from the total number of data points. Then, find this number in the first column. The lower number of runs is found in the second column. The upper number of runs can be found in the third column. If the number of runs in your data falls below the lower limit or above the upper limit then this is a signal of a special cause.

# of Useful Lower Number Upper Number Observations of Runs of Runs

14 4 1215 5 1216 5 1317 5 1318 6 1419 6 1520 6 1621 7 1622 7 1723 7 1724 8 1825 8 1826 9 1927 10 1928 10 2029 10 2030 11 21

Total data points

So, for 25 useful

observations we should observe

between 8 and 18 runs

Two data points on the median = 27 useful

observations

Source: Swed, F. and Eisenhart, C.

(1943) “Tables for Testing

Randomness of Grouping in a

Sequence of Alternatives.” Annals of

Mathematical Statistics. Vol. XIV, pp.

66-87, Tables II and III.

37

Rule 3 – RunsToo many runs – what is this data telling you?

Rule 3 Too Many Runs

0

1

2

3

4

5

6

7

Jan-0

8

Feb-0

8

Mar-08

Apr-08

May-0

8

Jun-0

8

Jul-08

Aug-0

8

Sep-0

8

Oct-08

Nov-0

8

Dec-0

8

Jan-0

9

Feb-0

9

Mar-09

Apr-09

May-0

9

Jun-0

9

Jul-09

Aug-0

9

Sep-0

9

Measure

of C

hara

cte

ristic

Rule 4: An Astronomical ValueFor detecting unusually large or small numbers:

• Data that is Blatantly Obvious different value

• Everyone studying the chart agrees that it is unusual

• Remember: Every data set will have a high and a low - this does not mean the high or low are astronomical

Rule 4

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Measure

ment or C

hara

cte

ristic

Remember, this rule is detecting only ONE astronomically high or low data point.

38

The Run Chart Rules for identifying non-random patterns

Source: The Data Guide by L. Provost and S. Murray, Jossey-Bass Publishers, 2011.

A Shift: 6 or more

An astronomical data point

Too many or too few runs

A Trend

5 or more

Me

as

ure

Time

X (CL)~

Did you find a shift, a trend, too many or too few runs or an astronomical data point?

Apply the Run Chart Rules to the Chart

39

So, let’s practice…

60

65

70

75

80

85

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Percent Compliance

ExercisePercent Compliance with Proper Hand Hygiene

Median = 81

How many runs on this chart

Week

Perc

ent C

om

plia

nce

40

60

65

70

75

80

85

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Percent Compliance

Week

Perc

ent C

om

plia

nce

15 runs

NOTE: 27 data points with 3 on the median gives you 24 “useful observations.” For 24 useful observations you expect between 8 and 18 runs.

Percent Compliance with Proper Hand HygieneNow apply the Run Chart Rules

Week

Median = 81

Did you find a shift, a trend, too many or too few runs or an astronomical data point?

• What can you explain about Shewhart (control) charts?

• What do you need to think more about to be effective with applying Shewhart charts?

SO,

Let’s continue to understand

variation statistically

41

What are 4 reasons why Shewhart Charts are preferred over Run Charts?

Because Control Charts…

1. Are more sensitive than run charts:� A run chart cannot detect special causes that are due to point-to-point

variation (median versus the mean)

� Tests for detecting special causes can be used with control charts

2. Have the added feature of control limits, which allow us to

determine if the process is stable (common cause variation)

or not stable (special cause variation).

3. Can be used to define process capability.

4. Allow us to more accurately predict process behavior and

future performance.

Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02

Month

Nu

mb

er

of

Co

mp

lain

ts

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

A

B

C

C

B

A

UCL=44.855

CL=29.250

LCL=13.645

Elements of a Shewhart Control Chart

X (Mean)

Measure

Time

An indication of a

special cause

(Upper Control Limit)

(Lower Control Limit)

42

There Are 5 Basic Control Charts

Variables Charts Attributes Charts

• p-chart(the proportion or percent of defectives)

• c-chart(the number of defects)

• u-chart(defect rate, e.g., falls per 1000 patient days)

• X & S chart (Average & SD chart)

• XmR chart (Individual & moving range chart)

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004, Chap.6

Deciding which chart is most appropriate starts with knowing what type of data you have collected

Variables Data

Attributes DataDefects

(occurrences only)Defectives

(occurrences plus non-occurrences)

Nonconforming UnitsNonconformities

43

The Control Chart Decision Tree

Variables Data Attributes Data

More than one

observation per

subgroup?

X bar & SXmR

Is there an equal area of opportunity?

Occurrences & Non-

occurrences?

p-chartu-chartc-chart

Decide on the type of data

Yes

Yes

Yes

No

No

The percent of

Defective Units

The number

of DefectsThe Defect

Rate

Individual

Measurement

Average and

Standard

Deviation

No


86

Type of Chart Medication Production Analysis

X bar & S Chart TAT for a daily sample of 25 medication orders

Individuals Chart(XmR)

The number of medication orders processed each week

C-ChartUsing a sample of 100 medication orders each week, we count the errors (defects) on each order

U-ChartOut of all medication orders each week, we calculate the number of errors (defects) per 10k orders

P-ChartFor all medication orders each week, we calculate the percentage that have 1 or more errors (i.e., are defective)

The choice of a control chart depends on the measure you have defined!

45

89

AIM* (How good? By when?)

Concept

Measure



Data Collection

Analysis ACTION



90

But the Charts Don’t Tell You…

• The reasons(s) for a Special Cause.

• Whether or not a Common Cause process

should be improved (is the performance of

the process acceptable?)

• How the process should actually be

improved or redesigned.

46

You need a Framework forPerformance Improvement

• Establish appropriate measures.

• Set an aim and goal for each measure.

• Develop theories and predictions on how you plan on achieving the aim and an appropriate time frame for testing.

• Test your theory, implement the change concepts, follow the measures over time and analyze the results.

• Revise the strategy as needed.

311

Finally, remember that data is a necessary part of the Sequence of Improvement

Sustaining improvements and Spreading changes to other locations

Developing a change

Implementing a change

Testing a change

Theory and Prediction

Test under a variety of conditions

Make part of routine operations

47

and the

93

1. What is your current level of knowledge about quality measurement?

2. What is your motivation for measuring?

3. Do you know the milestones in the Quality Measurement Journey (QMJ)?

4. Do you understand variation conceptually?

5. Do you understand variation statistically?

6. How well do you link measurement to improvement

Our Objectives for today…To answer Six Key Questions on Measurement

94

Appendices

•Appendix A: General References on Quality

• Appendix BC: References on Measurement

• Appendix C: Operational Definition Worksheet

• Appendix D: Faculty Bios

48


It must be remembered that there is nothing more difficult to plan, more doubtful of success, nor more dangerous to manage than the creation of a new system.

For the initiator has the enmity of all who would profit by the preservation of the old institution and merely lukewarm defenders in those who would gain by the new one.

A closing thought…

Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513

96

Appendices

•Appendix A: General References on Quality

• Appendix BC: References on Measurement

• Appendix C: Operational Definition Worksheet

• Appendix D: Faculty Bios

49

97

Appendix AGeneral References on Quality

• The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. G. Langley, K. Nolan, T. Nolan, C. Norman, L. Provost. Jossey-Bass Publishers., San Francisco, 1996.

• Quality Improvement Through Planned Experimentation. 2nd edition. R. Moen, T. Nolan, L. Provost, McGraw-Hill, NY, 1998.

• The Improvement Handbook. Associates in Process Improvement. Austin, TX, January, 2005.

• A Primer on Leading the Improvement of Systems,” Don M. Berwick, BMJ, 312: pp 619-622, 1996.

• “Accelerating the Pace of Improvement - An Interview with Thomas Nolan,”Journal of Quality Improvement, Volume 23, No. 4, The Joint Commission, April, 1997.

98

Appendix BReferences on Measurement

• Carey, R. and Lloyd, R. Measuring Quality Improvement in healthcare: A Guide to Statistical Process Control Applications. ASQ Press, Milwaukee, WI, 2001.

• Lloyd, R. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, Sudbury, MA, 2004.

• Nelson, E. et al, “Report Cards or Instrument Panels: Who Needs What?Journal of Quality Improvement, Volume 21, Number 4, April, 1995.

• Provost, L. and Murray, S. The Health Care Data Guide. Jossey-Bass Publishers, 2011.

• Solberg. L. et. al. “The Three Faces of Performance Improvement: Improvement, Accountability and Research.” Journal of Quality Improvement23, no.3 (1997): 135-147.

50

Team name: _____________________________________________________________________________

Date: __________________ Contact person: ____________________________________

WHAT PROCESS DID YOU SELECT?

WHAT SPECIFIC MEASURE DID YOU SELECT FOR THIS PROCESS?

OPERATIONAL DEFINITIONDefine the specific components of this measure. Specify the numerator and denominator if it is a percent or a rate. If it is an average, identify the calculation for deriving the average. Include any special equipment needed to capture the data. If it is a score (such as a patient satisfaction score) describe how the score is derived. When a measure reflects concepts such as accuracy, complete, timely, or an error, describe the criteria to be used to determine “accuracy.”

Appendix COperational Definition Worksheet


DATA COLLECTION PLANWho is responsible for actually collecting the data?How often will the data be collected? (e.g., hourly, daily, weekly or monthly?)What are the data sources (be specific)?What is to be included or excluded (e.g., only inpatients are to be included in this measure or only stat lab requests should be tracked).How will these data be collected?Manually ______ From a log ______ From an automated system

BASELINE MEASUREMENTWhat is the actual baseline number? ______________________________________________What time period was used to collect the baseline? ___________________________________

TARGET(S) OR GOAL(S) FOR THIS MEASUREDo you have target(s) or goal(s) for this measure?Yes ___ No ___

Specify the External target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)

Specify the Internal target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)

Appendix C: Operational Definition Worksheet (cont’d)


51

Robert Lloyd, PhD is Vice President at the Institute for Healthcare Improvement (IHI).

Dr. Lloyd provides leadership in the areas of performance improvement strategies,

statistical process control methods, development of strategic dashboards and building

capacity and capability for quality improvement. He also serves as lead faculty for

various IHI initiatives and demonstration projects in the US, the UK, Sweden, Denmark,

Norway, New Zealand, Australia and Africa.

Before joining the IHI, Dr. Lloyd served as the Corporate Director of Quality Resource

Services for Advocate Health Care (Oak Brook, IL). He also served as Senior Director of

Quality Measurement for Lutheran General Health System (Park Ridge, IL), directed the

American Hospital Association's Quality Measurement and Management Project

(QMMP) and served in various leadership roles at the Hospital Association of

Pennsylvania. The Pennsylvania State University awarded all three of Dr. Lloyd’s

degrees. His doctorate is in agricultural economics and rural sociology.

Appendix D: Robert Lloyd, PhD

Dr. Lloyd has written many articles and chapters in books. He is also the co-author of the internationally

acclaimed book, Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control

Applications (American Society for Quality Press, 2001, 5th printing) and the author of Quality Health Care: A

Guide to Developing and Using Indicators, 2004 by Jones and Bartlett (Sudbury, MA).

Dr. Lloyd lives in Chicago with his wife Gwenn and amusing dog Cricket. The Lloyds have a 22 year old

daughter Devon who is in her final year of university majoring in performance dance and choreography.

[email protected]

@rlloyd66

Appendix D: Gary Sutton, BSc (Honors) P102

Gary Sutton, is a Statistician and Improvement Advisor for

the Scottish Government. He holds a BSc (Honors) in Applied

Statistics. He has been employed as a statistician in the UK

and Scottish Government for over 20 years where he has

been involved in the management of a number of government

statistical surveys. Between July 2012 and December 2015,

Gary has specialized in applying Quality Improvement (QI) to

the Early Years Collaborative (EYC). He is a graduate of the

IHI Improvement Advisor Development Programme (2013)

and has also been an IA Grad for the IHI IA Programme in

Scotland (2015). Gary has presented QI topics to both large

and audiences and QI teams. He has also provided support

to IHI Faculty in the training and development of Scottish

faculty.

Building a Measurement System That...

Documents

Transcript of Building a Measurement System That...