1B - Why Measure.150119.RS - IHIapp.ihi.org/Events/Attachments/Event-2590/Document... · Microsoft...

Advanced Measurement for Improvement

Cambridge, MA • March 26-27, 2015

1

1B - Why Measure?

Advanced Measurement for Improvement Seminar

March 26-27, 2015

Health Care

Produces these

outcomes

“The Gap”

Intention

to

improve

A Vision

A Plan

to

improve

Changes

Patients

Patients/Care seekers want these

outcomes

Advanced

Measurement

Seminar

Improvement Capability



2

Pre-work

Introduced three faces of performance

measurement:

�Research

�Accountability/Judgment

�Improvement

P3

Prework_Three_Types_of_Measurement.pptx

Research,Accountability,Improvement

Three faces of measurement

Translational research

Cooking the numbers

Why? Who?



3

3 Faces of Performance Measurement

Improvement Accountability Research

Aim Improvement of care Comparison, choice, reassurance, spur for change

New knowledge

Test Observability Test observable No test, evaluate current performance

Test blinded or controlled

Bias Accept consistent bias Measure and adjust to reduce bias

Design to eliminate bias

What to Measure “Just enough” data, small sequential samples

Obtain 100% of available, relevant data

“Just in case” data

Flexibility of

Hypothesis

Hypothesis flexible, changes as learning takes place

No hypothesis Fixed hypothesis

Testing Strategy Sequential tests No tests One large test

Determining if a

Change is an

Improvement

Run charts or Shewhart control charts

No change focus Hypothesis, statistical tests (t-test, F-test, chi square), p-

vlaues

Response to Context Variation

Utilize to test resilience of process design

Ignore, acknowledge, or use risk adjustment

Control or eliminate effects of confounding vbls

Confidentiality of

the Data

Data used only by those involved with improvement

Data available for public consumption and review

Research subjects’ identities protected

Solberg, L. I., G. Mosser, et al. (1997)

Translational Research

Westfall, J. M., J. Mold, et al. (2007), Khoury, M. J., M. Gwinn, et al. (2007).

Quality Improvement is a T3

enterprise



4

Measurement for Improvement

What's the logic of Measurement for Improvement?

(1) "If we measure, we have to improve."

Or

(2) "If we improve, we have to measure."

?

P8

Source: Kevin Little



5

Three ways to get better numbers P9

1. Improve the system.

2. Distort the system.

3. Distort the numbers.

Brian Joiner, Ph.D., statistician,

founder of Joiner Associates,

Kevin’s boss, teacher and friend.

He sketched his "better numbers"

analysis 30 years ago.


Campbell's Insight on Distortion

“The more any quantitative social

indicator is used for social decision-

making, the more subject it will be to

corruption pressures and the more apt

it will be to distort and corrupt the

social processes it is intended to

monitor.”

"Campbell's Law" from Assessing the Impact

of Planned Social Change, 1976

P10

Donald T. Campbell, Ph.D.

social psychologist

(1916-1996)




6

Distorting the system & the numbers

Measurement Use: Judge

schools and students as

successful based on the

scores from a competency test.

"Unless fifty-eight per cent of

students passed the math

portion of the test and sixty-

seven per cent passed in

language arts, the state could

shut down the school."

P11

www.newyorker.com/magazine/2014/0

7/21/wrong-answer


CMS Five-Star Nursing Home Ratings: Measurement Challenges



7

P13

http://www.medicare.gov/NursingHomeCompare/About/Ratings.html accessed 19 Nov 2014

Self-reported by

Nursing Homes

before 2015


P14

"Receiving a high star rating has never been more important to nursing homes.

When nurses and doctors discharge patients from hospitals, they often use the

ratings in referral decisions, and insurers consider them when setting up preferred

networks. The ratings are also often a first stop for investors and lenders, who

consult them to decide whether a nursing home company is a safe bet."



8

Visual Story: Text, numbers and repeated stacked bar graphs, over time

P15

http://www.nytimes.com/2014/10/07/business/medicare-alters-its-

nursing-home-rating-system.html

More Background-1

Up through 2014, staff ratings are derived from an annual self-report, completed at same time as the annual inspection.

Raising staff levels for the month when the annual inspection occurs can yield a higher score on the staff component and so boost the star rating.

CMS announced in October 2014 that starting in 2015 they will change the staffing component to a quarterly report, linked to electronic payroll data, potentially auditable.

(based on New York Times articles)

P16




9

More Background-2 P17

"Advocates for patients and for

large employers that pay for

healthcare services argue that the

government should not slow the

rollout. They acknowledge that

performance ratings are an

imperfect science, but counter that

more transparency and better tools

are needed to help consumers

make informed choices."

"Dr. Patrick Conway, deputy

administrator for innovation and

quality for the CMS, said in a June

blog post that ratings empower

consumers and encourage

providers to strive for higher levels

of quality."

http://www.modernhealthcare.com/article/20140830/NEW

S/308299954 accessed 19 Nov 2014


More Background-3 P18

Risk-

adjusted

http://www.cms.gov/Medicare/Provider-Enrollment-and-

Certification/CertificationandComplianc/Downloads/usersguide.pdf p. 10 accessed 19 Nov 2014



10

Exercise: CMS Five-Star Ratings

CMS describes the Five-Star ratings this way:

“The primary goal of this rating system is to provide residents and their families with an easy way to understand assessment of nursing home quality, making meaningful distinctions between high and low performing nursing homes.”

(Design for Nursing Home Compare Five-Star Quality Rating System: Technical Users’ Guide, July 2012, p. 1)

1. Our view: the CMS Five-Star rating is a measurement system primarily for judgment/accountability.

Do you agree or disagree? Why?

2. Many people argue that transparent reporting of measures can drive better performance, reflecting improvements in nursing homes.

How could the Five-Star rating system be amended so that people will improve nursing homes rather than distort nursing homes or distort the reported numbers?

Discuss in pairs & share!

P19


Insights? Questions?

1B - Why Measure.150119.RS - IHIapp.ihi.org/Events/Attachments/Event-2590/Document... · Microsoft...

Documents

Transcript of 1B - Why Measure.150119.RS - IHIapp.ihi.org/Events/Attachments/Event-2590/Document... · Microsoft...