Mission, Strategy, Values - IHI

10/20/2015

1

IHI ExpeditionPutting your Patient Experience Data to Work

Session 2: Understanding Potential Pitfalls and

How to Avoid Data Craziness: Formal Survey Data

October 20, 2015

These presenters have

nothing to disclose

Kevin Little PhDKristine White RN, BSN, MBAAngela Zambeaux

Today’s Host2

Rebecca Goldberg, Project Coordinator, Institute for

Healthcare Improvement (IHI), coordinates multiple

projects focused on increasing value in health care by

improving quality and reducing costs. Currently,

Rebecca’s primary responsibility is coordinating and

hosting IHI’s Expeditions, monthly virtual support

programs focused on specific topic areas. Rebecca is a

recent graduate of Georgetown University in

Washington, D.C., where she obtained her Bachelor of

Science degree in human science with a minor in public

health.

10/20/2015

2

Audio Broadcast3

You will see a box

in the top left hand

corner labeled

“Audio broadcast.”

If you are able to

listen to the

program using the

speakers on your

computer, you

have connected to

the audio

broadcast.

Phone Connection (Preferred)4

To join by phone:

1) Click on the “Participants”

and “Chat” icon in the top,

right hand side of your

screen to open the

necessary panels

2) Click the button on

the right hand side of the

screen.

3) A pop-up box will appear

with the option “I will call

in.” Click that option.

4) Please dial the phone

number, the event

number and your attendee

ID to connect correctly .

10/20/2015

3

WebEx Quick Reference

• Please use chat to

“All Participants”

for questions

• For technology

issues only, please

chat to “Host”

5

Enter Text

Select Chat recipient

Raise your hand

6

Chat

6

Name and the Organization you represent

Example: Sam Jones, Midwest Health

Please send your message to All Participants

10/20/2015

4

Expedition Director

7

Angela G. Zambeaux, Senior Project Manager, Institute

for Healthcare Improvement (IHI), has managed a wide

variety of IHI projects, including a project funded by the US

Department of Health and Human Services that partnered

with the design and innovation consulting firm IDEO

around shared decision-making and patient-centered

outcomes research, the STAAR (STate Action to Reduce

Avoidable Rehospitalizations) initiative, virtual

programming for office practices, and in-depth quality and

safety assessments for various hospitals and hospital

systems. Prior to joining IHI, Ms. Zambeaux provided

project management support to a small accounting firm

and spent a year in France teaching English to elementary

school students.

Expedition Objectives

At the conclusion of this Expedition, participants will be able to:

List the variety of patient experience data available in your organization

Identify and avoid wasted effort in use of required data

Discuss the use of complaint data for improvement

Place patient stories in context

Define fast and slow feedback and provide examples of when each is appropriate

Explain the role of leaders in interpreting and using data to drive improvement

8

10/20/2015

5

Expedition Sessions

Session 1 – Data Sources: What’s Out There and What Do

You Have?

Session 2 – Understanding Potential Pitfalls and How to

Avoid Data Craziness

Session 3 –Using Surveys, Letters, and Complaints as Data

Session 4 –Storytelling: Patient, Clinician, and Staff Stories

Session 5 – Fast and Slow Feedback – Best Practices for

Both

Session 6 – Leadership from Where You Are and Where You

Are Going

9

Session Objectives

List the variety of patient experience data

available in your organization (continuation

from Session 1)

Identify and avoid wasted effort in use of

required survey data

10/20/2015

6

Faculty

11

Kevin Little, PhD, Improvement Advisor, Institute for Healthcare

Improvement (IHI), is a statistician specializing in the use of

information to study, understand, and improve system

performance. His experience in application of statistical methods

includes direct work with scientists and engineers in a range of

disciplines. He has also coached improvement teams in a range of

industries. Dr. Little served as Improvement Advisor to the

National Health Disparities Collaboratives from 2001 to 2006, and

to IHI's Hospital Portfolio of projects from 2010 to 2012. Recently,

he has worked on the measurement strategy for the Healthier

Hospitals Initiative and led a pilot to improve physician

communication behaviors.

Faculty

12

Kristine K. S. White, RN, BSN, MBA, Principal, Aerate Consulting, and

Co-Founder, Aefina Partners, LLC, previously served in senior leadership

roles at Spectrum Health. Her areas of expertise include leadership and

system design for cultures of excellence and innovation, integrating

innovation practices and skills into organizations, and readying cultures

and organizations to solve problems and identify new tools and

processes for the future. Ms. White has worked with physicians to

increase the effectiveness of physician communication efforts and with

leaders and teams to drive meaningful improvement in the patient and

family experience in organizations of all types and understand and utilize

patient experience data sets. She has also coached senior teams to

strategically focus and prioritize efforts that yield value to patients within

their systems. Ms. White is passionate about integrating patient and

family advisors into the design and evaluation of health care and has

helped many organizations build the infrastructure and processes to do

so. Her aim is to connect leaders and health care teams to a clear

purpose, with measurable and sustainable impact and value to patients

and their families.

10/20/2015

7

Agenda

Review: Patient Experience Data Sources and Action Period Assignment #1

KL's Measurement Perspective

Jargon Check: Top Box

Survey Data: Top Four Advice to maximize impact and reduce wasted effort

Case Vignette: Using Survey Data to Assess Impact of a Change Guest Speaker Amanda Griffiths

Percentiles, Correlations, How n Matters

Assignment for Session 3

Review: Sources of Patient Experience Data

10/20/2015

8

Patient Experience Data Overview

No single perfect measure of patient experience

exists

“Good enough” data, multiply sourced, drives

improvement

Focus on things that matter

Understand organizational opportunities AND

team specific opportunities

15

Patient experience is multi-faceted

Each data source gives

you one view of patient

experience.

Multiple sources provide

different views and

details, enable you to

build a coherent picture.

10/20/2015

9

Sources of Patient Experience Data

CAHPS: respecting its influence, understanding its limitations

Press Ganey, NRC Picker, Gallup, Avatar, etc.

Focus groups

Patient Relations

Patient/Family advisors

Billing

Physicians

Safety culture surveys

Staff and provider engagement surveys

“Hot” comments- a gold mine

18

Data Source Data TypeDirect or Indirect

Patient Experience

CAHPS surveys (national government-sponsored patient experience surveys in U.S.)

Survey data Direct3rd party formal surveys, linked to common set of

questions across multiple organizations

In-house Comment Cards/Open Ended questions of patients

Staff vitality surveys, safety culture surveys Indirect

Patient/Family AdvisorsFocus groups, conversations

DirectPatients and Families

StaffIndirect

Physicians

Front line process/service performance data Workplace (“Gemba”) data

Indirect/Direct

Rounding observations Indirect

Patient Relations data (grievances, complaints and positive letters)

Admin/Operations data

DirectBilling complaints and issues (U.S.)

Dashboard metrics: LWBS, errors, safety performance etc.

Indirect

A Table to Organize Patient Experience Data

10/20/2015

10

Review of Action Period Assignment #1

Looking at the survey data at a high level, it is clear that

while the majority of organizations have the data listed,

as we drill down to accessibility, understanding and

summary by senior leaders, and link back to point of

care staff, it is clear that there are opportunities for

improvement.

Most units/departments are reviewing data monthly,

many organizations are also reviewing data monthly but

there are also a large number doing so annually.

For nearly all groups of people, you are reviewing more

than 30 surveys in each time period listed.

Measurement Perspective

Measurement is necessary but only a means to a greater purpose

We focus on formal, designed surveys of patient experience

20

10/20/2015

11

STOPDATA

CRAZINESS

Three faces of Measurement

– Research

– Accountability/Judgment

– Improvement

P22

More explanation in Appendix 1

10/20/2015

12

Informing Ecological Design, LLC • Madison, WI

Lord Kelvin's advice

• "… when you can measure what

you are speaking about, and

express it in numbers, you know

something about it; but when you

cannot measure it, when you

cannot express it in numbers,

your knowledge is of a meager

and unsatisfactory kind…"

• "the more you understand what is

wrong with a figure, the more

valuable that figure becomes.“

Sir William Thomson, Baron Kelvin of Largs

What's "wrong" with government

mandated survey figures?

Low and variable response rate opens door for bias in

estimates

– a challenge for judgment, ok for improvement

Differential responses by demographic groups may

obscure patterns and opportunities

– Patients by payer type (in U.S.!)

– Preferred Language

– Ethnicity

– Comorbidities

Sampling variation: Is 78.6% really different from

81.9%?

24

10/20/2015

13

Jargon Check: Top Box

Why does it matter? What are the implications?

25

Top Box

For survey data,

“top box” refers

to most positive

choice on a

ordered scale*

*Exception: On the CAHPS survey questions that use a 0 to 10 point scale, top box refers to evaluation of a

service as 9 or 10 out of 10 point scale, with 10 the best score possible.

https://cahpsdatabase.ahrq.gov/CAHPSIDB/Public/Files/Doc4_How_Results_are_Calculated_CG_2014.pdf

https://cahpsdatabase.ahrq.gov/CAHPSIDB/Public/Files/Doc4_How_Results_are_Calculated_CG_2014.pdf

10/20/2015

14

Meaning and Use of "Top Box"

Common way for U.S. government agencies* and 3rd-party vendors to report information

Top-box responses are linked to strong customer loyalty

Allows a simple focus on best service and performance

27

*AHRQ, CMS….what about other countries??

Formal Survey Data: Top Four Advice

28

10/20/2015

15

CAHPS* survey data:

"Top Four" to Know and Do

29

In U.S. *Consumer Assessment of Healthcare Providers and Systems" https://www.cahps.ahrq.gov/ …our

points apply to every other formal patient experience survey data we know

Understand Percentiles

Interpret Correlations

Remember effect of "n"

Plot dots in time order

Case Vignette: Using Formal Survey Data in Improvement

Amanda Griffiths, Patient Experience Leader

UnityPoint Health Trinity

Rock Island, IL

Case Vignette: Using Formal Survey Data in Improvement

10/20/2015

16

Context for Our Intervention

HCAHPS scores are low

We give good clinical care

Communication isn’t good

Rounding can improve this

6th floor rounding pilot program

Intervention June 2015Communication with Nurses

Responsiveness of Staff

Pain Management

10/20/2015

17

How should the June 2015 data be interpreted?

Degree of Belief?

11 Month Avg

thru May 2015

June 2015 Point Increase

over 11 mo.

average

%age

increase

Communication with Nurses 70.7 97 26.3 37%

Responsiveness of staff 53.8 82.5 28.7 53.4%

Pain Management 65.2 85.0 19.8 30.4%

Hospital Environment 52.2 68.2 16.0 30.6%

Communication with Doctors 80.5 84.8 4.3 5.2%

Communication about

Medications

51.2 68.8 17.6 34.4%

Percentiles

Why do percentiles matter? What are the implications?

35

10/20/2015

18

Why percentiles matter

You get a sense of what other organizations are able to achieve. Can be a shock and motivator.

Survey performance is improving by "everybody" on most measures. 80% Top Box can be relatively poor; interpret raw scores in context.

36

Alice running with the Red Queen, just to stay in one place

38

https://cahpsdatabase.ahrq.gov/CAHPSIDB/Public/CG/CG_Topscores.aspx#Percentile

accessed 15 October 2015

Excerpt: CAHPS Clinician and Group (Primary Care)

2014 Visit Adult 2.0 Top Box Scores

https://cahpsdatabase.ahrq.gov/CAHPSIDB/Public/CG/CG_Topscores.aspx#Percentile

10/20/2015

19

39

https://data.medicare.gov/data/hospital-compare accessed 11 Oct 2015; data

tables updated 8 Oct 2015 for calendar year 2014

Example using a picture: U.S. national HCAHPS 2014 survey results





10/20/2015

20

79



A six point change in Top Box % above or below the median translates to a ~40% change in percentile score!

Variability by Service Type 90th Percentile42

Data is based on the Press-Ganey Means and Ranks report for FY11Q4

Survey Type Mean Score

Outpatient Oncology 95.1

Outpatient Services 94.5

Ambulatory Surgery 94.2

Home Health 93.3

Adult Medical Practices 92.9

Pediatric Medical Practice 92.6

NICU 91.5

Urgent Care 91.3

Pediatric Inpatient 89.4

Emergency Services 88.8

Adult Inpatient 88.2

LTACH 87.6


10/20/2015

21

43Percentiles are increasing over time: HCAHPS example

Based on summary tables 2008-2014,

http://www.hcahpsonline.org/SummaryAnalyses.aspx accessed 10 October 2015

44

2014

Wait times in

primary care not

improving;

confounded with

increasing numbers

of providers

reporting CG-

CAHPS

http://www.hcahpsonline.org/SummaryAnalyses.aspx

10/20/2015

22

Correlations

While "correlation does not imply causation", "Causation does not

exist without correlation."

45

Why correlations matter

Correlations are a first step to making sense of

relations among multiple survey questions

The lowest score on a panel of questions may not

be strongly associated with overall evaluation

Tackling the lowest score may not be good use of

organization resources

46

See the Data Tools Self Assessment for more details about correlation

10/20/2015

23

What is correlation?

Correlation, based on either scores or ranks, measures strength of association and ranges from 1 (perfect positive linear or rank order relationship) to 0 (no linear or rank relationship) to -1 (perfect negative linear or reverse rank order relationship.)

Here’s a picture that shows some invented data, with the correlation coefficient ranging from 0.96 to 0.55

47

Specific Case of General Pattern48

http://circoutcomes.ahajournals.org/content/3/2/188.full.pdf+html accessed 15 Oct 2015

…we found that hospitals that score high on questions such as “skill of nurses (physician),” “how well the nurses (physician) kept you informed,” “amount of attention paid to your special or personal needs,” “how well your pain was controlled,” “the degree to which the hospital staff addressed your emotional needs,” “physician’s concern for your questions and worries,” “time physician spent with you,” and “staff efforts to include you in decisions about your treatment” also tended to score high on patient overall satisfaction. In contrast, there was no association with scoring high on questions concerned with the room (eg, “room temperature and pleasantness of room decor”), meals (eg, “quality of food, temperature of food”), tests (eg, “waiting time for tests or treatment”), and discharge (eg, “speed of discharge process”) and the patient overall

satisfaction score. Moreover, patient satisfaction with nursing care was the most important determinant of patient overall satisfaction,

thus highlighting an important area for further quality improvement efforts and underscoring the role of the entire health care team in the in-hospital treatment of patients with AMI.” (p. 193, emphasis added.)

http://circoutcomes.ahajournals.org/content/3/2/188.full.pdf+html

10/20/2015

24

Adult Inpatient Correlations

Press Ganey National database – through June 30, 2012

Staff addressed emotional needs .79

Staff sensitivity to inconvenience .78

Teach/instruct self-care, med, treatment .78

Staff attitude toward visitors .74

Nurses kept you informed .73

How well your pain was controlled .69

Skill of physician .67

Room cleanliness .62

Noise level in and around room .52

50

http://www.hcahpsonline.org/Files/Report_April_2015_Corrs.pdf

http://www.hcahpsonline.org/Files/Report_April_2015_Corrs.pdf

10/20/2015

25

How n matters

51

Why "n" matters

Survey samples have built-in variation, just from

sampling

– This is in addition to the variation that arises from different

experiences of care, the variation you need to control

When you examine formal survey responses for

departments or units within your organization, n can get

"small."

Knowledge of n should inform comparisons, month to

month or across units or providers ("control chart

thinking")

10/20/2015

26

Web app:https://iecodesign.shinyapps.io/survey_simulator

53

Knowledge of the “n effect” should

Dampen or eliminate management cycles of

despair or celebration, based on a single

reported percent.

Cause you to interpret one month unit-level

results (really small n!?) with caution

Help make the case for plotting survey results in

time order

Inspire you to learn and use control charts--see

Provost and Murray (2012)

54

10/20/2015

27

Return to UnityPoint Example

For June 2015, 18 surveys were returned in the latest report for a med/surg acute care unit.

One of the questions of the nursing communications composite is "During this hospital stay, how often did nurses listen carefully to you?"

17 of 18 patients responded "Always" (Top Box) = 94.4%.

If the Top Box average for this question for the previous year was 82%, how unusual is it to observe 94.4%?

55

How do you interpret these data?Table for 4 Quarters

Chart of Top Box ValuesPhys IDN

surveys

count of

Top BoxTB %

4 2 1 50.0

13 2 2 100.0

16 3 1 33.3

17 4 4 100.0

3 5 5 100.0

5 7 6 85.7

9 8 7 87.5

6 9 7 77.8

11 11 9 81.8

2 11 10 90.9

19 13 11 84.6

14 15 11 73.3

8 21 16 76.2

1 22 18 81.8

7 22 20 90.9

12 23 15 65.2

18 23 18 78.3

15 25 21 84.0

10 27 19 70.4

10/20/2015

28

Plot survey data in time order

Start with run charts, move on to control charts

57

Why plot data in time order?

Single survey numbers provide very little useful

guidance for improvement

You need time order to make “before and after”

comparisons to assess progress

If “n” is about the same for each survey number

in the series, you can look for striking patterns

over time to signal improvement with run charts

See Perla et al. (2011) for run chart shift and trend “rules” useful for fewer than 20 time periods

10/20/2015

29

59

n = 20 each month, sampled from adjusted model, 10% sampling fraction

No

change,

points

just

bounce

around!

Plot survey data in time order 60

Collabstart

Baseline median

10/20/2015

30

61

Compressed

Percentile

scale is

good

news/bad

news.

Do you know

which is

which?

Return to UnityPoint Example

Strengths Enhancements to consider

Showed three survey items plotted in one page

You can show many items plotted on one page; use medians.

Showed 12 monthly values Show 24 months to help you understand seasonal effects

Asked for survey data by month of service not month of survey completion

Annotate chart with notes of interventions

Recorded the number of surveys, month by month

Construct a control chart to account for variation in sample sizes

10/20/2015

31

To learn more

Perla, R., Provost, L., and Murray, S. (2011), “The run chart: a simple analytical tool for learning from variation in healthcare processes”, BMJ Quality & Safety. 2011 Jan; 20(1):46-51. Note: the shift and trend rules cited are appropriate only for data series no more than 20 points long. See http://www.iecodesign.com/index.php/217-run-charts-in-quality-improvement-work

Provost, L. and Murray, S. The Health Care Data Guide: Learning from Data for Improvement. Jossey-Bass Publishers, 2011, especially chapter 12.

http://www.iecodesign.com/index.php/223-rank-and-funnel-plots

Data Tools Self-Assessment with Answers

63


64

http://www.ncbi.nlm.nih.gov/pubmed/21228075

http://www.iecodesign.com/index.php/217-run-charts-in-quality-improvement-work

http://www.iecodesign.com/index.php/223-rank-and-funnel-plots

10/20/2015

32


1. Make a simple flow chart (three to five steps) that shows

how formal patient complaints are handled by your

organization.

2. Annotate the flow chart so that it is clear:

who receives the complaints

who reviews the complaints

who determines actions

how the information is stored for analysis and reference

3. Are verbal or written comments that do not rise to the level

of formal complaint handled the same way as formal

complaints? Why or why not?

Appendix 1: Three faces of measurement

10/20/2015

33

Why are you measuring?

The answer to this question will guide your entire quality

measurement journey!

Improvement?(improving the effectiveness or

efficiency of a process or system)

Accountability Judgment?

(making comparisons; no change focus)

Research?(testing theory and building

new knowledge)

The Three Faces of Performance Measurement

Aspect Improvement Accountability Research

Aim Improvement of care

(efficiency & effectiveness)

Comparison, choice,

performance management

New knowledge

(efficacy)

Methods:

• Test Observability Test are observable No test, evaluate current

performance

Test blinded or controlled

• Bias Accept consistent bias Measure and adjust to

reduce bias

Design to eliminate bias

• Sample Size “Just enough” data, small

sequential samples

Obtain 100% of available,

relevant data

“Just in case” data

• Flexibility of

Hypothesis

Flexible hypotheses,

changes as learning takes

placeNo hypothesis

Fixed hypothesis

(null hypothesis)

• Testing Strategy Sequential tests No tests One large test

• Determining if achange is animprovement

Run charts or Shewhart

control charts

(statistical process control)

No change focus

(maybe compute a percent

change or rank order)

Hypothesis, statistical

tests (t-test, F-test,

chi square, p-values

• Confidentiality ofthe data

Data used only by those

involved with improvement

Data available for public

consumption and review

Research subjects’

identities protected

Reference: Solberg, L., Mosser, G., and McDonald, S. “The Three Faces of Performance Measurement:

Improvement, Accountability and Research” Journal on Quality Improvement vol. 23, no. 3, (March 1997), 135-147.

10/20/2015

34

Expedition Communications

• All sessions are recorded

• Materials are sent one day in advance

• Listserv address for session communications:

[email protected]

• To add colleagues, email us at [email protected]

69

Session 370

Using Surveys, Letters, and Complaints as DataTuesday, November 3rd, 2015, 1:00-2:00 PM ET

mailto:[email protected]


10/20/2015

35

Thank You!

71

Angela Zambeaux

[email protected]

Rebecca Goldberg

[email protected]

Please let us know if you have any questions or

feedback following today’s Expedition webinar.

Action Period 1 Survey Results

72



10/20/2015

36

Survey Results 73

Does your organization have this information?

Yes– No– I'm not sure–

Government-defined patient experience survey data (e.g. CAHPS in the U.S.)

80.00%28

17.14%6

2.86%1

Externally designed and managed patient experience data other than government defined surveys (e.g. Press-Ganey surveys)

52.94%18

38.24%13

8.82%3

Internally designed and managed patient survey data (e.g. 90-day follow up phone call on patient experience)

68.57%24

20.00%7

11.43%4

Formal patient complaints 100.00%35

0.00%0

0.00%0

Patient letters 85.71%30

2.86%1

11.43%4

Leadership rounding/direct observation 71.43%25

17.14%6

11.43%4

Spoken patient complaints or comments

83.33%30

2.78%1

13.89%5

Survey ResultsIs the information organized and accessible to you?



83.33%25

6.67%2

10.00%3


60.00%15

28.00%7

12.00%3


41.94%13

45.16%14

12.90%4


20.59%7

5.88%2


21.21%7

15.15%5

Leadership rounding/direct observation

39.39%13

57.58%19

3.03%1


61.76%21

26.47%9

11.76%4

10/20/2015

37

Survey ResultsIs the information summarized and understood by senior leaders?



63.33%19

6.67%2

30.00%9


48.00%12

28.00%7

24.00%6


32.26%10

35.48%11

32.26%10


11.76%4

20.59%7


21.88%7

25.00%8


50.00%16

25.00%8

25.00%8


51.52%17

18.18%6

30.30%10

Is the information linked back to front-line staff?

–Yes– No– I'm not sure–


58.06%18

25.81%8

16.13%5


46.15%12

34.62%9

19.23%5


31.25%10

40.63%13

28.13%9


22.86%8

42.86%15


27.27%9

30.30%10


42.42%14

30.30%10

27.27%9


42.42%14

24.24%8

33.33%11

Survey Results

10/20/2015

38

Additional Data SourcesPress Ganey, Leadership Rounds, looking into a new administration and management patient experience rounding app.

NSQIP data for 30 day patient outcomes after surgery

Community Health Assessment-informs programs and services regarding population health needs.

Patient satisfaction surveys

Discharge phone calls 24-48 hrs post discharge include service issues and names of staff

Patient focus groups run regularly with inpatients on subacute site, information gathered is themed and fed back to frontline teams. Patient stories are also used in a similar way.

We have a Patient Advisory Council that meets every other month who provide us with feedback on "focus areas" and we have consumer board members who also provide feedback.

A Incident-reporting system

Additional mock CAHPS/HOS Survey data

We accept comments (complaints and commendaitons) via a web application from all pts and families - managed similar to patient letters

Automated discharge phone calls to all inpatients

FaceBook comments, local press

The information may be inconsistently shared among front line staff.

78Survey ResultsTime Period: How frequently are these surveys reviewed by the groups/people in the left column?

Annual– Quarterly– Monthly–

Whole organization 29.03%9

22.58%7

48.39%15

Unit or department 9.38%3

15.63%5

75.00%24

Care team or individual provider

7.41%2

29.63%8

62.96%17

1-10– 11-20– 21-30– More than 30–

Whole organization 12.90%4

3.23%1

0.00%0

83.87%26

Unit or department 16.13%5

6.45%2

12.90%4

64.52%20

Care team or individual provider

30.77%8

7.69%2

11.54%3

50.00%13

Typical number of surveys completed and analyzed in each time period

Mission, Strategy, Values - IHI

Documents

Transcript of Mission, Strategy, Values - IHI