1

Developing a Dashboard Measurement and Evaluation:

Understanding the Impact of Innovation

October 29, 2013

2

Objectives

Describe dashboard development process in the context of MGH Innovation Unit work.

Identify barriers and facilitators in developing, implementing, and sustaining the Innovation Unit Dashboard performance measurement tool.

3

Agenda

Overview of Innovation Units Empirical Outcomes Dashboard Development Rationale Innovation Unit Dashboard:

Metric selection Data sources & relevant benchmarks

Using data to tell stories Future considerations

Positioning MGH for The Future

Technology Application: Partners E-Care, Outcomes Registries

ThePatientJourney

Direct Patient Care:

ED, Periop, Inpatient

(Innovation Units)

Population Manageme

nt:Reducing the

Trend of Healthcare

Costs, Long-term

Outpatient Care

Patient Affordability For MGH & Payers:

Overhead (Non-Labor

costs)

Care Redesign: Multidisciplina

ry Services, Large Patient

Population, Big $$$

Incentives: Intrinsic and Extrinsic

5

Innovating Care at MGH

Innovation Units are tests of change that will help us quickly identify what works and what does not work to improve the quality of care delivered to our patients.

High performing interdisciplinary teams that deliver safe, effective, efficient, timely, equitable care, that is patient- and family-centered

Standardization of processes and care reduces variation and introduces a systematic approach to improving quality and safety in the inpatient setting

Identify and prioritize hazards and opportunities for standardization, then implement evidence based methods to rectify the problem

We are attempting transformational change.We are attempting transformational change.

6

Three Key Areas of Focus and Four Desired Outcomes

Focus

1. New Culture through Relationship Based Care

2. New Role of Attending Nurse; Domains of Practice

3. Standardized Processes

Throughput and LOS Reduction

Technology

Controlling Variation

Implementing Evidence Based Practice

Outcomes

1. Patient Satisfaction: care is equitable and patient- and family-focused

2. Clinical Quality: to improve quality and to make care safer

3. Unit Cost Reductions: to make care more cost effective

4. Staff Satisfaction: to remain a great place to practice

7

Before During Post

Where Are There Opportunities to Reduce Costs Across These Processes of Care?

Admission Process: ED,

Direct Admits,

Transfers

Patient Stay; Direct Patient Care, Tests, Treatments, Procedures,

Clinical Support, Operational Support

Discharge Process

Post Discharge Care

Preadmission Care

Support Functions: Finance, Information Systems, HR

Goal: High-performing interdisciplinary teams that deliver safe, effective, timely, efficient and equitable care that is patient and family centered.

“Patient Journey” Framework

- 8 -

Before During After

Admission process: ED, direct admits,

transfers

Patient stay; direct patient care; tests; treatments; procedures;

clinical support; operational support

Discharge process

Post-discharge

care

Pre-admission

care

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Inte

rve

nti

on

Innovations in Care Delivery “Patient Journey” Framework – Initial 15 Interventions

Relationship-based care ♦ The Attending Nurse role ♦ Hand-Over Rounding Checklist

Discharge Planning: -Est. discharge date -Discharge disposition

Welcome Packet (notebook and discharge envelope)

Domains of PracticeDaily Interdisciplinary Team RoundsElectronic Unit WhiteboardsIn-Room WhiteboardsSmart Phones Wireless laptop computers/tabletsBusiness cards Hourly roundingQuiet hours

Discharge -Follow-up Call Program

Goal: High-performing, inter-disciplinary teams that deliver safe, effective, timely, efficient, and equitable care that is patient- and family-centered

Copyright MGH 2012Copyright MGH 2012

9

Focus on Empirical Outcomes

• Focus on “What difference have you made?”

• Shift from structure and process to outcomes.

• Key indicators that paint a picture of the organization.

10© American Nurses Credentialing Center

Donabedian Model

Donabedian, 1966; 1990

Innovation Cluster Focus Areas *

Communication

Patient Engagement

Roles & Structures

Interventions ** Throughout Admission

Relationship-Based CareAttending Nurse Handover Rounding Checklist

Pre-AdmissionPre-Admit Data CollectionWelcome Packet

During AdmissionDomains of Practice Interdisciplinary RoundsBusiness CardsQuiet HoursHourly RoundingElectronic White BoardsIn Room White BoardsSmart PhonesHand Held/ Tablets

Post-DischargeDischarge Follow-up Phone Calls

Others as identified

Ed

uc

ation

Evaluation(Pre, During, Post)

Quantitative Qualitative•HCAHPS

•Leadership Influence over Professional Practice Environments (LIPPES)

•LOS

•Quality Indicators

•Patients Perceptions of Feeling Known (PPFKN)

•Readmissions

•Revised Perceptions of Practice Environment Scale (RPPE)

•Cost per Case Mix

•Staff Retention

•Focus Groups (Staff, Patients, Families, etc)

•Observations

•Narratives

•Survey of the Innovation Unit Expectations (SIUE-pre)

•Survey of the Innovation Unit Experiences (SIUE-post)

* The clusters are a lens with which we gain perspective on any particular intervention.

** May apply to any or all 3 of the cluster focus areas June 2013

Other measures as identifiedOther measures as identified

– Evaluation– Evaluation

12

Why a Dashboard

“Rapid Improvement in any field requires measuring results…”

Porter, Lee

13

Data and Information

Data are individual facts, statistics or items of information. (http://dictionary.com)

Information is the result of processing, manipulating and organizing data in a way that adds to the knowledge of the person receiving it. (http://en.wikiquote.org)

InformationInformationDataData

14

Challenge

Create an easy to use dashboard tool Implement quickly Consolidate Key Performance Indicators (KPIs) from

multiple hospital sources Provide visibility of data across Innovation Units Use current benchmarks to measure performance Foster data transparency Drive improvement through PDSA Cycle

Supporting change with data Testing changes Spreading improvements

15

Dashboard strategy

Begin with end in mind Know your customers/understand how they use

information Know questions dashboard is trying to answer When thinking about metrics, make sure you can

actually collect the data Develop a draft, engage users, and iterate Disseminate to users Refine as needed Periodically revisit dashboard needs as they relate to

ongoing measurement plan.

16

Dashboard tactics Create shell, simple mockup

Make sure all functions and levels of info are represented. Make sure it is feasible to obtain all the data. Figure out who will be doing the data aggregation and preparation. These steps will help determine scope.

Fill in shell with metrics Complete prototype (in Excel) Consolidate draft metrics from multiple sources into single, concise, printable view Highlight performance relative to benchmarks with visual displays Refine structure and design, including time periods for reporting

Demo/pilot dashboard Helps set expectations with users Validates format and metrics

Document business requirements Includes calculation of metrics and description of benchmarks and data sources

Plan for updates

17

QUALITY AND SAFETY

Patient-Centered Outcome MeasuresFalls per 1,000 Patient Days

Total Fall Rate 2.78 3.15 3.55 0.00 0.95 0.00 3.14 1.55 0.00 0.47 1.36 Worse NA Better

Observed 7 9 10 0 1 0 7 2 0 1 3 NA NA NA

Falls with Injury per 1,000 Patient DaysFalls with Injury Rate 0.79 0.70 0.35 0.00 0.00 0.00 0.00 0.77 0.00 0.00 0.00 Worse NA Better

Observed 2 2 1 0 0 0 0 1 0 0 0 NA NA NA

Hospital Acquired (HA) Pressure UlcersTotal HA Pressure Ulcer Prevalence Rate 0.0% 3.3% 0.0% 0.0% 0.0% 0.0% 0.0% 7.1% NA 0.0% 0.0% Worse NA Better

Observed 0 1 0 0 0 0 0 1 0 0 NA NA NA

Hospital Acquired (HA) Pressure Ulcers Type II or GreaterTotal HA Pressure Ulcer Type II or Greater Prevalence Rate0.0% 3.3% 0.0% 0.0% 0.0% 0.0% 0.0% 7.1% NA 0.0% 0.0% Worse NA Better

Observed 0 1 0 0 0 0 0 1 0 0 NA NA NA

RestraintsTotal Restraint Prevalence Rate 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 7.1% NA 0.0% 0.0% Worse NA Better

Observed 0 0 0 0 0 0 0 1 0 0 NA NA NA

Peripheral Intravenous (PIV) Infiltrations - Pediatric/NeonatalTotal PIV Infiltration Prevalence NA NA NA 0.0% 0.0% 0.0% NA NA NA NA NA Worse NA Better

Observed 0 0 0 NA NA NA

Central Line-associated Bloodstream Infections per 1,000 Line Days (CLABSI)Total CLABSI Rate 0.00 1.73 0.00 3.40 0.00 0.00 2.55 2.84 NA 0.00 4.85 >1 NA 0 or 1

Observed 0 1 0 1 0 0 1 3 0 1

Color Shading relative to Benchmark:

Massachusetts General Hospital - PCS Innovation Unit Dashboard At a Glance

Unit Unit

Rate is better (lower) than benchmark.

Rate is worse (higher) than benchmark.

UnitUnitICU ICUMeasures

NDNQI

NDNQI

NDNQI

NDNQI

NHSN

Unit UnitUnit Unit UnitBench-mark

NDNQI

Color Shading Relative to Benchmark

NDNQI

Surg Adult Benchmark 2.57

Critical Care Adult Benchmark 0.99

17

Metric categories: Throughput and Efficiency Patient & Staff Satisfaction Quality and Safety Infection Control Patient Satisfaction Staff Satisfaction

Dashboard Development

Innovation Unit Dashboard sample

Goal: Measure the impact of Innovation Units’ interventions Tactic:Reliably store & communicate evaluation data

Goal: Measure the impact of Innovation Units’ interventions Tactic:Reliably store & communicate evaluation data

18

NSI Reporting, Examples Pre-Innovation unit launch (Summer 2011)

Reporting for select metrics through Excellence Every Day Portal Quarterly unit level data and charts developed and stored in Shared File

Area for RN Leadership (Printed color copies delivered to units)

Sample unit level report for CAUTI metricSample unit level report for CAUTI metric

Sample Shared File area folder structure.

Sample Shared File area folder structure.

Sample Excellence Every Day Portal PageSample Excellence Every Day Portal Page

19

What we did: Pre-Innovation unit launch

Identified need for robust, comprehensive, tool for Nursing Sensitive Indicator (NSI) reporting.

PCS had an initial Executive Committee Dashboard in place. Talked with internal experts for Strategic Performance Indicator

reporting.

J un 10 Jun 10 Jun 10 Jun 10 Jun 10 Jun 10 Jun 10Sep 10 Mar 10 Sep 10 Mar 10 Sep 10 Mar 10 Sep 10 Mar 10 Sep 10 Mar 10 Sep 10 Mar 10 Sep 10 Mar 10

Dec 09 Dec 09 Dec 09 Dec 09 Dec 09 Dec 09 Dec 0997% 100% 99% 97% 98% 96% 100%99% 100% 94% 98% 98% 95% 99%97% 100% 99% 100% 99% 96% 97%95% 96% 94% 98% 92% 96% 94%95% 92% 95% 99% 94% 93% 93%95% 89% 90% 93% 95% 98% 94%97% 97% 98% 99% 97% 98% 97%99% 99% 94% 97% 96% 98% 98%96% 96% 96% 98% 98% 94% 97%98% 100% 100% 100% 100% 98% 95%98% 94% 100% 96% 97% 90% 80%

100% 100% 100% 100% 100% 91% 93%98% 100% 99% 100% 98% 93% 95%98% 94% 94% 96% 97% 90% 80%99% 100% 100% 100% 100% 91% 93%

3.4 SCIP Wound Infection Composite

95%

92% 97%

97%

100% 96%

90%

99%

96%93%100%

97% 99% H 94%97% 98% 97%

94%

96%

3.5 SCIP-VTE, Rate of VTE Prophylaxis:

3.3 PNE Composite

H99%

HPM Metric

H/L

97%

90%

99% 100%

Hospital Hospital

97%98% 98%

Leader

HospitalHospital Hospital Hosptial Hospital

Leader Leader Leader

H

94%

3.2 CHF Composite

better

Current PeriodColor Scoring

Thresholds

@ orH=Higher is better; L=Lower is better

97% H

Leader Leader

99%

97%98%

96%

99% 97%97%

100%

100%

98%

92% 97%99%

H 97% 100%

100%

3.

Un

ifo

rm H

igh

Qu

ali

ty

Ordered

Received

Note: Sample dashboard for demonstration purposes only.Note: Sample dashboard for demonstration purposes only.

20

What we did: Metric selection

Interventions Source1. Improve Patient Experience

Nurse CommunicationQuiet at NightResponsivenessCleanlinessPain ManagementOverall ratingDischarge Information

2. Improve Quality Decrease Hospital Acquired Conditions Relationship-based care

CAUTI Attending NurseFalls with Injury Handover Rounding checklistCLABSI Domains of PracticeCentral Line Infections Interdisciplinary RoundsPressure Ulcers Business Cards

Restraint Utilization Quiet hoursPeripheral Intravenous (PIV) Infiltrations Hourly Rounding

Electronic WhiteboardsSmart Phones

Service Excellence, HCAHPS dataHourly RoundingQuiet HoursPain Tiger TeamHotel Style CleaningSmart Phones and WhiteboardsPatient/Family notebookDischarge EnvelopeDischarge phone callsUnit based patient advocate

Goals and Metrics

Infection Control, Patient Care Services Office of Quality and Safety

21

What we did: Metric selection, continued

Interventions Source3. Reduce Costs

Direct Cost per Case mix adjusted discharge Follow up phone callsLabor Expense HandoversHours Worked per Equiv Patient Day White boardsMedical Supply Expense Smart PhonesTotal Expense Discharge Envelope

4. Maintain Staff SatisfactionProfessional Practice Environment Staff Relationship-based care

Staff Perception Survey Mean Scores Attending Nurse RoleNDNQI RN Survey Practice Environment Domains of Practice

Scale--Nursing Work Index Mean Score

Readmissions Standardized processesALOS Use of TechnologyAdmission/ ED Admit Volume Controlling Variation

Implementing Evidence-based practiceSafe Handover Communication

Goals and Metrics

Finance Department, Admitting Department, Center for Quality and Safety

Optimize Efficiency/Throughput

Institute for Patient Care

PHS Finance, MGH Finance

22

What we did: Data Sources & Benchmarks

PHS Finance (EPSI)PHS Finance (EPSI)

Patient Care Services(NDNQI)

Patient Care Services(NDNQI)

Infection Control (CDC)

Infection Control (CDC)

PCS Institute for Patient Care

(Staff Perception Surveys)

PCS Institute for Patient Care

(Staff Perception Surveys)

Admitting (PATCOM)Admitting (PATCOM)

ED Information System(EDIS)

ED Information System(EDIS)

MGH Finance (EPSI/Action OI)MGH Finance

(EPSI/Action OI)

Service Excellence(HCAHPS)

Service Excellence(HCAHPS)

Ellison 17 Ellison 18

QUALITY AND SAFETY

Patient-Centered Outcome MeasuresFalls per 1,000 Patient Days

Total Fall Rate 4.50 1.46 4.95 0.77 1.92 1.32 2.16 1.79 TBD 0.65 4.85 0.45Observed (N) 11 3 13 1 2 2 5 2 2 10 1

Falls with Injury per 1,000 Patient DaysFalls with Injury Rate 0.41 0.49 1.52 0.00 0.96 0.00 0.00 0.89 TBD 0.00 1.45 0.45Observed (N) 1 1 4 0 1 0 0 1 0 3 1

Hospital Acquired (HA) Pressure UlcersTotal HA Pressure Ulcer Prevalence Rate 0.0% 0.0% 6.9% 0.0% 0.0% 0.0% 0.0% 7.7% TBD NA 4.8% 4.2%Observed (N) 0 0 2 0 0 0 0 1 1 1

Hospital Acquired (HA) Pressure Ulcers Type II or GreaterTotal HA Pressure Ulcer Type II or Greater Prevalence Rate0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 7.7% TBD NA 4.8% 4.2%Observed (N) 0 0 0 0 0 0 0 1 1 1

RestraintsTotal Restraint Prevalence Rate 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 7.7% TBD NA 0.0% 0.0%Observed (N) 0 0 0 0 0 0 0 1 0 0

Peripheral Intravenous (PIV) Infiltrations - Pediatric/NeonatalTotal PIV Infiltration Prevalence NA NA NA 0.0% 0.0% 0.0% NA NA NA NA NA NAObserved (N) 0 0 0

Central Line-associated Bloodstream Infections per 1,000 Line Days (CLABSI)Total CLABSI Rate 6.54 NA 1.36 2.90 4.76 0.00 1.10 1.70 TBD NA 0.00 0.00Observed (N) 1 1 1 1 0 1 2 0 0

Note: metrics to be reported beginning FY 2012 Color Shading relative to Benchmark:Catheter-associated Urinary Tract Infections per 1,000 Device DaysVentilator-associated Pneumonia per 1,000 Vent Days

Massachusetts General Hospital - PCS Innovation Units Dashboard

Rate is better (lower) than benchmark.

Rate is worse (higher) than benchmark.

VascularBigelow 14

ObstetricsBlake 13

ICUBlake 12

NICUBlake 10

CICUEllison 9

MeasuresOrtho

White 6OncologyLunder 9

MedicineEllison 16

Pediatrics SurgeryWhite 7

PsychBlake 11Individual Units listed across topIndividual Units listed across top

Service Excellence(Pediatric Survey)Service Excellence(Pediatric Survey)

CQS(EPSI - Readmissions)

CQS(EPSI - Readmissions)

Many sources and contacts Many (many) data formats

23

What we did: Dashboard notes

Defines metric, source, contact, frequency.

Notes: Calculation Benchmark Data Source Contact FrequencyTotal Fall Rates Number of Patient Falls with Injury is the

number of events reported that resulted in patient injury, calculated per 1000 patient days. Lower values are better performance. At this time there are no unit-based benchmarks for patient falls so we are using Benchmarks from Partners HPM. In September 2010, PCS Quality & Safety began to submit data to NDNQI and unit based benchmarks will soon be available.

The number of events reported, calculated per 1000 patient days.

NDNQI Incident reports from RL Solutions

Nancy McCarthy Quarterly

Falls w/ Injury Rates Number of Patient Falls with Injury is the number of events reported that resulted in patient injury, calculated per 1000 patient days. Lower values are better performance. At this time there are no unit-based benchmarks for patient falls so we are using Benchmarks from Partners HPM. In

The number of events reported that resulted in patient injury, calculated per 1000 patient days.

NDNQI Incident reports from RL Solutions

Nancy McCarthy Quarterly

HA - Pressure Ulcer Rates--ALL

Quarterly pressure ulcer incidence rate data is collected in a one-day prevalence study. # of hospital acquired pressure ulcers (any stage). Lower values are better.

Numerator = # of hospital acquired, stage II or greater pressure ulcers. Denominator = Total # of discharges?

NDNQI One-day prevalence / incidence study conducted by nursing reps from PCS Office of Quality and Safety.

Nancy McCarthy Quarterly

HA - Pressure Ulcer Rates Stage II or greater

Quarterly pressure ulcer incidence rate data is collected in a one-day prevalence study. # of hospital acquired, stage II or greater pressure ulcers. Lower values are better.

Numerator = # of hospital acquired, stage II or greater pressure ulcers. Denominator = Total # of discharges?

NDNQI One-day prevalence / incidence study conducted by nursing reps from PCS Office of Quality and Safety.

Nancy McCarthy Quarterly

Restraint Rate Restraint prevalence % of Patients in restraints NDNQI One-day prevalence / incidence study conducted by nursing reps from PCS Office of Quality and Safety.

Office of Quality and Safety

Quarterly

Peripheral Intravenous Inflitrations

NEW; Pediatric and Neonatal populations Total PIV Infiltration Point Prevalence. Total number of PIV infiltrations (Grades 2-4) on the unit divided by the total number of PIV sites on a unit. For children less than age 10, a Grade 1 infiltration is defined identically to a Grade 2.

NDNQI One-day prevalence / incidence study conducted by nursing reps from PCS Office of Quality and Safety.

Office of Quality and Safety

Quarterly

CLABSI Infection Rate Quarterly Line Infection incidence rate Numerator = # Line Infections (hospital acquired). Denominator = 1000 line days

NHSN Pooled Mean

Infection Control Paula Wright, Irene Goldenshtein

Quarterly

Massachusetts General Hospital - PCS Innovation Unit Dashboard Notes

Metric

24

Success of Dashboard Tool

Initial dashboard pushed out when (12) Innovation Units launched. Expanded dashboard as other phases rolled out.

Accessed centrally in Shared File Areas and on Intranet.

“I post the dashboard on our Communication Board on our unit.”

“I love the opportunity to be transparent with my staff—it facilitates ownership of the clinical practice and an understanding of the global picture.”

“It inspires my staff to ask questions about what we could be doing differently.”

“I used the dashboard as part of the rollout of interventions.”

“Data are key for having conversations with my staff.”

25

Using Data to Tell Stories

Quantitative data never tell the full story

Project outcomes measured with qualitative themes from interviews and observations as well

Promote narrative culture

Not everything that can be counted counts,

and not everything that counts can be

counted.

Albert Einstein,

Physicist

Not everything that can be counted counts,

and not everything that counts can be

counted.

Albert Einstein,

Physicist

265

27

28

Future Considerations

Involve stakeholders in development process Review and revise list of metrics

Simplify Identify “need to know” vs. “nice to know” data

Connect dashboard with trend information Look beyond red/yellow/green Include graphical and visual representations of data Provide detailed notes and caveats Maintain data integrity Automate (to extent possible)

29

Critical Success Factors

Shared vision Leadership engagement Agreement on metrics and data definitions Accountability

30

Resources

Edward Tufte, www.edwardtufte.comEdward Tufte, www.edwardtufte.com

31

Questions?

“Discovery consists in

seeing what everyone else has seen

and thinking what no one else has thought.”Albert Szent-Gyorgyi, Hungarian Biochemist, 1937 Nobel Prize Winner