Day 2: Session III Considerations in Comparative and Public Reporting Presenters: Patrick Romano, UC...

Day 2: Session IIIDay 2: Session IIIConsiderations in Comparative and Public ReportingConsiderations in Comparative and Public Reporting

Presenters: Patrick Romano, UC DavisPresenters: Patrick Romano, UC DavisShoshanna Sofaer, Baruch CollegeShoshanna Sofaer, Baruch College

AHRQ QI User MeetingAHRQ QI User MeetingSeptember 26-27, 2005September 26-27, 2005

Selecting AHRQ Quality IndicatorsSelecting AHRQ Quality Indicatorsfor public reporting and pay-for-performancefor public reporting and pay-for-performance

Type or conceptual frameworkType or conceptual framework Face validity or salience to providersFace validity or salience to providers Impact or opportunity for improvementImpact or opportunity for improvement Reliability or precisionReliability or precision Coding (criterion) validityCoding (criterion) validity Construct validityConstruct validity Susceptibility to biasSusceptibility to bias

Types of provider-level quality indicatorsTypes of provider-level quality indicators

Structure: the conditions under which care is providedStructure: the conditions under which care is provided Volume (AAA repair, CEA, CABG, PCI, esophageal or Volume (AAA repair, CEA, CABG, PCI, esophageal or

pancreatic resection, pediatric heart surgery) pancreatic resection, pediatric heart surgery) Process: the activities that constitute health careProcess: the activities that constitute health care

Use of desirable/undesirable procedures (C/S, VBAC, Use of desirable/undesirable procedures (C/S, VBAC, bilateral cardiac cath, incidental appendectomy, bilateral cardiac cath, incidental appendectomy, laparoscopic cholecystectomy) laparoscopic cholecystectomy)

Outcome: changes attributable to health careOutcome: changes attributable to health care Risk-adjusted mortality (AMI, CHF, GI hemorrhage, Risk-adjusted mortality (AMI, CHF, GI hemorrhage,

hip fracture, pneumonia, stroke, AAA repair, CABG, hip fracture, pneumonia, stroke, AAA repair, CABG, craniotomy, esophageal resection, pancreatic craniotomy, esophageal resection, pancreatic resection, THA, pediatric heart surgery)resection, THA, pediatric heart surgery)

Risk-adjusted complications or “potential safety-Risk-adjusted complications or “potential safety-related events” (Patient Safety Indicators) related events” (Patient Safety Indicators)

Key features of structural measuresKey features of structural measures

Enabling factors that make it easier (harder) for Enabling factors that make it easier (harder) for professionals to provide high-quality care (i.e., facilitators professionals to provide high-quality care (i.e., facilitators or markers)or markers)

Weakly associated with process/outcome measuresWeakly associated with process/outcome measures Easy to measure, but hard to modifyEasy to measure, but hard to modify Few intervention studies, causal relationships unclear – Few intervention studies, causal relationships unclear –

do better structures lead to different processes, or do do better structures lead to different processes, or do better processes lead to different structures?better processes lead to different structures?

Use structural indicators when acceptable process or Use structural indicators when acceptable process or outcome measures are not available (“free ride” problem)outcome measures are not available (“free ride” problem)

Focus on modifiable structures OR settings in which Focus on modifiable structures OR settings in which hospitals that cannot modify structures are allowed to hospitals that cannot modify structures are allowed to close (excess capacity)close (excess capacity)

Minimum hospital volume needed to detect Minimum hospital volume needed to detect doubling of mortality rate (doubling of mortality rate (αα=0.05, =0.05, ββ=0.2)=0.2)

Ref: Dimick, et al. Ref: Dimick, et al. JAMA.JAMA. 2004;292:847-851. 2004;292:847-851.

Impact: Estimated lives saved by implementing Impact: Estimated lives saved by implementing hospital volume standards (NIS)hospital volume standards (NIS)

Birkmeyer et al., Surgery 2001;130:415-22Birkmeyer et al., Surgery 2001;130:415-22

Volume indicatorVolume indicator RR mortalityRR mortalityLVH vs HVHLVH vs HVH

Patients at Patients at LVHs in MSAsLVHs in MSAs

Potential lives Potential lives saved by volume saved by volume

standardsstandards

CABGCABG 1.381.38 164,261164,261 1,4861,486

Coronary Coronary angioplasty/PCIangioplasty/PCI 1.331.33 121,292121,292 345345

AAA repairAAA repair 1.601.60 18,53418,534 464464

Carotid Carotid endarterectomyendarterectomy 1.281.28 82,54482,544 118118

EsophagectomyEsophagectomy 3.013.01 1,6961,696 168168

Key features of process measuresKey features of process measures

Directly actionable by health care providers Directly actionable by health care providers (“opportunities for intervention”)(“opportunities for intervention”)

Highly responsive to changeHighly responsive to change Validated – or potentially “validatable” – in Validated – or potentially “validatable” – in

randomized trials (but NOT the AHRQ QIs)randomized trials (but NOT the AHRQ QIs) Illustrate the pathways by which interventions Illustrate the pathways by which interventions

may lead to better patient outcomesmay lead to better patient outcomes Focus on modifiable processes that are salient to Focus on modifiable processes that are salient to

providers, and for which there is clear opportunity providers, and for which there is clear opportunity for improvementfor improvement

Key features of outcome measuresKey features of outcome measures What really matters to patients, families, communitiesWhat really matters to patients, families, communities Intrinsically meaningful and easy to understandIntrinsically meaningful and easy to understand Reflect not just what was done but how well it was done Reflect not just what was done but how well it was done

(difficult to measure directly)(difficult to measure directly) Morbidity measures tend to be reported inconsistently (due Morbidity measures tend to be reported inconsistently (due

to poor MD documentation and/or coding)to poor MD documentation and/or coding) Outcome measures may be confounded by variation in Outcome measures may be confounded by variation in

observation units, discharge/transfer practices, LOS, observation units, discharge/transfer practices, LOS, severity of illnessseverity of illness

Many outcomes of interest are rare or delayedMany outcomes of interest are rare or delayed Are outcomes sufficiently under providers’ control?Are outcomes sufficiently under providers’ control? Focus on outcomes that are conceptually and empirically Focus on outcomes that are conceptually and empirically

attributable to providers (e.g., process linkages), and for attributable to providers (e.g., process linkages), and for which established benchmarks demonstrate opportunity which established benchmarks demonstrate opportunity for improvement.for improvement.

Literature review (all)Literature review (all)– To identify quality concepts and potential indicators To identify quality concepts and potential indicators – To find previous work on indicator validityTo find previous work on indicator validity

ICD-9-CM coding review (all)ICD-9-CM coding review (all)– To ensure correspondence between clinical concept and coding To ensure correspondence between clinical concept and coding

practicepractice Clinical panel reviews (PSI’s, pediatric QIs)Clinical panel reviews (PSI’s, pediatric QIs)

– To refine indicator definition and risk groupingsTo refine indicator definition and risk groupings– To establish face validity when minimal literature To establish face validity when minimal literature

Empirical analyses (all)Empirical analyses (all)– To explore alternative definitionsTo explore alternative definitions– To assess nationwide rates, hospital variation, relationships among To assess nationwide rates, hospital variation, relationships among

indicatorsindicators– To develop methods to account for differences in riskTo develop methods to account for differences in risk

AHRQ QI development: AHRQ QI development: General processGeneral process

AHRQ QI development: ReferencesAHRQ QI development: References

AHRQ Quality Indicator documentation web page atAHRQ Quality Indicator documentation web page at http://http://www.qualityindicators.ahrq.gov/downloads.htmwww.qualityindicators.ahrq.gov/downloads.htm – Refinement of the HCUP Quality Indicators (Technical Review)Refinement of the HCUP Quality Indicators (Technical Review), May , May

2001 2001 – Measures of Patient Safety Based on Hospital Administrative Data - Measures of Patient Safety Based on Hospital Administrative Data -

The Patient Safety IndicatorsThe Patient Safety Indicators, August 2002, August 2002 Peer-reviewed literature (examples):Peer-reviewed literature (examples):

– AHRQ’s Advances in Patient Safety: From Research to AHRQ’s Advances in Patient Safety: From Research to ImplementationImplementation (4-volume compendium) (4-volume compendium)

– Romano, et al. Health Aff (Millwood). 2003; 22(2):154-66. Romano, et al. Health Aff (Millwood). 2003; 22(2):154-66. – Zhan and Miller. JAMA. 2003; 290(14):1868-74. Zhan and Miller. JAMA. 2003; 290(14):1868-74. – Sedman, et al. Pediatrics. 2005; 115(1):135-45.Sedman, et al. Pediatrics. 2005; 115(1):135-45.– Rosen et al., Med Care. 2005; 43(9):873-84.Rosen et al., Med Care. 2005; 43(9):873-84.

http://www.qualityindicators.ahrq.gov/downloads.htm

http://www.qualityindicators.ahrq.gov/downloads.htm

Face validity: Clinical panel reviewFace validity: Clinical panel review

Intended to establish consensual validityIntended to establish consensual validity Modified RAND/UCLA Appropriateness MethodModified RAND/UCLA Appropriateness Method Physicians of various specialties/subspecialties, nurses, Physicians of various specialties/subspecialties, nurses,

other specialized professionals (e.g., midwife, pharmacist)other specialized professionals (e.g., midwife, pharmacist) Potential indicators were rated by 8 multispecialty panels; Potential indicators were rated by 8 multispecialty panels;

surgical indicators were also rated by 3 surgical panelssurgical indicators were also rated by 3 surgical panels All panelists rated all assigned indicators (1-9) on: All panelists rated all assigned indicators (1-9) on:

– Overall usefulnessOverall usefulness– Likelihood of identifying the occurrence of an adverse event Likelihood of identifying the occurrence of an adverse event

or complication (i.e., not present at admission)or complication (i.e., not present at admission)– Likelihood of being preventable (i.e., not an expected result Likelihood of being preventable (i.e., not an expected result

of underlying conditions) of underlying conditions) – Likelihood of being due to medical error or negligence (i.e., Likelihood of being due to medical error or negligence (i.e.,

not just lack of ideal or perfect care)not just lack of ideal or perfect care)– Likelihood of being clearly charted Likelihood of being clearly charted – Extent to which indicator is subject to case mix biasExtent to which indicator is subject to case mix bias

Medical error and complications continuum

Evaluation framework for PSIsEvaluation framework for PSIs

Pre-conference ratings and Pre-conference ratings and comments/suggestionscomments/suggestions

Individual ratings returned to panelists with Individual ratings returned to panelists with distribution of ratings and other panelists’ distribution of ratings and other panelists’ comments/suggestionscomments/suggestions

Telephone conference call moderated by PI, Telephone conference call moderated by PI, with note-taker, focusing on high-variability with note-taker, focusing on high-variability items and panelists’ suggestions (90-120 mins)items and panelists’ suggestions (90-120 mins)

Suggestions adopted only by consensusSuggestions adopted only by consensus Post-conference ratings and comments/ Post-conference ratings and comments/

suggestionssuggestions `̀

Medical error UnavoidableComplications

Example reviews of PSIsExample reviews of PSIsMultispecialty panelsMultispecialty panels

Overall ratingOverall rating

Not present on Not present on admissionadmission

PreventabilityPreventability

Due to medical Due to medical error error

Charting by Charting by physiciansphysicians

Not biased by Not biased by case mixcase mix

(5)

(7)

(4)

(2)

(6)

(3)

(8)

(8)

(8)

(8)

(7)

(7)

Postop Pneumonia Decubitus Ulcer

Final selection of PSIsFinal selection of PSIs

Retained indicators for which “overall usefulness” Retained indicators for which “overall usefulness” rating was “Acceptable” or “Acceptable-”rating was “Acceptable” or “Acceptable-” – Median score 7-9; ANDMedian score 7-9; AND– Definite agreement (“acceptable”) if no more than 1 or Definite agreement (“acceptable”) if no more than 1 or

2 panelists rated indicator below 72 panelists rated indicator below 7– Indeterminate agreement(“acceptable-”) if no more Indeterminate agreement(“acceptable-”) if no more

than 1 or 2 panelists rated indicator in 1-3 rangethan 1 or 2 panelists rated indicator in 1-3 range 48 indicators reviewed (15 by 2 separate panels)48 indicators reviewed (15 by 2 separate panels) 20 “accepted” based on face validity20 “accepted” based on face validity

– 2 dropped due to operational concerns2 dropped due to operational concerns 17 “experimental” or promising indicators17 “experimental” or promising indicators 11 rejected11 rejected

Panel ratings of PSI “preventability”Panel ratings of PSI “preventability”

Acceptable Acceptable (-) Unclear Unclear (-) Decubitus ulcer Complications of

anesthesia Death in low mortality DRG

Failure to rescue

Foreign body left in Selected infections due to medical care

Postop hemorhage/ hematoma

Postop physiologic/ metabolic derangement

Iatrogenic pneumothoraxa

Postop PE or DVTb Postop respiratory failure

Postop hip fracturea Transfusion reaction Postop abdominopelvic wound dehiscence

Technical difficulty with procedure

Birth trauma Postop sepsis

Obstetric trauma (all delivery types)

a Panel ratings were based on definitions different than final definitions. For “Iatrogenic pneumothorax,” the rated denominator was restricted to patients receiving thoracentesis or central lines; the final definition expands the denominator to all patients (with same exclusions). For “In-hospital fracture” panelists rated the broader Experimental indicator, which was replaced in the Accepted set by “Postoperative hip fracture” due to operational concerns. b Vascular complications were rated as Unclear (-) by surgical panel; multispecialty panel rating is shown here.

International expert panel ratings of PSIsInternational expert panel ratings of PSIsOrganization for Economic Cooperation and DevelopmentOrganization for Economic Cooperation and Development

PSIs recommended

PSIs not recommended

Experimental or rejected PSIs recommended

Selected infections due to medical care

Death in low mortality DRG Postop wound infection

Decubitus ulcer Postop hemorhage/ hematoma In-hospital hip fracture or fall Complications of anesthesia Iatrogenic pneumothorax Postop PE or DVT Postop abdominopelvic wound

dehiscence

Postop sepsis Failure to rescue Technical difficulty with procedure

Postop physiologic/ metabolic derangement

Transfusion reaction Postop respiratory failure Foreign body left in Postop hip fracture Birth trauma Obstetric trauma (all delivery types)

Impact: Estimated cases in 2000 (NIS)Impact: Estimated cases in 2000 (NIS)Romano et al., Health AffRomano et al., Health Aff 2003;22(2):154-66 2003;22(2):154-66

IndicatorIndicator FrequencyFrequency±±95% CI95% CI Rate/100Rate/100

Postoperative septicemiaPostoperative septicemia 14,055 ± 106014,055 ± 1060 1.0911.091

Postoperative thromboembolismPostoperative thromboembolism 75,811 ± 4,15675,811 ± 4,156 0.9190.919

Postoperative respiratory failurePostoperative respiratory failure 12,842 ± 93812,842 ± 938 0.3590.359

Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement

4,003 ± 4194,003 ± 419 0.0890.089

Decubitus ulcerDecubitus ulcer 201,459 ± 10,104201,459 ± 10,104 2.1302.130

Selected infections due to medical careSelected infections due to medical care 54,490 ± 2,65854,490 ± 2,658 0.1930.193

Postoperative hip fracturePostoperative hip fracture 5,207 ± 3275,207 ± 327 0.0800.080

Accidental puncture or lacerationAccidental puncture or laceration 89,348 ± 5,66989,348 ± 5,669 0.3240.324

Iatrogenic pneumothoraxIatrogenic pneumothorax 19,397 ± 1,02519,397 ± 1,025 0.0670.067

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 17,014 ± 96817,014 ± 968 0.2060.206

Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)

NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74

IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)

Postoperative septicemiaPostoperative septicemia 21.921.9 10.910.9 $57,700$57,700

Postoperative thromboembolismPostoperative thromboembolism 6.66.6 5.45.4 21,70021,700

Postoperative respiratory failurePostoperative respiratory failure 21.821.8 9.19.1 53,50053,500


19.819.8 8.98.9 54,80054,800

Decubitus ulcerDecubitus ulcer 7.27.2 4.04.0 10,80010,800

Selected infections due to medical careSelected infections due to medical care 4.34.3 9.69.6 38,70038,700

Postoperative hip fracturePostoperative hip fracture 4.54.5 5.25.2 13,40013,400

Accidental puncture or lacerationAccidental puncture or laceration 2.22.2 1.31.3 8,3008,300

Iatrogenic pneumothoraxIatrogenic pneumothorax 7.07.0 4.44.4 17,30017,300

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 3.03.0 3.93.9 21,40021,400

Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)

VA PTF analysis by Rosen et al., Med Care 2005;43:873-84VA PTF analysis by Rosen et al., Med Care 2005;43:873-84


Postoperative septicemiaPostoperative septicemia 35.735.7 1818 $39,531$39,531

Postoperative thromboembolismPostoperative thromboembolism 10.210.2 77 12,85612,856

Postoperative respiratory failurePostoperative respiratory failure 29.329.3 1919 39,84839,848


44.544.5 1515 37,46037,460

Decubitus ulcerDecubitus ulcer 10.910.9 55 5,8875,887

Selected infections due to medical careSelected infections due to medical care 9.89.8 1111 18,70618,706

Postoperative hip fracturePostoperative hip fracture 17.917.9 1010 18,90618,906

Accidental puncture or lacerationAccidental puncture or laceration 3.93.9 33 11,62611,626

Iatrogenic pneumothoraxIatrogenic pneumothorax 10.110.1 55 8,0398,039

Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 8.18.1 66 14,38414,384

Impact: Estimated cases in 2000 (NIS)Impact: Estimated cases in 2000 (NIS) Romano et al., Health AffRomano et al., Health Aff 2003;22(2):154-66 2003;22(2):154-66

IndicatorIndicator Frequency±95% CI Rate per 100

Birth traumaBirth trauma 27,035 ± 5,674 0.667

Obstetric trauma –cesareanObstetric trauma –cesarean 5,523 ± 597 0.593

Obstetric trauma - vaginal w/out Obstetric trauma - vaginal w/out instrumentationinstrumentation

249,243 ± 12,570 8.659

Obstetric trauma - vaginal w Obstetric trauma - vaginal w instrumentationinstrumentation

60,622 ± 3,104 24.408

Postoperative abdominopelvic wound Postoperative abdominopelvic wound dehiscencedehiscence

3,858 ± 289 0.193

Transfusion reactionTransfusion reaction 138 ± 49 0.0004

Complications of anesthesiaComplications of anesthesia 5,305 ± 455 0.056

Foreign body left during procedureForeign body left during procedure 2,710 ± 204 0.008

Impact of patient safety events in 2000Impact of patient safety events in 2000Zhan & Miller, JAMA 2003; replicated by Rosen et al., 2005Zhan & Miller, JAMA 2003; replicated by Rosen et al., 2005


Birth traumaBirth trauma -0.1 (NS)-0.1 (NS) -0.1 (NS)-0.1 (NS) 300 (NS)300 (NS)

Obstetric trauma –cesareanObstetric trauma –cesarean -0.0 (NS)-0.0 (NS) 0.40.4 2,7002,700

Obstetric trauma - vaginal w/out Obstetric trauma - vaginal w/out instrumentationinstrumentation

0.0 (NS)0.0 (NS) 0.050.05 -100 (NS)-100 (NS)

Obstetric trauma - vaginal w Obstetric trauma - vaginal w instrumentationinstrumentation

0.0 (NS)0.0 (NS) 0.070.07 220220

Postoperative abdominopelvic wound Postoperative abdominopelvic wound dehiscencedehiscence

9.69.6 9.49.4 40,30040,300

Transfusion reaction*Transfusion reaction* -1.0 (NS)-1.0 (NS) 3.4 (NS)3.4 (NS) 18,900 (NS)18,900 (NS)

Complications of anesthesia*Complications of anesthesia* 0.2 (NS)0.2 (NS) 0.2 (NS)0.2 (NS) 1,6001,600

Foreign body left during procedureForeign body left during procedure†† 2.12.1 2.12.1 13,30013,300

* All differences NS for transfusion reaction and complications of anesthesia in VA/PTF.

† Mortality difference NS for foreign body in VA/PTF.

National trends in PSI rates, 1994-National trends in PSI rates, 1994-20022002

Rare events (<0.1%)Rare events (<0.1%)

0.00%

0.01%

0.02%

0.03%

0.04%

0.05%

0.06%

0.07%

0.08%

0.09%

0.10%

1994 1997 2000 2001 2002

Foreign body left during procedure

Anesthesia reactions and complications

Death in low-mortality DRGs

Postop hip fracture

HCUPNet at http://www.hcup.ahrq.gov/, accessed 9/19/05.

http://www.hcup.ahrq.gov/


Low-frequency medical complications (0.05-0.5%)Low-frequency medical complications (0.05-0.5%)

0.05%

0.10%

0.15%

0.20%

0.25%

0.30%

0.35%

0.40%

0.45%

1994 1997 2000 2001 2002

Postop physiologic/metabolic derangements

Postop respiratory failure

Infection due to medical care



National trends in PSI rates, 1994-2002National trends in PSI rates, 1994-2002High-frequency medical complications (0.5-2.5%)High-frequency medical complications (0.5-2.5%)

0.5%

0.7%

0.9%

1.1%

1.3%

1.5%

1.7%

1.9%

2.1%

2.3%

1994 1997 2000 2001 2002

Decubitus ulcer

Postop septicemia

Postop thromboembolism




Surgical/technical complicationsSurgical/technical complications

0.00%

0.05%

0.10%

0.15%

0.20%

0.25%

0.30%

0.35%

0.40%

1994 1997 2000 2001 2002

Accidental puncture or laceration

Iatrogenic pneumothorax

Postop hemorrhage or hematoma

Postop abdominopelvic wound dehiscence



National trends in PSI rates, 1994-2002National trends in PSI rates, 1994-2002Obstetric complicationsObstetric complications

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

22%

1994 1997 2000 2001 2002

Obstetric trauma: cesarean delivery

Obstetric trauma: vaginal delivery w/out instrumentation

Obstetric trauma: vaginal delivery w instrumentation

Birth trauma



Reliability or precision: signal ratioReliability or precision: signal ratio

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0Foreign Body

Tranf. Reaction

Postop AP wound dehis

Postop hemorr/hemat

Postop physio/metab

Iatrogenic PTX

Postop hip fracture

Ob trauma –cesarean

Failure to rescue

Anesth complications

Postop resp failure

Postop sepsis

Postop DVT/PE

Death low mort DRGs

Ob trauma - vag forc/vac

Accid puncture/lac

Selected infection

Decubitus ulcer

Ob trauma - vag w/out

Birth trauma

Source: 2002 State Inpatient Data. Average Signal Ratio across all hospitals (N=4,428)

Year-to-year correlation of hospital effectsYear-to-year correlation of hospital effects

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0Foreign Body

Tranf. ReactionPostop AP wound dehis

Postop hemorr/hemat

Postop physio/metabIatrogenic PTX

Postop hip fracture

Ob trauma –cesareanFailure to rescue

Anesth complicationsPostop resp failure

Postop sepsis

Postop DVT/PEDeath low mort DRGs

Ob trauma - vag forc/vac

Accid puncture/lacSelected infection

Decubitus ulcer

Ob trauma - vag w/outBirth trauma

Source: 2001-2002 State Inpatient Data, hospitals with at least 1,000 discharges (N=4,428). Risk-adjusted unsmoothed rates.

Coding (criterion) validity based on Coding (criterion) validity based on literature review (MEDLINE/EMBASE)literature review (MEDLINE/EMBASE)

Validation studies of Iezzoni et al.’s CSPValidation studies of Iezzoni et al.’s CSP– At least one of three validation studies (coders, At least one of three validation studies (coders,

nurses, or physicians) confirmed PPV at least nurses, or physicians) confirmed PPV at least 75% among flagged cases75% among flagged cases

– Nurse-identified process-of-care failures were Nurse-identified process-of-care failures were more prevalent among flagged cases than more prevalent among flagged cases than among unflagged controlsamong unflagged controls

Other studies of coding validityOther studies of coding validity– Very few in peer-reviewed journals, some in Very few in peer-reviewed journals, some in

“gray literature”“gray literature”

Validation (%) of Complications Screening ProgramValidation (%) of Complications Screening ProgramMed Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18Med Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18

CSP IndicatorCSP Indicator PSIPSI Coder:Coder:ComplicComplicPresentPresent

RN: RN: process process problemproblem

MD:MD:ComplicComplicpresentpresent

MD:MD:QualityQuality

problemproblem

Postprocedural Postprocedural hemorrhage/ hemorrhage/ hematomahematoma

#9 narrower: #9 narrower: requires proc requires proc code + dxcode + dx

83 (surg)83 (surg)49 (med)49 (med)

66 vs 4666 vs 4613 vs 513 vs 5


37 vs 237 vs 231 vs 231 vs 2

Postop Postop pulmonary pulmonary compromisecompromise

#11 narrower:#11 narrower:includes only includes only resp failureresp failure

7272 52 vs 4652 vs 46 7575 20 vs 220 vs 2

DVT/PEDVT/PE #12 surgical only#12 surgical onlySlight changesSlight changes


72 vs 4672 vs 4669 vs 569 vs 5


50 vs 250 vs 220 vs 220 vs 2

In-hosp hip frx In-hosp hip frx and fallsand falls

#8 surgical only, #8 surgical only, no E codesno E codes


76 vs 4676 vs 4654 vs 554 vs 5


24 vs 224 vs 25 vs 25 vs 2

Criterion validity of PSIs linked to NSQIP, VA hospitalsCriterion validity of PSIs linked to NSQIP, VA hospitalsTsilimingras, Romano, et al., AcademyHealth 2005Tsilimingras, Romano, et al., AcademyHealth 2005

IndicatorIndicator

SensitivitySensitivity PPVPPV

CurrentCurrentInpatientInpatient

BetterBetterInpatientInpatient

CurrentCurrentInpatientInpatient

BetterBetterInpatientInpatient

Postop sepsisPostop sepsis 32%32% 37%37% 44%44% 45%45%

Postop Postop thromboembolismthromboembolism 56%56% 58%58% 22%22% 22%22%

Postop respiratory Postop respiratory failurefailure 19%19% 67%67% 74%74% 66%66%

Postop physiologic/ Postop physiologic/ metabolic derangementmetabolic derangement 44%44% 48%48% 54%54% 63%63%

Postop abdominopelvic Postop abdominopelvic wound dehiscencewound dehiscence 29%29% 61%61% 72%72% 57%57%

Construct validity based on literature Construct validity based on literature review (MEDLINE/EMBASE)review (MEDLINE/EMBASE)

Approaches to assessing construct validityApproaches to assessing construct validity– Is the outcome indicator associated with explicit Is the outcome indicator associated with explicit

processes of care (e.g., appropriate use of processes of care (e.g., appropriate use of medications)?medications)?

– Is the outcome indicator associated with implicit Is the outcome indicator associated with implicit process of care (e.g., global ratings of quality)?process of care (e.g., global ratings of quality)?

– Is the outcome indicator associated with nurse Is the outcome indicator associated with nurse staffing or skill mix, physician skill mix, or other staffing or skill mix, physician skill mix, or other aspects of hospital structure?aspects of hospital structure?

Summary of construct validity evidence in literatureSummary of construct validity evidence in literatureIndicatorIndicator Explicit processExplicit process Implicit processImplicit process StaffingStaffing

Complications of anesthesiaComplications of anesthesia

Death in low mortality DRGsDeath in low mortality DRGs ++

Decubitus ulcerDecubitus ulcer ±±

Failure to rescueFailure to rescue ++++

Foreign body left during procedureForeign body left during procedure

Iatrogenic pneumothoraxIatrogenic pneumothorax

Selected infections due to medical careSelected infections due to medical care

Postop hip fracturePostop hip fracture ++ ++

Postop hemorrhage or hematomaPostop hemorrhage or hematoma ±± ++

Postop physiologic/metabolic derangementsPostop physiologic/metabolic derangements ––--

Postop respiratory failurePostop respiratory failure ±± ++ ±±

Postop thromboembolismPostop thromboembolism ++ ++ ±±

Postop sepsisPostop sepsis ––--

Accidental puncture or lacerationAccidental puncture or laceration

Transfusion reactionTransfusion reaction

Postop abdominopelvic wound dehiscencePostop abdominopelvic wound dehiscence

Birth traumaBirth trauma

Obstetric trauma – vaginal birth w instrumentationObstetric trauma – vaginal birth w instrumentation

Obstetric trauma – vaginal w/out instrumentationObstetric trauma – vaginal w/out instrumentation

Obstetric trauma – cesarean birthObstetric trauma – cesarean birth

Construct validity: Do indicators track together?Construct validity: Do indicators track together?Factor loadings from 2001 VA/PTFFactor loadings from 2001 VA/PTF

-0.3-0.2-0.10.00.10.20.30.40.50.60.70.80.91.0

Factor 1 Factor 2 Factor 3

PSI risk adjustment methodsPSI risk adjustment methods

Must use only administrative dataMust use only administrative data APR-DRGs and other canned packages APR-DRGs and other canned packages

may adjust for complicationsmay adjust for complications Final model Final model

– DRGs (complication DRGs aggregated) DRGs (complication DRGs aggregated) – Modified Comorbidity Index based on list Modified Comorbidity Index based on list

developed by Elixhauser et al.developed by Elixhauser et al.– Age, Sex, Age-Sex interactions Age, Sex, Age-Sex interactions

Susceptibility to bias at the hospital level:Susceptibility to bias at the hospital level:Impact of risk-adjustment, 1997 SID (summary)Impact of risk-adjustment, 1997 SID (summary)

High BiasHigh Bias Medium BiasMedium Bias Low BiasLow Bias

Failure to rescue Failure to rescue (44% change 2 deciles)(44% change 2 deciles)

Postoperative respiratoryPostoperative respiratoryfailure (11%)failure (11%)

Postop abdominopelvicPostop abdominopelvicwound dehiscence (4%)wound dehiscence (4%)

Accidental puncture orAccidental puncture orlaceration (24%)laceration (24%)

Postoperative hip fracturePostoperative hip fracture(8%)(8%)

Obstetric trauma –Obstetric trauma –cesarean birth (2%)cesarean birth (2%)

Decubitus ulcer (26%)Decubitus ulcer (26%)Iatrogenic pneumothoraxIatrogenic pneumothorax(14%)(14%)

Postop hemorrhagePostop hemorrhageor hematoma (4%)or hematoma (4%)

Postop thromboembolismPostop thromboembolism(14%)(14%)

Postop physio/metabolicPostop physio/metabolicderangement (5%)derangement (5%)

Complications ofComplications ofanesthesia (<1%)anesthesia (<1%)

Death in low mortalityDeath in low mortalityDRGs (13%)DRGs (13%)

Obstetric trauma – Obstetric trauma – vaginal w instrument (5%)vaginal w instrument (5%)

Obstetric trauma –Obstetric trauma –vaginal w/out instrumentvaginal w/out instrument

Postoperative sepsisPostoperative sepsis(11%)(11%)

Selected infections due toSelected infections due tomedical care (10%)medical care (10%) Birth trauma (0%)Birth trauma (0%)

Measurement for quality-based purchasing Measurement for quality-based purchasing and public reporting: Conclusionsand public reporting: Conclusions

Quality-based purchasing and public reporting Quality-based purchasing and public reporting may stimulate improvement in quality of care, or may stimulate improvement in quality of care, or at least more attention to quality indicatorsat least more attention to quality indicators

Measures/indicators must be selected based on Measures/indicators must be selected based on local priorities and limitations of available data – local priorities and limitations of available data – AHRQ QIs appropriate for public reporting may AHRQ QIs appropriate for public reporting may differ across states and regionsdiffer across states and regions

Results must be presented and disseminated in a Results must be presented and disseminated in a manner that earns the confidence of providers, manner that earns the confidence of providers, purchasers/consumers, and other stakeholderspurchasers/consumers, and other stakeholders

Reference: Remus D, Fraser I. Reference: Remus D, Fraser I. Guidance for Using the AHRQ Quality Guidance for Using the AHRQ Quality Indicators for Hospital-level Public Reporting or PaymentIndicators for Hospital-level Public Reporting or Payment. . AHRQ Publication No. 04-0086-EF.AHRQ Publication No. 04-0086-EF.

AcknowledgmentsAcknowledgments

Funded by AHRQFunded by AHRQ Support for Quality Indicators II (Contract No. 290-04-0020) Mamatha Pancholi, AHRQ Project Officer Marybeth Farquhar, AHRQ QI Senior Advisor Mark Gritz and Jeffrey Geppert, Project Directors, Battelle Health

and Life Sciences

Data used for analyses:Data used for analyses:Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and

Utilization Project (HCUP), Agency for Healthcare Research and Utilization Project (HCUP), Agency for Healthcare Research and QualityQuality

State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and QualityResearch and Quality

AcknowledgmentsAcknowledgments

We gratefully acknowledge the data organizations in participating We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California study: the Arizona Department of Health Services; California Office of Statewide Health and Development; Colorado Health and Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Containment Council; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Association; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Department Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Health Systems; Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Washington State Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Department of Health and Department of Health; and Wisconsin Department of Health and Family Service.Family Service.

Questions?Questions?

Day 2: Session III Considerations in Comparative and Public Reporting Presenters: Patrick Romano, UC...

Documents

Transcript of Day 2: Session III Considerations in Comparative and Public Reporting Presenters: Patrick Romano, UC...