AHRQ Quality Indicators Developed by Stanford-UCSF Evidence Based Practice Center Funded by the...

AHRQ Quality IndicatorsAHRQ Quality IndicatorsDeveloped by Stanford-UCSF Evidence Based Practice Developed by Stanford-UCSF Evidence Based Practice

CenterCenter

Funded by the Agency for Healthcare Research and QualityFunded by the Agency for Healthcare Research and Quality

EPC Team (PSI Development)

PI: Kathryn McDonald, M.M., StanfordPI: Kathryn McDonald, M.M., StanfordPatrick Romano, M.D., M.P.H, UC DavisPatrick Romano, M.D., M.P.H, UC DavisJeffrey Geppert, J.D., Ed.M., StanfordJeffrey Geppert, J.D., Ed.M., StanfordSheryl Davies, M.A., Stanford Sheryl Davies, M.A., Stanford Bradford Duncan, M.D., M.A., Stanford Bradford Duncan, M.D., M.A., Stanford Kaveh G. Shojania, M.D., UCSFKaveh G. Shojania, M.D., UCSF

Support of Quality Indicators

PI: Kathryn McDonald, M.M., StanfordPI: Kathryn McDonald, M.M., StanfordSheryl Davies, M.A., StanfordSheryl Davies, M.A., StanfordPatrick Romano, M.D., M.P.H, UC DavisPatrick Romano, M.D., M.P.H, UC DavisJeffrey Geppert, J.D. Ed.M., StanfordJeffrey Geppert, J.D. Ed.M., StanfordMark Gritz, PhD, Battelle Mark Gritz, PhD, Battelle Greg Hubert, BattelleGreg Hubert, BattelleDenise Remus, RN PhD, AHRQ Project Denise Remus, RN PhD, AHRQ Project OfficerOfficer

Acknowledgements Acknowledgements

Funded by AHRQFunded by AHRQ Contract No. 290-97-0013 Contract No. 290-97-0013 Support of Quality Indicators Contract No. 290-02-0007Support of Quality Indicators Contract No. 290-02-0007Presentation funded by AHRQ Presentation funded by AHRQ

Data used for analyses:Data used for analyses:Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost

and Utilization Project (HCUP), Agency for Healthcare Research and Utilization Project (HCUP), Agency for Healthcare Research and Qualityand Quality

State Inpatient Databases (SID), 1997 (19 states). Healthcare State Inpatient Databases (SID), 1997 (19 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and QualityResearch and Quality

AcknowledgementsAcknowledgements We gratefully acknowledge the data organizations in We gratefully acknowledge the data organizations in

participating states that contributed data to HCUP and that we participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; used in this study: the Arizona Department of Health Services; California Office of Statewide Health and Development; California Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Department of Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Tennessee Hospital Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Washington State Department of Health; and Wisconsin Department of Health and Family Service.Department of Health and Family Service.

OutlineOutline

1.1. Administrative data and quality Administrative data and quality indicatorsindicators

2.2. AHRQ Quality Indicators (QI)AHRQ Quality Indicators (QI)

3.3. Development of AHRQ QIsDevelopment of AHRQ QIs

4.4. Risk adjustment & MSX smoothing Risk adjustment & MSX smoothing methodsmethods

5.5. Application of QIs to research and Application of QIs to research and qualityquality

History of AHRQ QIs/PSIsHistory of AHRQ QIs/PSIs

Healthcare Cost and Utilization Project Healthcare Cost and Utilization Project (HCUP)(HCUP)

HCUP discharge data collection (1988)HCUP discharge data collection (1988) HCUP Quality IndicatorsHCUP Quality Indicators

– Mortality for Inpatient ProceduresMortality for Inpatient Procedures– Complication RatesComplication Rates– Potentially Inappropriate UtilizationPotentially Inappropriate Utilization– Potentially Avoidable Hospital AdmissionsPotentially Avoidable Hospital Admissions

Refinement of HCUP QIsRefinement of HCUP QIs Refinement commissioned by AHRQ in 1999Refinement commissioned by AHRQ in 1999 Completed by UCSF-Stanford EPCCompleted by UCSF-Stanford EPC Two related projectsTwo related projects Two technical reviewsTwo technical reviews

– Refinement of the HCUP Quality IndicatorsRefinement of the HCUP Quality Indicators– Measures of Patient Safety Based on Administrative DataMeasures of Patient Safety Based on Administrative Data

Three indicator sets, AHRQ QIsThree indicator sets, AHRQ QIs– Inpatient Quality Indicators (IQIs)Inpatient Quality Indicators (IQIs)– Prevention Quality Indicators (PQIs)Prevention Quality Indicators (PQIs)– Patient Safety Indicators (PSIs)Patient Safety Indicators (PSIs)

Administrative Data & Administrative Data & Quality ImprovementQuality Improvement

OpportunitiesOpportunities– Coding practices Coding practices

improvingimproving– Data availability Data availability

improving (e.g., less improving (e.g., less truncation) truncation)

– More specific codesMore specific codes

– Large data sets improve Large data sets improve precisionprecision

– Comprehensive: all Comprehensive: all hospitalshospitals

– Quality screening Quality screening feasiblefeasible

ObstaclesObstacles– Coding errors introduce Coding errors introduce

noisenoise– Lack of information on Lack of information on

timing, comorbidity vs. timing, comorbidity vs. adverse events adverse events

– Varying number of Varying number of secondary diagnoses secondary diagnoses fields can cause bias fields can cause bias

– Heterogeneous severity Heterogeneous severity within single codewithin single code

Administrative DataAdministrative Data

State Inpatient DatabasesState Inpatient Databases Includes ICD-9-CM dx and procedure codes, Includes ICD-9-CM dx and procedure codes,

DRG, dates, age, sex, payer, race, discharge DRG, dates, age, sex, payer, race, discharge disposition, hospital and/or patient zip codesdisposition, hospital and/or patient zip codes

1995-20021995-2002 33 States 33 States 80% + of all U.S. hospital discharges 80% + of all U.S. hospital discharges 18 states available for purchase18 states available for purchase In 27 state sample, approximately 3200 In 27 state sample, approximately 3200

hospitalshospitals

Administrative DataAdministrative Data

Nationwide Inpatient Sample (NIS)Nationwide Inpatient Sample (NIS) Sampling of State Inpatient DatabasesSampling of State Inpatient Databases 1988-20011988-2001 7.5 million discharges/1,000 hospitals/33 7.5 million discharges/1,000 hospitals/33

StatesStates Approximates 20% sample of nonfederal Approximates 20% sample of nonfederal

acute care hospitalsacute care hospitals Discharge level weights applied for Discharge level weights applied for

national estimatesnational estimates Available for purchaseAvailable for purchase

HCUPnetHCUPnet

http:///http:///hcupnet.ahrq.govhcupnet.ahrq.gov// Web-based tool to query NIS and KIDS Web-based tool to query NIS and KIDS

databases, 1993-2001databases, 1993-2001 Pre-run tables for 1997-2001Pre-run tables for 1997-2001 Query based on ICD-9-CM, DRG or CCSQuery based on ICD-9-CM, DRG or CCS Information on hospitalizations, charges, Information on hospitalizations, charges,

length of stay, mortality, discharge statuslength of stay, mortality, discharge status Stratification by age, sex, race, income, Stratification by age, sex, race, income,

insurance, hospital characteristicsinsurance, hospital characteristics Rank order hospitalizationsRank order hospitalizations

OutlineOutline






Sample AHRQ QI definitionSample AHRQ QI definitionDECUBITUS ULCERRelationship to quality: Relationship to quality: Identifies cases of decubitus ulcers that develop during Identifies cases of decubitus ulcers that develop during hospitalization. hospitalization.

Indicator definition: Indicator definition: Number of patients with decubitus ulcer (see definition and exclusions Number of patients with decubitus ulcer (see definition and exclusions below) per 100 eligible admissions (population at risk).below) per 100 eligible admissions (population at risk).

Definition of decubitus ulcer: Definition of population at risk:Patients eligible to be included in this indicator:

Secondary diagnosis code of decubitus Secondary diagnosis code of decubitus ulcer:ulcer:decubitus ulcer [707.0]decubitus ulcer [707.0]

a. All patients (medical and surgical)a. All patients (medical and surgical)c. Patient must have a length of stay of c. Patient must have a length of stay of

more than 4 days.more than 4 days.d. Patient must not be in MDC 9 (Diseases d. Patient must not be in MDC 9 (Diseases

and Disorders of the Skin Subcutaneous and Disorders of the Skin Subcutaneous Tissue and Breast), or have any diagnosis of Tissue and Breast), or have any diagnosis of hemiplegia, paraplagia, or quadriplagia. hemiplegia, paraplagia, or quadriplagia.

Prevention Quality Prevention Quality Indicators (PQIs)Indicators (PQIs)

Defined using area population as Defined using area population as denominatordenominator

Potentially avoidable hospitalizations or Potentially avoidable hospitalizations or ambulatory care sensitive conditionsambulatory care sensitive conditions

Conditions for which good outpatient care Conditions for which good outpatient care can potentially prevent the need for can potentially prevent the need for hospitalization or for which early intervention hospitalization or for which early intervention can prevent complications or more severe can prevent complications or more severe diseasedisease

Public health, comprehensive health care Public health, comprehensive health care systemssystems

Based on existing, validated indicators set, Based on existing, validated indicators set, but modified and updatedbut modified and updated

Prevention Quality Prevention Quality Indicators (PQIs)Indicators (PQIs)

Bacterial pneumoniaBacterial pneumonia DehydrationDehydration Pediatric gastroenteritisPediatric gastroenteritis Urinary tract infectionUrinary tract infection Perforated appendix Perforated appendix Low birth weightLow birth weight Angina without procedureAngina without procedure Congestive heart failureCongestive heart failure HypertensionHypertension Adult asthma Adult asthma Pediatric asthmaPediatric asthma Chronic obstructive pulmonary diseaseChronic obstructive pulmonary disease Diabetes short-term complicationDiabetes short-term complication Diabetes long-term complication Diabetes long-term complication Uncontrolled diabetesUncontrolled diabetes Lower-extremity amputation among patients with diabetesLower-extremity amputation among patients with diabetes

Inpatient Quality Indicators Inpatient Quality Indicators (IQIs)(IQIs)

Defined using both hospital admissions and area Defined using both hospital admissions and area population as denominatorpopulation as denominator

Inpatient mortality for certain procedures and Inpatient mortality for certain procedures and medical conditionsmedical conditions

Utilization of procedures for which there are Utilization of procedures for which there are questions of overuse, underuse, and misusequestions of overuse, underuse, and misuse

Volume of certain procedures Volume of certain procedures Risk-adjusted using APR-DRGsRisk-adjusted using APR-DRGs Potential for internal quality improvement Potential for internal quality improvement

purposespurposes Based on existing, validated indicators Based on existing, validated indicators

Inpatient Quality Indicators Inpatient Quality Indicators (IQIs)(IQIs)

Mortality Rates for ConditionsMortality Rates for Conditions Acute myocardial infarction (2 Acute myocardial infarction (2

versions)versions) Congestive heart failureCongestive heart failure Gastrointestinal hemorrhageGastrointestinal hemorrhage Hip fractureHip fracture Pneumonia Pneumonia StrokeStroke

Mortality Rates for ProceduresMortality Rates for Procedures Abdominal aortic aneurysm repairAbdominal aortic aneurysm repair Coronary artery bypass graftCoronary artery bypass graft CraniotomyCraniotomy Esophageal resectionEsophageal resection Hip replacementHip replacement Pancreatic resectionPancreatic resection Pediatric heart surgeryPediatric heart surgery

Hospital-level Procedure Utilization RatesHospital-level Procedure Utilization Rates Cesarean section delivery (primary and total)Cesarean section delivery (primary and total) Incidental appendectomy in the elderlyIncidental appendectomy in the elderly Bi-lateral cardiac catheterizationBi-lateral cardiac catheterization Vaginal birth after Cesarean section (2 versions)Vaginal birth after Cesarean section (2 versions) Laparoscopic cholecystectomyLaparoscopic cholecystectomy

Area-level Utilization RatesArea-level Utilization Rates Coronary artery bypass graftCoronary artery bypass graft HysterectomyHysterectomy Laminectomy or spinal fusionLaminectomy or spinal fusion PTCAPTCA

Volume of ProceduresVolume of Procedures Abdominal aortic aneurysm repairAbdominal aortic aneurysm repair Carotid endarterectomyCarotid endarterectomy Coronary artery bypass graftCoronary artery bypass graft Esophageal resectionEsophageal resection Pancreatic resectionPancreatic resection Pediatric heart surgeryPediatric heart surgery PTCAPTCA

Patient Safety Indicators Patient Safety Indicators (PSIs)(PSIs)

Defined using hospital admissions as Defined using hospital admissions as denominatordenominator

Inpatient complications of care and Inpatient complications of care and potential patient safety eventspotential patient safety events

Potential for internal quality improvement Potential for internal quality improvement purposes, monitoring of patient safety purposes, monitoring of patient safety events events

Novel indicators, based on concepts Novel indicators, based on concepts reported in the literature reported in the literature

Patient Safety Indicators Patient Safety Indicators (PSIs)(PSIs)

Provider-level Patient Safety IndicatorsProvider-level Patient Safety Indicators Accidental puncture or laceration during Accidental puncture or laceration during

procedureprocedure Complications of anesthesiaComplications of anesthesia Death in low mortality DRGsDeath in low mortality DRGs Decubitus ulcerDecubitus ulcer Failure to rescueFailure to rescue Foreign body left in during procedureForeign body left in during procedure Iatrogenic pneumothoraxIatrogenic pneumothorax Selected infection due to medical careSelected infection due to medical care Postoperative hemorrhage or hematomaPostoperative hemorrhage or hematoma Postoperative hip fracturePostoperative hip fracture Postoperative physiologic and metabolic Postoperative physiologic and metabolic

derangementsderangements Obstetric trauma – vaginal delivery with Obstetric trauma – vaginal delivery with

instrumentinstrument Obstetric trauma – vaginal delivery Obstetric trauma – vaginal delivery

without instrumentwithout instrument Obstetric trauma – cesarean section Obstetric trauma – cesarean section

deliverydelivery

Postoperative pulmonary Postoperative pulmonary embolism or deep vein thrombosis embolism or deep vein thrombosis

Postoperative respiratory failurePostoperative respiratory failure Postoperative sepsisPostoperative sepsis Transfusion reactionTransfusion reaction Postoperative wound dehiscence Postoperative wound dehiscence

in abdominopelvic surgical in abdominopelvic surgical patientspatients

Birth trauma – injury to neonate Birth trauma – injury to neonate

Area-level Patient Safety Area-level Patient Safety IndicatorsIndicators

Foreign body left in during Foreign body left in during procedureprocedure

Iatrogenic pneumothoraxIatrogenic pneumothorax Infection due to medical careInfection due to medical care Technical difficulty with medical Technical difficulty with medical

carecare Transfusion reactionTransfusion reaction Postoperative wound dehiscence Postoperative wound dehiscence

in abdominopelvic surgical in abdominopelvic surgical patientspatients

PQI RatesPQI Rates

Source: SID, 2000. AHRQ Prevention Quality Indicators SAS Software Version 2.1 Revision 3.

PQI Per 100,000 Area SD

DIABETES SHORT TRM COMPLICATION 43.2 33.3

PERFORATED APPENDIX 311.6 157

DIABETES LONG TERM COMPLICATION 100.4 83.1

PEDIATRIC ASTHMA 166.4 191.2

CHRONIC OBSTRUCTIVE PULMONARY DISEASE 371.6 342.0

PEDIATRIC GASTROENTERITIS 128.8 189.6

HYPERTENSION 50.2 67.0

CONGESTIVE HEART FAILURE 526.6 488.3

LOW BIRTH WEIGHT 38.0 39.0

DEHYDRATION 158.9 143.6

BACTERIAL PNEUMONIA 506.8 360.0

URINARY INFECTION 169.0 142.1

ANGINA 108.8 128.2

DIABETES UNCONTROLLED 33.7 39.3

ADULT ASTHMA 102.3 95.6

LOWER EXTREMITY AMPUTATION 27.9 32.1

IQI RatesIQI RatesIQIIQI Per 100Per 100 Provider SDProvider SD

IN-HOSP MORT ESOPHAGEAL RESECTION IN-HOSP MORT ESOPHAGEAL RESECTION 11.5111.51 28.8828.88

IN-HOSP MORT PANCREATIC RESECTION IN-HOSP MORT PANCREATIC RESECTION 10.5310.53 25.1125.11

IN-HOSP MORT PEDIATRIC HEART SURG IN-HOSP MORT PEDIATRIC HEART SURG 4.904.90 11.8711.87

IN-HOSP MORT AAA REPAIR IN-HOSP MORT AAA REPAIR 16.8716.87 22.9722.97

IN-HOSP MORT CABG IN-HOSP MORT CABG 3.913.91 4.354.35

IN-HOSP MORT CRANIOTOMY IN-HOSP MORT CRANIOTOMY 9.749.74 12.3512.35

IN-HOSP MORT HIP REPLACEMENT IN-HOSP MORT HIP REPLACEMENT 0.380.38 2.322.32

IN-HOSP MORT AMI IN-HOSP MORT AMI 15.4115.41 13.1613.16

IN-HOSP MORT CHF IN-HOSP MORT CHF 5.035.03 4.394.39

IN-HOSP MORT STROKE IN-HOSP MORT STROKE 11.1911.19 8.358.35

IN-HOSP MORT GI HEMORRHAGE IN-HOSP MORT GI HEMORRHAGE 3.463.46 5.275.27

IN-HOSP MORT HIP FRACTURE IN-HOSP MORT HIP FRACTURE 3.453.45 6.526.52

IN-HOSP MORT PNEUMONIA IN-HOSP MORT PNEUMONIA 8.118.11 4.864.86

CESAREAN SECTION DELIVERY CESAREAN SECTION DELIVERY 20.3320.33 8.598.59

PRIMARY CESAREAN SECTIONPRIMARY CESAREAN SECTION 13.1213.12 7.457.45

VAGINAL BIRTH AFTER C-SECTION VAGINAL BIRTH AFTER C-SECTION 26.5426.54 15.2815.28

LAPAROSCOPIC CHOLECYSTECTOMY LAPAROSCOPIC CHOLECYSTECTOMY 73.2573.25 18.6518.65

INCIDENTAL APPENDECTOMY INCIDENTAL APPENDECTOMY 2.832.83 5.085.08

BI-LATERAL CATHETERIZATION BI-LATERAL CATHETERIZATION 11.1911.19 13.9613.96

Source: SID, 2000. AHRQ Inpatient Quality Indicators SAS Software Version 2.1, Revision 3. Release pending.

PSI RatesPSI Rates

Source: NIS, 2000. AHRQ Patient Safety Indicators SAS Software Version 2.1 Revision 2. Release pending.

PSI Rate per 1000

Complications Of Anesthesia 0.71

Death In Low Mortality Drgs 0.60

Decubitus Ulcer 21.96

Failure To Rescue 143.82

Foreign Body Left In During Proc 0.08

Iatrogenic Pneumothorax 0.81

Infection Due To Medical Care 2.00

Postoperative Hip Fracture 0.31

Postop Hemorrhage Or Hematoma 2.24

Postop Physio Metabol Derangmnt 0.87

Postop Respiratory Failure 3.48

Postoperative Pe Or Dvt 7.29

Postoperative Sepsis 10.75

Postoperative Wound Dehiscence 1.89

Accidental Puncture/Laceration 3.25

Transfusion Reaction 0.005

Birth Trauma 6.58

Ob Trauma - Vaginal W Instrument 222.63

Ob Trauma - Vaginal Wo Instrument 83.69

Ob Trauma - C-Section 5.95

OutlineOutline






MethodsMethods Evaluation frameworkEvaluation framework Literature reviewLiterature review

– Identification of indicatorsIdentification of indicators Gray literature/interviews Gray literature/interviews

– Identification of indicatorsIdentification of indicators Literature review Literature review

– Evidence for indicatorsEvidence for indicators Empirical analysesEmpirical analyses ICD-9-CM coding review (PSI only)ICD-9-CM coding review (PSI only) Clinical panel reviews (PSI only)Clinical panel reviews (PSI only)

Evaluation FrameworkEvaluation Framework Face validityFace validity: does the indicator capture an aspect of quality that is : does the indicator capture an aspect of quality that is

widely regarded as important and subject to provider or public health widely regarded as important and subject to provider or public health system control?system control?

Precision:Precision: is there a substantial amount of provider or community level is there a substantial amount of provider or community level variation that is not attributable to random variation? variation that is not attributable to random variation?

Minimum Bias:Minimum Bias: is there either little effect on the indicator of variations is there either little effect on the indicator of variations in patient disease severity and comorbidities, or is it possible to apply in patient disease severity and comorbidities, or is it possible to apply risk adjustment and statistical methods to remove most or all bias? risk adjustment and statistical methods to remove most or all bias?

Construct validity:Construct validity: does the indicator perform well in identifying true does the indicator perform well in identifying true (or actual) quality of care problems?(or actual) quality of care problems?

Fosters real quality improvement:Fosters real quality improvement: Is the indicator insulated from Is the indicator insulated from perverse incentives for providers to improve their reported performance perverse incentives for providers to improve their reported performance by avoiding difficult or complex cases, or by other responses that do not by avoiding difficult or complex cases, or by other responses that do not improve quality of care?improve quality of care?

Application:Application: Has the measure been used effectively in practice? Does Has the measure been used effectively in practice? Does it have potential for working well with other indicators? it have potential for working well with other indicators?

Literature ReviewLiterature ReviewIdentification of IndicatorsIdentification of Indicators

Systematic review to identify indicators Systematic review to identify indicators Thousands of articles screenedThousands of articles screened Over 200 abstractedOver 200 abstracted Only 20 + articles actually described Only 20 + articles actually described

indicators, most of which had indicators, most of which had overlapping indicatorsoverlapping indicators

Grey literature searched to identify Grey literature searched to identify over 200 indicatorsover 200 indicators

Empirical AnalysesEmpirical Analyses

Used novel statistical methods to Used novel statistical methods to measuremeasure– Precision/ReliabilityPrecision/Reliability– BiasBias– Inter-relatedness of indicatorsInter-relatedness of indicators

Precision criteria of 1.0% or more Precision criteria of 1.0% or more systematic variation among providerssystematic variation among providers– Then, literature review conductedThen, literature review conducted

Literature ReviewLiterature ReviewEvidence for Each IndicatorEvidence for Each Indicator

Identified and reported evidence for:Identified and reported evidence for:– Face validityFace validity– Precision and reliabilityPrecision and reliability– Potential biasPotential bias– Construct validityConstruct validity– Fosters true quality improvement Fosters true quality improvement

(gaming)(gaming)– Current useCurrent use

PSIs Methods PSIs Methods Development of Candidate Indicator Development of Candidate Indicator

ListList Background literature reviewBackground literature review

– Little evidence in peer reviewed journalsLittle evidence in peer reviewed journals Complications Screening ProgramComplications Screening Program Miller et al. Patient Safety IndicatorsMiller et al. Patient Safety Indicators Review of ICD-9-CM code bookReview of ICD-9-CM code book Codes from above sources grouped into Codes from above sources grouped into

indicators and assigned denominatorsindicators and assigned denominators Review of CSP evidence to retain Review of CSP evidence to retain

indicatorsindicators Final refinements of indicators Final refinements of indicators

PSIs Methods PSIs Methods Review of Candidate IndicatorsReview of Candidate Indicators

Literature review of potential Literature review of potential indicatorsindicators– Coding validity/consistency Coding validity/consistency – Construct validityConstruct validity

ICD-9-CM coding reviewICD-9-CM coding review Clinical panel review (face validity)Clinical panel review (face validity)

– Results used to define final set of Results used to define final set of indicatorsindicators

PSIs Methods PSIs Methods Clinical Panel ReviewClinical Panel Review

Intended to establish consensual validityIntended to establish consensual validity Modified RAND/UCLA Appropriateness MethodModified RAND/UCLA Appropriateness Method Doctors of various specialties/subspecialties, Doctors of various specialties/subspecialties,

nurses, specialized (e.g., midwife, pharmacist)nurses, specialized (e.g., midwife, pharmacist) Initial rating, followed by conference call, followed Initial rating, followed by conference call, followed

by final ratingby final rating Rated indicator on:Rated indicator on:

– Overall usefulnessOverall usefulness– Present on admission Present on admission – Preventability of complication Preventability of complication – Likelihood due to medical errorLikelihood due to medical error– Extent indicator subject to biasExtent indicator subject to bias

Eight multispecialty panels, three surgical panels Eight multispecialty panels, three surgical panels (5-9 members on each panel) (5-9 members on each panel)

Example reviewsExample reviewsMultispecialty PanelsMultispecialty Panels

Overall ratingOverall rating

Not present on Not present on admissionadmission

Preventability (4)Preventability (4)

Due to medical Due to medical error (2)error (2)

Charting by Charting by physicians (6)physicians (6)

Not biased (3)Not biased (3)

(5)

(7)

(4)

(2)

(6)

(3)

(8)

(8)

(8)

(8)

(7)

(7)

Postop Pneumonia Decubitus Ulcer

PSIs Methods PSIs Methods Final Selection of IndicatorsFinal Selection of Indicators

Indicators for which “overall Indicators for which “overall usefulness” rating was highusefulness” rating was high

Some changes in indicator set based Some changes in indicator set based on coding review and on coding review and operationalization concerns (e.g., operationalization concerns (e.g., reopening of surgical site)reopening of surgical site)

Empirical analyses of nationwide rates, Empirical analyses of nationwide rates, variation, impact of risk adjustment, variation, impact of risk adjustment, and relationship between indicatorsand relationship between indicators

OutlineOutline






Risk-Adjustment CriteriaRisk-Adjustment Criteria

User-specified criteria for evaluating User-specified criteria for evaluating risk-adjustment systemsrisk-adjustment systems

1) “Open” systems preferred1) “Open” systems preferred

2) Data collection costs minimized 2) Data collection costs minimized and well-justifiedand well-justified

3) Multiple-use coding system3) Multiple-use coding system

4) Official recognition4) Official recognition

Evidence on DRG-based Evidence on DRG-based SystemsSystems

Open systemsOpen systems Widely adopted by state agenciesWidely adopted by state agencies Based on existing data collection systemsBased on existing data collection systems Use for reimbursement ensures improved Use for reimbursement ensures improved

data qualitydata quality Evidence suggests at least equivalent Evidence suggests at least equivalent

performance across broad spectrum of performance across broad spectrum of conditionsconditions

Studies underway to examine alternativesStudies underway to examine alternatives

3M APR-DRG3M APR-DRG

All-patient refined (956 categories in All-patient refined (956 categories in version 15.0, including pediatrics)version 15.0, including pediatrics)

Severity of illness subclass that Severity of illness subclass that reflect presence of co reflect presence of co morbidity/complication and levelmorbidity/complication and level

Risk of mortality subclassRisk of mortality subclass Differential impact of secondary Differential impact of secondary

diagnosis by conditiondiagnosis by condition

Evidence on 3M APR-DRGEvidence on 3M APR-DRG

All-patient refined (956 categories in All-patient refined (956 categories in version 15.0, including pediatrics)version 15.0, including pediatrics)

Severity of illness subclasses that Severity of illness subclasses that reflect presence of co reflect presence of co morbidity/complication and levelmorbidity/complication and level

Risk of mortality subclassesRisk of mortality subclasses Differential impact of secondary Differential impact of secondary

diagnosis by conditiondiagnosis by condition

Evidence on 3M APR-DRGEvidence on 3M APR-DRG

Better empirical performance than Better empirical performance than DRG-based alternatives on predicting DRG-based alternatives on predicting mortality mortality (especially for surgical patients; (especially for surgical patients; patients at large, urban, teaching hospitals)patients at large, urban, teaching hospitals)

Better empirical performance than Better empirical performance than DRG-based alternatives on predicting DRG-based alternatives on predicting resource use resource use (especially for medical patients; (especially for medical patients; patients over 65, at children, teaching hospitals)patients over 65, at children, teaching hospitals)

Better at reflecting the distribution of Better at reflecting the distribution of patient severity at the extremespatient severity at the extremes

Risk-Adjustment Risk-Adjustment ConclusionsConclusions

No single system based on No single system based on administrative or clinical data is administrative or clinical data is clearly superior clearly superior

DRG-based systems perform as well, DRG-based systems perform as well, and often better, than alternativesand often better, than alternatives

Data enhancements may improve Data enhancements may improve performance (e.g., condition present performance (e.g., condition present on admission, key clinical variables)on admission, key clinical variables)

Risk-Adjustment ModelRisk-Adjustment ModelInpatient Quality IndicatorsInpatient Quality Indicators

Direct standardizationDirect standardization Indirect standardizationIndirect standardization

RA = (OR / ER) * PRRA = (OR / ER) * PR

(RA – risk adjusted; OR – observed; ER (RA – risk adjusted; OR – observed; ER – expected; PR – population)– expected; PR – population)

Risk-Adjustment ModelRisk-Adjustment Model

Expected rate – Assuming the Expected rate – Assuming the hospital’s case-mix and the hospital’s case-mix and the population ratespopulation rates

Risk-adjusted rate – Assuming the Risk-adjusted rate – Assuming the population’s case-mix and the population’s case-mix and the hospital’s rateshospital’s rates


Linear regression model:Linear regression model:

observed rate = hospital effect +observed rate = hospital effect +

demographic effect demographic effect + +

condition effect + condition effect + errorerror

Model estimated on the SID, 2000 Model estimated on the SID, 2000 (25 million discharges)(25 million discharges)


IQI – Age, sex, APR-DRG (with risk of IQI – Age, sex, APR-DRG (with risk of mortality or severity of illness mortality or severity of illness subclass)subclass)

(linear with hospital fixed effects)(linear with hospital fixed effects) PQI – Age and sexPQI – Age and sex

(linear with area fixed effects)(linear with area fixed effects) PSI – Age, sex, modified CMS DRG PSI – Age, sex, modified CMS DRG

and AHRQ comorbidity (logistic)and AHRQ comorbidity (logistic)

How it Works: CABG How it Works: CABG MortalityMortality

CovariateCovariate Freq.Freq. EffectEffect CovariateCovariate Freq.Freq. EffectEffect

InterceptIntercept 0.0000.000 16521652 0.2400.240 -0.001-0.001

SexSex 0.2790.279 0.0050.005 16531653 0.1310.131 0.0200.020

Age1Age1 0.2040.204 0.0010.001 16541654 0.0360.036 0.3010.301

Age2Age2 0.1790.179 0.0090.009 16611661 0.1270.127 0.0000.000

Age3Age3 0.1950.195 0.0130.013 16621662 0.1340.134 -0.001-0.001

Age4Age4 0.2450.245 0.0310.031 16631663 0.0860.086 0.0140.014

Sex*Age1Sex*Age1 0.0490.049 0.0100.010 16641664 0.0140.014 0.3500.350

Sex*Age2Sex*Age2 0.0490.049 0.0110.011 16311631 0.0100.010 0.0100.010

Sex*Age3Sex*Age3 0.0600.060 0.0180.018 16321632 0.0130.013 0.0130.013

Sex*Age4Sex*Age4 0.0850.085 0.0340.034 16331633 0.0160.016 0.0510.051

16341634 0.0040.004 0.4780.478

OtherOther 0.0770.077 0.1630.163

How it Works: CABG How it Works: CABG MortalityMortality

SIDSIDEffectEffect

SIDSIDFreq.Freq.

HospHospFreq.Freq. DiffDiff

Effect*Effect*DiffDiff

SexSex 0.0130.013 0.2790.279 0.2620.262 -0.016-0.016 0.0000.000

56 to 6356 to 63 0.0050.005 0.2040.204 0.2560.256 0.0520.052 0.0000.000

64 to 6864 to 68 0.0160.016 0.1790.179 0.1610.161 -0.018-0.018 0.0000.000

69 to 7369 to 73 0.0270.027 0.1950.195 0.1550.155 -0.039-0.039 -0.001-0.001

74 to high74 to high 0.0540.054 0.2450.245 0.1650.165 -0.079-0.079 -0.004-0.004

Demographic EffectDemographic Effect -0.005-0.005

Observed RateObserved Rate 0.0470.047

Risk-adjusted RateRisk-adjusted Rate 0.0520.052

MSX Smoothing ModelMSX Smoothing Model

Observed quality measure = true Observed quality measure = true quality (signal) + error (noise)quality (signal) + error (noise)

Smaller hospitals and/or less Smaller hospitals and/or less frequent conditions have more noisefrequent conditions have more noise

Difficult to compare hospitals, trend Difficult to compare hospitals, trend over time, and identify best practicesover time, and identify best practices

Confidence intervals reflect but do Confidence intervals reflect but do not address the problemnot address the problem

Key Features of MSX Key Features of MSX ApproachApproach

Removes noise – uses redundancy over Removes noise – uses redundancy over time and among measurestime and among measures

Improves forecasts – predicting current Improves forecasts – predicting current quality based on past performancequality based on past performance

Reduces dimensionality appropriately - Reduces dimensionality appropriately - allows meaningful summary measuresallows meaningful summary measures

Reveals and helps reduce biases, Reveals and helps reduce biases, identify best practicesidentify best practices

OutlineOutline






Caveats of UseCaveats of Use

Validity of dataValidity of data– Validity of codingValidity of coding– Present on admissionPresent on admission– Outpatient careOutpatient care– Linking of admissions and impact of LOSLinking of admissions and impact of LOS

Incomplete risk adjustment Incomplete risk adjustment

Using the AHRQ QIUsing the AHRQ QI

State monitoring of ratesState monitoring of rates Hospital quality improvementHospital quality improvement National Healthcare Quality ReportNational Healthcare Quality Report

– PQIs and PSIsPQIs and PSIs CMS Pay for Performance Demonstration CMS Pay for Performance Demonstration

ProjectProject– Postoperative Hemorrhage or HematomaPostoperative Hemorrhage or Hematoma– Postoperative Metabolic and Physiologic Postoperative Metabolic and Physiologic

Derangement Derangement Romano et alRomano et al

– PSI National trends, (HA, Mar/Apr ’03) PSI National trends, (HA, Mar/Apr ’03)

Using the AHRQ QIUsing the AHRQ QI KovnerKovner

– QI and nurse staffingQI and nurse staffing MillerMiller

– PSI and Costs and LOS PSI and Costs and LOS Alexander/Shortell Alexander/Shortell

– PSI and Quality improvement culturePSI and Quality improvement culture BakerBaker

– PSI and Patient safety culture; hospital characteristicsPSI and Patient safety culture; hospital characteristics RosenRosen

– VA hospitals, QI and other measures (NSQuIP)VA hospitals, QI and other measures (NSQuIP) VolppVolpp

– QIs and new resident work hoursQIs and new resident work hours

Technical ReportsTechnical Reports

Development of Quality Indicators, risk adjustment and MSX methods documented in:

Davies S, Geppert J, McClellan M, McDonald KM, Romano PS, Shojania KG. Refinement of the HCUP Quality Indicators. Technical Review Number 4. Rockville, MD: (Prepared by the UCSF-Stanford Evidence-based Practice Center under Contract No. 290-97-0013) Agency for Healthcare Research and Quality; 2001. Report No.: 01-0035

McDonald KM, Romano PS, Geppert J, Davies S, Shojania KG. Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators. Technical Review Number 5. Rockville, MD: (Prepared by the UCSF-Stanford Evidence-based Practice Center under Contract No. 290-97-0013) Agency for Healthcare Research and Quality; August 2002. Report No.: 01-0038.

For More Information on AHRQ For More Information on AHRQ QIsQIs

Quality Indicators: Additional information and Quality Indicators: Additional information and

assistanceassistance

E-mail: E-mail:

[email protected]@qualityindicators.ahrq.gov

Website: Website: http://qualityindicators.ahrq.gov/http://qualityindicators.ahrq.gov/– QI technical reports, documentation and QI technical reports, documentation and

software is available on the websitesoftware is available on the website

User Support is provided under contract by User Support is provided under contract by Battelle Memorial Institute, Stanford Battelle Memorial Institute, Stanford University and University of California at University and University of California at DavisDavis

AHRQ Quality Indicators Developed by Stanford-UCSF Evidence Based Practice Center Funded by the...

Documents

Transcript of AHRQ Quality Indicators Developed by Stanford-UCSF Evidence Based Practice Center Funded by the...