Risk scoring - Cardiff PICU scoring.pdf · zPRISM (Pediatric RISk of Mortality) zPIM (Paediatric...
Transcript of Risk scoring - Cardiff PICU scoring.pdf · zPRISM (Pediatric RISk of Mortality) zPIM (Paediatric...
Risk scoring
Allan Wardhaugh
A tale of two units
Holby City, Holby
Sacred Heart
PICU mortality last year
Holby City500 admissions15 deaths
Sacred Heart1200 admissions90 deaths
‘Crude’ mortality
Holby City 3%
Sacred Heart 7.5%
What else do we need to know?
How sick were the patients?Did the patients at Sacred Heart undergo more complex procedures?Does Holby City send sick patients away?Does Holby City discharge patients to die somewhere else?What is the mortality in other units?
Case Mix
Standardised Mortality Ratio
Measured mortalityPredicted mortality – risk adjustment tool
SMR = Measured/ Predicted
∴ SMR > 1 performing poorlySMR < 1 performing well
Predicted mortality
Historical dataNational ‘average’
What about case mix?
Risk adjustment
Predicts mortality for a population based on a statistical model derived from a similar populationExamples
APACHE (Acute Physiology and Chronic Health Evaluation )
CRIB (Clinical Risk Index for Babies)
PRISM (Pediatric RISk of Mortality)
PIM (Paediatric Index of Mortality)
Regression statistics
Target variable – ‘dependent variable’Predictors – ‘independent variables’Regression statistics use association between variables to predict one (DV) from another (IV).Simplest form y = b0 + b1(x)where y = predicted value, b0= regression constant, b1= regression coefficient
Multiple regressiony = b0+b1(x1)+b2(x2)+…bn(xn)
PICU
Mortality is target variable
Predictor variables?
Example
1000 patients 100 died900 survivedProbability of death 0.1
Odds of death 0.11Admitted electively
10 of 100 deaths500 of 900 deathsProbability of death 0.02
Odds of death 0.02
Probability and Odds
ppOdds−
=1
Probabilty = 0 – 1
Odds = 0 - ∞
Regression statistics - logistic
For non-quantitative DV (e.g. dead/alive), logistic regression is usedFor each IV, odds are calculated for likelihood of having DVOdds assymetrical
very small number (0 – 1) if event unlikelyvery large if event likely (>1 - ∞)
Rectified by using natural log of odds – called logit – makes it a linear function
Regression statistics - logit
Log odds = b0+b1(x1)+b2(x2)+…bn(xn)
Probability = odds/(1 + odds)Logit = ln odds
logit
logit
e1ep+
=
PRISM
Pediatric Risk of Mortality Score14 physiological variables
Worst measurement in first 24 hoursNow on PRISM III – relies on scores in first 12 or 24 hours
Probability of PICU death= eR/1 + eR
Where R = 0.207 × PRISM – 0.005 × age(mo) – 0.433 × operative status – 4.782
PRISM III
PRISM – example60 month old non-surgical patient
PRISM Score Mortality Risk (%)
3 1.6
6 2.7
9 5
12 8.9
15 15.3
18 25.2
21 38.6
24 46.1
27 68.5
30 80.2
PRISM - disadvantages
Data collection cumbersome (14 variables over a 24 hour period)May diagnose death rather than predict it (40% deaths occur in first 24 hours)Score may not allow comparison between units – patients poorly managed in first 24 hours will develop high PRISM score, so disease severity will appear to be greater
PIM – Paediatric Index of Mortality –initial cohorts
678 consecutive admissions PICU RCHM 1988814 consecutive admissions RCHM 19901412 consecutive admissions 1994–5 RCHM
PIM – identifying variables
Data collected for admission (for most) and first 24 hours34 Physiological Stability Index measurementsMAP, PIP, PEEP, and othersWorst value in first 24 hours for all
PIM – derivation of model
All PRISM data collected plus additional informationUnivariate analysis carried out on all factors to test for association with mortalityFactors not associated (p>0.1) excluded from further analysisLogistic regression analysis used to derive preliminary model.
PIM – testing the model
Learning and Test cohorts1994 – 96 5695 patients in 8 PICUs (Australia, Birmingham)
Enough patients in each unit to include 20 deaths.Learning sample data analysed to calculate regression coefficientsModel then tested on test sample, and examined for goodness of fit.Regression coefficients re-estimated using all 8 units for final model.
PIM - coefficients
PIM – final equations
elogit/(1+elogit)
Logit = (2.357.pupils)+(1.826.specified diagnosis)+(–1.552.elective admission)+(1.342.mechanical ventilation)+(0.021.(SBP–120))+(0.071.Baseex)+(0.415.(100.FiO2/PaO2))–4.873
PIM - recalibration
PICU outcomes change with timeReferral patterns change with timeAttitudes to withdrawing and limiting care may change with time
Calibration findings
Specific diagnosisResp illness O:E 160:212Non-cardiac post-op O:E 48:82
293 coded diagnostic categories examinedIn-hospital cardiac arrest associated with increaedrisk of deathAsthma. Bronchiolitis, croup, obstructive sleep apnoea, DKA associated with reduced risk
New ‘high risk’ and ‘low risk’ categories introducedPost – op subdivided into with or without CBP.IQ <35 omitted (difficult to code reliably)
SMR
Australia and New Zealand SMR 0.84 (0.76–0.92) UK 0.89 (0.77–1.00).
PRISM vs PIM
PRISM predicted 66% more deaths in this sampleScore altered by treatment in the first 24 hoursMay diagnose rather than predict deathPRISM III data requires 96 measured variablesLicense requiredNote that neither are adequate for individual case prediction – apply to populations only
United KingdomPaediatric Intensive Care Outcome Study
UK PICOS
PIM mortality ratio (observed/expected unit deaths) by unit. Generated using UK PICOS recalibration
.5
1
2M
orta
lity
ratio
200 400 600 800 1000Number of admissions
Your unit Other units Control limits
Mortality ratio calculated using the UK PICOS calibration of PIM in the UK.
Upper and lower control limits represent a 99.9% confidence interval around a mortality ratio of 1 based on the UK PICOS overall mortality of 6.2%..
Outcome
PRISM III 24 hour score re-calibrated for UKPerformance of PIM-2 and PRISM III very similarPIM – 2 recommended as model of choice as data easier to collect
DoH/ WAG fundedRun from Universities of Sheffield, Leicester and LeedsFirst annual report March 2003 – February 2004
2009-11 Unadjusted
2009-11 adjusted (PIM-2R)
League tables
Governments like themJournalists like themLocal politicians like themPatients groups like them
Do any of the above understand them?
9 NICUs over 6 yearsCrude and risk adjusted (CRIB score) mortalityHospitals ranked in league tables each year according to W score
W= 100 × (observed - expected deaths)/No of admissions.Mortality lower than expected if W < 0
Results
Conclusions
Hospitals varied annually in their league positionConfidence intervals for W scores overlapped for all hospital every year except year 3‘Overall, hospital 1 did perform significantly better than expected but it is debatable whether this makes it a model hospital since its performance was inconsistent’.
Summary
PIM/ PIM 2 data easy to collect Useful in comparing unit performanceInterpret with care if number of deaths low (especially <20).Not for use as an individual prediction testImportant to complete as accurately as possiblePICANET randomly check to ensure data qualityLeague tables are unreliable