50 shades of greyhow (in)accurate are risk prediction scores

Dr Doug CampbellMeasuring, managing and minimising riskPerioperative SIG, Grand Hyatt, Melbourne

October 27th 2018

Declarations and Acknowledgements

• No relevant conflicts of interest• Funded by Precision Driven Health

Those who make many species are the 'splitters,' and those who make few are the 'lumpers.'

Myth of individualised prediction

…with apologies to XKCD

Validation 101- easy as ABCD

Validation 101- easy as ABCD

A=0B=0.96C=0.922

nzRISK – CalibrationIntercept = 0Slope = 0.97McFadden’s R2 = 0.28AUROC = 0.92

Intercept = 0Slope = 0.96McFadden’s R2 = 0.31AUROC = 0.91

Intercept = 0Slope = 0.98McFadden’s R2 = 0.37AUROC = 0.90

1 month

1 year 2 year

nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed

N (%) N % % N N

1 9481 0 0 0.9 1

2 8528 0 0 1.5 0

3 9303 0 0 2.9 1

4 8713 0.1 0.1 5.4 5

5 9026 0.1 0.1 9.5 5

6 9192 0.2 0.1 14.8 8

7 (0.2-0.3) 8784 0.2 0.3 21.6 26

8 (0.3—0.5) 9008 0.4 0.2 35.9 21

9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78

10 (>1.5) 8996 5.6 5.4 508.1 489

nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed

N (%) N % % N N

1 9481 0 0 0.9 1

2 8528 0 0 1.5 0

3 9303 0 0 2.9 1

4 8713 0.1 0.1 5.4 5

5 9026 0.1 0.1 9.5 5

6 9192 0.2 0.1 14.8 8

7 (0.2-0.3) 8784 0.2 0.3 21.6 26

8 (0.3—0.5) 9008 0.4 0.2 35.9 21

9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78

10 (>1.5) 8996 5.6 5.4 508.1 489

Low risk, NZRISK < 0.5%, 11% of deaths

High risk, NZRISK > 1.5%, 77% of deaths

Intermediate risk, NZRISK 0.5-1.5%, 12% of deaths

Decision curve analysis

Glance et al. Impact of the choice of model for identifying low-risk patients using the 2014 ACC/AHA perioperative guidelines. Anesthesiology 2018; 129(5): 889-900

Generalisability

SORT internal validation

AUROC = 0.91

AUROC = 0.90

SORT external validation

Generalisability

SORT internal validation

AUROC = 0.91

AUROC = 0.90

SORT external validation

Protopapa KL et al. Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surgery 2014; 101: 1774-83Campbell D et al. Development and validation of a multivariable prediction model of perioperative mortality in noncardiac surgery (NZRISK). Br J Surgery 2018; in press

What risk tool for CADENZA?

Eugene N et al. Development of the NELA risk model. BJA 2018; 121(4): 739-48

ACS-NSQIP - morbidity

DAH90 – the median is not the message

Campbell D, Boyle L, Djamali K, Cumin D, Weller J, Short T, Merry A. Unpublished data.

New Zealand 2011-6, n = 141331-month mortality = 4.2%3-month mortality = 5.7%

83 days48 days

Checklist for risk tools

• Does it model the correct outcome?– 1-month mortality, morbidity, long-term mortality

• Is it accurate?– calibration (SHARED DECISION-MAKING)– discrimination (TRIAGE)

• Is it generalizable to my patient?– validated in Australia or New Zealand?

www.nzrisk.com

www.nzrisk.com

www.nzrisk.com

Comparison - general

Calculator Covariates Country N AUROC Calibration

ACS-NSQIP 22 US 1.4 million 0.944 Brier 0.011POSPOM 17 France 5.5 million 0.929 Brier 0.004nzRISK 8 NZ 360140 0.922 Brier 0.007SORT 6 UK 16,788 0.91 H-L 12.16P-POSSUM 18 UK 10,648 total 0.83 (0.68-0.92)

H-L = Hosmer-Lemeshow statistic, McF = McFaddens pseudo rho-squared statistic, Brier = Brier score, O:E observed to expected ratio

High risk calibration