© Imperial College LondonPage 1

A method for estimating the cost of reducing the false alarm rate in multi-

institution performance monitoring using CUSUM charts

Alex Bottlerobert.bottle@imperial.ac.uk

Imperial College LondonDr Foster Unit

© Imperial College LondonPage 2

Overview

• Background: cumulative sum charts and nationwide NHS mortality monitoring tool

• Extent of multiple testing

• Factors affecting false alarm rate

• Simulation for false alarm and successful detection rates

• Estimation of ‘cost’: worked example for AMI

• Summary

© Imperial College LondonPage 3

CUSUM chart essentials

• Plots one patient at a time

• Chart statistic (log-likelihood ratio) goes up if patient dies and down if patient survives

• Chart rises faster if low-risk patient dies

• If crosses preset threshold, chart ‘signals’

• Threshold choice involves consideration of type I and type II error rates

© Imperial College LondonPage 4

CUSUM charts

© Imperial College LondonPage 5

Mortality monitoring tool

• In use in ~100 acute hospitals in England

• Compares each hospital’s case-mix adjusted mortality rate with national average

• Tests for an odds ratio of at least 2

• Displayed using cumulative sum charts

• Data are updated monthly

© Imperial College LondonPage 6

Mortality monitoring tool: opening screen

© Imperial College LondonPage 7

Extent of multiple testing

Over time: threshold handles this elementBut…At each hospital trust each month:• 78 diagnosis groups• >100 procedure groups

National monitoring incurs further ‘cost’:• ~150 acute hospital trusts• Consultant-level monitoring?

© Imperial College LondonPage 8

Factors affecting the false alarm rate

• Threshold: the higher this is set, the lower the false alarm rate

• Length of monitoring: number of patients varies by hospital and diagnosis

• Expected mortality rate: e.g. 5% rates will have high FAR than 1% rates

• Size of increase (OR) to be detected (not considered here)

© Imperial College LondonPage 9

Research question

• A higher chart threshold -> lower FAR but slower detection of high mortality rates

• Compared with the conventional 5% false alarm rate, what is the ‘cost’ of having a lower false alarm rate (1% or 0.1%) to deal with all the multiple testing?

© Imperial College LondonPage 10

Simulation: FAR and SDR

• For FAR, generate 5,000 artificial hospitals with mortality rate p

• Do this for various p, p=0.1% to 30%

• Calculate FAR after t patients, t in steps of 5 from 5 to 20,000

• Do this for different thresholds h, h=0.5 to 15

• For SDR, generate hospitals with rate 2p/(1-p)

© Imperial College LondonPage 11

Using the simulation to estimate ‘cost’

• For each dx, work out the threshold h needed for FAR of 5% at average hosp

• Find number of monitored patients t needed for SDR of 80% using threshold h

• Knowing the dx’s expected death rate and OR to be detected, convert t into a number of deaths

• Repeat for FAR of 1% and 0.1%• Find the difference in number of deaths

between the pairs of FAR values

© Imperial College LondonPage 12

‘Cost’ calculation for AMI in England (1)

• National death rate=11.8%. Average number of AMIs per hospital=467

• For FAR=5%, h=5.2 -> t=185 for SDR=80%

• At rate p, this means 21.8 deaths

• At rate 2p/(1-p), this means 39.0 deaths

• ‘Excess’ deaths: 39.0 – 21.8 = 17.2

© Imperial College LondonPage 13

‘Cost’ calculation for AMI in England (2)

• For FAR=0.1%, h=8.6 -> t=305 for SDR=80%

• At rate p, this means 36.0 deaths

• At rate 2p/(1-p), this means 64.4 deaths

• ‘Excess’ deaths: 64.4 – 36.0 = 28.4

• ‘Cost’ of lowering FAR to 0.1% =

28.4 – 17.2 = 11.2 extra deaths at average hosp

© Imperial College LondonPage 14

Findings for AAA repair and CABG

• AAA repair: less common but high risk

• ‘Cost’=6.3 for FAR=0.1%, 2.4 for FAR=1%

• CABG: common but low risk

• ‘Cost’= 6.8 for FAR=0.1%, 2.8 for FAR=1%

• These are all figures for an average hospital

© Imperial College LondonPage 15

Summary

• Multiple testing can be addressed by lowering the false alarm rate: raise the threshold for CUSUM charts

• Other approaches include minimising ‘loss function’ or maximising ‘desirability function’

• The proposed measure of ‘cost’ depends on mortality rate and hospital volume

• The ‘cost’ can be derived from simulation and is intuitive to less-technical users