Use of prognostic scoring
systems to predict outcomes of
critically ill patients
Dr. Kwok Ming HO
MBBS, Postgrad Dip (Echo), MPH, FRCP (Glasg), FANZCA, FJFICM
Staff Specialist, Intensive Care Unit
Royal Perth Hospital
This thesis is presented for the degree of Doctor of Philosophy of
The University of Western Australia
Schools of Medicine, Pharmacology, and Population Health
2008
Acknowledgement
I would like to thank Drs. Geoffrey Clarke and John Weekes for their part in
initiating the clinical database in the Intensive Care Unit of Royal Perth Hospital and also all
the consultants who have been recording the Acute Physiology and Chronic Health
Evaluation (APACHE) data for every admission to the Intensive Care Unit. I would like to
thank my PhD supervisors (Clinical Associate Professor Steven Webb, Clinical Professor
Geoffrey Dobb, Professor Matthew Knuiman, and Professor Judith Finn) and my colleagues
(Drs. Kok Yeng Lee and Simon Towler) in the Intensive Care Unit of Royal Perth Hospital
for their contributions to the studies generated from this thesis. I would also like to thank
BUPA Foundation for funding the cost of data linkage. Finally, I would like to express my
special thanks to my wife, Kayo, and my son, Akio, for their patience and support. Without
their support, completion of this thesis would not be possible.
Certification by the co-authors
The following co-authors certify that Dr. K.M. Ho is the main author in initiating the
original idea, designing and analysing the data, and drafting of the manuscripts and thesis.
All co-authors agree that the published manuscripts be included in this PhD thesis.
Name of co-author Signature Date
1. Prof. Matthew Knuiman
2. Prof. Judith Finn
3. Clin. A/Prof Steven Webb
4. Clin. A/Prof Geoffrey Dobb
5. Dr. Kok Y. Lee
6. Dr. Simon Towler
7. Ms Teresa Williams
CONTENT
Abbreviations P. 1
Summary (overview) of the thesis P. 2-4
Publications arising from the thesis P. 5-6
Section one: Background, rationale, materials, and methods of the study
Chapter 1. Background, motivation and rationale P. 7-13
Chapter 2. Characteristics of the cohort and statistical methods P. 14-28
Section two: Assessment of the APACHE II scoring system in an Australian context
Chapter 3. The worst first 24-hour and admission APACHE II P. 29-39
scoring system
Chapter 4. The use of the APACHE II scoring system for indigenous P. 40-46
patients
Chapter 5. Assessing calibration by meta-analytic techniques P. 47-55
Section three: Relationship between the APACHE II scoring system, organ failure scores,
and co-morbidities in determining hospital mortality and ICU readmission
Chapter 6. Comparing the APACHE II scoring system with organ failure
scores to predict hospital mortality P. 56-66
Chapter 7. Combining the APACHE II scoring system with Sequential
Organ Failure Assessment (SOFA) scores to predict hospital mortality P. 67-76
Chapter 8. Combining the APACHE II scoring system with co-morbidity
data to predict hospital mortality P. 77-85
Chapter 9. The effect of co-morbidity on risk of unplanned ICU
readmission P. 86-94
Chapter 10. Evaluating the APACHE II scoring system in predicting
hospital mortality of ICU readmissions P. 95-106
Section four: The use of inflammatory markers in addition to organ failure score and the
APACHE II scoring system in predicting post-ICU hospital mortality and ICU readmission
Chapter 11. Inflammatory markers and risk of unplanned ICU
readmission P. 107-15
Chapter 12. Inflammatory markers and prediction of hospital mortality P. 116-24
Section five: Predicting long term survival after hospital discharge
Chapter 13. The PREDICT model P. 125-35
Chapter 14. The effect of socioeconomic status on long term survival P. 136-43
Section six: Conclusion
Chapter 15. Summary and directions for future research P. 144-6
References P. 147-55
Appendices: Ethics approval forms and correspondence P. 156-60
1
Abbreviations
ACHS Australian Council of Healthcare Standards
APACHE Acute Physiology and Chronic Health Evaluation
ARIA Area of Remoteness Index of Australia
ANZICS Australian & New Zealand Intensive Care Society
CHIC Confidentiality of Health Information Committee
CI Confidence Interval
CRP C-reactive protein
DLU Data Linkage Unit
HMD Hospital Morbidity Databases
ICD International Classification of Diseases
ICU Intensive Care Unit
MPM Mortality Prediction Model
ROC Receiver Operating Characteristic
RPH Royal Perth Hospital
RPHICU Royal Perth Hospital Intensive Care Unit
SAPS Simplified Acute Physiology Score
SEIFA Socio-Economic Indices for Areas
SES Socioeconomic Status
SMR Standardised Mortality Ratio
SOFA Sequential Organ Failure Assessment
SPSS Statistical Package for the Social Sciences
SUPPORT Study to Understand Prognoses and Preferences for Outcomes
and Risks of Treatments
WA Western Australia
2
Summary (overview) of the thesis
This research thesis consists of five sections. Section one provides the
background information (chapter 1) and a description of characteristics of the cohort
and the methods of analysis (chapter 2).
The Acute Physiology and Chronic Health Evaluation (APACHE) II scoring
system is one of commonly used severity of illness scoring systems in many intensive
care units (ICUs). Section two of this thesis includes an assessment of the
performance of the APACHE II scoring system in an Australian context. First, the
performance of the APACHE II scoring system in predicting hospital mortality of
critically ill patients in an ICU of a tertiary university teaching hospital in Western
Australia was assessed (Chapter 3). Second, a simple modification of the traditional
APACHE II scoring system, the ‘admission APACHE II scoring system’, generated
by replacing the worst first 24-hour data by the ICU admission physiological and
laboratory data was assessed (Chapter 3). Indigenous and Aboriginal Australians
constitute a significant proportion of the population in Western Australia (3.2%) and
have marked social disadvantage when compared to other Australians. The difference
in the pattern of critical illness between indigenous and non-indigenous Australians
and also whether the performance of the APACHE II scoring system was comparable
between these two groups of critically ill patients in Western Australia was assessed
(Chapter 4).
Both discrimination and calibration are important indicators of the
performance of a prognostic scoring system. Meta-analytic techniques were used in
this thesis to illustrate the uniformity of fit in the calibration of the APACHE II
scoring system across different diagnostic and age subgroups and the results of these
3
techniques were compared with the results from assessment of the slope and intercept
of the calibration curve of the APACHE II scoring system (chapter 5).
There are other factors and alternative scoring systems that may be useful and
may improve the performance of the APACHE II scoring system if they are
incorporated. Section three of this thesis includes a comparison of the performance of
the APACHE II scoring system with two organ failure scoring systems (Chapter 6),
and an evaluation of whether the performance of the APACHE II scoring system
could be enhanced by incorporating organ failure data (Chapter 7) and more detailed
co-morbidity data (Chapter 8).
Unplanned ICU readmission is one of the quality indicators adopted by the
Australian Council of Healthcare Standards (ACHS). This undesirable in-hospital
outcome was further explored by assessing whether the APACHE II scoring system
and co-morbidity can be used to predict unplanned ICU readmission during the same
hospitalisation (Chapter 9). The APACHE II scoring system (and in fact all existing
ICU scoring systems) had excluded patients who were readmitted to the ICU during
the same hospitalisation, and as such, its performance in predicting mortality of ICU
readmission remained unknown. The use of the APACHE II scoring system in
patients readmitted to ICU during the same hospitalisation was evaluated and also
whether incorporating events prior to the ICU readmission to the APACHE II scoring
system would improve its ability to predict hospital mortality of ICU readmission was
assessed in chapter 10.
Whilst there have been a number of studies investigating predictors of post-
ICU in-hospital mortality none have investigated whether unresolved or latent
inflammation and sepsis may be an important predictor. Section four examines the
role of inflammatory markers measured at ICU discharge on predicting ICU re-
4
admission (Chapter 11) and in-hospital mortality during the same hospitalisation
(Chapter 12) and whether some of these inflammatory markers were more important
than organ failure score and the APACHE II scoring system in predicting these
outcomes.
Section five describes the development of a new prognostic scoring system
that can estimate median survival time and long term survival probabilities for
critically ill patients (Chapter 13). An assessment of the effects of other factors such
as socioeconomic status and Aboriginality on the long term survival of critically ill
patients in an Australian ICU was assessed (Chapter 14).
Section six provides the conclusions. Chapter 15 includes a summary and
discussion of the findings of this thesis and outlines possible future directions for further
research in this important aspect of intensive care medicine.
5
Publications arising from the thesis
Chapter 3.
Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, Webb SA. A comparison of
admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II
scores in predicting hospital mortality: a retrospective cohort study. Critical Care
2006;10:R4.
Chapter 4.
Ho KM, Finn J, Dobb GJ, Webb SA. The outcome of critically ill Indigenous patients.
Medical Journal of Australia 2006;184:496-9.
Chapter 5.
Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model: a
descriptive study. Journal of Clinical Epidemiology 2007;60:746-51.
Chapter 6.
Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of Acute
Physiology and Chronic Health Evaluation (APACHE) II score with organ failure
scores to predict hospital mortality. Anaesthesia 2007;62:466-73.
Chapter 7.
Ho KM. Combining sequential organ failure assessment (SOFA) score with acute
physiology and chronic health evaluation (APACHE) II score to predict hospital
mortality of critically ill patients. Anaesthesia & Intensive Care 2007;35:515-21.
Chapter 8.
Ho KM, Finn J, Knuiman M, Webb SA. Combining multiple comorbidities with
Acute Physiology Score to predict hospital mortality of critically ill patients: a linked
data cohort study. Anaesthesia 2007;62:1095-100.
6
Chapter 9.
Ho KM, Dobb GJ, Finn J, Knuiman M, Webb SA. The effect of co-morbidities on
risk of intensive care readmission during the same hospitalisation: a linked data cohort
study. Journal of Critical Care 2008 (published online in April 2008).
Chapter 10.
Ho KM, Knuiman M. Bayesian approach to predict hospital mortality of intensive
care readmissions during the same hospitalisation. Anaesthesia & Intensive Care
2008;36:38-45.
Chapter 11.
Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein concentration
as a predictor of intensive care unit readmission: a nested case-control study. Journal
of Critical Care 2006;21:259-65.
Chapter 12.
Ho KM, Lee KY, Dobb GJ, Webb SA. C-reactive protein concentration as a predictor
of in-hospital mortality after ICU discharge: a prospective cohort study. Intensive
Care Medicine 2008;34:481-7.
Chapter 13.
Ho KM, Knuiman M, Finn J, Webb SA. Estimating long-term survival of critically ill
patients: the PREDICT model. Public Library of Science One 2008;3:e3226.
Chapter 14.
Ho KM, Dobb GJ, Knuiman M, Finn J, Webb SA. The effect of socioeconomic
inequalities on outcomes of seriously ill patients: a linked data cohort study. Medical
Journal of Australia 2008;189:26-30.
7
Section one: Background, rationale, materials, and
methods of the study
Chapter 1. Background, motivation and rationale
The significance of intensive care services
ICUs are specially staffed and equipped hospital wards that provide advanced
life support for patients with life-threatening illnesses or after major surgery. The
major role of ICUs is to save lives that might otherwise be lost to acute life-
threatening illnesses such as severe infection, trauma, burns, drug overdose,
cerebrovascular accidents, or acute respiratory failure. Because of the staffing and
equipment requirements in ICUs, intensive care service is much more expensive than
many other health care services. There are more than 6,000 ICUs in the US, with
between 75,000 and 90,000 beds.1,2
The expenditures on health care were $2.1 trillion
in 2006 and accounted for 16% of the gross domestic product, of which
10% (of the
total health expenditures) is estimated to be spent on critically ill patients in the
United States.3
The cost of adult ICUs in the United Kingdom has been estimated at
£700 million, which represents 0.1% of GDP.4
In 2001, there were 172 ICUs in
Australia, providing a total of 1,272 beds to care for the 137,598 critically ill patients.5
The total cost of ICU care in Australia is not known. In 2003, the average costs of an
ICU day and total cost of ICU stay per patient in a tertiary ICU in Australia were
estimated to be about A$2,670 and A$9,852, respectively.6
Demand for ICU services is increasing,7 and at a rate that is higher than the
average for all health care services.2 Increase in treatment and monitoring technology,
patients’ expectations, and ageing population all contribute to this increased demand
for ICU services.7 The current data suggest that intensive care services are reasonably
8
cost effective when compared to other medical and surgical interventions,8-10
but this
may change significantly if intensive care services are provided without any rationing
or monitoring of the outcome data. Intensive care is indeed increasingly being
provided to many older and sicker patients, whom in the past were not treated in the
ICUs.11
Undesirable outcomes following critical illness
Although most patients survive their critical illness after intensive care therapy
without any undesirable events, unplanned ICU readmission and in-hospital death
after ICU discharge during the same hospitalisation are not uncommon in many
ICUs.12,13
Some patients may also die within a short period of time after their hospital
discharge or survive with a very poor quality of life.14
If modifiable risk factors of
these undesirable outcomes after critical illness can be identified, perhaps patient
outcomes can be improved by improving the process of care to reduce these risk
factors. From a clinical perspective, many patients and clinicians may also be
interested to know the risk factors of these poor outcomes, even if they are not
modifiable, when making difficult triage or treatment decision in ICUs. Furthermore,
survival after hospital discharge is increasingly being used as an end-point in
assessing cost effectiveness of expensive new technology and treatments in critical
illness.15
If tools that can predict short and long term outcomes after critical illness are
available, the accuracy of risk adjustment and cost effectiveness analysis of any new
treatments or technology can potentially be improved.
A quest for a cost effective high quality intensive care service coupled with an
increasing demand provides a strong rationale for improved modelling of prediction
of outcomes in ICUs.
9
The existing prediction or scoring systems in intensive care
Predictions and prognostications of outcomes of critically ill patients are often
made on a daily basis by many clinicians working in the ICUs or critical care
environments to triage ICU admissions and treatment.16-18
It will be desirable if
clinicians have accurate and consistent information regarding the patients likely
outcomes. Several prediction and prognostic scoring systems have been developed,
and these include scoring systems that evaluate co-morbidities,19
organ failure,20-22
or
a combination of factors including age, chronic health status, and severity of
physiological derangement. The latter type of scoring systems, such as the Acute
Physiology and Chronic Health Evaluation (APACHE) scoring system, Mortality
Prediction Model (MPM), and Simplified Acute Physiology Score (SAPS), are
commonly used in many ICUs for audit and research purposes.23-25
The performances of these scoring systems are quite variable, especially when
applied to different cohorts of critically ill patients in different ICUs. The APACHE
scoring system, initially described in 1981 in USA, is the most widely used method in
assessing severity of illness in ICUs. The APACHE II scoring system, published in
1985, is a revised version that uses the worst physiological measurements within the
first 24 hours of ICU admission, age and previous health status coupled with
diagnostic information to estimate the risk of hospital death.24
An increasing score
(range 0 to 71) has been shown to be closely correlated with the subsequent risk of
death across a range of life threatening diseases, with an area under the receiver
operator characteristic (ROC) curve in the initial validation study of 86%.24
The
APACHE III scoring system, published in 1991, is a further modification of the
APACHE scoring system and has incorporated more diagnostic categories to improve
its calibration.25
The performance of the APACHE III scoring system has
10
subsequently been evaluated in Brazil, the United Kingdom, Germany, US, and
Australia with variable results.26,27
The use of APACHE III scoring system was
further extended to predict 6-month survival of critically ill patients and the area
under receiver operating characteristic (ROC) curve in the validation cohort was
found to be 78%.14
In 2006, a further update on the APACHE scoring system was
made and the APACHE IV scoring system was published.28
Despite the availability of
the newer versions of the APACHE scoring system, the APACHE II scoring system
remains very popular and commonly used in many ICUs for audit and clinical
research purposes. This may be because of its ease of use and its long history in
clinical use that allows easy comparison between different ICUs or time periods.29,30
Perhaps somewhat surprising, the APACHE II scoring system has not been
thoroughly evaluated in any Australian ICUs and its applicability in predicting
hospital mortality of critically ill Aboriginal patients has also not been evaluated.
The APACHE II scoring system has significant limitations. First, the
APACHE II scoring system uses the worst physiological derangement of a patient
within the first 24 hours of ICU admission. In collecting the worst physiological and
laboratory data, the data collector has to compare all the physiological and laboratory
data over a 24-hour period to generate the correct (maximised) APACHE II score.31
Studies have shown that accuracy of the data can be very variable depending on
expertise of the data collector and the process of data collection can be very time
consuming. In using the APACHE II (or III and IV) scoring system as a tool to
stratify participants into different risk categories in a clinical trial in ICUs, enrolment
of the participants may have to be delayed to wait for complete collection of the first
24 hours of physiological and laboratory data. Alternatively, the worst physiological
and laboratory data until the point of enrolment within the first 24 hours of ICU
11
admission are used. Another possible and more straightforward alternative would be
to use the physiological and laboratory data at the time of ICU admission. This
approach is easier and the results may be potentially less variable because comparing
physiological and laboratory data obtained within the first 24 hours of ICU admission
is not necessary. The admission data have in fact been used by some studies to
calculate the APACHE II score for risk adjustment purposes.32,33
Whether this
modified use of the APACHE II scoring system is reliable or as accurate as the
original APACHE II scoring system has not been assessed.
Second, the APACHE II scoring system was primarily designed to estimate
hospital mortality of critically ill patients who have not been treated in an ICU during
the same hospitalisation. In the original cohort used to derive the APACHE II scoring
system, patients readmitted to the ICU during the same hospitalisation were
excluded.24
No other existing ICU scoring systems have studied this subgroup of
critically ill patients and there is currently no risk adjustment tool that can estimate the
risk of hospital death of these patients. The Australian Council of Healthcare
Standards (ACHS) has adopted unplanned ICU readmission within 72 hours of ICU
discharge as an indicator of the quality of care of an ICU because unplanned ICU
readmissions are associated with an increase in health care costs, patients morbidities
and mortality.34,35
Very little epidemiological data on this quality indicator or
unexpected death after ICU discharge during the same hospitalisation are available
from Australian ICUs. Whether the APACHE II scoring system, either alone or in
combination with other prognostic factors such as co-morbidities or organ failure, can
be used to predict these undesirable in-hospital outcomes has not been thoroughly
evaluated in an Australian ICU population.36,37
12
Third, it is possible that many patients and ICU clinicians make ICU triage
and treatment decisions based on their perception of the most likely long term
survival and quality of life following a critical illness.38,39
Epidemiological data on the
long term outcomes following critical illness are sparse. Most published
epidemiological data on long term outcomes of critically ill patients are limited by a
relatively short duration and significant loss to follow up, small sample size, cohort
with limited range of diagnoses, or absence of information on severity of acute illness
or pre-existing comorbidities.40
The use of the APACHE II and III scoring systems in predicting survival after
hospital discharge was evaluated by two studies. The SUPPORT investigators from
the United States of America and Wright et al. from United Kingdom published two
scoring systems that can provide an estimation of 6-month and 5-year survival
probabilities of critically ill patients, respectively.14,41
The latter scoring system
provides only three survival probabilities if a patient’s risk score falls into either <70,
70-80, or >90.41
As such, its utility is limited. There is currently no prognostic scoring
systems that can give long term survival estimate of more than 5 years.
Broad aims of the study
In this thesis, the strengths and weaknesses of one of the most commonly used
scoring systems in ICUs, the APACHE II scoring system, were evaluated in an
Australian context including its application to critically ill indigenous patients. An
assessment was made as to whether this scoring system can be further improved by
incorporating other data including organ failure and detailed co-morbidity data.
13
The incidence for two undesirable in-hospital outcomes, unplanned ICU
readmission and unexpected death after ICU discharge, was described and whether
the APACHE II scoring system was useful to predict these outcomes was assessed.
As a first attempt to develop a tool that can estimate long term survival
probabilities following critical illness, a new scoring system (the PREDICT model)
was developed. Evidence suggests that ethnicity and socio-economic status may have
a significant association with long term survival after some life-threatening
diseases.42,43
Whether Aboriginality, socioeconomic status, and accessibility to
essential services may have an independent association with long term survival
outcome of critically ill patients, over and beyond the usual biological factors such as
co-morbidities and severity of illness as measured by the APACHE II scoring system,
were assessed in this thesis.44
14
Section one: Background, rationale, materials, and
methods of the study
Chapter 2. Characteristics of the cohort and statistical methods
Patient selection and data sources
(1) Royal Perth Hospital Intensive Care Unit (RPHICU) databases
Royal Perth Hospital (RPH) intensive care unit is the largest ICU in Western
Australia and it provides over 40% of all intensive care services in Western Australia.
Since 1987, demographics and clinical information such as the admission APACHE II
score and its components on admission, admission diagnoses, daily assessment of
organ failure (RPH organ failure score), and daily administration of common
therapeutic modalities, have been collected for all patients admitted to the RPHICU.
Since 1989, the worst APACHE II scores within the first 24 hours of ICU admission
have been collected. Since 2004, daily Sequential Organ Failure Assessment (SOFA)
score have been collected. The data were collected prospectively on a pre-printed data
collection form by the senior medical staff of the unit and entered into a database
(.dbf) by one designated clerical officer.
Between 1987 and 2002 (i.e. 16 years), there were 26,021 RPHICU
admissions for 22,990 patients (over 3,000 patients had more than one ICU
admissions). The patient population was comprised of more males (67%) than
females, had a mean age of 57.2 (± standard deviation [SD] 17.4) years, with only
0.7% under the age of 16 years (RPH provides only limited paediatric services) and
4.8% were 80 years or older. The median length of stay in ICU and hospital was 2.6
and 12 days, respectively. The mean APACHE II score for ‘elective’ admissions was
15
10.0 (± SD 4.1), while the mean APACHE II score for ‘emergency’ patients was 13.5
(± SD 7.5). The overall crude ICU and hospital survival was 92.1% and 89.2%,
respectively.
(2) Western Australian Data Linkage
Details of each individual patient can theoretically be summarised in a more
comprehensive manner by linking all the available health and administrative
databases from different sources. Data Linkage WA at the Department of Health
collaborates with the Centre for Health Services Research at the University of
Western Australia, the Division of Health Sciences at Curtin University of
Technology, and the Telethon Institute for Child Health Research to provide
information for valuable medical and population health research. The unit was
established in 1995 to develop and maintain a system of linkages connecting data
about health events for individuals in WA. The unit manages the Western Australian
Data Linkage System which links the WA's core population health data sets.
Operations depend on access to personal identifying information derived from each of
the contributing data sources, but the actual health details are stored and managed
separately by delegated data custodians. These linkages are created and maintained
using rigorous, internationally accepted privacy preserving protocols, extensive
clerical review, and probabilistic matching.45
The probabilistic matching technique is
based on six Automatch (software package) passes. These six Automatch passes
include unit medical record number (unique only to teaching hospitals), surname &
first name, initial, data of birth, sex and address of the patient. Clerical checking of
additional information for possible matches that fall within a ‘grey area’ between
16
definite matches and definite non-matches is undertaken to improve the accuracy of
the data.
The quality of the WA hospital morbidity databases (HMD) linked data was
assessed by a sampling technique a few years ago, and both the percentage of invalid
links (false positives) and missed links (false negatives) were estimated to be 0.11%.46
The HMD have demographic information including date of birth, gender, clinical
diagnoses recorded in all public and private hospitalisations coded according to the
International Classification of Diseases (ICD-9-CM and ICD-10CM), and details of
the hospital length of stay and hospital discharge status of all patients in WA.47
The
WA Death Registry has information on date, causes (text and Australian Bureau of
Statistics (ABS) coded), and place of all deaths (e.g. hospital or residential address) in
WA.
The data used in this thesis were obtained by linking the RPHICU database
with the WA HMD and WA Death Registry. For consistency reasons, the first RPH
ICU admission on or following January 1st 1987 was classed as that person’s index
admission and taken as time zero for estimation of their survival time after hospital
discharge. The survival status of this cohort was assessed on 31st December 2003 and
the mean duration of follow-up of the cohort was about 6 years. A minimum of 12
months follow-up period was available for the whole cohort (22,990 patients), and 5-
year, 10-year, and 15-year follow-up was available for 18,048 patients, 11,265
patients, and 3,070 patients, respectively.
The RPHICU database contained information regarding the demographic
factors of the patients such as age, gender, and ethnicity, the APACHE II score,
APACHE II predicted mortality, elective/emergency status, source of admission,
admission diagnostic categories (as classified in the APACHE II scoring system),
17
length of ICU stay, and length of hospital stay. In this thesis, the Western Australian
Data Linkage Unit provided linked information from HMD and WA Death Registry
regarding specific co-morbidities (from which Charlson co-morbidity index was
estimated),19
socioeconomic factors, and long term mortality data of all ICU
admissions.
The Centre for Health Services Research at The University of Western
Australia developed the SPSS syntax based on the Dartmouth-Manitoba algorithm to
generate a ‘Charlson co-morbidity index’ for the patients in the ICU database.48
The
co-morbidities identified within HMD records where the hospital admission date was
within a five-year period prior to the index ICU admission were all included. A
relatively long ‘look back’ period was used in this thesis to try to capture all pre-
existing co-morbidities of the patients. It is possible, that an ICU patient may not have
had any previous WA hospitalisations prior to the index ICU admission, especially if
the patient was young (and previously healthy) or had been living in other states of
Australia or overseas prior to the index ICU admission. Those ICU patients with no
link to the WA HMD were assumed to have had no previous hospitalisations and their
Charlson co-morbidity index was estimated as zero.
The post code of each ICU patient’s usual place of residence was used to
classify patients into different socioeconomic groups, using the Socio-Economic
Indices for Areas (SEIFA) of the closest Census year to the year of index ICU
admission.49
The degree of accessibility to essential services was classified by Area of
Remoteness Index of Australia (ARIA) also using the post code of the patient’s usual
place of residence. In the ARIA system, accessibility to essential services is different
among the following categories, Major Cities of Australia; Inner Regional Australia;
Outer Regional Australia; Remote Australia; and Very Remote Australia.49,50
In the
18
studies of the long term outcome of critically ill patients, patients with residential
addresses outside Western Australia were excluded from analysis because follow up
on their survival outcome outside WA would be impossible.
The cohort of patients admitted to the RPHICU between 1987 and 2002 was
used to investigate the effects of demographic factors, socioeconomic status, co-
morbidities, severity of acute illness (as measured by the APACHE II scoring system)
on risk of ICU readmission, hospital mortality, and also long term survival outcome
after hospital discharge. For the studies that compared the performance of the
APACHE II scoring system with organ failure scoring systems and also studies on
predictors of unplanned ICU readmission and unexpected death after ICU discharge,
only the RPHICU data between 2004 and 2005 were used because the daily SOFA
score was not available before 2004.
(3) Comparing RPHICU cohort to patients in other Australian ICUs
RPHICU is staffed by fully trained intensivists who have postgraduate
specialist qualifications in intensive care medicine. The formal training of intensive
care specialists in Australia was started over 30 years ago in Australia and the quality
of intensive care services is in general very high when compared to other countries.51
RPHICU and most ICUs in Australia are often regarded as a ‘closed’ unit. The
intensive care team has a strong administrative and clinical role in deciding ICU
admission, treatment options, and discharge decisions on patients who are critically ill
in the hospital.51,52
Similar staff training and administrative model of the ICUs have
created very similar clinical practices and case mix, and a supportive environment for
multi-centre research and collaborations between different Australian ICUs.53-55
19
In this thesis the characteristics of patients in three diagnostic subgroups
including sepsis, community acquired pneumonia and non-operative trauma were
compared to assess whether the RPHICU cohort in 2001 and 2002 was comparable to
patients admitted to 55 other Australian ICUs during the same time period. These
three major diagnoses were selected because they were easily matched between the
APACHE II and III scoring system that was used for the RPHICU cohort and other
Australian ICUs, respectively. The comparisons showed that patients in the RPHICU
were younger and with less co-morbidities when compared to the patients in the other
Australian ICUs. The severity of acute illness (i.e. the APACHE II predicted
mortality) and in-hospital survival function were, however, comparable between the
RPHICU cohort and other Australian ICUs in patients with these three major
diagnoses (Table 1 to 3 and Figure 1 to 3).56
With these limitations in mind, not all the
results of this thesis may be generalisable to other Australian ICUs.
20
Table 1. Characteristics of severe sepsis or septic shock admissions to Royal Perth
Hospital ICU (RPHICU) and other Australian ICUs.
Variable RPHICU (n=111) Other Australian ICUs (n=1,429) P value
#
Age, yrs (SD) 54.6 (16.9) 60.1 (17.9) 0.001
Male / female, no. (%) 54 (48.6) / 57 (51.4) 792 (55.4) / 637 (44.6) 0.198
APACHE II score 22.0 22.0 0.900
(SD, median, IQR) (7.9, 22.0, 11.0) (9.8, 21.0, 13.7)
APACHE II predicted 45.7 45.6 0.776
mortality, % (23.2, 45.2, 37.6) (26.5, 41.6, 43.6)
(SD, median, IQR)
Chronic respiratory 2 (1.8) 126 (8.8) 0.006
disease, no. (%)
Chronic cardiovascular 1 (0.9) 140 (9.8) 0.001
disease, no. (%)
Chronic renal disease, 3 (2.7) 105 (7.3) 0.079
no. (%)
Chronic liver disease, 0 (0) 59 (4.1) 0.019
no. (%)
Immunosuppressed state, 7 (6.3) 185 (12.9) 0.05
no. (%)
Length of ICU stay, 9.9 5.1 0.001
days (SD, median, IQR) (13.1, 5.1, 7.0) (7.7, 2.4, 4.9)
Length of hospital stay, 26.4 17.3 0.001
days (SD, median, IQR) (23.6, 18.0, 24.0) (23.7, 9.9, 16.1)
ICU mortality, no. (%) 24 (21.6) 319 (23.0) 0.815
28-day in-hospital 28 (23.4) 355 (27.9) 0.582
mortality, no. (%)
Hospital mortality, 35 (31.5) 417 (30.7) 0.832
no. (%)
# P values were generated by either Mann-Whitney or chi-square test. IQR, interquartile range.
21
Figure 1. Kaplan Meier survival curve of the patients with severe sepsis or septic
shock from Royal Perth Hospital ICU (RPHICU) and other Australian ICUs.
25 20 15 10 5 0
Days since ICU admission
1.0
0.8
0.6
0.4
0.2
0.0
RPHICU
Other Australian ICUs
Cumulative survival
Survival difference between
the two cohorts was
insignificant, p=0.194
by log rank test
22
Table 2. Characteristics of pneumonia admissions to Royal Perth Hospital ICU
(RPHICU) and other Australian ICUs.
Variable RPHICU (n=82) Other Australian ICUs (n=1,066) P value #
Age, yrs (SD) 56.1 (15.7) 61.1 (17.8) 0.003
Male / female, no. (%) 47 (57.3) / 35 (42.7) 588 (55.2) / 477 (44.7) 0.731
APACHE II score 19.0 19.3 0.798
(SD, median, IQR) (7.2, 20.0, 9.3) (8.1, 19.0, 10.0)
APACHE II predicted 35.4 35.7 0.798
mortality, % (19.9, 35.5, 28.3) (22.2, 32.2, 31.0)
(SD, median, IQR)
Chronic respiratory 8 (9.8) 206 (19.3) 0.038
disease, no. (%)
Chronic cardiovascular 1 (1.2) 93 (8.7) 0.011
disease, no. (%)
Chronic renal disease, 3 (3.7) 27 (2.5) 0.469
no. (%)
Chronic liver disease, 0 (0) 25 (2.3) 0.251
no. (%)
Immunosuppressed state, 5 (6.1) 101 (9.4) 0.427
no. (%)
Length of ICU stay, 10.2 6.9 0.001
days (SD, median, IQR) (12.6, 7.0, 8.3) (9.9, 3.6, 6.6)
Length of hospital stay, 21.4 18.7 0.008
days (SD, median, IQR) (21.0, 15.0, 11.8) (39.0, 11.4, 13.5)
ICU mortality, no. (%) 13 (15.9) 169 (16.2) 1.000
28-day in-hospital 18 (22.0) 190 (20.2) 0.671
mortality, no. (%)*
Hospital mortality,* 20 (24.4) 230 (23.0) 0.786
no. (%)
# P values were generated by either Mann-Whitney or chi-square test. * ICU and hospital mortality
outcome of other Australian ICUs cohort was available only in 1,039 and 997 patients, respectively.
IQR, interquartile range.
23
Figure 2. Kaplan Meier survival curve of the patients with pneumonia from Royal
Perth Hospital ICU (RPHICU) and other Australian ICUs.
25 20 15 10 5 0
Days since ICU admission
1.0
0.8
0.6
0.4
0.2
0.0
RPHICU Other Australian ICUs
Cumulative survival
Survival difference between
the two cohorts was insignificant, p=0.860
by log rank test
24
Table 3. Characteristics of non-operative head and multiple trauma admissions to
Royal Perth Hospital ICU (RPHICU) and other Australian ICUs.
Variable RPHICU (n=176) Other Australian ICUs (n=2,114) P value #
Age, yrs (SD) 35.9 (16.3) 42.6 (19.3) 0.001
Male / female, no. (%) 137 (77.8) / 39 (22.2) 1599 (75.6) / 515 (24.4) 0.583
APACHE II score 14.6 12.4 0.001
(SD, median, IQR) (7.2, 13.0, 9.8) (7.6, 11.0, 10.0)
Glasgow Coma Scale 9.8 11.7 0.001
within first 24 hrs (4.7, 14.0, 7.0) (4.4, 15.0, 6.0)
(SD, median, IQR)
APACHE II predicted 12.9 11.8 0.124
mortality, % (15.1, 6.3, 12.1) (14.1, 6.2, 12.4)
(SD)
Chronic respiratory 1 (0.6) 58 (2.7) 0.084
disease, no. (%)
Chronic cardiovascular 0 (0) 43 (2.0) 0.074
disease, no. (%)
Chronic renal disease, 0 (0) 3 (0.1) 1.000
no. (%)
Chronic liver disease, 0 (0) 12 (0.6) 0.616
no. (%)
Immunosuppressed state, 0 (0) 38 (1.8) 0.113
no. (%)
Length of ICU stay, 8.9 4.7 0.001
days (SD, median, IQR) (9.3, 4.0, 9.8) (7.1, 2.0, 4.7)
Length of hospital stay, 25.4 19.4 0.001
days (SD, median, IQR) (60.7, 18.0, 26.8) (47.5, 8.0, 16.7)
ICU mortality, no. (%)* 17 (9.7) 163 (8.0) 0.472
28-day in-hospital 18 (10.2) 195 (9.7) 0.791
mortality, no. (%)*
Hospital mortality,* 20 (11.4) 210 (10.5) 0.701
no. (%)
# P values were generated by either Mann-Whitney or chi-square test. * ICU and hospital mortality
outcome of other Australian ICUs cohort was available only in 2,031 and 2,010 patients, respectively.
IQR, interquartile range.
25
Figure 3. Kaplan Meier survival curve of the patients with non-operative head and
multiple trauma from Royal Perth Hospital ICU (RPHICU) and other Australian
ICUs.
Cumulative survival
2520151050
Days since ICU admission
1.0
0.8
0.6
0.4
0.2
0.0
RPHICU
Other Australian ICUs
Survival difference between
the two cohorts was insignificant, p=0.470
by log rank test
Cumulative survival
26
Statistical Methods
A variety of standard statistical methods were used to model the effects of
different risk factors on outcomes of critically ill patients and to assess model
performance in this thesis.
Logistic regression was used to generate predicted probability (or risk) of any
predictive models that model on a categorical outcome variable. Propensity score
method was used to assess and adjust for the effect of selection bias in one of the
studies when missing data was a significant problem.57
The discrimination of a
scoring system was assessed by area under the receiver operating characteristic
(ROC) curve, and the difference in area under the ROC curves derived from the same
cases was assessed according to the method suggested by Hanley and McNeil.58
The calibration of a scoring system was assessed by the shape of the
calibration curve and Hosmer-Lemeshow Chi-square statistics.59
The slope and
intercept of the calibration curve derived from patients of different diagnostic
subgroups were computed to assess the uniformity of fit of the scoring system across
different diagnostic subgroups. The uniformity of fit of a prognostic model across
different subgroups of patients was then compared with the results of meta-analytic
techniques that utilised funnel and forest plots to assess calibration of the model.60
These latter techniques in assessing uniformity of fit in calibration across different
subgroups of patients have not been previously reported in the literature.
In developing a scoring system to predict median survival time and long term
survival probabilities of critically ill patients, a Cox proportional hazards regression
model was fitted.61
The proportional hazards assumption of the predictors in the Cox
model was checked by plotting the logarithm of the negative logarithm of the Kaplan
Meier survivor estimates. Predictors were pre-selected according to clinical
27
plausibility and also data from the literature instead of significance of the p-value of a
predictor in the univariable analysis.62,63
During the modelling process, categorising
continuous predictors was avoided and a non-linear relationship with hazard of death
was allowed by using a 6-knot restricted cubic spline function.62,63
The discrimination
performance of the Cox model was assessed with the c-index, which is a
generalisation of the c-statistic, that allowed for censored data and was computed and
adjusted for optimism (arising from using the same data to develop the model and
assess its performance) by a bootstrap technique to penalise for possible over-fitting,
with 200 re-samples and at least 200 patients per risk group.63,64
Model calibration of
the Cox model (similarity of predicted risks and proportions actually dying) was
assessed graphically and used a bootstrap re-sampling to construct a bias-corrected
calibration curve.63
Splitting of the sample into development and validation data sets
was not used in this thesis because this technique was regarded as data ‘inefficient’
and not as accurate as bootstrapping technique.63
A nomogram was presented to
illustrate how the Cox model can be used to generate median survival time and long
term survival probabilities of a heterogenous group of critically ill patients.63
Whenever possible, the overall performance of the scoring system was assessed by
Nagelkerke R2
and Brier’s score.65-67
A p-value less than 0.05 was regarded as significant and all tests were two-
tailed in this thesis. No adjustment was made for multiple comparisons in the
subgroup analyses because of the small sample size of the subgroups. All statistical
analyses were performed by SPSS statistical software (version 13.0 for Windows,
SPSS Inc. USA) and the Cox model was constructed by using the Design library in S-
PLUS software (version 8.0, 2007, Insightful Corp.; Seattle, Washington, USA).
28
There were a total of 26,021 admissions to RPHICU between 1987 and 2002
with a hospital survival rate of 89.2%. The power of the survival analysis of ICU
cohorts depends on the total number of outcome events, that is, mortality. Power
calculations for the survival analysis showed that a total of 200 events provides an
88% power to detect a relative risk (RR) of 1.25 for a continuous risk factor (eg
APACHE score or predicted mortality, Charlson co-morbidity index), a total of 200
events provided around 95% power to detect a relative risk of 2.0 for a binary risk
factor (eg the presence of a certain co-morbidity) that has a prevalence of 10%, and a
total of 250 events provides an 86% power to detect a RR of 1.5 for a binary risk
factor that has a prevalence between 25 and 50% (eg gender, received mechanical
ventilation). Therefore, the sample size of this cohort would be adequate for
predictive modelling with multiple predictive variables.
Ethics approval
The RPHICU database contains clinical data that are identified by patient
medical record numbers. Patient medical record numbers were used to retrieve the
data on inflammatory markers of some ICU admissions in two studies that evaluated
the association between inflammatory markers and outcomes after ICU discharge.
The conduct of the studies was reviewed and found to meet with the approval of the
Hospital Ethics Committee or representative on the basis that they were clinical
audits. For studies involving linked data and data from the Aboriginal patients, they
were approved by the WA Confidentiality of Health Information Committee (CHIC)
and the WA Aboriginal Health Information and Ethics Committee, respectively.
(Appendices on page 156-160).
29
Section two: Assessment of the APACHE II scoring
system in an Australian context
Chapter 3. The worst first 24-hour and admission APACHE II
scoring system
The APACHE II scoring system is widely used in many Australian ICUs for audit
and clinical research purposes.54,55
The APACHE II scoring system has, however, not
been thoroughly evaluated in an Australian ICU. In this chapter, the performance of the
APACHE II scoring system in a tertiary Australian ICU was evaluated. The results
showed that the performance of the APACHE II scoring system was similar to the
original APACHE II cohort and reports from other ICUs.24
The overall discrimination as
measured by area under the ROC curve was 0.85 with 95% confidence interval [CI]:
0.84-0.86 and the Standardised Mortality Ratio [SMR] was 0.84 with 95%CI: 0.80-0.88.
The performance of the APACHE II model in RPHICU also appeared to be stable
between 1993 and 2003 without significant changes over time as reported by some other
ICUs. This was different from what had been described in other ICUs. The possible
explanations may include the high quality data collection process of the RPHICU since
the inception of the database in 1987, stability in case mix at RPHICU, or changes in case
mix being offset by improvement in care.68
When the performance of the APACHE II scoring system in different diagnostic
subgroups was considered, the discrimination of the scoring system was least satisfactory
in two major diagnostic subgroups; the subgroup with sepsis, pneumonia, gastrointestinal
obstruction / perforation for which the area under ROC curve was 0.68 and also the
30
cardiac arrest subgroup for which the area under the ROC curve was 0.74. Using the
SMR as a guide to assess the model calibration across different subgroups, the APACHE
II scoring system appeared to be least well calibrated in the subgroups of patients with
multiple trauma (SMR 1.24, 95%CI: 1.17-1.31) and those transferred directly to RPHICU
from another hospital (SMR 0.71, 95%CI: 0.67-0.75).
The traditional APACHE II scoring system uses physiological data values
recorded as the worst values over the first 24 hours after ICU admission. The collection
of physiological data on admission only is logistically easier and is used by some ICUs.
The performance of a modified APACHE II scoring system using the admission
physiology and laboratory data to replace the worst first 24-hour data (the admission
APACHE II scoring system) was assessed and compared with the traditional worst first
24-hour APACHE II scoring system. The discrimination of the admission APACHE II
scoring system both overall and within different diagnostic subgroups was considered
satisfactory. The calibration of the admission APACHE II scoring system, as illustrated
by the calibration curve and SMR, was better than the worst first 24-hour APACHE II
scoring system in most patient subgroups. The only exception was the group of patients
with multiple trauma among whom the model calibration deteriorated by using the
admission physiology and laboratory data.
While there are some limitations with the admission APACHE II scoring
system as discussed in this study, the results showed that it is valid to use this modified
APACHE II scoring system as an alternative risk adjustment tool for critically ill non-
trauma patients.
31
In conclusion, the APACHE II scoring system had a satisfactory overall
performance in a major tertiary Australian ICU and its performance was stable between
1993 and 2003. Its performance was, however, not uniform across all different patient
subgroups. The scoring system’s discrimination appeared to be least satisfactory among
patients with sepsis, pneumonia, gastrointestinal obstruction / perforation and cardiac
arrest. Using admission physiology and laboratory data to replace the worst first 24-hour
data to generate the admission APACHE II predicted mortality (the admission APACHE
II scoring system) appeared to be a viable and simple alternative to the worst 24-hour
APACHE II scoring system as an audit and risk adjustment tool for critically ill non-
trauma patients.
The details of this study are contained in the following published article:
Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, Webb SA. A comparison of
admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II scores
in predicting hospital mortality: a retrospective cohort study. Critical Care 2006;10:R4.
Available online http://ccforum.com/content/10/1/R4
Open AccessVol 10 No 1ResearchA comparison of admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II scores in predicting hospital mortality: a retrospective cohort studyKwok M Ho1,2,3, Geoffrey J Dobb4,5, Matthew Knuiman6, Judith Finn7, Kok Y Lee8 and Steven AR Webb8,9
1Consultant Intensivist Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia2PhD candidate, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia3PhD candidate, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia4Acting Head of the Department, Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia5Associate Professor, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia6Professor, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia7Senior Lecturer, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia8Consultant Intensivist, Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia9Senior Lecturer, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia
Corresponding author: Kwok M Ho, [email protected]
Received: 17 Aug 2005 Revisions requested: 26 Sep 2005 Revisions received: 6 Oct 2005 Accepted: 26 Oct 2005 Published: 25 Nov 2005
Critical Care 2006, 10:R4 (doi:10.1186/cc3913)This article is online at: http://ccforum.com/content/10/1/R4© 2005 Ho et al.; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
Introduction The Acute Physiology and Chronic HealthEvaluation (APACHE) II score is widely used in the intensivecare unit (ICU) as a scoring system for research and clinicalaudit purposes. Physiological data for calculation of theAPACHE II score are derived from the worst values in the first24 hours after admission to the ICU. The collection ofphysiological data on admission only is probably logisticallyeasier, and this approach is used by some ICUs. This studycompares the performance of APACHE II scores calculatedusing admission data with those obtained from the worst valuesin the first 24 hours.
Materials and Methods This was a retrospective cohort studyusing prospectively collected data from a tertiary ICU. Therewere no missing physiological data and follow-up for mortalitywas available for all patients in the database. The admission andthe worst 24-hour physiological variables were used to generatethe admission APACHE II score and the worst 24-hourAPACHE II score, and the corresponding predicted mortality,respectively.
Results There were 11,107 noncardiac surgery ICU admissionsduring 11 years from 1 January 1993 to 31 December 2003.The mean admission and the worst 24-hour APACHE II scorewere 12.7 and 15.4, and the derived predicted mortalityestimates were 15.5% and 19.3%, respectively. The actualhospital mortality was 16.3%. The overall discrimination ability,as measured by the area under the receiver operatingcharacteristic curve, of the admission APACHE II model(83.8%, 95% confidence interval = 82.9–84.7) and the worst24-hour APACHE II model (84.6%, 95% confidence interval =83.7–85.5) was not significantly different (P = 1.00).
Conclusion Substitution of the worst 24-hour physiologicalvariables with the admission physiological variables to calculatethe admission APACHE II score maintains the overalldiscrimination ability of the traditional APACHE II model. Theadmission APACHE II model represents a potential alternativemodel to the worst 24-hour APACHE II model in critically illnontrauma patients.
IntroductionScoring systems such as Acute Physiology and ChronicHealth Evaluation (APACHE), the Therapeutic Intervention
Scoring System, and Mortality Probability Models (MPM) havebeen developed and used as quality assurance tools and forrisk stratification in research involving critically ill patients [1,2].
Page 1 of 8(page number not for citation purposes)
APACHE = Acute Physiology and Chronic Health Evaluation; CI = confidence interval; ICU = intensive care unit; MPM = Mortality Probability Models; SAPS = Simplified Acute Physiology Score.
Critical Care Vol 10 No 1 Ho et al.
Each scoring system has its own strengths and weaknesses,and the choice depends on the system's ease of use andgoodness of fit for that particular intensive care unit (ICU) orpatient group.
The traditional APACHE II model utilises the worst values of12 physiological variables during the first 24 hours followingICU admission, along with an evaluation of the patient'schronic health and admission diagnosis to calculate theAPACHE II predicted mortality [3]. The APACHE II model hasbeen widely validated and used by many ICUs to classify theseverity of illness and to predict hospital mortality [2,4-7].APACHE II has now been modified to APACHE III; however,some studies have shown that APACHE III may underestimatethe number of deaths [8,9]. Although the APACHE II model isquite old, and other scoring systems have been developedusing more recent cohorts, APACHE II is still widely used forresearch and clinical audit purposes. APACHE II is easier touse than APACHE III and has been in use for a long period,which allows consistency [2,10].
A potential problem with these methods is that the worst 24-hour physiological data used to derive APACHE II scores andAPACHE III scores can be treatment-dependent and thereforeit may reflect poor clinical management rather than sickerpatients [11-13]. Collection of the admission physiologicalvariables rather than the worst 24-hour physiological variablesis a standard practice in some ICUs to calculate the APACHEII predicted mortality, and may theoretically overcome thispotential problem [14,15]. The use of only admission physio-logical variables may make data collection easier as the datacollector does not need to peruse all the blood tests and phys-iological variables over 24 hours to work out the worst score.However, the performance of APACHE II scores using admis-sion data has not been thoroughly assessed [3,16].
When the APACHE III scoring system was developed, theeffect of using admission physiological variables rather thanthe worst 24-hour physiological variables was assessed. Theabsolute difference between the mean scores, derived fromthe admission and worst 24-hour physiological data, was notstatistically significantly different from zero [16]. However, theproportion of missing values favoured the worst 24-hour val-ues over the admission values, as did the maximum explana-tory power. Some other scoring systems use only admissiondata (MPM II0 and Simplified Acute Physiology Score [SAPS]III), and it is therefore established that scoring systems usingphysiological data from the time of admission to the ICU canprovide valid assessment of the severity of illness and out-come prediction [17,18].
In the present study we evaluated the performance of theAPACHE II model using physiological data at the time of ICUadmission with the model using data obtained from the worstvalues in the first 24 hours.
Materials and methodsThis was a retrospective cohort study that utilised prospec-tively collected data. The study was conducted in the medical–surgical ICU at Royal Perth Hospital, an 800-bed universityteaching hospital. The 22-bed ICU is a 'closed' ICU thatadmits critically ill adult patients of all specialties and is staffedby fully trained intensivists. The unit database contains de-identified information for components of the APACHE II scorefor physiological data collected at admission and for the worstvalues in the first 24 hours – admission diagnosis and source,age, ethnicity, ICU mortality and hospital mortality. The admis-sion and the worst 24-hour physiological data were used togenerate the admission APACHE II score and the worst 24-hour APACHE II score, respectively. The admission APACHEII score and the worst 24-hour APACHE II score were thenused to calculate the admission APACHE II predicted mortal-ity (admission APACHE II model) and the worst 24-hour pre-dicted mortality (worst 24-hour APACHE II model), using thepublished APACHE II mortality prediction equation coeffi-cients [3].
The data were collected by the duty ICU consultant on papersheets and updated on a daily basis by the duty consultantwhile the patient remained in the ICU. After the patient wasdischarged from the ICU, the data were checked for transcrip-tion errors and completeness by a designated trained clericalstaff member using data from the computerised laboratorydatabase, going through the ICU vital signs flow chart againbefore the data were transferred to the computer. A total of 12consultants were involved in collecting data, of which sevenwere involved throughout the study period, using a standard-ised data dictionary. The worst 24-hour APACHE II score wasdetermined precisely as described by Knaus and colleagues[3].
Measurement of all 12 physiological variables on admissionand over the first 24 hours in the ICU was mandatory in theAPACHE data recording form. If the patient was anaesthe-tised before ICU admission, the Glasgow coma score wasassessed using the available clinical information prior toanaesthesia. Acute renal failure was defined as oliguria withurine output less than 135 ml over a consecutive 8-hour periodwith abnormal serum creatinine concentrations over 133µmol/l. Other than the Glasgow coma score and urinary out-put, pre-ICU physiological data were not used in the calcula-tion of APACHE II scores. Arterial blood gas measurementswere judged to be inappropriate in some patients, and in thesepatients the serum bicarbonate concentration was used to cal-culate the physiological score [3]. One data custodian wasresponsible for ensuring data quality throughout the studyperiod. The data were reviewed for internal consistency beforeannual lockdown, and there were no patients with missingphysiological data or who were lost to mortality follow-up. Thestudy utilised de-identified data only and was deemed to be a
Page 2 of 8(page number not for citation purposes)
Available online http://ccforum.com/content/10/1/R4
'Clinical Audit' by the Hospital Ethics Committee and as suchthe need for formal ethics committee approval was waived.
The performance of the admission APACHE II model in pre-dicting hospital mortality was compared with the performanceof the worst 24-hour APACHE II model with respect to theirdiscrimination ability and calibration. Because the originalAPACHE II prediction model did not include cardiac surgicalpatients, we have included only the data from noncardiac sur-gery ICU admissions. All patients in the database in the studyperiod were considered, including those patients who diedwithin 24 hours of ICU admission.
The discrimination ability of each of the scoring systems wasassessed by the area under the receiver operating character-istic curve: above 90% was regarded as excellent, above 80%
was regarded as good, and below 80% was regarded as poorin this study. Calibration was assessed by comparing absoluteobserved mortality with predicted mortality in fixed risk strata(for example 0–0.099, 0.1–0.199, and so on) using the Hos-mer-Lemeshow chi-square H statistic. P < 0.05 in the Hos-mer-Lemeshow chi-square H statistical test infers a significantdeparture from the null hypothesis of good calibration. Therelationship between the admission APACHE II predicted hos-pital mortality risk and the worst 24-hour APACHE II predictedhospital mortality risk was assessed by the two-tailed Pearsoncorrelation coefficient. The ratio of total observed to predictedmortality is the standardised mortality ratio (SMR).
The discrimination ability was further analysed for differentdiagnostic and patient subgroups to test the uniformity of fit ofboth models. The diagnostic subgroups analysed included
Table 1
Characteristics of the cohort
Variables Mean (SD)
Age (years) 53.5 (19.5)
Male/female (%) 6,871/4,236 (61.9/38.1)
Admission source (%)
Operating room 4,885 (44.0)
Recovery room 638 (5.7)
Emergency department 2,976 (26.8)
Ward 1,481 (13.3)
Another hospital 1,127 (10.1)
Primary organ failure (%)
Cardiovascular 3,693 (33.2)
Neurological 3,893 (35.0)
Respiratory 2,682 (24.1)
Gastrointestinal 401 (3.6)
Renal 167 (1.5)
Metabolic 217 (2.0)
Haematological 49 (0.4)
ICU stay (days) 5.1 (7.8)
Hospital stay (days) 21.1 (29.3)
Admission APACHE II score 12.7 (7.3)
Worst 24-hour APACHE II score 15.4 (7.9)
Admission APACHE predicted mortality (%) 15.5 (19.1)
Worst 24-hour APACHE predicted mortality (%) 19.3 (22.1)
Actual ICU mortality (%) 12.0
Actual hospital mortality (%) 16.3
All data in parentheses are standard deviations unless stated otherwise. APACHE, Acute Physiology and Chronic Health Evaluation; ICU, intensive care unit; SD, standard deviation.
Page 3 of 8(page number not for citation purposes)
Critical Care Vol 10 No 1 Ho et al.
patients with different major diagnoses such as sepsis, pneu-monia, and gastrointestinal perforation or obstruction, intracra-nial haemorrhage, multiple trauma, cardiac arrest, and electivesurgery. The patient subgroups analysed included aboriginalpatients, patients transferred from another hospital, patientsadmitted to the ICU before or after early 1999, patients whostayed in the ICU longer than 24 hours, and patients who sur-vived longer than 24 hours of hospitalisation. P < 0.05 wasregarded as significant in all analyses and no adjustment wasmade for multiple comparisons in the subgroup analyses. Allstatistical analyses were performed by SPSS statistical soft-ware (version 11.0 for Windows; SPSS Inc., Chicago, IL,USA] and confidence intervals were generated by ConfidenceInterval Analysis (version 2.0.0; BMJ 2000, UK).
ResultsThe time for collecting and checking the admission physiolog-ical data manually required an average of 5 minutes per patient(range, 3–7 minutes), and the average for the worst 24-hourphysiological data was 20 minutes per patient (range, 10–40minutes). The time required to work out the worst 24-hourAPACHE II score was longer when more blood tests had beenperformed for the patient.
There were 11,107 noncardiac surgery ICU admissions in the11-year period from 1 January 1993 to 31 December 2003.The characteristics of the ICU cohort are presented in Table 1.The difference in the admission APACHE II score and theworst 24-hour APACHE II score was small in most patients(Figure 1). The mean admission APACHE II score and theworst 24-hour APACHE II scores were 12.7 and 15.4, and thederived predicted hospital mortality estimates were 15.5%and 19.3%, respectively. The admission APACHE II predictedmortality and the worst 24-hour APACHE II predicted mortalitywere closely correlated (Pearson correlation coefficient =0.955, P = 0.0001). The actual hospital mortality was 16.3%.The overall standardised mortality ratio was 1.05 (95% confi-dence interval [CI] = 1.00–1.10) and was 0.84 (95% CI =0.80–0.88) using the admission APACHE II predicted mortal-ity and the worst 24-hour APACHE II predicted mortality as thedenominator, respectively.
The overall discrimination abilities, as measured by the areaunder the receiver operating characteristic curve, of the admis-sion APACHE II model (83.8%, 95% CI = 82.9–84.7) and theworst 24-hour APACHE II model (84.6%, 95% CI = 83.7–85.5) with the entire cohort were not significantly different (P
Table 2
The discriminating ability of the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model to predict inhospital mortality in different diagnostic and patient subgroups
Different diagnostic and patient subgroups
Number of patients
Mean area under the ROC curve (%) (95% confidence interval)a
Standardised mortality ratio (95% confidence interval)
Admission model Worst 24-hour model Admission model Worst 24-hour model
Sepsis, pneumonia, gastrointestinal perforation or obstruction
1,474 68.3 (65.4–71.3) 68.5 (65.6–71.4) 0.94 (0.90–0.98) 0.77 (0.75–0.80)
Intracranial, subdural or subarachnoid haemorrhage
851 79.5 (76.3–82.7) 80.4 (77.2–83.5) 1.29 (1.22–1.36) 1.03 (0.98–1.08)
Multiple trauma 1,299 87.0 (84.1–89.9) 87.3 (84.4–90.1) 1.73 (1.63–1.84) 1.24 (1.17–1.31)
Cardiac arrest (nonoperative or intraoperative)
395 73.9 (69.1–78.8) 73.9 (69.0–78.8) 0.92 (0.88–0.96) 0.82 (0.79–0.85)
Elective surgery (excluding cardiac surgery)
3,012 78.6 (74.8–82.4) 80.8 (77.3–84.4) 1.04 (1.00–1.09) 0.79 (0.76–0.83)
Aboriginal patients 863 77.8 (74.2–81.4) 78.8 (75.2–82.3) 1.02 (0.95–1.09) 0.82 (0.77–0.87)
Patients transferred from another hospital
1,127 79.4 (76.3–82.4) 80.4 (77.4–83.5) 0.87 (0.82–0.92) 0.71 (0.67–0.75)
Patients admitted between 1993 and early 1999
5,553 85.4 (84.0–86.7) 86.1 (84.8–87.4) 1.05 (1.01–1.09) 0.85 (0.82–0.88)
Patients admitted between early 1999 and 2003
5,554 83.3 (82.0–84.5) 84.1 (82.8–85.3) 1.09 (1.06–1.13) 0.88 (0.86–0.91)
Patients stayed in the ICU longer than 24 hours
8,461 80.4 (79.2–81.5) 81.2 (80.1–82.3) 0.99 (0.97–1.02) 0.79 (0.77–0.81)
Patients survived longer than 24 hours of hospitalisation
10,733 82.2 (81.1–83.2) 83.0 (82.0–84.0) 0.93 (0.91–0.95) 0.74 (0.73–0.76)
aThere was no significant difference in the areas under the receiver operating characteristic (ROC) curves between the admission APACHE II model and the worst 24-hour APACHE II model (P = 1.00).
Page 4 of 8(page number not for citation purposes)
Available online http://ccforum.com/content/10/1/R4
= 1.00) (Figure 2). The discrimination abilities of the admissionAPACHE II model and the worst 24-hour APACHE II modelwere also not significantly different within all subgroups ana-lysed (Table 2).
The Hosmer and Lemeshow goodness of fit chi-square H sta-tistic was 66.7 for the admission APACHE II model and was189.3 for the worst 24-hour APACHE II model indicating abetter fit for the admission APACHE II model but both P valueswere very small (P < 0.0001). The calibration curve of the two
APACHE II models is displayed in Figure 3 and shows the bet-ter fit of the admission APACHE II model especially in the highrisk strata. The overall correct classification rate (based onclassifying a patient to die if his/her predicted mortality riskexceeded 50%) for the admission APACHE II model and theworst 24-hour APACHE II model were both 85.4% (Table 3).
DiscussionThe advantages of the admission APACHE II modelOur results showed that the performance of the admissionAPACHE II model is no worse than the traditional worst 24-hour APACHE II model when there are no significant missingdata. These results were consistent with the results of otherstudies that assessed or utilised the admission APACHE IIscore to calculate the APACHE II predicted mortality [15-17].
The use of the admission APACHE II score to calculate theAPACHE II predicted mortality (admission APACHE II model)has a few potential advantages and may represent a viablealternative to the traditional APACHE II model. First, it canassess the risk of hospital death at ICU admission, as in theMPM II0 and SAPS III scoring systems that assess the risk ofhospital death at ICU admission [17,18]. The admissionAPACHE II model also shares these systems' advantages ofease of use, and, since they are independent of ICU treatment,may be more applicable for risk stratification in clinicalresearch and triage decisions [19]. The ability of a scoring sys-tem to stratify patient risk on admission to the ICU mayfacilitate stratification of patients into trials that assess earlyinterventions in critically ill patients.
Second, the data collection for the admission APACHE IImodel is less laborious than the worst 24-hour APACHE IImodel, as demonstrated in our data. It may also reduce errorsbecause it does not require perusal of a series of values toobtain the worst score. Nevertheless, this potential advantageis important only when a computerised information system isnot available and the data are collected manually.
Third, the admission APACHE II model may be a better reflec-tion of quality of care in the ICU because risk assessmentoccurs before any ICU therapy is instituted [12-14].
Finally, poor calibration with the worst 24-hour APACHE IImodel has been reported in many studies [20-22]. Our resultsconfirmed this problem of the worst 24-hour APACHE IImodel, with the predicted mortality being much higher than theactual mortality in the high-risk strata. The admission APACHEII model appeared to have reduced the overestimation of mor-tality in the high-risk strata and improved the calibration of theAPACHE II model in the present study. However, data oncalibration of the admission APACHE II model from otherstudies are lacking [15-17] and further studies in other set-tings will be needed to confirm this finding.
Figure 1
The difference in APACHE II scores using the admission and worst 24-hour physiological dataThe difference in APACHE II scores using the admission and worst 24-hour physiological data. AP, Acute Physiology and Chronic Health Evaluation.
Figure 2
The receiver operating characteristic (ROC) curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model in predicting hospital mortalityThe receiver operating characteristic (ROC) curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model in predicting hospital mortal-ity. Area under ROC curves: worst 24-hour APACHE II model, 84.6% (95% CI = 83.7–85.5); admission APACHE II model, 83.8% (95% CI = 82.9–84.7). No significant difference between the two areas under the ROC curves (P = 1.00).
Page 5 of 8(page number not for citation purposes)
Critical Care Vol 10 No 1 Ho et al.
Limitations of the admission APACHE II modelThe admission APACHE II model is a minor modification of theworst 24-hour APACHE II model and retains many intrinsicweaknesses and problems of the worst 24-hour APACHE IImodel. These weaknesses include errors arising from impre-cise principal diagnosis, lead time bias, and poor uniformity offit of the model. The admission APACHE II model, as withother ICU scoring systems such as the APACHE III model,needs an accurate diagnosis to accurately predict the hospitalmortality. The admission APACHE II model does not eliminatethis requirement.
The performance of the worst 24-hour APACHE II model isaffected by the source and timing of patient referral to the ICU,and it tends to underestimate the mortality of the patientsreferred from other ICUs or hospitals [23,24]. Our results weredifferent from these reports. This may be because manypatients were transferred from remote Western Australia andwere not fully resuscitated when they were admitted to theICU. The standardised mortality ratio of the patients trans-ferred from other hospitals, based on the admission APACHEII model in this study, was closer to unity than that of the worst24-hour APACHE II model (Table 2). The admission APACHEII model was associated with a lower lead time bias in thisstudy. The uniformity of fit in the discrimination ability of theadmission APACHE II model and the worst 24-hour APACHEII model was similarly poor in patients with sepsis, pneumonia,gastrointestinal perforation, and cardiac arrest, and also in theaboriginal patients. Both the worst 24-hour APACHE II modeland the APACHE III model were not well calibrated in predict-ing mortality in trauma patients [23,25,26]. Our results con-firmed this problem of the worst 24-hour APACHE II model,and the admission APACHE II model did not improve the per-formance of the worst 24-hour APACHE II model in this sub-group of patients.
Limitations of the studyThis was a single-centre study and these results may not begeneralisable to other ICUs [23]. Our observation that thestandardised mortality ratio calculated with the admissionphysiological variables was closer to unity than that calculatedwith the worst 24-hour values may be different in other units.Further evaluation of the admission APACHE II model in otherICUs is essential.
Also, this study did not directly compare the admissionAPACHE II model with other scoring systems that assess therisk of hospital mortality at ICU admission such as the MPM II0
Table 3
Classification table for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model to predict hospital mortality
Observed hospital mortality Predicted hospital mortality
No (n) Yes (n) % correct
Using the worst 24-hour APACHE II model
No 8,899 394 95.8
Yes 1,229 585 32.2
Overall percentage 85.4
Using the admission APACHE II model
No 8,966 327 96.5
Yes 1,293 521 28.7
Overall percentage 85.4
The cutoff value is 0.50.
Figure 3
Calibration curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II score and the worst 24-hour APACHE II score in predicting hospital mortality across different risk strataCalibration curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II score and the worst 24-hour APACHE II score in predicting hospital mortality across different risk strata. The Hosmer-Lemeshow goodness of fit chi-square H statistic for the admis-sion APACHE II predicted mortality and for the worst 24-hour APACHE II predicted mortality were 66.9 and 189.3, respectively (both P < 0.0001).
Page 6 of 8(page number not for citation purposes)
Available online http://ccforum.com/content/10/1/R4
and SAPS III models [17,18]. Whether the performance of theadmission APACHE II model is comparable with these scoringsystems remains uncertain and will be further investigated.
Critical illness is a dynamic process and therefore outcomeprediction based on a single time point such as ICU admis-sion, as in the admission APACHE II model, does not considerchanges in patients' clinical status over time and theirresponse to treatment. Serial predictions over a period of time,as in the APACHE III model, may improve prediction accuracyand clinical utilities, although acquiring these data continu-ously will be difficult in practice [27,28].
Finally, the admission APACHE II model, as with most otheroutcome prediction models, does not consider functional out-comes beyond survival [9].
ConclusionIn conclusion, substituting the worst 24-hour physiologicalvariables with the admission physiological variables to calcu-late the admission APACHE II score and the APACHE II pre-dicted mortality does not result in significantly worsecalibration or discrimination compared with the traditionalAPACHE II model. The admission APACHE II modelrepresents a potential alternative model to the worst 24-hourAPACHE II model in critically ill nontrauma patients.
Competing interestsThe authors declare that they have no competing interests.
Authors' contributionsKMH performed the statistical analysis and drafted the manu-script. GJD initiated the original idea of the study and helpedto draft the manuscript. MK, JF, and SARW helped analyse thedata and draft the manuscript. KYL was the data-collectionquality controller and helped to draft the manuscript. Allauthors read and approved the final manuscript.
AcknowledgementsThe authors would like to thank Dr Geoffrey Clarke and Dr John Weekes for their part in initiating the Royal Perth Hospital ICU database, and thank all ICU consultants who have been recording APACHE II data for every admission to the ICU. This study was solely funded by the Depart-ment of Intensive Care, Royal Perth Hospital.
References1. Knaus WA: APACHE 1978–2001: the development of a quality
assurance system based on prognosis: milestones and per-sonal reflections. Arch Surg 2002, 137:37-41.
2. Gunning K, Rowan K: ABC of intensive care: outcome data andscoring systems. BMJ 1999, 319:241-244.
3. Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: aseverity of disease classification system. Crit Care Med 1985,13:818-829.
4. Oh TE, Hutchinson R, Short S, Buckley T, Lin E, Leung D: Verifi-cation of the Acute Physiology and Chronic Health Evaluationscoring system in a Hong Kong intensive care unit. Crit CareMed 1993, 21:698-705.
5. Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD:Assessment of the performance of five intensive care scoringmodels within a large Scottish database. Crit Care Med 2000,28:1820-1827.
6. Breen D, Churches T, Hawker F, Torzillo PJ: Acute respiratoryfailure secondary to chronic obstructive pulmonary diseasetreated in the intensive care unit: a long term follow up study.Thorax 2002, 57:29-33.
7. Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP:Intensive Care Society's Acute Physiology and Chronic HealthEvaluation (APACHE II) study in Britain and Ireland: a prospec-tive, multicenter, cohort study comparing two methods for pre-dicting outcome for adult intensive care patients. Crit CareMed 1994, 22:1392-1401.
8. Buist M, Gould T, Hagley S, Webb R: An analysis of excess mor-tality not predicted to occur by APACHE III in an Australianlevel III intensive care unit. Anaesth Intensive Care 2000,28:171-177.
9. Angus DC: Scoring system fatigue...and the search for a wayforward. Crit Care Med 2000, 28:2145-2146.
10. Konarzewski W: Continuing to use APACHE II scores ensuresconsistency. BMJ 2000, 321:383-384.
11. Shann F: Mortality prediction model is preferable to APACHE.BMJ 2000, 320:714.
12. Boyd O, Grounds RM: Physiological scoring systems and audit.Lancet 1993, 341:1573-1574.
13. Knaus W, Draper E, Wagner D: APACHE III study design: ana-lytic plan for evaluation of severity and outcome in intensivecare unit patients. Introduction. Crit Care Med 1989,17:S176-S180.
14. Khilnani G, Banga A, Sharma S: Predictors of mortality ofpatients with acute respiratory failure secondary to chronicobstructive pulmonary disease admitted to an intensive careunit: a one year study. BMC Pulm Med 2004, 4:12. it is a full arti-cle but no page span because it does not have printed version,only Internet version
15. Goel A, Pinckney RG, Littenberg B: APACHE II predicts long-term survival in COPD patients admitted to a general medicalward. J Gen Intern Med 2003, 18:824-830.
16. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M,Bastos PG, Sirio CA, Murphy DJ, Lotring T, Damiano A, et al.:APACHE III prognostic system. Risk prediction of hospitalmortality for critically ill hospitalized adults. Chest 1991,100:1619-1636.
17. Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, CamposRA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR, on behalf ofthe SAPS 3 Investigators: SAPS 3-From evaluation of thepatient to evaluation of the intensive care unit. Part 1: Objec-tives, methods and cohort description. Intensive Care Med2005, 31:1336-1344.
18. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, RapoportJ: Mortality Probability Models (MPM II) based on an interna-tional cohort of intensive care unit patients. JAMA 1993,270:2478-2486.
19. Joynt GM, Gomersall CD, Tan P, Lee A, Cheng CA, Wong EL: Pro-spective evaluation of patients refused admission to an inten-sive care unit: triage, futility and outcome. Intensive Care Med2001, 27:1459-1465.
20. Carson SS, Bach PB: Predicting mortality in patients sufferingfrom prolonged critical illness: an assessment of four severity-of-illness measures. Chest 2001, 120:928-933.
Key messages
• Modifying the APACHE II model using admission physi-ological variables instead of worst 24-hour physiological variables to calculate the APACHE II score and pre-dicted mortality (admission APACHE II model) does not result in significantly worse calibration and discrimina-tion compared with the traditional APACHE II model in critically ill nontrauma patients.
Page 7 of 8(page number not for citation purposes)
Critical Care Vol 10 No 1 Ho et al.
21. Tan IK: APACHE II and SAPS II are poorly calibrated in a HongKong intensive care unit. Ann Acad Med Singapore 1998,27:318-322.
22. Arabi Y, Al Shirawi N, Memish Z, Venkatesh S, Al-Shimemeri A:Assessment of six mortality prediction models in patientsadmitted with severe sepsis and septic shock to the intensivecare unit: a prospective cohort study. Crit Care 2003,7:R116-R122.
23. Cowen JS, Kelly MA: Errors and bias in using predictive scoringsystems. Crit Care Clin 1994, 10:53-72.
24. Combes A, Luyt CE, Trouillet JL, Chastre J, Gibert C: Adverseeffect on a referral intensive care unit's performance ofaccepting patients transferred from another intensive careunit. Crit Care Med 2005, 33:705-710.
25. Zimmerman JE, Wagner DP, Draper EA, Wright L, Alzola C, KnausWA: Evaluation of acute physiology and chronic health evalu-ation III predictions of hospital mortality in an independentdatabase. Crit Care Med 1998, 26:1317-1326.
26. Chawda MN, Hildebrand F, Pape HC, Giannoudis PV: Predictingoutcome after multiple trauma: which scoring system? Injury2004, 35:347-358.
27. Afessa B, Keegan MT, Mohammad Z, Finkielman JD, Peters SG:Identifying potentially ineffective care in the sickest critically illpatients on the third ICU day. Chest 2004, 126:1905-1909.
28. Wagner DP, Knaus WA, Harrell FE, Zimmerman JE, Watts C: Dailyprognostic estimates for critically ill adults in intensive careunits: results from a prospective, multicenter, inception cohortanalysis. Crit Care Med 1994, 22:1359-1372.
Page 8 of 8(page number not for citation purposes)
40
Section two: Assessment of the APACHE II scoring
system in an Australian context
Chapter 4. The use of the APACHE II scoring system for the
indigenous patients
Indigenous Australians are over-represented in ICU admissions in the Northern
Territory (28% of the population but 45% of all ICU admissions),69
but there is little
information on their pattern of critical illness and outcomes from other parts of Australia.
The APACHE II scoring system has been used for risk adjustment purposes for many
critically ill patients and has a reasonable discrimination with the RPHICU cohort. This
study hypothesised that the APACHE II scoring system will have a similar performance
when applied to critically ill indigenous and non-indigenous patients.
This chapter examines and compares the patterns of critical illness of the
indigenous patients with the non-indigenous patients in the Western Australia RPHICU
cohort and assesses whether the APACHE II scoring system is a reasonable risk
adjustment tool for critically ill indigenous patients. First, indigenous Australians were
over-represented in RPHICU admissions comprising 3.2% of the population but 6.4% of
all ICU admissions. The pattern of their critical illness was also very different from other
patients. Compared with non-indigenous patients, the indigenous patients were younger
and more likely to have been transferred from another hospital. ICU admissions due to
respiratory or renal failure, sepsis, pneumonia, trauma, and cardiopulmonary arrest with a
higher severity of acute illness were more common among indigenous patients leading to
41
a significantly longer length of ICU stay. Chronic liver and renal diseases were also more
common among the indigenous patients.
Second, during the 11 year-period from 1993 to 2003 there was a progressive
increase in emergency ICU admissions among indigenous patients. If emergency ICU
admission is regarded as an “ambulance at the bottom of a cliff” with many patients
admitted to the ICU only after other layers of the health care system have failed to
reverse or prevent the critical illness, this result raised the concern that primary and
preventive care services for the indigenous patients might have deteriorated over this
period. Another possible explanation for the increase in emergency admissions would be
better access to intensive care service in WA.
Third, with the difference in sample size in mind, the APACHE II scoring system
has a similar calibration curve for indigenous and non-indigenous patients, especially in
patients with less severe acute diseases (predicted mortality <50%). The discrimination of
the model for critically ill indigenous patients (area under the ROC curve 0.79, 95%CI:
0.75-0.82) was, however, slightly less satisfactory than for non-indigenous patients. This
was most likely due to the different disease pattern of the indigenous patients, among
whom sepsis, pneumonia, cardiac arrest, and transfer from another hospital were very
common. As discussed in Chapter 3, the overall discrimination of the APACHE II
scoring system was also less satisfactory among these diagnostic and patient subgroups.
In conclusion, as for many other illnesses, the pattern of ICU utilisation differs
between indigenous and non-indigenous Australians. Critical illnesses requiring
emergency ICU admission are increasing among indigenous Western Australians. The
performance of the APACHE II scoring system appears to be less satisfactory among the
42
indigenous Australians, and this is most likely due to a larger proportion of them being
transferred from another hospital and the different disease pattern. Therefore, the
limitations of the APACHE II scoring system should be considered when the model is
used as a risk adjustment tool for critically ill indigenous Australians.
Further details of this study are contained in the following published article:
Ho KM, Finn J, Dobb GJ, Webb SA. The outcome of critically ill Indigenous
patients. Medical Journal of Australia 2006;184:496-9.
47
Section two: Assessment of the APACHE II scoring
system in an Australian context
Chapter 5. Assessing calibration by meta-analytic techniques
It is important to understand that the performance of a scoring system may vary
significantly depending on the characteristics of the cohort among whom the model is
applied. This may explain why a prognostic scoring system can be reported to perform
very well in some ICUs and not so well in other ICUs. The results in Chapters 3 and 4
demonstrated that the performance of the APACHE II scoring system was less
satisfactory in some diagnostic and patient subgroups in the RPHICU cohort.
Most researchers use Standardised Mortality Ratio (SMR) of different subgroups
of patients to illustrate whether a prognostic scoring system is well calibrated across these
different subgroups of patients or with good uniformity of fit. This method generates a
complicated table of numerical data that can be difficult to interpret and hard to see clear
patterns.
Forest plots are commonly used in meta-analyses to combine quantitative results
from several different studies and funnel plots are useful in identifying publication bias.
In this chapter, the variation in the calibration of the APACHE II scoring system across
different diagnostic and patient subgroups is explored through the use of forest and
funnel plots. As discussed in Chapter 3, patients with multiple trauma and to a lesser
extent sepsis or pneumonia appeared to be least well-calibrated in the APACHE II
scoring system according to the SMR method. In this chapter the application of forest and
funnel plots to these data demonstrated visually that the calibration of the APACHE II
48
scoring system was poor for these subgroups of patients. The fact that the APACHE II
scoring system was poorly calibrated for patients with multiple trauma was also
confirmed by computing the slope and intercept of the calibration curve for these
subgroups of patients. The applications of the forest and funnel plots were further
expanded to other grouping classifications which revealed that the APACHE II scoring
system was not well calibrated for patients older than 80 years with the RPHICU patients.
This study demonstrates the feasibility and utility of using forest and funnel plots
in testing uniformity of fit in calibration of a scoring system as an alternative to
complicated tables of SMR data. It should, however, be noted that a type II error could
affect the interpretation of forest and funnel plots. That is, the ability of a funnel plot to
detect poor calibration among some subgroups of patients could be unreliable when the
sample size of the cohort is small, a problem similar to using Hosmer-Lemeshow chi-
square statistics to assess model calibration.
In conclusion, this study has introduced an alternative method to illustrate
uniformity of fit or variation in the calibration of a prognostic scoring system across
different subgroups of patients. The funnel plot illustrated visually that the APACHE II
scoring system was not well calibrated in patients with multiple trauma and patients older
than 80 years with the RPHICU cohort. This alternative approach may be useful in
situations when the sample size of the cohort is not too small.
Further details of this study are contained in the following published article:
Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model: a
descriptive study. Journal of Clinical Epidemiology 2007;60:746-51.
56
Section three: Relationship between the APACHE II
scoring system, organ failure scores, and co-morbidities in
determining hospital mortality and ICU readmission
Chapter 6. Comparing the APACHE II scoring system with organ
failure scores to predict hospital mortality
It has been suggested that the intensity and duration of organ failure in critically
ill patients is important in determining their outcomes.20-22
The APACHE II prognostic
scoring system considers age, chronic health status, and physiological derangement
within the first 24 hours of ICU admission only. The APACHE II scoring system
considers two important prognostic factors (age and chronic health status) that most
organ failure scores do not consider. Conversely, most organ failure assessment scores
consider the response of the patients to treatment and the progression of organ failure
during the ICU stay that are not considered by the APACHE II score.24
The natural
question to ask is whether the APACHE II scoring system is better than some of these
organ assessment scores in predicting hospital mortality of critically ill patients.
Since the inception of the RPHICU database in 1987, daily organ failure
assessment has been recorded for every ICU admission by a locally developed RPHICU
organ failure score.70,71
There are also a number of other organ failure assessment scores
developed by different ICUs. Sequential Organ Failure Assessment (SOFA) score was
developed in 1996 after a consensus conference,20
and since then it has been evaluated
and used by many ICUs as a risk adjustment tool.21,22
In this chapter, the predictive
performance of the RPHICU organ failure score was assessed. Specifically, the
57
performance of the APACHE II scoring system and two organ failure scores (the
RPHICU organ failure score and the SOFA score) were compared in relation to
predicting hospital mortality of critically ill patients.
The results of this study showed that the APACHE II scoring system has better
discrimination, calibration and overall predictive performance, as assessed by the
Nagelkerke R2
or Brier’s score, than either the RPHICU organ failure score or SOFA
score in predicting hospital mortality. This result suggests that age and chronic health
status are important prognostic factors and should be considered in a prognostic scoring
system that predicts hospital mortality of critically ill patients. The locally developed
organ failure score (RPHICU organ failure score) had a reasonably good predictive
performance (area under ROC of the first day or cumulative 5-day RPHICU organ failure
score was 0.822 and 0.819, respectively) and its performance was indeed comparable to
the widely used SOFA score in determining hospital mortality. This latter result supports
the validity of using the RPHICU organ failure score (or intensity and duration of organ
failure as defined by the RPHICU organ failure score) as a predictor in the modelling of
outcomes of critically ill patients in the subsequent studies of this thesis.
In conclusion, the APACHE II scoring system is better than organ failure
assessment scores alone in predicting hospital mortality of critically ill patients. Age and
chronic health status are important factors in determining hospital mortality of critically
ill patients.
Further details of this study are contained in the followed published article:
58
Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of
Acute Physiology and Chronic Health Evaluation (APACHE) II score with organ failure
scores to predict hospital mortality. Anaesthesia 2007;62:466-73.
67
Section three: Relationship between the APACHE II
scoring system, organ failure scores, and co-morbidities in
determining hospital mortality and ICU readmission
Chapter 7. Combining the APACHE II scoring system with
Sequential Organ Failure Assessment (SOFA) scores to predict
hospital mortality
In Chapter 6, the APACHE II scoring system was shown to be better than two
organ failure scores in predicting hospital mortality. It is, however, possible that the two
scoring systems can be supplementary to each other because they consider the risk of
death of a critically ill patient by using data at different time points. Although both the
APACHE II and SOFA scoring system have been used together for risk adjustment in
many clinical trials, whether using both scoring systems together will improve the
accuracy of risk adjustment has never been assessed. If most of the information relevant
to the risk of death of a critically ill patient are already completely captured by the
APACHE II scoring system, it can be argued that the additional use of an organ failure
score for further risk adjustment will only increase the unnecessary work-load of data
collection. On the other hand, if the two scores can supplement each other then the use of
both scoring systems together may improve the accuracy of risk prediction and
adjustment.
The daily SOFA scores can be summarised into three scores; the Admission
SOFA score (i.e. SOFA score on the first day of ICU admission), Max SOFA
(summation of the maximum SOFA score of each organ failure during the entire ICU
68
stay), and the Delta SOFA (the difference between Max SOFA and Admission SOFA).21
In this chapter, combining each of these three SOFA scores with the APACHE II scoring
system was evaluated in relation to predicting hospital mortality of critically ill patients.
The results of this study showed that combining the Max SOFA (area under ROC
curve 0.875 vs. 0.858, P = 0.014; Nagelkerke R2: 0.411 vs. 0.371; Brier Score: 0.086 vs.
0.090) or Delta SOFA score (area under ROC curve 0.874 vs. 0.858, P = 0.003;
Nagelkerke R2: 0.412 vs. 0.371; Brier Score: 0.086 vs. 0.090) with the APACHE II score
did improve the discrimination and overall performance of the predictions when
compared with using the APACHE II scoring system alone, especially in the emergency
ICU admissions. This improvement was not apparent when the Admission SOFA was
combined with the APACHE II scoring system. These results suggest that intensity and
duration of organ failure do play a significant part in determining hospital mortality of
critically ill patients, over and beyond the risks associated with age, chronic health status,
and physiological derangement within the first 24 hours of ICU admission modelled by
the APACHE II scoring system. The improvement in performance was, however,
relatively small and may not be clinically significant when compared to the APACHE II
scoring system alone. With the exceptions of patients who are transferred from another
ICU or readmitted to ICU during same hospitalisation, it can be argued that most, but not
all, of the information relevant to mortality risk of a critically ill patient is captured by
age, chronic health status, and severity of physiology derangement at the onset of critical
illness.
In conclusion, intensity and duration of organ failure after the first 24 hours of
ICU admission does play a small part in determining mortality of critically ill patients.
69
Combining the Max SOFA or Delta SOFA score with the APACHE II scoring system
may improve the accuracy of risk adjustment if additional daily organ failure data
collection is possible. However, the degree of improvement is small and so the additional
data collection effort may not be justified or worthwhile.
Further details of this study are contained in the following published article:
Ho KM. Combining sequential organ failure assessment (SOFA) score with acute
physiology and chronic health evaluation (APACHE) II score to predict hospital
mortality of critically ill patients. Anaesthesia & Intensive Care 2007;35:515-21.
77
Section three: Relationship between the APACHE II
scoring system, organ failure scores, and co-morbidities in
determining hospital mortality and ICU readmission
Chapter 8. Combining the APACHE II scoring system with co-
morbidity data to predict hospital mortality
Co-morbidities have been reported to be associated with hospital outcomes of
critically ill patients.19,24
The APACHE II scoring system assesses severe chronic health
condition in addition to age and physiological derangement in predicting mortality of
critically ill patients. There is a suggestion that the mortality risk associated with co-
morbidities may not be fully captured by the APACHE II scoring system. There are at
least three criticisms concerning how co-morbidities are considered in the APACHE II
model. First, only severe co-morbidities that are debilitating or requiring organ
supportive therapy (e.g. dialysis, home oxygen) are considered in the APACHE II scoring
system. Other co-morbidities such as diabetes mellitus, cerebrovascular accident, and
ischaemic heart disease are not considered. Second, the APACHE II scoring system does
not allow extra weighting if a patient has two or more severe co-morbidities. Third, the
APACHE II scoring system will give a larger co-morbidity weighting (or score) if a
patient is admitted after an emergency than following an elective surgical procedure.
There is in fact very little epidemiological data to support the decision to perform this
differential weighting.24
This chapter provides an assessment of the effects of severe co-morbidities in the
APACHE II scoring system and also whether incorporating more detailed co-morbidity
78
data, including minor co-morbidity from the Elixhauser co-morbidities and Charlson co-
morbidity index will improve the performance of the APACHE II scoring system.19,72
This was done overall and also for subgroups formed according to older or younger
patients (>75 vs <75 years old), elective surgery or emergency admission, and cardiac
surgery or non-cardiac surgery patients.
The results of this study showed that minor co-morbidities as described in the
Charlson co-morbidity index and Elixhauser co-morbidities were prevalent in critically ill
patients. Among 24,303 ICU admissions, 3,615 (14.9%), 10,223 (42.1%), and 11,597
(47.7%) patients had at least one co-morbidity as defined in the APACHE II scoring
system, Charlson co-morbidity index, and Elixhauser co-morbidities, respectively. While
the ability of co-morbidity alone to discriminate between hospital survivors and non-
survivors was poor (areas under ROC <0.610), severe co-morbidity was a significant
component in the APACHE II scoring system in predicting mortality of non-cardiac-
surgical admissions. Replacing the weighted co-morbidity data in the APACHE II
scoring system with other more comprehensive measures of co-morbidity, such as
Charlson co-morbidity index or counts of minor co-morbidities, did not significantly
improve the discrimination of the APACHE II scoring system in non-cardiac surgical
patients. For cardiac surgical patients neither the severe co-morbidity component in the
APACHE II scoring system, nor more comprehensive measures, contributed significantly
to the APACHE II scoring system in predicting their hospital mortality.
In conclusion, severe co-morbidity was a significant component of the APACHE
II scoring system for non-cardiac surgical patients. Further improvement in the predictive
performance of the APACHE II scoring system was not observed by incorporating more
79
detailed co-morbidity data. The APACHE II scoring system appeared to have captured
most, if not all, of the hospital mortality risk due to co-morbidity. Clinicians should not
put undue emphasis on the total number of co-morbidities when they make
prognostication on hospital mortality of critically ill patients.
Further details of the study are contained in the following published article:
Ho KM, Finn J, Knuiman M, Webb SA. Combining multiple comorbidities with
Acute Physiology Score to predict hospital mortality of critically ill patients: a linked
data cohort study. Anaesthesia 2007;62:1095-100.
86
Section three: Relationship between the APACHE II
scoring system, organ failure scores, and co-morbidities in
determining hospital mortality and ICU readmission
Chapter 9. The effects of co-morbidity on risk of unplanned ICU
Readmission
The previous chapter examined co-morbidity as a predictor of hospital mortality.
In this chapter, the effects of co-morbidity on risk of unplanned ICU readmission are
examined.
It has been hypothesised that ICU readmission may represent an intermediate
event for an intrinsically sicker group of patients.73
If co-morbidities are a risk factor for
ICU readmission then this may explain why ICU readmission is associated with a higher
hospital mortality. It is also of interest to determine whether multiple minor co-
morbidities, as measured by the Charlson co-morbidity index but not by the APACHE II
scoring system, are a risk factor for ICU readmission and explain the excess mortality
associated with ICU readmission.12,73
Unplanned ICU readmission within 72 hours of ICU discharge is one of the
quality indicators adopted by the ACHS.34,35
As such, we had stratified the analyses into
patients who were readmitted within 72 hours or after 72 hours of ICU discharge in this
study. Our results showed that the total number (or count) of Charlson co-morbidities was
found to be an independent risk factor for late (>72 hours) but not early (< 72 hours)
unplanned ICU readmission during the same hospitalisation. The severity of illness on
87
first admission to ICU as measured by the APACHE II scoring system was not
independently associated with either early or late unplanned ICU readmission.
These results suggest that ICU readmission occurred because of factors that were
not primarily related to the severity of acute illness leading to the first ICU admission.
Co-morbidities not measured by the APACHE II scoring system might explain why some
patients were readmitted to the ICU after 72 hours of ICU discharge but not those who
were readmitted earlier.
The multivariate analyses showed that either early (<72 hours) or late (>72 hours)
ICU readmission and the severity of acute illness as measured by the APACHE II scoring
system were significant risk factors for hospital mortality. Co-morbidities were not a
significant independent risk factor of hospital mortality. These findings imply that co-
morbidities could not account for why patients with either early or late ICU readmissions
had higher hospital mortality. The excess mortality associated with ICU readmissions
could be due to factors or events that occurred after the onset of critical illness such as
nosocomial infections or complications (e.g. thromboembolism).
Further details of this study are contained in the following published article:
Ho KM, Dobb GJ, Finn J, Knuiman M, Webb SA. The effect of co-morbidities on
risk of intensive care readmission during the same hospitalisation: a linked data cohort
study. Journal of Critical Care 2008 (published online in April 2008).
95
Section three: Relationship between the APACHE II
scoring system, organ failure scores, and co-morbidities in
determining hospital mortality and ICU readmission
Chapter 10. Evaluating the APACHE II scoring system for
predicting hospital mortality in ICU readmissions
The results in the previous chapter showed that co-morbidity could be a
significant risk factor for late ICU readmission, but it could not explain why ICU
readmission was associated with an increase in hospital mortality. There is currently no
risk adjustment scoring system for patients who are readmitted to the ICU during the
same hospitalisation. This study aimed to assess whether the APACHE II scoring system
was useful to predict mortality of ICU readmission when it was applied to patients during
their readmission.
In this chapter, the use of the APACHE II predicted mortality measured at the
time of ICU readmission (the Readmission APACHE II predicted mortality) together
with information collected prior to ICU readmission were assessed for their ability to
predict hospital mortality in ICU readmission during the same hospitalisation. The prior
information considered included the time interval between ICU discharge and
readmission, admission source, elective surgery status, length of stay of the first ICU
admission and the APACHE II predicted mortality for the first ICU admission. ICU
readmissions were stratified into two groups (7 days or >7days) to assess whether these
factors would have different effects. A relatively long period before readmission (7 days)
was chosen as a cut point in this study because any prior events before ICU readmission
96
are theoretically less likely to have a significant effect on the course of very late ICU
readmission (> days) when compared to earlier readmission (7 days).
Using two alternative approaches, logistic regression and multilevel likelihood
ratio, the information prior to ICU readmission was found to be not as important as the
Readmission APACHE II predicted mortality in determining the hospital mortality of
these patients. For patients who were readmitted after 7 days of ICU discharge, the
severity of acute illness leading to the first ICU admission had no relationship to their
subsequent hospital mortality (odds ratio 1.05, 95% confidence interval 0.27-1.25,
p=0.602). Combining the APACHE II predicted mortality of the first ICU admission with
the Readmission APACHE II predicted mortality also did not improve the
discrimination between survivors and non-survivors further (area under the ROC curve
0.694 vs 0.699, p=0.682). Other factors such as the time interval between ICU discharge
and readmission, admission source, elective surgery status, and length of stay of the first
ICU admission were also not significant in determining the hospital mortality of ICU
readmissions.
The Readmission APACHE II predicted mortality only had a moderate ability
to discriminate survivors and non-survivors and its performance was slightly better when
applied to ICU readmissions within 7 days of ICU discharge (area under ROC curve
0.785 vs 0.694). The results of this study have at least two significant clinical
implications. First, the mortality risk associated with ICU readmission might not be
related to the patient’s events prior to their readmission including factors related to their
first ICU admission during the same hospitalisation. It is highly possible that ICU
readmission occurred because of an adverse event closer to the time of readmission rather
97
than because they were intrinsically a sicker group of patients in their first ICU
admission.71
It is possible that some nosocomial complications such as pneumonia,
thromboembolism, catheter or wound-related infections may precipitate the readmissions.
This explanation also suggests that some ICU readmissions and their associated
attributable mortality could be potentially preventable if these precipitating events were
identified early and modified in time.
Second, the fact that the performance of the Readmission APACHE II predicted
mortality was not as good as the APACHE II scoring system when applied to the first
ICU admission (in all patients) implies that the model was not capturing the excess
mortality risk of ICU readmission. It is possible that the effect of a disease may have a
stronger mortality effect when it occurs in a hospital setting (that is, nosocomial event) or
when there is a delay in clinical recognition than when a similar disease that occurs
without recent critical illness. Also, a certain degree of physiological derangement from a
nosocomial complication (e.g. deep vein thrombosis, hospital acquired pneumonia,
wound infection, catheter related sepsis) may have a stronger mortality effect on a patient
who has not completely recovered from their critical illness. These findings suggest that
patients who are readmitted to ICU after a recent critical illness are very different from
patients without prior critical illness. Any prognostic model for ICU readmission should
consider a different weighting coefficient for a similar diagnosis and physiological
derangement when compared to patients without prior ICU admission.
In conclusion, the readmission diagnosis and physiological derangement within
the first 24 hours of the readmission explained most of the mortality risk of ICU
readmission. The modified use of the APACHE II model as the Readmission APACHE
98
II predicted mortality only had a moderate ability to discriminate survivors and non-
survivors in ICU readmissions. Events prior to ICU readmission only had a very small
effect on mortality of early ICU readmissions.
Further details of this study are contained in the following published article:
Ho KM, Knuiman M. Bayesian approach to predict hospital mortality of intensive
care readmissions during the same hospitalisation. Anaesthesia & Intensive Care
2008;36:38-45.
107
Section four: The use of inflammatory markers in
predicting hospital mortality and ICU readmission
Chapter 11. Inflammatory markers and risk of unplanned ICU
readmission
The results of Chapter 9 suggest that it is not patients who were sicker at the time
of their first ICU admission that have greatest risk of ICU readmission. The implication is
that factors associated with the management of the patient in ICU or adverse events
following first admission to ICU may be risk factors for ICU readmission. It is possible
that ICU readmission is more likely when the initial ICU discharge was pre-mature, that
is, the patient was discharged from the ICU when they had not adequately recovered from
their critical illness and were placed in a less intensive environment. For example,
nocturnal ICU discharge, discharge to a general ward instead of a high dependency unit,
persistent organ dysfunction (e.g. a high SOFA score on the day of discharge), and a lack
of follow-up team have been identified as risk factors for unplanned ICU readmission and
support the hypothesis that at least some ICU discharges may be premature and
potentially preventable.37,74,75
This study explores whether residual organ dysfunction (as measured by the
Discharge SOFA score) and/or an occult infection or inflammatory process as suggested
by an increase in serum inflammatory markers such as fibrinogen, white cell counts, and
C-reactive protein (CRP) at the time of ICU discharge may increase the risk of ICU
readmission during the same hospitalisation.76
In this study only inflammatory markers
routinely used in daily clinical practice in RPHICU were assessed. As the laboratory data
108
and SOFA scores were required, only the ICU readmissions in the most recent year
(2004) were assessed. The propensity score method was used to adjust for the potential
effects of selection bias due to missing laboratory data.
The results of this study showed that the SOFA score at ICU discharge was not
predictive of subsequent unplanned ICU readmission and CRP concentration at ICU
discharge may be a risk factor for unplanned ICU readmission. These results support the
hypothesis that some readmissions might be due to an occult infection or persistent
inflammation without obvious organ failure that may not be apparent to the ICU
clinicians before the ICU discharge. The strong association between CRP and unplanned
ICU readmission reported in our study was recently confirmed by a large prospective
cohort study in Germany.77
Further details of this study are contained in the following published article:
Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein
concentration as a predictor of intensive care unit readmission: a nested case-control
study. Journal of Critical Care 2006;21:259-65.
116
Section four: The use of inflammatory markers in
predicting hospital mortality and ICU readmission
Chapter 12. Inflammatory makers and prediction of hospital
mortality
In the last chapter, CRP concentration at ICU discharge was found to be
associated with unplanned ICU readmission. In this chapter, the use of CRP
concentration at ICU discharge to predict unexpected hospital mortality after ICU
discharge was examined.
This prospective study was based on the cohort of 603 patients first admitted to
RPH ICU between 1 June and 31 December 2005 and who survived to ICU discharge.
The incidence of unexpected death in this cohort was about 4.3%. The APACHE II
predicted mortality, different SOFA scores (Admission, Max, Delta, and Discharge
SOFA score), and CRP concentrations within 24 hours of ICU discharge were all related
to unexpected death after ICU discharge in the univariate analysis. Multivariate analysis
showed that Delta SOFA score (which signifies progression of organ failure during the
ICU stay) and CRP concentrations at ICU discharge were the only significant predictors
of unexpected hospital death.
The findings from this study and also from the previous study (Chapter 11)
suggest that persistent or new onset inflammation near ICU discharge is a risk factor for
undesirable in-hospital outcomes after ICU discharge.57
Most ICU clinicians decide when
to discharge their patients based on their clinical judgement on the risks and benefits of
keeping or discharging their patients from the ICU. There is, so far, no predictive model
that can be used to assist ICU clinicians in making this decision. If the results of this
117
study are confirmed by other ICUs, it is possible that the intensity of organ failure during
the ICU stay and markers of inflammation near ICU discharge are main determinants of
short term undesirable outcomes after ICU discharge. As such, these risk factors should
be considered when prognostic scoring systems that aim to assist ICU clinicians to
improve their discharge decisions are developed.
Further details of this study are contained in the following published article:
Ho KM, Lee KY, Dobb GJ, Webb SA. C-reactive protein concentration as a
predictor of in-hospital mortality after ICU discharge: a prospective cohort study.
Intensive Care Medicine 2008;34:481-7.
125
Section five: Predicting long term survival after hospital
discharge
Chapter 13. The PREDICT model
There are several potential uses for a tool that is capable of predicting survival
over longer time-periods. Long term survival after critical illness is increasingly being
recognised as an important outcome in assessing effectiveness of new therapy.15
In order
to control for confounding and bias in assessing long term survival of critically ill
patients in a clinical trial, a risk adjustment tool that can objectively estimate long term
survival is essential. From a clinical perspective, many patients and clinicians are also
interested in knowing the long term survival outcome after critical illness, in addition to
other information such as burden of treatment and quality of life after recovery, when
making treatment decisions in the ICU.38,39
There are, so far, only two long term survival
scoring systems and they are limited to estimation of 6-month to 5-year survival of
critically ill patients after hospital discharge.14,41
The RPHICU clinical database was linked to the Western Australian Data
Linkage System (WADLS) in order to obtain information on long term survival
following discharge from hospital. The WADLS contains all hospital admission and
death records for all people resident in Western Australia. The geographical isolation of
Western Australia and the low emigration rate indicate that long term survival data are
available on almost all RPHICU patients.76
In this study, a new prognostic model, based on seven pre-selected predictors
(APACHE II Predicted Risk, Existing Diseases, and Intensive Care Therapy: the
126
PREDICT model), to estimate median survival time and long term survival probabilities
of patients after their critical illness was developed. The seven pre-selected predictors
included age, gender, pre-existing disease (as measured by the Charlson co-morbidity
index), severity of acute illness (as measured by the APACHE II predicted mortality),
and intensity and duration of intensive care therapy. Among these seven predictors, we
found that age, co-morbidities, and severity of acute illness were most important in
determining long term survival of critically ill patients. These three predictors also had a
relatively linear relationship to the probability of long term survival and fulfilled the
proportional hazards assumption over a 15-year period. To facilitate the use of this
model, a nomogram was derived which allows estimation of a patient’s median survival
time and long term survival probabilities by summing scores of each of the seven
predictors.
The current prognostic model, the PREDICT model, provides a framework for
risk adjustment purposes in assessing long term survival of critically ill patients.
However, this prognostic model has some limitations. Firstly, patients’ wishes and the
anticipated quality of life before and after their critical illness are important factors in
making treatment decisions. The median survival time (or long term survival
probabilities) is only one of many factors that patients and clinicians may consider in
making treatment decisions. Furthermore, the c-statistic of this model is only about 0.76
and this leaves considerable uncertainty in its applicability in predicting long term
survival of individual patients. As such, the predicted survival probabilities of this
prognostic model should only be considered as an average estimate of patients with
similar characteristics and not be used for individual patients.
127
Secondly, evidence suggests that combining an objective prognostic scoring
system with physicians’ intuition of a patient’s prognosis may improve the accuracy of
outcome prediction.14,16
Whether combining this current prognostic model with
physicians’ intuition will improve its predictive performance further remains uncertain,
but this merits further investigation. Thirdly, although a large cohort of critically ill
patients was studied and the case-mix of this cohort was, to some extent, similar to other
Australian ICUs (Chapter 2)56
, validation of this model in other ICUs is essential to
assess its generalisability. Finally, the performance of the current model may be
improved if more predictors were considered in the model. The next chapter considers
other potential prognostic factors such as socioeconomic status, accessibility to essential
services, and indigenous status in relation to long term survival of critically ill patients.
Further details of this study are contained in the following published article:
Ho KM, Knuiman M, Finn J, Webb SA. Estimating long-term survival of critically ill
patients: the PREDICT model. Public Library of Science One 2008;3:e3226.
Estimating Long-Term Survival of Critically Ill Patients:The PREDICT ModelKwok M. Ho1,2*, Matthew Knuiman3, Judith Finn3, Steven A. Webb1,2
1 School of Population Health, University of Western Australia and Royal Perth Hospital, Perth, Western Australia, Australia, 2 School of Medicine, University of Western
Australia and Royal Perth Hospital, Perth, Western Australia, Australia, 3 School of Population Health, University of Western Australia, Crawley, Western Australia, Australia
Abstract
Background: Long-term survival outcome of critically ill patients is important in assessing effectiveness of new treatmentsand making treatment decisions. We developed a prognostic model for estimation of long-term survival of critically illpatients.
Methodology and Principal Findings: This was a retrospective linked data cohort study involving 11,930 critically illpatients who survived more than 5 days in a university teaching hospital in Western Australia. Older age, male gender, co-morbidities, severe acute illness as measured by Acute Physiology and Chronic Health Evaluation II predicted mortality, andmore days of vasopressor or inotropic support, mechanical ventilation, and hemofiltration within the first 5 days of intensivecare unit admission were associated with a worse long-term survival up to 15 years after the onset of critical illness. Amongthese seven pre-selected predictors, age (explained 50% of the variability of the model, hazard ratio [HR] between 80 and60 years old = 1.95) and co-morbidity (explained 27% of the variability, HR between Charlson co-morbidity index 5 and0 = 2.15) were the most important determinants. A nomogram based on the pre-selected predictors is provided to allowestimation of the median survival time and also the 1-year, 3-year, 5-year, 10-year, and 15-year survival probabilities for apatient. The discrimination (adjusted c-index = 0.757, 95% confidence interval 0.745–0.769) and calibration of thisprognostic model were acceptable.
Significance: Age, gender, co-morbidities, severity of acute illness, and the intensity and duration of intensive care therapycan be used to estimate long-term survival of critically ill patients. Age and co-morbidity are the most importantdeterminants of long-term prognosis of critically ill patients.
Citation: Ho KM, Knuiman M, Finn J, Webb SA (2008) Estimating Long-Term Survival of Critically Ill Patients: The PREDICT Model. PLoS ONE 3(9): e3226.doi:10.1371/journal.pone.0003226
Editor: Jeffrey A. Gold, Oregon Health & Science University, United States of America
Received July 27, 2008; Accepted August 25, 2008; Published September 17, 2008
Copyright: � 2008 Ho et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricteduse, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: A grant was received from BUPA to provide support for the cost of linking the databases used in this study. The funding agency has no involvement inthe study design, collection of data, analysis and interpretation of data, writing of the report, or in the decision to submit the article for publication.
Competing Interests: The authors have declared that no competing interests exist.
* E-mail: [email protected]
Introduction
Demand for intensive care unit (ICU) services is increasing [1],
and at a rate that is higher than the average for all health care
services [2]. Increase in treatment and monitoring technology,
patients’ expectations, and ageing population all contribute to this
increased demand for intensive care services [1]. Indeed, intensive
care is increasingly being provided to older and sicker patients,
whom in the past were not treated in intensive care [3]. Intensive
care services accounted for 10% of the US$2.1 trillion total health
expenditures on health care in the United States in 2006 [4] and
has been estimated to cost more than £700 million in the United
Kingdom in 1999 [5]. The cost of intensive care services coupled
with increasing demand provides the rationale for improved
modelling of outcomes of critically ill patients.
Long-term survival after critical illness is increasingly being
recognized as an important outcome in assessing effectiveness of
new therapy [6]. In order to control for confounding and bias in
assessing long-term survival of critically ill patients in a clinical
trial, a risk adjustment tool that can objectively estimate long-term
survival is essential. From a clinical perspective, many patients and
clinicians are also interested in knowing the long-term survival
outcome after critical illness, in addition to other information such
as burden of treatment and quality of life after recovery, when
making treatment decisions in the ICU. Although many clinicians
may foretell patient hospital survival outcome more accurately
than some objective prognostic models [7], treatment decisions
made by clinicians do vary considerably with their practice style
and work experience [8–10]. The strategy of continuing intensive
treatment for all patients until death will reduce the need for
patients and clinicians to make difficult treatment decisions and
may improve the survival time of some. This strategy is, however,
expensive and undesirable by imposing excessive burden of
treatment on those who have a very poor prognosis [11]. For
example, initiating acute renal replacement therapy in critically ill
patients with less than 10% probability of 6-month survival was
estimated to cost US$274,000 (£137,000) per quality-adjusted life
year saved [12].
The SUPPORT investigators from the United States and
Wright et al. from the United Kingdom published two prognostic
models that were based on age, severity of acute illness and
admission diagnosis to estimate 6-month and 5-year survival of
PLoS ONE | www.plosone.org 1 September 2008 | Volume 3 | Issue 9 | e3226
critically ill patients, respectively [13,14]. The utility of latter
model is, however, limited by its ability to classify 5-year survival
probabilities only into three risk categories when the calculated
risk score is either ,70, 70–80, or .80 [14]. This model also did
not consider the potential effect of detailed co-morbidity data on
long-term survival of critically ill patients beyond the usual
assessment included in the Acute Physiology and Chronic Health
Evaluation (APACHE) score [14,15]. There is currently no
prognostic model that is available to estimate the survival of
critically ill patients beyond 5 years after the onset of critical
illness. Furthermore, the relative importance of age, co-morbidity,
and severity of acute illness in determining long-term prognosis of
critically ill patients also remains unknown. In this study we
examined the long-term survival of 11,930 critically ill adult
patients who survived at least 5 days and developed a new
prognostic model (Predicted Risk, Existing Diseases, and Intensive
Care Therapy: the PREDICT model) to estimate their median
survival time and long-term survival probabilities.
Methods
The characteristics of the cohortThis cohort study utilized the clinical database of the ICU at
Royal Perth Hospital (RPH) in Western Australia. RPH is the
largest tertiary university teaching hospital in Western Australia
and the 22-bed ICU admits patients of all specialties except liver
transplantation and captures over 40% of all critically ill patients
in Western Australia. The database analyzed in this study includes
details of all ICU admissions between 1989 and 2002, including
demographic factors, admission diagnosis, admission source,
severity of acute illness as measured by APACHE II scores based
on the worst first 24-hour ICU data [15], daily organ failure
assessment and supportive therapy required [16], and ICU and
hospital survival outcome.
In this study the patients with a diagnosis excluded from the
original APACHE II cohort (e.g. coronary artery graft surgery,
burns, snake bite)[15], those who resided outside Western
Australia at the time of ICU admission (who could not be
followed for survival), readmissions after the first ICU readmission,
patients who were younger than 16 years old, and patients who
did not survive more than 5 days during their hospitalization of
the index ICU admission were excluded. The data were reviewed
for internal consistency annually, and there were no patients with
missing hospital mortality data. Some of the other details of this
cohort have been described in our previous publications [16–18].
The ICU clinical database was linked to the Western Australian
hospital morbidity and mortality databases using probabilistic
matching [16], providing information on patients’ co-morbidities
as recorded in all private and public hospital admissions including
any prior ICU admissions up to 5 years before the index ICU
admission. A relatively long five-year ‘look back’ period was
chosen in this study to capture all existing co-morbidities of each
patient. We ascertained the presence of co-morbidities in the
Charlson co-morbidity index (Table 1) using the published ICD-
9-CM and ICD-10-AM coding algorithms [16,19]. We did not
assign a co-morbidity to a patient if that condition was diagnosed
during the hospitalization of the index ICU admission. The
proportions of invalid (false positive) and missed links (false
negatives) between Western Australian hospital morbidity and
mortality databases were evaluated several years ago, and both
false positives and negatives were estimated to be 0.11% [20].
The survival status of the patients was assessed on 31 December
2003 and the length of follow up ranged from 1 year to 15 years
with an average of 6 years. Western Australia is geographically
isolated and has a very low rate of emigration (,0.03% in
2002)[16], and as such, lost to long-term survival follow-up by the
Western Australian mortality database is likely to be very low. The
data analyzed had the patient name and address removed and the
study was approved by the RPH Ethics Committee and the
Western Australian Confidentiality of Health Information Com-
mittee (CHIC).
Development of the modelThe prognostic model was fitted using Cox proportional
hazards regression [21], restricting predictors to factors that were
likely to be important predictors of long-term survival of
hospitalized patients [13,14,22,23]. These pre-selected factors
included age [14,22], gender [22], APACHE II predicted
mortality risk [13–15], Charlson co-morbidity index [19,23], days
of mechanical ventilation, acute renal replacement therapy or
hemofiltration, and vasopressor or inotropic therapy during the
first 5 days of the index ICU admission [13]. The APACHE II
predicted mortality was chosen as a measure of severity of acute
illness because it is widely used and summarizes the diagnosis,
acute physiologic derangement within the first 24 hours of ICU
admission, severe co-morbidities, and whether the ICU admission
is after elective or emergency surgery. Our previous study also
showed that the APACHE II predicted mortality has a very stable
performance in this cohort over the past 10–15 years [17].
Although age and severe co-morbidities are already used to
calculate the APACHE II predicted mortality [15], these two
factors may still have a profound effect on long-term survival over
and beyond the weightings used in the APACHE II predicted
mortality [14,22,23]. As such, both age and Charlson co-morbidity
index were used as separate predictors in additional to the
APACHE II predicted mortality in this prognostic model. These
seven predictors were also chosen because they are often recorded
Table 1. Charlson co-morbidity index component and itsweighting.
Co-morbidity Weighting
Myocardial infarction 1
Congestive heart failure 1
Peripheral vascular disease 1
Cerebrovascular disease 1
Dementia 1
Chronic pulmonary disease 1
Connective tissue disease 1
Peptic ulcer disease 1
Mild liver disease 1
Diabetes mellitus 1
Hemiplegia 2
Moderate or severe renal disease 2
Diabetes with end-organ damage 2
Any tumour 2
Leukemia 2
Lymphoma 2
Moderate to severe liver disease 3
Metastatic solid tumour 6
AIDS 6
doi:10.1371/journal.pone.0003226.t001
PREDICT Model
PLoS ONE | www.plosone.org 2 September 2008 | Volume 3 | Issue 9 | e3226
in the administrative databases of many ICUs, and as such, it is
possible for other ICUs to validate this model using their data [24].
The proportional hazards assumption of the continuous
predictors in the Cox model was assessed and found to be
acceptable (Figure 1a, 1b, 1c). During the modelling process, we
avoided categorizing continuous predictors [24,25] and allowed a
non-linear relationship with hazard of death using a 6-knot
restricted cubic spline function [25]. The relative contribution of
each predictor was assessed using the chi-square statistic minus the
degrees of freedom [25]. The discrimination performance of the
model was assessed with the c-index, which is a generalization of
the c-statistic or the area under the receiver-operating character-
istic curve, allowing for censored data [25,26]. In this study, a c-
index between 0.70 and 0.80 was regarded as acceptable and a c-
index above 0.80 was regarded as excellent [27]. Using the Design
library in S-PLUS software (version 8.0, 2007. Insightful Corp.,
Seattle, Washington, USA), the c-index was computed and
adjusted for optimism (arising from using the same data to
develop the model and assess its performance) by a bootstrap
technique to penalise for possible over-fitting, with 200 re-samples
and at least 200 patients per risk group [25,28]. The bootstrapping
technique was used in this study instead of splitting the data into
development and validation data set because this method is
regarded as most data ‘efficient’ and accurate in developing a
prognostic model [25]. Model calibration (similarity of predicted
risks and proportions actually dying) was assessed graphically and
used a bootstrap re-sampling to construct a bias-corrected
calibration curve and its slope [25,29]. Nagelkerke’s R2 (a
generalized measure of the percentage of the variance in survival
accounted for by the model) was computed to assess the overall
performance of the model [25,30]. The performance of the model
was assessed over the full 15 years of follow-up, when follow-up
was restricted to a maximum of 5 years for each patient, and also
when only patients admitted after 1997 were considered.
A nomogram was developed for the model that generates the
median survival time and selected annual survival probabilities by
adding up the scores for each of the seven predictors [25]. The
use of the nomogram and how each predictor may affect a
patient’s long-term prognosis is described for a selection of typical
patient scenarios. In particular, these scenarios were selected to
illustrate how the long-term prognosis of a patient can be
different from the short-term prognosis. Nevertheless, the results
of the nomogram should only be considered as an average
estimate of patients with similar characteristics and not be used
for individual patients.
Results
Characteristics of the cohortThe study cohort consisted of a heterogeneous group of
critically ill patients, with elective surgery including heart valve
surgery, urology, gastrointestinal and spinal surgery accounting for
36.2% of all ICU admissions. The emergency admissions consisted
of patients with multiple trauma (8.5%), isolated head trauma
(2.5%), acute myocardial infarction, congestive heart failure,
cardiac arrhythmias, or cardiogenic shock (7.4%), hypovolemic
or hemorrhagic shock (0.8%), drug overdoses (7.2%), subarach-
noid or intracranial hemorrhage (5.1%), sepsis (4.3%), pneumonia
or aspiration (3.7%), obstructive airway diseases (2.1%), cardiore-
spiratory arrest (4.0%), gastrointestinal hemorrhage, perforation or
obstruction (2.4%), and other medical and surgical emergencies.
Details of this cohort including demographic factors, co-morbid-
ities, severity of acute illness, and the length of ICU and hospital
stay are described in Table 2.
Figure 1. The proportional hazards assumption of the predictors in the Cox model was assessed by plotting the logarithm of thenegative logarithm of the Kaplan Meier survivor estimates and the assumption was found to be acceptable for the three pre-selected continuous predictors; APACHE II predicted mortality, Charlson co-morbidity index, and age. (a) Graph assessing theproportionality of hazards associated with severity of acute illness measured by the APACHE II predicted mortality risk categories (0–20%, 20–40%,40–60%, 60–80%, 80–100%). (b) Graph assessing the proportionality of hazards associated with co-morbidities measured by Charlson co-morbidityindex categories (0, 1, 2, 3, 4–5, .5). (c) Graph assessing the proportionality of hazards associated with age measured by age categories (16–30, 30–50, 50–60, 60–70, 70–80, .80 years old)doi:10.1371/journal.pone.0003226.g001
PREDICT Model
PLoS ONE | www.plosone.org 3 September 2008 | Volume 3 | Issue 9 | e3226
Effect of the Predictors on Hazard of DeathAmong all the seven pre-selected predictors in the model, age
(50%), co-morbidity as measured by Charlson co-morbidity index
(27%), and severity of acute illness as measured by the APACHE
II predicted mortality (20%) made the strongest contributions in
predicting survival time (Figure 2). After adjusting for other
predictors, the log hazard of death increased linearly with age,
Charlson co-morbidity index, and the number of days of
vasopressor or inotropic therapy, mechanical ventilation, or
hemofiltration therapy (Figure 3). The relationship between the
APACHE II predicted mortality and log hazard of death was non-
linear with a steep effect when the APACHE II predicted mortality
was less than 10% and a smaller effect when it was more than
10%. The estimated (adjusted) hazard ratios for the seven
predictors are summarized in Figure 4.
Clinical Application of the ModelFigure 5 presents the model in the form of a nomogram that
provides the median survival time and long-term survival
probabilities corresponding to a particular total score. The total
score for a patient is obtained by adding up the scores for each of
the seven predictors. We use the following hypothetical but typical
patients to illustrate how the nomogram is used and how the short-
term prognosis of a patient can be quite different from the long-
term prognosis.
Patient A:
A 40-year old male, without pre-existing co-morbidities (ie
Charlson co-morbidity index = 0), was admitted to the ICU
because of septic shock with an APACHE II predicted mortality of
80%. He required vasopressor or inotropic therapy, mechanical
ventilation, and hemofiltration therapy during the first 5 days in
the ICU.
The gender of this patient scores 5 points, age scores 28 points,
Charlson co-morbidity scores zero points, the APACHE II
predicted mortality or risk scores 30 points, 5 days of vasopressor
or inotropic therapy scores 7 points, 5 days of mechanically
ventilation scores 15 points, and 5 days of hemofiltration scores 20
points. The total score of this patient is therefore 105 which gives
an estimated median survival time of about 4 years, .70% 1-year
survival probability, .50% 3-year survival probability, .45% 5-
year survival probability, and .20% 10-year survival probability.
Patient B:
A 70-year old female, with chronic obstructive airway disease
and non-insulin dependent diabetes mellitus with no end-organ
damage (ie Charlson co-morbidity index = 2), was admitted to the
ICU because of severe community acquired pneumonia with an
APACHE II predicted mortality of 30%. She required vasopressor
or inotropic therapy and mechanical ventilation but not
hemofiltration during the first 5 days in the ICU.
The gender of this patient scores zero points, age scores 70
points, Charlson co-morbidity index scores 12 points, the
APACHE II predicted mortality scores 16 points, 5 days of
mechanical ventilation scores 15 points, and 5 days of vasopressor
or inotropic therapy scores 7 points. The total score of this patient
is therefore 120 which gives an estimated median survival time of
about 2 years, 60% 1-year survival probability, 40% 3-year
survival probability, 30% 5-year survival probability, and 10% 10-
year survival probability.
Patient C:
A 80-year old male, with a history of myocardial infarction,
congestive heart failure, peripheral vascular disease, cerebrovas-
cular disease, and dementia (ie Charlson co-morbidity index = 5),
was admitted to an ICU with bowel perforation and peritonitis
with an APACHE II predicted mortality of 30%. He required
vasopressor or inotropic therapy and mechanical ventilation but
not hemofiltration during the first 5 days in the ICU.
The gender of this patient scores 5 points, age scores 85 points,
Charlson co-morbidity index scores 30 points, the APACHE II
predicted mortality scores 16 points, 5 days of mechanical
ventilation scores 15 points, and 5 days of vasopressor or inotropic
therapy scores 7 points. The total score of this patient is therefore
158 which gives an estimated median survival time of ,0.5 years,
25% 1-year survival probability, and 10% 3-year survival
probability.
Discrimination and Calibration of the Prognostic ModelThe adjusted c-index for this prognostic model was 0.757 (95%
confidence interval 0.745–0.769), Nagelkerke’s R2 was 0.255 and
the bias-corrected calibration of the model over a 15-year period
was reasonable (slope of the calibration = 0.98)(Figure 6). The
Nagelkerke’s R2 remained unchanged and the adjusted c-index
only increased marginally when the analysis was restricted to a
maximum of 5 years follow up (c-index = 0.759, slope = 0.97) or
data after 1997 (c-index = 0.762, slope of the calibration = 0.97).
Table 2. Characteristics of the cohort (n = 11,930).
Variables
Mean (median,standard deviation),unless statedotherwise
Age, yrs 53.8 (57.0, 19.0)
Gender (male/female), no. (%) 7489 (62.8)/4441 (37.2)
Elective surgery admission, no. (%) 4318 (36.2)
APACHE II score 13.7 (13.0, 6.8)
APACHE II predicted mortality, % 14.5 (7.0, 17.8)
No. of APACHE co-morbidities 0.1 (0, 0.3)
(a) Cardiovascular, no. (%) 592 (5.0)
(b) Respiratory, no. (%) 210 (1.8)
(c) Renal, no. (%) 109 (0.9)
(d) Immunosuppressed, no. (%) 197 (1.7)
(e) Liver, no. (%) 76 (0.6)
No. of Charlson co-morbidities 0.8 (0, 1.2)
Charlson co-morbidity index 1.0 (0, 1.7)
Length of ICU stay, days 5.6 (3.0, 8.3)
Length of hospital stay, days 20.3 (13.0, 25.9)
No. of patients mechanically ventilated (%) # 8034 (67.3)
No. of patients on inotrope (%) # 3921 (32.9)
No. of patients on dialysis (%) # 608 (5.1)
No. of ICU survivor (%)* 11557 (96.9)
No. of hospital survivor (%)* 11101 (93.1)
No. of survivor/total no. of patients followed up (%)
(a) at 1-year 10334/11101 (93.1)
(b) at 3-year 8031/10019 (80.2)
(c) at 5-year 6109/8212 (74.4)
(d) at 10-year 2609/4238 (61.6)
(e) at 15-year 441/887 (49.7)
#During the first 5 days in ICU.*Excluding patients died within 5 days of ICU admission.ICU, intensive care unit.APACHE, Acute Physiology and Chronic Health Evaluation.doi:10.1371/journal.pone.0003226.t002
PREDICT Model
PLoS ONE | www.plosone.org 4 September 2008 | Volume 3 | Issue 9 | e3226
Figure 2. Contribution of each predictor in predicting the survival time in the Cox proportional hazards model.doi:10.1371/journal.pone.0003226.g002
Figure 3. The relationship between relative hazard and each predictor after adjusting for other predictors in the model.doi:10.1371/journal.pone.0003226.g003
PREDICT Model
PLoS ONE | www.plosone.org 5 September 2008 | Volume 3 | Issue 9 | e3226
Discussion
This study showed that age, gender, co-morbidities (Charlson
co-morbidity index), severity of acute illness (the APACHE II
predicted mortality), and duration of intensive care therapy or
organ support within the first 5 days of ICU admission are
important prognostic factors for long-term survival of critically ill
patients. To the best of our knowledge, this new prognostic model
(Predicted Risk, Existing Diseases, and Intensive Care Therapy:
the PREDICT model) is the first preliminary prognostic model
that can be used to estimate the median survival time and long-
term survival probabilities of critically ill patients up to 15 years
after the onset of critical illness.
The current prognostic model has confirmed that age, gender,
co-morbidities, severity of acute illness, and duration of intensive
care therapy or organ failure are important predictors of 6 months
to 5 years survival of hospitalized or critically ill patients
[13,14,19,22,23]. The current model is indeed built on the results
of these previous studies but further extended the significance of
these risk factors in predicting survival of critically ill patients
beyond 6 months to 5 years. This current model also demonstrat-
ed that most of these predictors have a relatively linear relationship
to the long-term survival probability. More importantly, our
results also showed that age and co-morbidities are the most
important determinants of long-term prognosis of critically ill
patients. This latter finding has at least two significant clinical
Figure 4. The estimated (adjusted) hazard ratios and multilevel confidence bars (0.70 as illustrated by the black bar to 0.99 asillustrated by the orange bar) for the effects of predictors in the model are summarized in the figure below. An increase of 20 years ofage and an increase in Charlson co-morbidity index from 0 to 5 approximately doubled the risk of death. Doubling the APACHE II predicted mortalityfrom 20% to 40% increased the relative risk of death by about 30 to 40%. Similarly, increased the number of days of intensive care therapy from 1 to 5increased the relative risk of death by between 10% and 50%.doi:10.1371/journal.pone.0003226.g004
Figure 5. Nomogram for predicting long-term survival probabilities and median survival time. Note: gender: 2 = female, 1 = male.Predicted.mortality = APACHE II predicted mortality in %.doi:10.1371/journal.pone.0003226.g005
PREDICT Model
PLoS ONE | www.plosone.org 6 September 2008 | Volume 3 | Issue 9 | e3226
implications. First, the factors that determine long-term survival of
a critically ill patient are different from those that affect short-term
prognosis. Previous evidence suggested that diagnosis and acute
physiological derangement of a patient are most important in
determining hospital survival [15,31]. In our three hypothetical
patients, Patient A has in fact the most severe form of acute critical
illness and worst short-term prognosis. Nevertheless, because this
patient is younger and has no co-morbidities, this patient has a
very reasonable and better long-term prognosis than Patient B and
C. If we use the prognostic model developed by Wright et al. [14]
to estimate the long-term survival of our three hypothetical
patients, Patient B will have the best 5-year prognosis (risk score is
estimated to be 68) followed by Patient C (risk score 75) and then
Patient A (risk score 87). The lack of detailed co-morbidity data
and a heavy emphasis on severity of acute illness in the model
developed by Wright et al. is the most likely explanation why our
results are different from theirs.
Many clinicians may intuitively consider the intensity of organ
failure as very important in affecting a patient’s prognosis [32,33].
Our findings suggest that the effect of acute organ failure on long-
term survival is not strong and mostly captured by age, co-
morbidities, and the APACHE II predicted mortality on admission
to ICU. Our previous studies have also showed that the intensity of
organ failure alone is not as important as the APACHE II score in
predicting hospital mortality [34,35]. Therefore, our findings
suggest that clinicians should be very careful not to place undue
emphasis on the severity of acute illness and intensity of organ
failure when making long-term prognostications of critically ill
patients.
Second, because the contributions by intensive care therapy are
relatively small when compared to age, Charlson co-morbidity
index, and the APACHE II predicted mortality, using the data
after the first 24 to 48 hours of ICU stay is unlikely to
underestimate the final total prediction score significantly (,20
points)(Figure 5). Therefore, early estimation of a slightly
‘optimistic’ long-term survival probability and median survival
time is feasible after the first 24 to 48 hours of ICU stay; and in
patients with either extremes of prognosis, this early estimation is
unlikely to be significantly different from the final prediction by
collecting all data after five days of intensive care therapy.
Nevertheless, the current prognostic model utilizes the APACHE
II predicted mortality after ICU admission as a predictor to
estimate long-term survival, as such, the model cannot be used, in
its current form, as a tool to triage ICU admission.
This study has significant limitations. First, patients’ wishes and
the anticipated quality of life before and after their critical illness
are important factors in making treatment decisions [36,37]. The
median survival time and long-term survival probabilities is only
one of the many factors that patients and clinicians may consider
in making treatment decisions. Furthermore, the c-statistics of this
model is only about 0.76 and this leaves considerable uncertainty
in its applicability in predicting long-term survival of individual
patients. As such, the predicted survival probabilities of this
prognostic model should only be considered as an average estimate
of patients with similar characteristics and should not be used for
individual patients. Second, evidence suggests that combining an
objective prognostic model with physicians’ intuition may improve
the accuracy of outcome prediction [13]. Whether combining this
current prognostic model with physicians’ intuition will improve its
predictive performance further remains uncertain, but this merits
further investigation. Third, although we studied a large cohort of
critically ill patients, and also the case-mix, severity of illness, and
Figure 6. Bootstrap estimate of calibration accuracy for 15-year estimates from the Cox proportional hazards model. Dotscorrespond to apparent predictive accuracy and x marks the bootstrap-corrected estimates.doi:10.1371/journal.pone.0003226.g006
PREDICT Model
PLoS ONE | www.plosone.org 7 September 2008 | Volume 3 | Issue 9 | e3226
in-hospital survival of this cohort is very similar to many other
ICUs in Australia [38], validation of this model by other ICUs that
have access to data linkage is essential to assess its generalizability.
Finally, although the APACHE II prognostic model is still widely
used for risk adjustment purposes in many ICUs [39,40], it is
possible that using newer prognostic models instead of the
APACHE II prognostic model may improve our current model
[41]. Similarly, the performance of the current model may be
improved if we consider more predictors in the model although
this will also increase the complexity of the model. In this regard,
we hope that the PREDICT model developed in this study will be
of value to others who aim to develop a new prognostic model to
enhance our understanding of long-term survival of critically ill
patients.
In summary, Age, gender, co-morbidities, severity of acute
illness, and the intensity and duration of intensive care therapy can
be used to estimate long-term survival of critically ill patients. Age
and co-morbidity are the most important determinants of the long-
term prognosis of critically ill patients. The current prognostic
model, the PREDICT model, provides a framework for
prognostications and risk adjustment when long-term survival of
critically ill patients is considered.
Author Contributions
Conceived and designed the experiments: KMH MK JF SAW. Analyzed
the data: KMH MK JF SAW. Contributed reagents/materials/analysis
tools: KMH. Wrote the paper: KMH MK JF SAW.
References
1. Acute Health Division DoHS (1997) Review of intensive care in Victoria [Phase
1 report]. Melbourne: Department of Human Services.2. Halpern NA, Bettes L, Greenstein R (1994) Federal and nationwide intensive
care units and healthcare costs: 1986–1992. Crit Care Med 22: 2001–2007.
3. Kvale R, Flaatten H (2002) Changes in intensive care from 1987 to 1997 - hasoutcome improved? A single centre study. Intensive Care Med 28: 1110–1116.
4. Poisal JA, Truffer C, Smith S, Sisko A, Cowan C, et al. (2007) Health spendingprojections through 2016: modest changes obscure part D’s impact. Health Aff
(Millwood) 26: w242–w253.
5. The Audit Commission (1999) Critical to Success. The place of efficient andeffective critical care services within the acute hospital. London: Audit
Commission for Local Authorities and the National Health Service in Englandand Wales.
6. Girard TD, Kress JP, Fuchs BD, Thomason JW, Schweickert WD, et al. (2008)Efficacy and safety of a paired sedation and ventilator weaning protocol for
mechanically ventilated patients in intensive care (Awakening and Breathing
Controlled trial): a randomised controlled trial. Lancet 371: 126–134.7. Sinuff T, Adhikari NK, Cook DJ, Schunemann HJ, Griffith LE, et al. (2006)
Mortality predictions in the intensive care unit: comparing physicians withscoring systems. Crit Care Med 34: 878–885.
8. Cook DJ, Guyatt GH, Jaeschke R, Reeve J, Spanier A, et al. (1995)
Determinants in Canadian health care workers of the decision to withdrawlife support from the critically ill. JAMA 273: 703–708.
9. Elstein AS, Christensen C, Cottrell JJ, Polson A, Ng M (1999) Effects ofprognosis, perceived benefit and decision style upon decision making in critical
care. Crit Care Med 27: 58–65.10. Garland A, Connors AF (2007) Physicians’ influence over decisions to forego life
support. J Palliat Med 10: 1298–1305.
11. Connors AF Jr (1999) The influence of prognosis on care decisions in thecritically ill. Crit Care Med 27: 5–6.
12. Hamel MB, Phillips RS, Davis RB, Desbiens N, Connors AF Jr, et al. (1997)Outcomes and cost-effectiveness of initiating dialysis and continuing aggressive
care in seriously ill hospitalized adults. SUPPORT Investigators. Study to
Understand Prognoses and Preferences for Outcomes and Risks of Treatments.Ann Intern Med 127: 195–202.
13. Knaus WA, Harrell FE Jr, Lynn J, Goldman L, Phillips RS, et al. (1995) TheSUPPORT prognostic model. Objective estimates of survival for seriously ill
hospitalized adults. Study to understand prognoses and preferences for outcomes
and risks of treatments. Ann Intern Med 122: 191–203.14. Wright JC, Plenderleith L, Ridley SA (2003) Long-term survival following
intensive care: subgroup analysis and comparison with the general population.Anaesthesia 58: 637–642.
15. Knaus WA, Draper EA, Wagner DP, Zimmerman JE (1985) APACHE II: aseverity of disease classification system. Crit Care Med 13: 818–829.
16. Williams TA, Dobb GJ, Finn JC, Knuiman M, Lee KY, et al. (2006) Data
linkage enables evaluation of long-term survival after intensive care. AnaesthIntensive Care 34: 307–315.
17. Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, et al. (2006) A comparison ofadmission and worst 24-hour Acute Physiology and Chronic Health Evaluation
II scores in predicting hospital mortality: a retrospective cohort study. Crit Care
10: R4.18. Williams TA, Dobb GJ, Finn JC, Knuiman MW, Geelhoed E, et al. (2008)
Determinants of long-term survival after intensive care. Crit Care Med 36:1523–1530.
19. Charlson ME, Pompei P, Ales KL, MacKenzie CR (1987) A new method ofclassifying prognostic comorbidity in longitudinal studies: development and
validation. J Chronic Dis 40: 373–383.
20. Holman CD, Bass AJ, Rouse IL, Hobbs MS (1999) Population-based linkage ofhealth records in Western Australia: development of a health services research
linked database. ANZ J Public Health 23: 453–459.
21. Cox DR (1972) Regression models and life tables (with discussion). Journal of the
Royal Statistical Society 34: 187–220.
22. Lee SJ, Lindquist K, Segal MR, Covinsky KE (2006) Development and
validation of a prognostic index for 4-year mortality in older adults. JAMA 295:
801–808.
23. Pompei P, Charlson ME, Ales K, MacKenzie CR, Norton M (1991) Relating
patient characteristics at the time of admission to outcomes of hospitalization.
J Clin Epidemiol 44: 1063–1069.
24. Wyatt JC, Altman DG (1995) Prognostic models: clinically useful or quickly
forgotten? BMJ 311: 1539–1541.
25. Harrell FE Jr (2001) Regression Modeling Strategies. New York: Springer.
26. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver
operating characteristic (ROC) curve. Radiology 143: 29–36.
27. Feringa HH, Bax JJ, Hoeks S, van Waning VH, Elhendy A, et al. (2007) A
prognostic risk index for long-term mortality in patients with peripheral arterial
disease. Arch Intern Med 167: 2482–2489.
28. Efron B, Tibshirani R (1993) An Introduction to the Bootstrap. New York:
Chapman & Hall.
29. Ho KM (2007) Forest and funnel plots illustrated the calibration of a prognostic
model: a descriptive study. J Clin Epidemiol 60: 746–751.
30. Nagelkerke NJ (1991) A note on a general definition of the coefficient of
determination. Biometrika 78: 691–692.
31. Ho KM, Finn J, Knuiman M, Webb SA (2007) Combining multiple
comorbidities with Acute Physiology Score to predict hospital mortality of
critically ill patients: a linked data cohort study. Anaesthesia 62: 1095–1100.
32. Cabre L, Mancebo J, Solsona JF, Saura P, Gich I, et al.; and the Bioethics
Working Group of the SEMICYUC (2005) Multicenter study of the multiple
organ dysfunction syndrome in intensive care units: the usefulness of Sequential
Organ Failure Assessment scores in decision making. Intensive Care Med 31:
927–933.
33. Nathens AB, Rivara FP, Wang J, Mackenzie EJ, Jurkovich GJ (2008) Variation
in the rates of do not resuscitate orders after major trauma and the impact of
intensive care unit environment. J Trauma 64: 81–88.
34. Ho KM, Lee KY, Williams T, Finn J, Knuiman M, et al. (2007) Comparison of
Acute Physiology and Chronic Health Evaluation (APACHE) II score with
organ failure scores to predict hospital mortality. Anaesthesia 62: 466–473.
35. Ho KM (2007) Combining sequential organ failure assessment (SOFA) score
with acute physiology and chronic health evaluation (APACHE) II score to
predict hospital mortality of critically ill patients. Anaesth Intensive Care 35:
515–521.
36. Ho KM, Liang J (2004) Withholding and withdrawal of therapy in New Zealand
intensive care units (ICUs): a survey of clinical directors. Anaesth Intensive Care
32: 781–786.
37. Ho KM, English S, Bell J (2005) The involvement of intensive care nurses in
end-of-life decisions: a nationwide survey. Intensive Care Med 31: 668–673.
38. Finfer S, Bellomo R, Lipman J, French C, Dobb G, et al. (2004) Adult-
population incidence of severe sepsis in Australian and New Zealand intensive
care units. Intensive Care Med 30: 589–596.
39. Harvey S, Harrison DA, Singer M, Ashcroft J, Jones CM, et al.; PAC-Man study
collaboration (2005) Assessment of the clinical effectiveness of pulmonary artery
catheters in management of patients in intensive care (PAC-Man): a randomised
controlled trial. Lancet 366: 472–477.
40. Finfer S, Bellomo R, Boyce N, French J, Myburgh J, et al. (2004) , SAFE Study
Investigators (2004) A comparison of albumin and saline for fluid resuscitation in
the intensive care unit. N Engl J Med 350: 2247–2256.
41. Zimmerman JE, Kramer AA, McNair DS, Malila FM (2006) Acute Physiology
and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for
today’s critically ill patients. Crit Care Med 34: 1297–1310.
PREDICT Model
PLoS ONE | www.plosone.org 8 September 2008 | Volume 3 | Issue 9 | e3226
136
Section five: Predicting long term survival after hospital
discharge
Chapter 14. The effect of socioeconomic status on long term
survival
Socioeconomic status (SES), as measured by individual-level indicators such as
education, income, and occupation or disadvantaged area-level indicators, is a
determinant of outcomes for many chronic diseases.78-81
Many seriously ill patients are
admitted to ICU only after other layers of the health care system have failed to prevent or
reverse the critical illness. It is possible that SES can have a significant effect on
outcomes of critical illnesses.82
So far, no prognostic scoring system for critically ill
patients has considered SES. This may be because it is difficult to classify patients into
different SES levels and perhaps also because such a prognostic scoring system would be
difficult to generalise or validate in other ICUs. Furthermore, the association between
SES and co-morbidities could also potentially confound the relationship between SES
and mortality outcome.
The current study hypothesised that SES of the patients can affect their hospital
and long term outcome after critical illness, over and above the usual biological
explanation such as severity of acute illness and co-morbidities. So as such, SES may
potentially affect the performance of a prognostic scoring system when it is applied to
cohorts with different SES. Specifically, this study assessed the effects of SES on
hospital and long term survival of critically or seriously ill patients, after adjustment for
demographic factors, pre-existing co-morbidities, type of ICU admission, severity of
137
acute illness, and geographical accessibility to essential services. In this study all data
from the RPHICU since 1988 were used in the modelling process initially. When the
APACHE II predicted mortality was used as a covariate in the latter stage of the
modelling, only patients with the APACHE II predicted mortality data since 1989 were
included.
The results of this study showed that SES was not significantly associated with
hospital mortality regardless of whether or not adjustments were made for age, elective
admission, co-morbidities, severity of acute illness, accessibility to essential services
(measured by ARIA), and indigenous status. SES was, however, independently
associated with the long term survival of the patients after adjusting for age, type of
admission, co-morbidities, severity of acute illness, ARIA, and indigenous status. In fact,
a progressive increase in risk of (long term) death was observed from SES group I (least
disadvantaged) to group VI (most disadvantaged). The relationship between SES and
long term survival remained unchanged and significant when the analysis was restricted
to hospital survivors only.
These findings suggest that SES is not a significant factor in determining hospital
mortality in RPHICU where universal free access to intensive care services was
available. SES was an important factor in determining patients’ long term survival after
critical illness. This study has confirmed that indigenous status is an important
determinant of long term survival after a critical illness. As such, SES and ethnicity may
need to be considered in the modelling of long term outcome of critically ill patients and
these factors may potentially affect the performance of a long term prognostic scoring
138
system when it is applied to cohorts with different SES and ethnicity backgrounds (e.g.
private hospital patients).
The reason why SES is important in determining long term survival remains
speculative. Although the relationship between long term survival and SES was apparent
even after adjustment for many potential confounders or predictors of long term survival,
there are still other factors that can affect long term survival of patients. Some possible
modifiable factors that can explain the relationship between SES and long term survival
may include financial and cultural barriers to specialist medical services, poor nutrition,
overcrowded accommodation, smoking or alcohol use, and physical inactivity. These
issues have significant public health implications and merit further investigation.
Further details of this study are contained in the following published article:
Ho KM, Dobb GJ, Knuiman M, Finn J, Webb SA. The effect of socioeconomic
inequalities on outcomes of seriously ill patients: a linked data cohort study. Medical
Journal of Australia 2008;189:26-30.
144
Section six: Conclusion
Chapter 15. Summary and directions for future research
The APACHE II scoring system was assessed within an Australian context and
found to perform well over the past 10 to 15 years, including when the model was applied
to critically ill indigenous patients. Its performance was also reasonable when modified to
use the admission physiological and laboratory data only and this simple modification of
the APACHE II scoring system (the Admission APACHE II scoring system) represents
a viable simpler alternative risk adjustment tool to the traditional APACHE II scoring
system. The APACHE II scoring system had a better performance than two organ failure
assessment scores (SOFA and RPHICU organ failure score), but its performance was
only marginally improved when combined with one of the organ failure scores (SOFA
score). The severe co-morbidity data modelled in the APACHE II also appeared to
capture most of the hospital mortality risk associated with co-morbidity and no
significant improvement in the model performance was achieved by incorporating
multiple severe or minor co-morbidities into the model.
The APACHE II scoring system has significant limitations. Firstly, using meta-
analytic techniques and also the slope and intercept of the calibration curve, the
APACHE II scoring system was found to be poorly calibrated in some subgroups of
patients such as patients with multiple trauma. This finding was consistent with other
studies.82
Secondly, the APACHE II scoring system had a limited ability to predict other
undesirable in-hospital outcomes such as unplanned ICU readmission or unexpected
death after ICU discharge in an institution where these events were uncommon. The
145
APACHE II scoring system did not perform well when applied to patients readmitted to
ICU during the same hospitalisation. While co-morbidity may be a risk factor for late
ICU readmission, it could not account for the excess hospital mortality associated with
ICU readmission.
Age, gender, co-morbidity, the severity of acute illness, as measured by the
APACHE II predicted mortality in the APACHE II scoring system, intensity and duration
of intensive care therapy, socioeconomic status, and indigenous status were all important
determinants of long term survival of patients after their critical illness. Using seven pre-
selected predictors, a new prognostic model, the PREDICT model, that can estimate
median survival time and long term survival probabilities of critically ill patients was
developed and presented. This is indeed the first prognostic model that can be used to
estimate median survival time and also long term (>5-year) survival probabilities after a
critical illness. This is a preliminary model and its performance needs to be validated (or
improved) by other ICUs where access to long term survival data is available. It is likely
that the PREDICT model may not be generalisable to all other ICUs because long term
survival of a patient can be affected by factors that are not considered in this model (eg
socioeconomic status and ethnicity).
There are more opportunities for further research in the use of prognostic scoring
systems to predict outcomes of critically ill patients. The APACHE II is still widely used
in many ICUs but has significant limitations. The latest version of the APACHE model
(version IV) was published in 2006 and a new Simplified Acute Physiology Score (SAPS
III) has also been published in 2005.28,84
It is likely that these newer prognostic models
will replace the older APACHE II scoring system with time. The studies presented in this
146
thesis have demonstrated how these new scoring systems can be assessed in an Australian
context and also in predicting other undesirable in-hospital outcomes such as unplanned
ICU readmission and unexpected death after ICU discharge. A new ICU discharge
prognostic scoring system, using predictors available at the time of ICU discharge such as
organ dysfunction or markers of inflammation, may also be useful to triage patients into
different risk categories before ICU discharge. Risk stratification of critically ill patients
before ICU discharge merits further investigation.
Long term outcomes of patients after critical illness are important, however,
survival is only one of the outcomes patients and clinicians will consider. Quality of life
after critical illness is also very important. A prognostic scoring system or model that can
estimate long term survival as well as quality of life will be extremely useful to patients
and clinicians in making difficult treatment decisions. In order to achieve this goal, we
need a population based multi-centre long term observational study with long term
follow-up visits to assess the quality of life after critical illness. This will require a
significant amount of resources but will generate a vast amount of useful information to
formulate our future health policy. Western Australia is geographically isolated and with
a very low emigration rate. The linkage of comprehensive ICU databases of all Western
Australia ICUs to long term survival information will put Western Australia in a very
unique position to achieve this important goal.
147
References of the whole thesis
(1) Angus DC, Sirio CA, Clermont G, Bion J. International comparisons of critical
care outcome and resource consumption. Crit Care Clin 1997;13:389-407.
(2) Halpern NA, Bettes L, Greenstein R. Federal and nationwide intensive care units
and healthcare costs: 1986-1992. Crit Care Med 1994;22:2001-7.
(3) Poisal JA, Truffer C, Smith S, et al. Health spending projections through 2016:
modest changes obscure part D's impact. Health Aff (Millwood) 2007;26:w242–53.
(4) The Audit Commission. Critical to Success. The place of efficient and effective
critical care services within the acute hospital. London: Audit Commission for Local
Authorities and the National Health Service in England and Wales, 1999.
(5) Clarke T, Hart LG. Review of ICU Resources & Activity 00/01: ANZICS, 2002.
(6) Rechner IJ, Lipman J. The costs of caring for patients in a tertiary referral
Australian intensive care unit. Anaesth Intensive Care 2005;33:477-82.
(7) Acute Health Division DoHS. Review of intensive care in Victoria [Phase 1
report]. Melbourne: Department of Human Services, 1997.
(8) Ridley S, Morris S. Cost effectiveness of adult intensive care in the UK.
Anaesthesia 2007;62:547-54.
(9) Colagiuri S, Walker AE. Using an economic model of diabetes to evaluate
prevention and care strategies in Australia. Health Aff (Millwood) 2008;27:256-68.
(10) Hynes N, Sultan S. A prospective clinical, economic, and quality-of-life analysis
comparing endovascular aneurysm repair (EVAR), open repair, and best medical
treatment in high-risk patients with abdominal aortic aneurysms suitable for EVAR:
the Irish patient trial. J Endovasc Ther 2007;14:763-76.
(11) Kvåle R, Flaatten H. Changes in intensive care from 1987 to 1997 - has outcome
improved? A single centre study. Intensive Care Med 2002;28:1110-6.
148
(12) Rosenberg AL, Watts C. Patients readmitted to ICUs: a systematic review of risk
factors and outcomes. Chest 2000;118:492-502.
(13) Azoulay E, Adrie C, De Lassence A, Pochard F, Moreau D, Thiery G, Cheval C,
Moine P, Garrouste-Orgeas M, Alberti C, Cohen Y, Timsit JF. Determinants of
postintensive care unit mortality: a prospective multicenter study. Crit Care Med
2003;31:428-32.
(14) Knaus WA, Harrell FE, Lynn J, et al. The SUPPORT Prognostic Model;
objective estimates of survival for seriously ill hospitalized adults. Ann Intern Med
1995;122:191-203.
(15) Girard TD, Kress JP, Fuchs BD, et al. Efficacy and safety of a paired sedation
and ventilator weaning protocol for mechanically ventilated patients in intensive care
(Awakening and Breathing Controlled trial): a randomised controlled trial. Lancet
2008;371:126-34.
(16) Rocker G, Cook D, Sjokvist P, et al. Clinician predictions of intensive care unit
mortality. Crit Care Med 2004;32:1149-54.
(17) Girbes ARJ. Dying at the end of your life. Intensive Care Med 2004;30:2143-4.
(18) Griffith L, Cook D, Hanna S, et al., for the Level of Care Investigators and the
Canadian Critical Care Trials Group. Clinician discomfort with life support plans for
mechanically ventilated patients. Intensive Care Med 2004;30:1783-90.
(19) Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying
prognostic comorbidity in longitudinal studies: development and validation. J Chronic
Dis 1987;40:373-83.
(20) Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure
Assessment) score to describe organ dysfunction/failure. On behalf of the Working
149
Group on Sepsis-Related Problems of the European Society of Intensive Care
Medicine. Intensive Care Med 1996;22:707-10.
(21) Ferreira FL, Bota DP, Bross A, Melot C, Vincent JL. Serial evaluation of the
SOFA score to predict outcome in critically ill patients. JAMA 2001;286:1754-8.
(22) Kajdacsy-Balla Amaral AC, Andrade FM, Moreno R, Artigas A, Cantraine F,
Vincent JL. Use of the sequential organ failure assessment score as a severity score.
Intensive Care Med 2005;31:243-9.
(23) Gunning K, Rowan K. Outcome data and scoring systems. BMJ 1999;319:241-4.
(24) Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of
disease classification system. Crit Care Med 1985;13:818-29.
(25) Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG,
Sirio CA, Murphy DJ, Lotring T, Damiano A, et al. APACHE III prognostic system.
Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991;
100:1619-36.
(26) Cook DA. Performance of APACHE III models in an Australian ICU. Chest
2000;118:1732-8.
(27) Buist MD, Gould T, Haglet S, Webb R. An analysis of excess mortality not
predicted to occur by Apache III in an Australian level III intensive care unit. Anaesth
Intensive Care 1999;28;171-7.
(28) Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and
Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's
critically ill patients. Crit Care Med 2006;34:1297-310.
(29) Konarzewski W. Continuing to use APACHE II scores ensures consistency. BMJ
2000;321:383-4.
(30) Knaus WA. APACHE 1978-2001: The development of a quality assurance
150
system based on prognosis. Milestones and personal reflections. Arch Surg
2002;137:37-41.
(31) Ledoux D, Finfer S, McKinley S. Impact of operator expertise on collection of
the APACHE II score and on the derived risk of death and standardized mortality
ratio. Anaesth Intensive Care 2005;33:585-90.
(32) Khilnani G, Banga A, Sharma S. Predictors of mortality of patients with acute
respiratory failure secondary to chronic obstructive pulmonary disease admitted to an
intensive care unit: A one year study. BMC Pulm Med 2004;4:12.
(33) Goel A, Pinckney RG, Littenberg B. APACHE II predicts long-term survival in
COPD patients admitted to a general medical ward. J Gen Intern Med 2003;18:824-
30.
(34) Duke G, Santamaria J, Shann F, et al. Outcome-based clinical indicators for
intensive care medicine. Anaesth Intensive Care 2005;33:303-10.
(35) The Australian Council on Healthcare Standards. ACHS clinical indicator
summary guide. An approach to demonstrating the dimensions of quality. http://
www.achs.org.au (accessed on 18 October 2008).
(36) Paratz J, Thomas P, Adsett J. Re-admission to intensive care: identification of
risk factors. Physiother Res Int 2005;10:154-63.
(37) Duke GJ, Green JV, Briedis JH. Night-shift discharge from intensive care unit
increases the mortality-risk of ICU survivors. Anaesth Intensive Care 2004;32:697-
701.
(38) Ho KM, Liang J. Withholding and withdrawal of therapy in New Zealand
intensive care units (ICUs): a survey of clinical directors. Anaesth Intensive Care
2004;32:781-6.
151
(39) Griffiths J, Fortune G, Barber V, Young JD. The prevalence of post traumatic
stress disorder in survivors of ICU treatment: a systematic review. Intensive Care
Med 2007;33:1506-18.
(40) Williams TA, Dobb GJ, Finn JC, Webb SA. Long-term survival from intensive
care: a review. Intensive Care Med 2005;31:1306-15.
(41) Wright JC, Plenderleith L, Ridley SA. Long-term survival following intensive
care: subgroup analysis and comparison with the general population. Anaesthesia
2003;58:637-42.
(42) Ding J, Diez Roux AV, Nieto FJ, et al. Racial disparity in long-term mortality
rate after hospitalisation for myocardial infarction: the Atherosclerosis Risk in
Communities study. Am Heart J 2003;146:459-64.
(43) Tonne C, Schwartz J, Mittleman M, et al. Long-term survival after acute
myocardial infarction is lower in more deprived neighborhoods. Circulation 2005;
111:3063-70.
(44) Trewin D. Information Paper: Outcomes of ABS views on remoteness
consultation, Australia. Canberra: Australian Bureau of Statistics, 2001. 30.
(45) Jaro MA. Probabilistic linkage of large public health data files. Stat Med
1995;14:491-8.
(46) Holman CD, Bass AJ, Rouse IL, Hobbs MS. Population-based linkage of health
records in Western Australia: development of a health services research linked
database. Aust N Z J Public Health 1999;23:453-9.
(47) Health information Centre (Health Statistics Section). Hospital Inpatient
Summary (HA22) Reference Manual. Perth: Health Department of Western Australia,
1996.
152
(48) Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use
with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol
1993;46:1075-1079; discussion 1081-90.
(49) Trewin D. Socio-economic indexes for areas, Australia 2001. Information paper
2039.0, census of population and housing. Australian Bureau of Statistics, Canberra.
(50) Australian Institute of Health and Welfare 2004. Rural, regional and remote
health: a guide to remoteness classifications. AIHW cat. No. PHE 53. Canberra:
AIHW.
(51) Dobb GJ. Intensive care in Australia and New Zealand. No nonsense "down
under". Crit Care Clin 1997;13:299-316.
(52) Worthley LI. The ideal intensive care unit: open, closed or somewhere between?
Crit Care Resusc 2007;9:219-20.
(53) Jones DA, Cooper DJ, Finfer SR, et al. Advancing intensive care research in
Australia and New Zealand: development of the binational ANZIC Research Centre.
Crit Care Resusc 2007;9:198-204.
(54) Finfer S, Bellomo R, Lipman J, French C, Dobb G, Myburgh J. Adult-population
incidence of severe sepsis in Australian and New Zealand intensive care units.
Intensive Care Med 2004;30:589-96.
(55) Finfer S, Bellomo R, Boyce N, French J, Myburgh J, Norton R; SAFE Study
Investigators. A comparison of albumin and saline for fluid resuscitation in the
intensive care unit. N Engl J Med 2004;350:2247-56.
(56) Taori G, Ho KM, George C, Webb SA, Bellomo R, Hart G. Optimal time frame
end-point for assessing survival of critically ill patients: a comparative cohort study
(submitted for publication).
153
(57) Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein
concentration as a predictor of intensive care unit readmission: a nested case-control
study. J Crit Care 2006;21:259-65.
(58) Hanley JA, McNeil BJ. A method of comparing the areas under receiver
operating characteristic curves derived from the same cases. Radiology 1983;148:839-
43.
(59) Lemeshow S, Hosmer DW. A review of goodness of fit statistics for use in the
development of logistic regression model. Am J Epidemiol 1982; 115:92-106.
(60) Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model:
a descriptive study. J Clin Epidemiol 2007;60:746-51.
(61) Cox DR. Regression models and life tables (with discussion). Journal of the
Royal Statistical Society 1972;34:187-220.
(62) Wyatt JC, Altman DG. Prognostic models: clinically useful or quickly forgotten?
BMJ 1995;311:1539-41.
(63) Harrell FE Jr. Regression Modeling Strategies. New York: Springer; 2001.
(64) Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chapman &
Hall; 1993.
(65) Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective
sample sizes were required for external validation studies of predictive logistic
regression models. J Clin Epidemiol 2005;58:475-83.
(66) Arkes HR, Dawson NV, Speroff T, et al. The covariance decomposition of the
probability score and its use in evaluating prognostic estimates. Med Decis Making
1995;15:120-31.
(67) Nagelkerke NJ. A note on a general definition of the coefficient of determination.
Biometrika 1991;78:691-2.
154
(68) Kramer AA. Predictive mortality models are not like fine wine. Crit Care
2005;9:636-7.
(69) Stephens D. Critical illness and its impact on the aboriginal people of the top end
of the Northern Territory, Australia. Anaesth Intensive Care 2003;31:294-9.
(70) Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of
Acute Physiology and Chronic Health Evaluation (APACHE) II score with organ
failure scores to predict hospital mortality. Anaesthesia 2007;62:466-73.
(71) Williams TA, Dobb GJ, Finn JC, et al. Data linkage enables evaluation of long-
term survival after intensive care. Anaesth Intensive Care 2006;34:307-15.
(72) Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with
administrative data. Medical Care 1998;36:8-27.
(73) Angus DC. Grappling with intensive care quality-does the readmission rate tell
us anything? Crit Care Med 1998;26:1779-80.
(74) Beck DH, McQuillan P, Smith GH. Waiting for the break of dawn? The effects
of discharge time, discharge TISS scores and discharge facility on hospital mortality
after intensive care. Intensive Care Med 2002;28:1287-93.
(75) Ball C, Kirkby M, Williams S. Effect of the critical care outreach team on patient
survival to discharge from hospital and readmission to critical care: non-randomised
population based study. BMJ 2003;327:1014-7.
(76) Reny JL, Vuagnat A, Ract C, et al. Diagnosis and follow-up of infections in
intensive care patients: value of C-reactive protein compared with other clinical and
biological variables. Crit Care Med 2002;30:529-35.
(77) Kaben A, Corrêa F, Reinhart K, et al. Readmission to a surgical intensive care
unit: incidence, outcome and risk factors. Crit Care 2008;12:R123.
(78) Marmot MG. Status syndrome: a challenge to medicine. JAMA 2006;295:1304-7.
155
(79) Turrell G, Mathers CD. Socioeconomic status and health in Australia. Med J Aust
2000;172:434-8.
(80) Turrell G, Mathers C. Socioeconomic inequalities in all-cause and specific-cause
mortality in Australia: 1985-1987 and 1995-1997. Int J Epidemiol 2001;30:231-9.
(81) Volkers AC, Westert GP, Schellevis FG. Health disparities by occupation,
modified by education: a cross-sectional population study. BMC Public Health
2007;7:196.
(82) Alter DA, Chong A, Austin PC, et al.; SESAMI Study Group. Socioeconomic
status and mortality after acute myocardial infarction. Ann Intern Med 2006;144:82-
93.
(83) Chawda MN, Hildebrand F, Pape HC, Giannoudis PV. Predicting outcome after
multiple trauma: which scoring system? Injury 2004;35:347-58.
(84) Moreno RP, Metnitz PG, Almeida E, et al; SAPS 3 Investigators. SAPS 3--From
evaluation of the patient to evaluation of the intensive care unit. Part 2: Development
of a prognostic model for hospital mortality at ICU admission. Intensive Care Med
2005;31:1345-55.
156
Appendices
1. Confidentiality of Health Information Committee (CHIC) approval letter:
(#200321) Outcomes of critically illness in intensive care unit patients: RPHICU
linked data project.
2. Western Australian Aboriginal Health Information and Ethics Committee
(WAAHIEC) approval letter: (Ho 93-02/05) The short term outcome of critically
ill Aboriginal patients in a tertiary intensive care unit in Western Australia.
3. Western Australian Aboriginal Health Information and Ethics Committee
(WAAHIEC) approval letter: (158-02/07) The impact of socioeconomic status on
outcomes of critically ill patients: a linked data cohort study.
4. Western Australian Aboriginal Health Information and Ethics Committee
(WAAHIEC) endorsement to publish manuscript letter: (Ho 93-02/05) The impact
of socioeconomic status on outcomes of critically ill patients: a linked data cohort
study.
Top Related