Iterative Development of Sepsis Detection Algorithms for the Emergency Department

Developing Diagnostic Systems for Clinical Care

Peter Haug, MD.Homer Warner Center for Informatics ResearchIntermountain Healthcare

Diagnostic Modeling and SepsisSubjects for todays presentation

• Electronic Data and Diagnostic Models• Two Active Sepsis Systems for the ED

– Supporting the Sepsis Bundle– Research on Septic Shock

• A Development Environment to Train Diagnostic Algorithms– A Semi-Automated Development Environment– Using “Ontologies” to Describe Diagnostic Relationships– Supporting Iterative Development of Diagnostic Algorithms– Built-In Evaluation Tools

• Example: Developing a Comprehensive Sepsis Model?

Multiple Episodes of Care

Research and Analysis

Data Warehouse: Available for

Analysis

Electronic Health Record

Clinical Data Serves Us Twice

Medical Decision Support

Supporting Care/Collecting

Data

Page 4

Introducing Bayesian Networks A Graphical Diagnostic Modeling Tool

• Development: Good Authoring Tools– Models Trained from data– Or Created Manually– Easily Displayed and Inspected– (Partially) consistent with Intuition

• Processing: Good Runtime Characteristics– Models Work well with Missing Data– Resilient with Redundant Data– Can Generate Multiple Outputs– Produces Probabilistic Outputs

Case 1: Detecting Sepsis for the Sepsis Bundle

• The “Sepsis Bundle” is a set of 11 clinical interventions necessary for quality care of the septic patient.

• Early detection is required to support the initial interventions in the Sepsis Bundle.

• Need to identify “Possible Sepsis” patients within 2 hours of admission.

• Use the earliest available data that can discriminate Septic patients.

• Data used: Heart Rate, Age, Temperature, WBC, MeanBP

Detecting Sepsis for the Sepsis BundleGoal: Identify Sepsis Patients within 2 hours of admission.

Simple Sepsis Model: Designed for Early Detection

Effectiveness of the Sepsis Detector• Developed on ED patients from 2006-2008• Validated on ED patients from 2009-2011.• Five features chosen from 100+.• Naïve Bayesian Model.• Compared to:

• SIRS Criteria at 1 hour.• SIRS Criteria at 12 hours.• Manual Triage Nurse alert.

Table 2: Comparative PerformanceThrshld Sens. FPR PPV AUC

Deterministic AlertsTriage Alert N/A 0.556 0.033 0.064 0.761SIRS 2+ by 1hr N/A 0.719 0.147 0.020 0.786SIRS 2+ by 12hr N/A 0.858 0.186 0.019 0.836Probabilistic ModelAs Deployed 0.050 0.781 0.049 0.062 0.866Probabilistic Model at Alternate Alert Thresholds…Match Triage Alert Sensitivity 0.329 0.556 0.017 0.119 0.769…Match Triage Alert False Pos. 0.096 0.716 0.033 0.082 0.841

A Second Sepsis ModelIdentifying Septic Shock Patients for a Research Study

• Developed to Identify Patients for a Clinical Trial

• Goal: Identify Sicker Septic Patients– Patients with Shock– Patients Who Will Require Resuscitation

• A Rule-Based Approach– Implemented Using Bayesian Network

Technologies

Page 8

Heart_Rate (HeartRate) = if(HeartRate >= 90, Met, Unmet)

Ontology-Driven Modeling System: Input and OutputUses an Ontology to Semi-automatically Create a Diagnostic System

• Inputs:– Analytic Health Repository/Enterprise Data Warehouse– A Diagnostic Ontology. – Diagnostic Scenario

• Outputs:– A diagnostic predictive model produced by the system.– A list of the features found to be relevant to the

diagnostic problem addressed.– An analysis of the quality of the predictive model.– The raw data generated by the system and used to

develop the diagnostic model.

Data Warehouse/Analytic Health

Repository

DiseaseOntology

Concept Retrieval (from Ontology

Concept Translation to EDW Representation

Output

20%

20%

20%20%

20%

iii dfPdP

dfPdPfdP

)|()(

)|()()|(

Prediction Algorithm

Analysis ResultsAnalytic Data

Relevant Ontologic Concepts

Analysis Design Utility

Analytic Workbench· Diagnostic Models· Model Comparisons· Model Explanation

(by reference to the Ontology)

Natural Language

Processing Subsystem

Structural Knowledge Retrieval from the Ontology

Data Retrieval from the Analytic Health Repository

Ontology-Driven Model DiscoveryUsing knowledge embedded in ontologies to automate research?

• Ontologies capture medical knowledge in a computable form

• They capture hierarchical relationships (staph pneumonia is

a bacterial pneumonia”)

• They capture clinical relationships (“disease causes finding”

• They capture synonyms (pneumococcal pneumonia = ICD9:

481)

• An Research Automation Tool using an Ontology Can:

• Help select research patients

• Identify and extract relevant data

• Condition preliminary analysis of the data

• Support visualization of this data

• Return Data and results to the user for further study

Ontologies Describe How Diseases Are Related(according to ICD9)

Pneumococcal pneumoniaPneumococcal pneumonia

ICD9: 481

Other Bacterial PneumoniaOther bacterial pneumonia

ICD9: 482

Streptococal PneumoniaPneumonia due to Other

StreptococcusICD9: 482.3

BronchopneumoniaBronchopneumonia,

organism unspecifiedICD9: 485

Viral PneumoniaViral pneumonia

ICD9: 480

Staphlococcal PneumoniaPneumonia due to

StaphylococcusICD9: 482.4

Hemophilus PneumoniaPneumonia due to

Hemophilus influenzae ICD9: 482.2

Pseudomonas PneumoniaPneumonia due to

PseudomonaICD9: 482.1

Pneumonia

More Bactierial Pneumonias

Staph Aureus PneumoniaPneumonia due to

Staphylococcus, unspecifiedICD9: 482.40

MSSA Staph PneumoniaMethicillin Susceptable

Staph Aureus (MSSA) Pneumonia

ICD9: 482.41

MRSA Staph PneumoniaMethicillin Resistant Staph Aureus (MRSA) Pneumonia

ICD9: 482.42

Other Staph PneumoniaOther Staphylococcus

pneumoniaICD9: 482.49

Bacterial Pneumonia More Pneumonias

Ontologies can Also Describe How Clinical Data are Related to Diseases

has_X-ray_Manifestation

PneumoniaPneumonia, Organism

unspecifiedICD9: 486

Pneumococcal pneumoniaPneumococcal pneumonia

ICD9: 481

Pneumonia

More Bacterial Pneumonias

Bacterial Pneumonia More Pneumonias

has_Sign

White Blood CountHematology: White

Blood CountLOINC: 62239-9

has_Altered_Lab_Value

Pulmonary RalesSigns: Chest

Auscultation-RalesPTXT:

28.1.3.22.34.2.1.32

TemperatureVital Signs:

TemperatureLOINC: 8310-5

has_Altered_VS

Localize InfitrateX-ray Finding:

Localized InfiltrateSNOMED: 128309002

has_Micro_Manifestation

Other Bacterial PneumoniaOther bacterial pneumonia

ICD9: 482

More Manifestations

has_??_Manifestation

Sputum Culture: Positive

SNOMED: 442773002

+ Sputum Culture

Building a New Sepsis DetectorUsing the Diagnostic Modeling Tools

• Example: a simple Sepsis model– Built from 186,725 patients and 91 features– Model restricted to 15 features– Uses the simplest Bayesian Model (Naïve Bayes)

• A completely automated process.

Using Modeling Tools to Refine a Sepsis Detector

• The Diagnostic Modeling System Provides Tools to Modify Models

• Two Pathways to Improve Accuracy– Refine Model– Extend Data

• Refining a Model– Changing Feature Relationships– Adding Feature Aggregation Constructs

• Extending Training Data– Deriving New Features from Existing

Data (Summary Features)

ODMS

Extracted Data

Analytic Health

Repository

Diagnostic Model

iii dfPdP

dfPdPfdP

)|()(

)|()()|(

5 4 3 2 1

Analytic Results

Modified Diagnostic Model

iii dfPdP

dfPdPfdP

)|()(

)|()()|(

Modified Data

Analytic Workbench· Process Data from AHR· Produce Predictive Model· Test Predictive Model

Ontology

Submit New Model for Evaluation

Submit Modified Data for

Evaluation

Adding Semantics to the ModelSaying “Findings are consistent with Diagnosis”

Orientation_Consistent

TrueFalse

11.388.7

Oriented_x_3

AbsentPresent

11.388.7

Alert_and_oriented_x_3

AbsentPresent

37.262.8

Oriented_to_situation

AbsentPresent

99.9.060

Not_oriented_to_time

AbsentPresent

94.15.92

Not_oriented_to_situation

AbsentPresent

99.9.052

Not_oriented_to_place

AbsentPresent

97.03.04

Not_oriented_to_person

AbsentPresent

99.60.44

Not_oriented_to_S_O

AbsentPresent

100.003

Not_Oriented_x_3

PresentAbsent

0.4999.5

SepsisPresentAbsent

1.7598.2

PneumoniaPresentAbsent

0.4999.5

CXR_Consistent

FalseTrue

94.55.47

Multi_Lobar_Infiltrates

PresentAbsentUnknown

33.333.333.3

CXR_Ordered

PresentAbsent

0 100

Single_Lobe_Infiltrate

AbsentPresentUnknown

33.333.333.3

“Chest X-ray Results consistent with Pneumonia” “Mental Status consistent with Sepsis”

The Results of Diagnostic Model DevelopmentProviding tools to critique model accuracy

• Inspecting model accuracy can be done Graphically.– ROC Curves– Precision/Recall Tradeoff Graphs

• Relative value of features can also be critiqued.– Graphical and Tabular views.

Comparing Two Models Using the ROC Curves

Inspecting a Model’s Tradeoffs in Accuracy

Feature Extraction/Selection• Sepsis Features returned by the Analytic Workbench

Page 17

• Relevant features identified using links in the Ontology– Links from Diagnoses to Observations– Standard naming conventions

• Features are evaluated and ranked• The most informative subsets are used for

analysis.

Sepsis Model: Extending Its Coverage

• Initial Model: Designed to function within the first 2 hours

• Revised Model: Designed to function anytime within the first 24 hours

Simple Sepsis Model: Designed for Early DetectionComplex Sepsis Model: Effective at Multiple Times?

Results of Analysis of New Sepsis Models Accuracy

• Data: 280,143 patients of whom 4976 had in ICD 9 diagnosis of sepsis. ‐• Divided into a training set (186,725 patients) and a test set (93,418 patients).• 91 features associated with sepsis which were available during the initial day of care.• 118 features were available after feature expansion.• Built 12 different Sepsis Diagnostic Models

Naïve Bayes-OS

Naïve Bayes-ES

TAN Bayes-OL

TAN Bayes-EM

Custom Bayes-N

aïve

Custom Bayes-T

AN0.000.100.200.300.400.500.600.700.800.901.00 0.95 0.97 0.95 0.97 0.97 0.97

0.300.38

0.27

0.37 0.39 0.40

AROCPrecision at 60%

Conclusion

“All models are wrong; some models are useful.” Attributed to statistician George Box

• Diagnostic Models have a role in clinical care

• Enterprise Data Warehouses can contribute to Model development

• Stored Medical Knowledge (in the form of Ontologies) can accelerate Modeling

• Bayesian Models can flexibly represent a variety of clinical conditions.

Questions???

Comments and Questions