Post on 25-Jul-2020
Going Beyond the Numbers The Midas+ Journey in Predictive Analytics
Vicky Mahn-DiNicola RN, MS, CPHQ,VP Research & Market Insights
Jim Kirkendall, MBA, VP Analytics
Learning Objectives
• Define “predictive analytics” and “predictive modeling”
• Explain “machine learning” and how it is used in predictive modeling
• Describe strategies for strengthening the power of predictive modeling
• Discuss the Midas+ Roadmap for Advanced Analytics
National Security and Law Enforcement
Numbers Geek Nate Silver
• Statistics guru
• Accurately predicted the
presidential winner in all
50 states in 2012
• Sports Statistics
• ESBN 548 Blog
Obama Campaign CTO Harper Reed
Predictive Analytics Sometimes Used in Spam Filtering
Systems to Identify the Probability that a Message is Spam
Self-driving Cars
Predictive Analytics: A Definition
•
•
How Will Advanced Analytics
Change Healthcare?
The “Big Question” about “Big Data”
Where can “big data” methods be best
applied to accelerate progress in
transforming healthcare delivery?
Can health care data be integrated in real
time and context aware?
Technical Limitations
Cultural Barriers to Access
Patient Engagement
Midas+ Advanced Analytics
Strategic Objective
11
• Create a scaling platform that combines clinical,
financial, claims, and public data from many sources
across many points of service for the purpose of
research and predictive capabilities.
• Deliver valuable ad hoc or subscription-based analytic
solutions across the healthcare industry.
5/14/2014
Advanced Analytics Foundational
Strategies at Midas+
• Build an analytics data warehouse
• Implement data modeling techniques
• Develop a data validation strategy
• Get the right people trained and focused
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
Effects of Data Volume on Machine Learning
Xerox Internal Use Only 14
• The best ML models perform worse with smaller data sets than more basic models with a vast amount of data.
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
2) Data Quality & Speed of Implementing Data
•
•
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
2) Data Quality & Speed of Implementing Data
3) HIPPA Laws
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
2) Data Quality & Speed of Implementing Data
3) HIPPA Laws
4) Codified Data from NLP to Achieve Universal Meaning
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
2) Data Quality & Speed of Implementing Data
3) HIPPA Laws
4) Codified Data from NLP
5) Decouple Analytics from Application
Analytics Data Warehouse Challenges
1) Data Volume and Marrying Many Data Sources
2) Data Quality & Speed of Implementing Data
3) HIPPA Laws
4) Codified Data from NLP
5) Decouple Analytics from Application
6) Include Descriptive Information with Predictive Alerts
Midas+ Advanced Analytics
Designed to be Solution Agnostic
Analytics Data Warehouse
Machine Learning Definition
• A branch of artificial intelligence
• The study of systems that can learn from data – Email filters to classify messages into spam and non-spam
– Amazon and Netflicks learn your preferences and preference of
others like you to suggest more movies and books you’ll like
– Cars that “learn” how to drive
• Some machine learning systems attempt to minimize the
need for human intuition in data analysis, but the intuition
of the data scientist is critical to specify how data will be
represented and which models will be used to search for
meaning
Null Hypothesis:
“Low Blood pressure does not increase a patient’s chances of
going to the ICU”
Null Hypothesis:
“Low Blood pressure does not increase a patient’s chances of
going to the ICU”
Null Hypothesis:
“Low Blood pressure does not increase a patient’s chances of
going to the ICU”
Statistical Learning can Help Us Compute
Correlations on LOTS of variables for a Prediction
Variable #1:
“Is Low Blood pressure a predictor for ICU admission? ”
Pr(>|t|)
Pr(>|t|) = .002 There is a strong correlation between low blood pressure
and ICU transfer
Modeling Techniques
Supervised Learning • Linear or Logistical Regression
• Lasso
• Neural Networks
• SVM (support vector machine)
• Random Forests/Boosted Trees
Unsupervised Learning • Hierarchical Clustering
Supervised Learning
= Socks
= Socks
= Socks = Shirts
= Shirts
= Shirts
Training Set
Supervised Learning
Shirt or Sock? ?
Result: 97% probability it’s a shirt.
Unsupervised Learning
Training Set
Unsupervised Learning
Group 1
Group 2 Group 2 Group 2
Group 1 Group 1
Training Set
Unsupervised Learning
Group 1 or Group 2? ?
Result: 97% probability it’s in Group 2.
How should we label Group 1 and Group 2?
Supervised Learning: Linear Regression
• Calculating the price of a home given age of the home,
square feet, school district, number of bedrooms
Supervised Learning: Logistical Regression
• Computes the probability of a
patient having cancer given
an examination of 800 cells.
• Designer of model can
determine the probability
threshold for when a
prediction of “yes” is made.
Supervised Learning: Lasso • Always better to use the fewest number of features in a model provided
removing features doesn’t lower your predictive abilities.
• When you are building a model and examining the use of a large
number of features, there are diminishing returns on using each
additional feature.
• Lasso helps you understand the performance impact of removing or
keeping each feature.
Supervised Learning: Neural Network
• The algorithm of choice for machine
driven cars.
• Works very well with audio, video,
and data.
• Can speed time manipulating data
since the benefit of the model is
that it learns important features of
the data through its hidden units.
• Training is computationally high.
Supervised Learning: Support Vector Machine
• High accuracy and with an
appropriate kernel they can work
well even if you’re data isn’t linearly
separable.
• Especially popular in text
classification problems where very
high-dimensional spaces are the
norm.
• Memory-intensive, hard to interpret,
and kind of annoying to run and
tune.
Supervised Learning: Random Forests &
Boosted Trees
• Random forests are starting to steal the crown.
• Used by Netflix to classify movies and predict user
preferences.
• They easily handle feature interactions and you
don’t have to worry about outliers or whether the
data is linearly separable.
Unsupervised Learning
• Hierarchical Clustering – Designer must choose number of clusters in advance
– Top-down or bottom-up approach. Top-down…all observations start in one
cluster, and splits are performed recursively
– Model measures the distance between pairs of observations and a
measures the dissimilarity of sets as a function
Two Considerations for Data Validation
• Warehouse – Data used to train the predictive
models must be accurate and
complete if we expect great
results
– Validate interfaced data and
structured data derived from
NLP engines
• Predictive Model – Need to validate the strength of
the prediction
– Need to demonstrate value to
clinicians
Warehouse Data Validation • Validity of data interfaced from the EMR or other sources
• Check the confidence of data sent across interface
• If outside of 2 Standard Deviations we convert it or don’t use it
• Performed on drugs, vitals, and other data elements
Hospital What Client Sends What Midas+ Stores
Hospital 1 98.6 F 98.6 F
Hospital 2 37 C 98.6 F
Hospital 3 377 C NULL
Hospital 4 37 F 98.6 F
Hospital 5 98.6 C 98.6 F
Warehouse Data Validation Check completeness of data by facility and encounter type
Encounter Type Hospital 1 Hospital 2 Hospital 3 Hospital 4
Inpatient 98% 99% 45% 99%
Emergency 22% 98% 88% 90%
Observation 78% 95% 90% 95%
1. What percentage of patients had a History & Physical?
2. What percentage of patients had a pain score?
Encounter Type Hospital 1 Hospital 2 Hospital 3 Hospital 4
Inpatient 95% 99% 95% 99%
Emergency 92% 95% 98% 0%
Observation 98% 95% 92% 94%
Data
completeness
monitoring
serves as a
“Force
Multiplier”.
Data Integrity
Oversight
for all Midas+
Applications
and Future
Predictive
Analytics!
Data Validation on
Predictive Model
• Sensitivity (True vs. False Positives)
• Specificity (True vs. False Negatives)
• How well are we handling missing data (Sparcity)
• How to validate alerts against clinical record
• How to demonstrate clinical value in the alert
Interpreting Predictive Classification Results
True Positive (Sensitivity)
False Positive
False Negative
True Negative (Specificity)
Four possible outcomes in Predictive Analytics
Statistical Primer for Predictive Analytics
True Positive (Sensitivity)
False Positive
False Negative
True Negative (Specificity)
Four possible outcomes in Predictive Analytics
Patient tests positive for the
disease but doesn’t really have it!
Statistical Primer for Predictive Analytics
True Positive (Sensitivity)
False Positive
False Negative
True Negative (Specificity)
Four possible outcomes in Predictive Analytics
Patient tests negative suggesting they are
healthy but they actually have the
disease
Statistical Primer for Predictive Analytics
True Positive (Sensitivity)
False Positive
False Negative
True Negative
(Specificity)
100%
0%
True Positive
Rate
False Positive Rate
ROC Curve is the Area Under the Line Also known as the C-Statistic
True Positive (Sensitivity)
False Positive
(Specificity)
False Negative
True Negative
100%
0%
True Positive
Rate
1.0 = Perfect
Line of no-discrimination
Points above the
Diagonal line Represent better
Than random
Points below the Diagonal line
Represent Worse Than random
0.5 = Random Coin Toss
False Positive Rate
CMS Readmission Calculator
C-Statistic = .60 to .63
100%
0%
True Positive
Rate
CMS Prediction Model Using Logistic Regression and Hierarchical logistic regression models on 2008 to 2010 data
performed better than random guessing
False Positive Rate
Acute MI = .63 Heart Failure = .60 Pneumonia = .63
Sparcity (missing data)
0700 am vitals:
BP 120/80
P 88
R 20
T (not documented)
We could use the previous temperature, use the next temperature, ignore it or
compute it based on a rule……
Nearest Neighbor Algorithm
• Modeling technique to
calculate the missing
data point.
• Take the mean of the
values with patients with
similar vital sign
“constellations”.
Developing Talent and Skills • Foundational Competencies
– Data warehousing
– Statistical Modeling
• Specialized Skills & New Learning – Learn new statistical software packages e.g. R, Matlab, Julia
– Learn PMML (Predictive Model Mark Up Language)
– Sytrue (NLP engine) integration
– Intersystems Solutions • DeepSee (cubing application for complex indicators)
• iKnow (NLP engine)
– Data Integration across multiple EMR systems in the US
• Insight Validation by Clinical Stakeholders
2014 Advanced Analytic Projects
50
Predictive ICU Admission Service
Predictive Readmission Service
Predictive Denied Days Service
Risk of Mortality and Severity of Illness
5/14/2014
Why the Urgency for a
Predictive ICU Admission Service?
• ICU admissions in U.S. have increased nearly 50% over six years
• 2.79 million in 2002-2003 to 4.14 million in 2008-2009
• ICU admissions contribute nearly 30% of all hospital costs
• 1 out of every 3 dollars spent on healthcare is “critical care”
• Traditional cost containment approaches to decrease critical care resource
utilization have been challenged as cost shifting instead of cost reducing: – Aggressive triage and discharge criteria
– Earlier discussion of end of life decision making and palliative care
– Earlier use of alternative care settings for longer term intensive care needs.
• A predictive algorithm is needed to improve patient outcomes, lower patient
comorbidities, and reduce hospital costs
• Clinicians need real-time actionable intelligence to impact decision making
52
Developing Our Hypothesis Using the
Modified Early Warning Score (MEWS)
1. Patients show clinical signs of
deterioration before going into
cardiac or respiratory arrest
2. Vital signs and lab
measurements can be used to
determine these signals of
clinical deterioration within 4
hours of the event
9 Month Pilot Using MEWS to Identify
Patients at risk for “Code Blue”
• Peninsula Regional Medical Center in Salisbury, Maryland (317 Bed
Tertiary Care)
• Over initial 9 month pilot there was a 67% decrease in code blues and
76% increase in rapid response team calls on initial medical surgical pilot
unit
• No code clues or mortalities in 3 months on a cardiac telemetry unit
• Pilot expanded to six medical surgical units resulting in 64% decrease in
code blues and 55% increase in rapid response calls
• PRMC estimates a potential annual savings of $3.2 million
54
http://www.mckesson.com/about-mckesson/newsroom/press-releases/2013/peninsula-
regional-medical-center-wins--mckesson-award-for-clinical-excellence
Modified Early Warning Score
• A simple guide used by hospital nursing and medical staff as well as emergency medical
services to quickly determine the degree of illness of a patient. It is based on data
derived from four physiological readings.
5/14/2014 55
• Alert-Voice-Pain-Unresponsive (will use in our 2nd iteration)
• Individual component scores are summed. Patients with a MEWS score of 5 or
higher are statistically linked to increased likelihood of death or admission to an ICU
(intensive care unit).
LTHT MEWS Leeds Teaching Hospitals Trust, England 2011
Can Machine Learning do better than MEWS?
5/14/2014 57
If a patient has a heart rate of 110, respiratory rate of 20, and temperature of
38.4 degrees they get 2 points. However, if that patient instead has a heart
rate of 111, respiratory rate of 21, and temperature of 38.5, they get 6 points.
Are these range cutoffs the best we can do?
Should the scoring model be identical regardless of the patient’s condition?
Can we incorporate additional pieces of data?
Next Iteration: Midas+ ICU Predictives Version 2
• Laboratory Results
– Hemoglobin
– Hematocrit
– Platelet Count
– Creatinine
– Sodium
– More
• Medications (all)
• Pain (normalized)
• Methodology Changes
– Temporal optimization
– Excluding patients that expired (used to be considered positive but we’re
concerned that the patients who died may
be skewing the learning….TBD)
– ICD-9 codes generated from NLP
– SNOMED codes
– MS-DRG assignment and relative
weights to active patients “in-
house” derived from ICD-9 codes
derived from NLP (used for real time
risk adjustment)
Initial Predictive ICU Admission Progress
• Developed temporal models for: – Blood Pressure -Temperature
– Heart Rate -Respiratory Rate
– 02 -Urine Output
• Only MEWS scoring on test set – identifies 217 of 862 ICU patients (25.17% sensitivity)
– 1215 false positives
Predictive ICU Admission Progress
with Machine Learning…
5/14/2014 60
• Using additional variables on test set
including attending physician specialty,
age, sex, time of admission.
– Identifies 433 of 862 ICU patients
(50.23% sensitivity)
– 190 false positives (99.1 % specificity)
– 7/10 alerts are true positives
Clinical Validation
• Three Groups – True Positives
• We predicted the patient needed ICU and went.
• If the alert had actually been “live in production”, would earlier
intervention likely have made a difference?
– False Negatives
• We did NOT predict the patients needed ICU but they went.
• Was there something special about this group that we missed?
– False Positives
• We predicated the patient needed ICU and they did NOT go
• Where there any interventions that we could see which would help us
train our learners?)
Training the Learners About Interventions that
May Reduce ICU Admissions
• Nasopharyngeal/oropharyngeal
suctioning
• Additional oxygen
• IV fluid bolus
• IV furosemide (Lasix) bolus
• Non-invasive positive pressure
ventilation
• Respiratory Treatments (albuterol)
• Narcan
• D50
• Vitamin K
Lessons Learned • Time of vital signs,
assessments or interventions
by nursing do not always match
other clinical data e.g. progress
notes, medical orders
• Times may reflect data
validation rather than actual
time of occurrence
• Data entry practices vary
across nursing units
• Workflow has to change so that
vitals and clinical observations
are entered into computer at
point of care immediately after
assessments
2014 Advanced Analytic Projects
64
Predictive ICU Admission Service
Predictive Readmission Service
Predictive Denied Days Service
Risk of Mortality and Severity of Illness
5/14/2014
Integrated Risk Adjustment
• Hierarchical clusters to create homogenous
clinical risk groups
• Risk of mortality and severity of illness scores
are computed for each individual encounter.
• Expected LOS and Expected Charges will be
computer for each encounter.
• Methodology can be run while the patient is in
the hospital and/or immediately after final
coding.
• Model is easily deployable within a Cache
infrastructure via a series of cache object script
routines for ease of maintenance.
We have the
data…..now what?
Time for Questions