Brittney Goldberg Caitlin Anderson Lincoln Anderson Patty Anderson.
The MD Anderson / IBM Watson Announcement: What does it mean for machine learning in healthcare?
-
Upload
health-catalyst -
Category
Healthcare
-
view
647 -
download
0
Transcript of The MD Anderson / IBM Watson Announcement: What does it mean for machine learning in healthcare?
Is it really a setback, in general, or not?March 1, 2017
Dale Sanders Executive Vice President, Software
Forbes Magazine: “MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine”
Let’s Make Things Very Clear• IBM and Watson are a frequent competitor to Health Catalyst• I do NOT celebrate the difficulties of our competitors, especially IBM• “There but by the grace of God, go I”• Watson’s success begets Health Catalyst’s success• Were it not for IBM, I wouldn’t have a career in information
technology• IBM was the backbone of the Air Force information systems that taught me so
very much
Opening Salvo to Stir Things Up • Tying Watson to a “cancer moonshot” created the peak of already inflated
expectations about Watson• Every executive and politician wants to be John F. Kennedy
• We have a generation of political and corporate executives who don’t understand technology and software, even though it’s running their world
• Executives are selling technology they don’t understand and executives are buying technology they don’t understand
• Information asymmetry always leads to an exploited consumer
• Technology professionals have a moral and ethical obligation to speak up when they see this happening
Agenda• My background as it relates to this topic• The fundamental data challenges of applying Watson to healthcare• Health Catalyst’s approach to machine learning and AI in healthcare
• I’m not selling here… I’m just informing you about a different approach• History will be the judge about whether the Catalyst approach works or not
• These slides are purposely bland… this webinar is not about selling Health Catalyst
Data, data, data… for decision supportMy Background
1983 2016
B.S. Chemistry, biology minor
US Air Force Command, Control, Communication, Computers & Intelligence (C4l) Officer
Reagan/Gorbachev Summits
TRW/National Security Agency• START Treaty• Nuclear Non-proliferation• Nuclear command & control
system threat protection• Knowledge Based Systems
Commercialization
Nuclear Warfare Planning and Execution-- NEACP & Looking Glass
Intel Corp, Enterprise Data Warehouse
• Chief Data Guy• Regional Director of
Medical Informatics, Intermountain Healthcare
• CIO, Northwestern
• Chief Data Warehousing Guy
CIO, Cayman Islands National Health System
Product Development, Health Catalyst
The Over-Hype of AI in the 1990s• I lived it. I hyped it.• Military and credit reporting systems managed the largest databases
in the world at the time• They pale in comparison to Silicon Valley data content today
• My team at TRW, the Knowledge Based Systems Group, was tasked with commercializing our military and intelligence technology in expert systems, fuzzy logic systems, neural nets, and genetic algorithms
• Our first target was healthcare. Sound familiar?
I presented the following six slides at a conference in Feb 2012, exactly one year after Watson’s victory on Jeopardy, when hopes for Watson were very high in medicine. I was a fairly lonely contrarian.
8
What About Watson?
Watson
First, a little background on Dale Sanders Natural Language Processing and Text Mining
Watson is revolutionary. It’s the first thing in my IT career that really excited me… everything
else has been incremental or variations of the same flavor
9
10Watson’s Technology
Apache Unstructured Information Management Architecture (UIMA) Hadoop Java, C++
Lexicals and ontologies DBPedia, WordNet, and Yago
IBM Content Analytics with Enterprise Search 90 IBM Power 750 servers enclosed in 10 racks 16 Terabytes of memory A 2,880 processor core Linux based
11What is Watson?
Near-word associations coupled with semantic mapping and zillions of sources of knowledge… digitized books, encyclopedias, news feeds, magazines, blogs, Wikipedia, etc.
Equivalent to approximately 240 million pages, in memory
Jeopardy answer “A famous red quaffed clown or just any incompetent fool”
Watson’s correct answer “Who is Bozo?”
Watson searched its indexes for near-word associations, recognized that Bozo was the most common word in the indexes that was missing from the question
12Watson’s Problem With Healthcare
Watson’s training set for Jeopardy was a HUGE collection of human wisdom, academic and otherwise, stretching back thousands of years
What’s the training set for healthcare wisdom? A few decades of clinical trials journals? Claims processing data from a dysfunctional healthcare system that doesn’t include patient
outcomes? Progress notes? Radiology reports? Pathology reports?
Watson is not going to impact healthcare in the near term like many hope it will
13Factoids
More than 50% of all medicines are prescribed, dispensed or sold inappropriately
Less than 40% of patients in the public sector and 30% in the private sector are treated according to clinical guidelines
World Health Organization, May 2010
Key Points• Watson is a text-centric, Natural Language Processing (NLP) engine
• Millions of “near word associations” are processed in seconds
• Although related at some level, that’s different than a generic pattern recognition approach to machine learning used for discrete data and images
• NLP: ”Find things for me faster in all this text.”
• Machine Learning: “Make decisions and suggestions for me, and learn from each decision and suggestion.”
Key Points• 80% of healthcare data is text-based clinical notes and diagnostic
reports, if you don’t count digital images, but that’s still not very much data in terms of sheer volumes, and the quality and consistency of that data varies considerably across clinicians
• The source of Watson’s primary knowledge base in healthcare-- peer-reviewed journals and clinical trials data-- is relatively small in terms of volume and has questionable value in day-to-day healthcare
• Watson’s training set for Jeopardy was at least 100x larger than what’s available to train Watson for healthcare
IBM Watson “Learning” Acquisitions• Phytel• Explorys• Truven• Merge• If the fundamental design of Watson is NLP and text-centric, will these
acquisitions help Watson learn?
Is training Watson on chemotherapy and radiation therapy protocols the right strategy for treating and
preventing cancer?I would argue that it’s not. Current cancer treatment strategies will go down in history alongside bloodletting and trepanation. We need to
apply Watson and similar technology breakthroughs on something other than optimizing the status quo, which is anything but great.
The Cancer Data EcosystemThis is the data you need to prevent and treat cancer. Do we have this data in high volume, across many patients, with reasonable quality and consistency?
No.
• Genomics
• Lifestyle
• Epigenetics
• Microbiome
• Environmental
• Traditional healthcare delivery data
• Quality and length of life outcomes data for long-term survivors
• All the above on healthy patients so we understand the target condition
Health Catalyst’s Approach to AI and Machine Learning
Semantics• Machine learning is one thing. Machine doing is another.• In my definition, it’s not Artificial Intelligence until the machine acts
on your behalf.• We’ll get there in healthcare, but it will take a long time.• In the meantime, I prefer “Suggestive Analytics” based on machine
learning.
Our Simple MissionOur mission is to organize the data in healthcare and make it accessible,
useful, and valuable to the clients, patients, and families we serve.
With data, all things are possible. Without it, not much.
Our fundamental strategy for Machine Learning:
Integrate text and discrete data to inform the vectors and clusters in our models
Your machine learning aspirations must be tempered by the data that’s available, both in breadth and depth.Ironically, it’s easier for us to model and predict bad things in healthcare right now, than good things. We have more data about bad outcomes than good outcomes.
No Data, No Machine Learning• Moore’s Law: Chips double in capacity every 18 months• Sanders’ Law: Machine learning models double in capability every 6
months• But without data content, the models are of no use
25
For the most part, this is the simple three-part pattern recognition model that we are building and that, I would argue, healthcare should broadly pursue
Patients like this [pattern]
Who were treated like this
[pattern]
Had these outcomes and costs [pattern]
The Human Health Data Ecosystem
And, by the way, we don’t have much of any data on healthy patients
We Are Not “Big Data” in Healthcare, Yet
27
Data Volume vs. Machine Learning Model
“But invariably, simple models and a lot of data trump more elaborate models based on less data.”
•“The Unreasonable Effectiveness of Data”, March 2009, IEEE Computer Society; Alon Halevy, Peter Norvig, and Fernando Pereira, of Google
28
29
Google’s Self Driving Car drove 80 million miles before it ever touched a road
Think of a computer sitting in the seat of this computerized driving simulator, not a human
30
Retina: The data collection system for
Feature extraction
Cerebral cortex: The data base and algorithms for Classification
& Clustering
The more times you go through this loop with different ”data”, the faster
and better you become at feature extraction and classifying “people”
31
Pattern recognition process
Data acquisition
Data reduction
Feature extraction
Classification & Clustering
Confidence evaluation
EHRs, billing, outcomes data, lab, meds, vitals, supply chain, et al
Cleaning out the noisy or bad data, identifying general patterns
These are properties of the object. Finding new and specific ways to identify new categories and representations of patient types, outcomes, events, encounters, episodes
Using the features to assign patterns to the categories and representations
Evaluating and correcting the confidence in the model’s output
32
The challenges in healthcareData acquisition
Data reduction
Feature extraction
Classification & Clustering
Confidence evaluation
Very limited data. We think we are big data, but we’re not and generally, what limited data we have, is about sick patients, not healthy patients.
How, then, do we extract Features that Classify a healthy patient so we know how to achieve that “Healthy Patient” pattern?
If we don’t collect outcomes data, how then do we identify the Features to Classify a healthy or sick patient with good or bad outcomes?
33
ess of Predictive AnalyticsThe Machine Learning loop
33
• In healthcare, we have, essentially, no outcomes data, so this is an open loop• If you don’t have a strategy for intervention, predicting something for the sake of
predicting has no value
Troubling factoid
• Of the 1,958 quality metrics in the National Quality Measures Clearinghouse, only 7% of those
measure clinical outcomes and less than 2% of those are based on patient reported outcomes
34N Engl J Med 2016; 374:504-506, February 11, 2016
Thank you for the graphs, PreSonus
Healthcare and patients are continuous flow, analog process and beings
But, if we sample that analog process enough, we can approximately recreate it with digital data
35
We are treating physicians and nurses as if they were digital sampling devices.
“Every new click of the mouse you guys ask me to do, all in the name of data, sucks another piece of my soul away.” --Beleaguered primary care physician
37
Predictive and suggestive analytics in the same
user interface
The efficacy and costs of antibiotic protocols
for inpatients
The Antibiotic Assistant at Intermountain Healthcare: The First Triple Aim
Antibiotic Protocol Dosage Route Interval Predicted
EfficacyAverageCost/Patient
Option 1 500mg IV Q12 98% $7,256
Option 2 300mg IV Q24 96% $1,236
Option 3 40mg IV Q6 90% $1,759
• Antiinfective drugs• Average Savings per Patient = $280
• Cost of Hospitalization• Average Savings per Patient = $13,759
• Annual Savings (12-bed ICU)• Est. Total Savings per Year = $7,925,184
New England Journal of Medicine January 22, 1998
Economic Impact
• 30% reduction in Adverse Drug Events• 27.4% reduction in Mortality• 99.1% “on-time” delivery of pre-operative antibiotics• 84.5% reduction in post-operative antibiotic use• Stabilized antibiotic resistance
Annals of Internal Medicine May 15, 1996
Quality of Care Impact
The Shark Tank Story
• Chicago-based healthcare IT startups• Three hours of 15 minute presentations• Incredibly creative ideas at the
application layer of technology• Absolutely no answer for, or conceptual
understanding of, the challenges at the healthcare data layer
41
This is not an HIE, Clinical Data Repository, or Enterprise Data Warehouse. It’s a little bit of all three but better.
Health Catalyst Data Operating System
Kernel
Metadata
Data Ingest
Real-time Streaming
Machine Learning
NLP
Source Connectors
Catalyst Analytics Engine Core Services
Data Processing
Secure Messaging
Security, Identity& Compliance
Health Catalyst Fabric
RegistriesTerminology & Groupers
EHR Integration ISVsPRBLeading Wisely
Catalyst Apps
Care Management
Apps
Alerting FHIR
Big Data
SAMD & SMD
Measures Patient & Provider Matching
Atlas
Risk Classifications
Patient Attribution
Data QualityData Governance
Data Pattern Recognition
Data Export
43
New Generation Product Briefing
Health Catalyst Data Operating SystemMachine Learning Foundation1
catalyst.ai
• Our machine learning models• Our strategy for embedding machine
learning into all of our products
2 healthcare.ai
• Our tools to automate machine learning tasks
• Democratizing machine learning by releasing as open-source
3
44
healthcare.ai
Our open-source machine learning software product
Automates key tasks in developing models, or customizing existing models using local
data
Makes deployment in an analytics
environment easy and ‘production quality’
45
New Generation Product BriefingScaling People
Data Architects
Great domain knowledge Often looking for opportunities to advance career/skills
With the right tools…
Data architects make great feature engineers Data architects can easily get started in predictive
analytics.
With healthcare.ai, you have the people to do data science right now.
46
The healthcare.ai project listCentral Line-Associated Bloodstream Infection (CLABSI) Risk – Clinical Decision SupportCongestive Heart Failure, Readmissions Risk – Clinical Decision SupportCOPD, Readmissions Risk – Clinical Decision SupportRespiratory (COPD, Asthma, Pneumonia, & Resp. Failure), Readmission Risk – Clinical Decision SupportPredictive Appointment No-shows – Operations and Performance ManagementPre-surgical Risk (Bowel) – Clinical Decision Support and client requestPropensity to Pay – Financial Decision SupportPatient Flight Path, Diabetes Future Risk – Clinical Decision SupportPatient Flight Path, Diabetes Future Cost– Clinical Decision SupportPatient Flight Path, Diabetes Top Treatments – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (Glaucoma) – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (Retinopathy) – Clinical Decision SupportPatient Flight Path, Diabetes Next Likely Complications (ESRD) – Clinical Decision Support
In Development
Built
Planned
Sepsis Risk – Clinical Decision SupportPost-surgical Risk (Hips and Knees) – Clinical Decision SupportCharge-denial Risk – Financial Decision SupportCharge-grouping Guidance – Financial Decision SupportPredictive ETL Batch Load Times – PlatformHospital Length of Stay – Operations and Performance ManagementHospital Census – Operations and Performance Management
CAUTI and VTE – Clinical Decision Support Risk-adjusted Comparisons Across Health Systems – CAFÉ1-yr Admission Risk – Population Health and Accountable CareBronchiolitis Admissions Risk – Clinical Decision SupportEmergency C-section Risk – Clinical Decision SupportPalliative Care vs Invasive Procedure Guidance – Clinical Decision SupportMortality Risk in Pre-term Births – Clinical Decision SupportRegistry Automation via Unsupervised Learning – Clinical Decision SupportMortality Risk in PICU – Clinical Decision Support
47
Predictive SeedlingsBronchiolitis Admissions Risk
Emergency C-section Risk
Palliative Care vs Invasive Procedure Guidance
Mortality Risk in Pre-term Births
Mortality Risk in PICU
Deep Learning for Large Tabular Data (1M+ rows)
Patients Like This – Modifiable Risk-factor Recommendation for Patient Attributes
Patients Like This – Optimal Treatment Recommendation
Registry Automation via Unsupervised Learning
Radiology Image Classification via Deep Learning
Pathology Image Classification via Deep Learning
Currently possible with healthcare.ai and the right data
Roadmap for healthcare.ai
In Summary• Watson was overhyped, overbought, oversold… Not maliciously, but
rather, probably naively• But it will have a big impact on society• Healthcare data ecosystem is just not quite ready for Watson,
especially the text content that Watson thrives on• We have a bright future ahead for machine learning in healthcare, if
you adjust your strategy and expectations according to the data content that’s available