February 19, 2013: I. Sim Decision Support Systems Medical Informatics – Epi 206 Decision Support...
-
Upload
dustin-roberts -
Category
Documents
-
view
215 -
download
2
Transcript of February 19, 2013: I. Sim Decision Support Systems Medical Informatics – Epi 206 Decision Support...
February 19, 2013: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision Support Systems
Ida Sim, MD, PhD
February 19, 2013
Division of General Internal Medicine, and the Center for Clinical and Translational Informatics
UCSF
Copyright Ida Sim, 2013. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Outline
• Decision support systems– background, definition
• How decision support systems “think”– rule-based systems
– neural networks
• CDSS effectiveness & adoption
• Decision support in the Age of Watson
3
Big Picture of Health Informatics
Virtual Patient
Transactions
Raw data
Medical knowledge
Clinical research
transactions
Raw research
data
Dec
isio
n su
ppor
t
Med
ical
logi
c
PATIENT CARE / WELLNES RESEARCH
Workflow modeling and support, usability, cognitive support, computer-supported cooperative work (CSCW), etc.
Clinical Decision Support Systems
EHR
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
What is a “Decision”? “Logic”?
• An action that consumes resources in the real world• Logic
– Oxford English Dictionary• reasoning conducted or assessed according to strict principles
of validity
– Merriam Webster• interrelation or sequence of facts or events when seen as
inevitable or predictable
• something that forces a decision apart from or in opposition to reason
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision or Logic?
Decision Logic
Diabetics with hypertension should be started on ACEI, ARB, or other
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision or Logic?
Decision Logic
Diabetics with hypertension should be started on ACEI, ARB, or other
X
I prescribe lisinopril for Mrs. Chan (diabetic, BP 156/92)
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision or Logic?
Decision Logic
Diabetics with hypertension should be started on ACEI, ARB, or other
X
I prescribe lisinopril for Mrs. Chan (diabetic, BP 156/92) X
I prescribe amlodipine for Mrs. Chan (diabetic, BP 156/92)
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision or Logic?
Decision Logic
Diabetics with hypertension should be started on ACEI, ARB, or other
X
I prescribe lisinopril for Mrs. Chan (diabetic, BP 156/92) X
I prescribe amlodipine for Mrs. Chan (diabetic, BP 156/92) X
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Clinical Decision Support
• Clinical decision support system (CDSS)– software that is designed to be a direct aid to clinical decision-
making
– in which the characteristics of an individual patient are matched to a computerized clinical knowledge base
– and patient-specific assessments or recommendations are then presented to the clinician and/or the patient for a decision (Sim et al, JAMIA, 2001)
• Examples of clinical decisions to be supported?
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Major Target Tasks of CDSSs• Diagnostic support
– DxPlain, QMR, iTriage• Drug dosing
– aminoglycoside, theophylline, warfarin• Preventive care
– reminders for vaccinations, mammograms• Disease management
– diabetes, hypertension, AIDS, asthma• Test ordering, drug prescription
– reducing daily CBCs in hospital, drug allergy checking• Utilization
– referral management, clinic follow up
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
What Isn’t a CDSS
• Medline• UpToDate• Static guideline repositories
– www.guideline.gov (National Guideline Clearinghouse)
• Online laboratory data, test results, chart notes
• Retrospective quality improvement reports– how your vaccination rates compare to your
colleagues’
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
A CDSS?
• Chief complaint: “Symptoms for $400 please”
• Symptom: Chest pain and shortness of breath
• Dr. Watson: What is pulmonary embolism!
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Outline
• Decision support systems– background, definition
• How decision support systems “think”– rule-based systems
– neural networks
• CDSS effectiveness & adoption• Decision support in the Age of Watson
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Basic Decision Support Task
• Decision support– given starting conditions and a defined set of action choices,
recommend or rank action choices for user• Requires some “thinking” to recommend or rank
– strictly deterministic thinking– thinking with fuzziness and probabilistic features
• in the starting data or the reasoning procedure
• in the outcomes (e.g. prob. of adverse reaction)– often involves thinking about concepts (e.g., “abnormal”) as
well as numbers• symbolic vs. quantitative computing
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Decision Support “Thinking”• Strictly deterministic, e.g.,
– first-order logic rule-based systems
– adhoc rule-based systems (non-mathemetical reasoning about probability)
• e.g., if high WBC AND cough AND fever AND abn. CXR then likelihood of pneumonia is 4 out of 5
• Probabilistic/fuzzy, e.g.,
– bayesian networks• formal probabilistic reasoning, extension of decision analysis
– neural networks
– fuzzy logic, genetic algorithms, case-based reasoning, etc., or hybrids of these
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Forward Chaining Rules
• Forward chaining/reasoning (data-driven)– start with data, execute applicable rules, see if
new conclusions trigger other rules, and so on– example
• if HIGH-WBC and COUGH and FEVER and ABN-CXR => PNEUMONIA
• if PNEUMONIA => GIVE-ANTIBIOTICS• if GIVE-ANTIBIOTICS => CHECK-ALLERGIES• if PNEUMONIA and GIVE-ANTIBIOTICS and NOT
(ALLERGIC-DOXYCYCLINE) => GIVE-DOXYCYCLINE
– use if sparse data
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Backward Chaining Rules
• Backward chaining/reasoning (goal-driven) – start with “goal rule,” determine whether goal rule
is true by evaluating the truth of each necessary premise
– example • patient with lots of findings and symptoms• is this lupus? => are 4 or more ACR criteria satisfied?
– malar rash?– discoid rash?– skin photosensitivity? etc
• if 4 or more ACR criteria true => systemic lupus– use if lots of data
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Rule Reasoning Problems
• Combinatorial explosion of rules– need rule for each contingency
• if MOD-WBC and COUGH and FEVER and ABN-CXR => PNEUMONIA
• Rules may be contradictory– if COUGH and ABN-CXR => INTERSTITIAL-LUNG-DZ
• Rules may be circular
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Representational Challenges
• Need to use standard vocabulary terms– need to manage evolution of vocabularies (e.g., changing
terminologies in psychiatry: no Asperger’s in new DSM-V)• Rules may involve complex semantic relationships
– if NEPHROPATHY caused-by DIABETES• caused solely by? predominantly by?
• what if I’m not sure? 20% sure? 80% sure?
– if SINUSITIS greater than 6 months• representing temporal relationships requires 2nd order logic
• Need knowledge engineering and clinical expertise to build and maintain the knowledge base over time– need to keep rules up-to-date with latest evidence
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Sharing Rules
• Why not have libraries of rules?• Reusable, central upkeep, evidence-based...• Many attempts, none yet successful
– AHRQ library of e-recommendations
– Morningside public-private partnership1
• included VA, Kaiser, DoD, AMIA, Partners, Intermountain, ASU, etc.
– Epic users• difficult to share rules and CDSSs across Epic
installations1http://www.tatrc.org/docs/2010-8-6-Morningside-Article.pdf
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Summary of Rule-Based Systems
• Deterministic, relatively simple reasoning• Combinatorial explosion even for small
domains• Requires extensive knowledge engineering
and clinical expertise • Rules are difficult to share• But remain most widely used method due to
simplicity for small problems
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Outline
• Decision support systems– background, definition
• How decision support systems “think”– rule-based systems
– neural networks
• CDSS effectiveness & adoption• Decision support in the Age of Watson
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Neural Networks• Finds a non-linear relationship between input parameters
and output state• Structure of network
– usually input, output, and 1-2 hidden fully connected layers
– each connection has a “weight”
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
NN for MI Diagnosis• Inputs (e.g., all patient characteristics in the EHR)
• EKG findings (ST elevation, old Q’s)
• rales (Yes, No)
• JVD (in cm)
• Outputs are the set of possible outcomes/diagnoses
EKG findings
Rales
JVD
Response to TNG
Acute MI
No Acute MI
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Training the Neural Network• Network gets “trained”
– give examples of known patients and diagnoses• can handle missing data
– system iteratively adjusts connection weights to find the network “pattern” that associates sets of input variables (patients) with right output state (MI or not)
• Test accuracy on another set of patients• In Baxt’s MI neural network
– training set: 130 pts with MI, 120 without– test set: 1070 UCSD ER patients with anterior chest
pain
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Baxt’s Acute MI Neural Net• Evaluation results: prevalence of MI 7% (Lancet, 1996)
• Results were driven by non-standard predictors– rales, jugular venous distention
• Why wasn’t this neural network used more widely?– “black box” nature limits explanatory ability and lessens
acceptance– users have to input the variables manually
• interfacing to EHRs would increase adoption– need to define and code “rales” and other input terms
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Outline
• Decision support systems
– background, definition• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS effectiveness & adoption• Decision support in the Age of Watson
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Is Decision Support Effective?
• 2005 systematic review of CDSS effectiveness1
– diagnosis: 4/10 (40%) studies beneficial
– reminder systems: 16/21 (76%)
– disease management systems: 23/37 (62%)
– drug dosing: 19/29 (66%)
– few studies improved patient outcomes: 7/52 (13%)
• Counted the number of systems in each category that were “effective” (p>0.05)– but CDSS not all the same! (apples and oranges)
1Garg et al. JAMA 2005 293(10):1223-1238
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
CPOE and Medication Safety• 1998: CPOE reduced medication errors 55%1
• 2005: Qualitative study found 22 error types promoted by CPOE, quite common2
• 2008: Systematic review of 10/543 citations, no RCTs3 – 5 studies (P <= .05) for ADE reduction, 5 n.s.
• 2011: CPOE part of Stage 1 Meaningful Use criteria– “more than 30% of patients with at least one medication on
their medication list have at least one medication ordered through CPOE”
1Bates JAMA 1998;280:1311-1316.2Koppel JAMA. 2005 Mar 9;293(10):1197-2033Wolfstadt J Gen Intern Med. 2008 Apr;23(4):451-8.
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
CDSS Running Example
• Hypertension treatment Clinical Decision Support System (CDSS)– Clinic has an EHR
– During patient visit, CDSS notes that BP and trend is too high
– CDSS checks patient’s Cr, diabetes status, cardiac status, current meds and allergies and recommends drug therapy change according to JNC VII guidelines and insurance coverage
– Presents e-prescription for MD to verify. If verified, order is sent directly to pharmacy and medication list updated
Taxonomy of CDSSs
OR
INFORMATION DELIVERY•Delivery format•Delivery mode•Action integration•Delivery interactivity/explanation availability
System user/Target decision
maker
DECISION SUPPORT•Reasoning method•Clinical urgency•Recommendation explicitness•Logistical complexity•Response requirement
CONTEXT•Target decision maker•Clinical setting•Clinical task•Unit of optimization•Relation to point of care•Potential external barriers to action
WORKFLOW•Degree of workflow integration
System user/Output
intermediary [ ]
Target decision maker
KNOWLEDGE/DATA SOURCEClinical knowledge source [ ]Patient data source [ ]Data source intermediary [ ]Degree of customizationUpdate mechanism
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
CDSS Effectiveness Summary
• Systematic review of systematic reviews on “Impact of eHealth on Quality & Safety”– “…many of the clinical claims made about the most
commonly deployed eHealth technologies cannot be substantiated by the empirical evidence.”1
• Findings limited by– methodological problems and design type of studies
– insufficient appreciation of workflow component of CDSSs
– insufficient appreciation of heterogeneity of systems
1Black et al, PLoS Med 2011 8(1):e1000387
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Low CDSS Adoption
• Adoption of CDSSs beyond simple reminders– < 10% of those with EHRs
• Reasons – informatics
– technical
– organizational / financial
– fundamental conundrum
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Informatics Barriers• Requires computation across Data, Information,
Knowledge– data is often qualitative, fuzzy
• how to represent “looks sick,” “severe pneumonia”
– information (meta-data) often not easily available• e.g., seen in another ER last week for same problem
– lots of tacit vs. explicit knowledge required• Most CDSSs are rule-based systems
– combinatorial explosion, rules not shared, updated...– inability to handle probabilistic outcomes, values
• Computer best at data-intensive simplistic deterministic decisions (augmenting intelligence) vs. knowledge-intensive, probabilistic, value-based decisions
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Technical Barriers
• CDSS has to interface to local data systems– manual double-entry input is a no-go
– Meaningful Use establishes CCD1 as standard EHR exchange format
• e.g., Problem List, Allergies, K+ value
• Exchange standard may not be “granular” enough for CDSS– e.g., Allergies as free text, vs. med and reaction
• Need standardized (i.e., coded) input data – e.g., what’s in the Past Medical History field?
1http://www.hl7.org/implement/standards/cda.cfm
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Organizational Barriers
• CDSSs are complex workflow interventions– high requirement for complementary innovations
– requires organizational change leadership and expertise
• Incentives/rewards for better quality still unclear under new ACO rules
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Outline
• Decision support systems
– background, definition• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS effectiveness & adoption• Decision support in the Age of Watson
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Artificial Intelligence Holy Grail
• Machine intelligence, like HAL-9000 in 2001 A Space
Odyssey, or Data in Star Trek
• IBM’s Deep Blue beat chess grandmaster Gary
Kasparov (1997)– chess is a highly structured game with defined rules and
solutions (just a lot of them)
– but Deep Blue didn’t help solve protein folding problems
• Watson beat all time Jeopardy! winner in 2011
• What kind of artificial intelligence do we need for health
care?
Watson in the Big Picture
Virtual Patient
Transactions
Raw data
Medical knowledge
Clinical research
transactions
Raw research
data
Dec
isio
n su
ppor
t
Med
ical
logi
c
PATIENT CARE / WELLNES RESEARCH
Workflow modeling and support, usability, cognitive support, computer-supported cooperative work (CSCW), etc.
Watson
Nuance voice recognition
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
The Jeopardy!/Medical Logic Problem
• Jeopardy, like chess, is a narrow game• a “question answering” game requiring “natural language”
processing” (= NLP)
• Question answering is a specific kind of intelligence– "The antagonist of Stevenson's Treasure Island” -- “Who is Long-
John Silver?,” vs.
– “What triggered the revolution in Egypt?” “What causes Chronic
Fatigue Syndrome?” vs.
– Book the cheapest, most convenient transportation for a 4-city trip
to Spain this summer
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
The Jeopardy!/Medical Logic Problem
• Voice recognition (picking out words from speech)– Watson: was given questions in written text
– Clinical: Dragon Dictate etc moderately good for restricted domains (e.g., radiology)
• Understand the sentence/question– Watson: “The antagonist of Stevenson’s Treasure Island”
– Clinical: “What antibiotics treat pertussis?”• Go look for candidate answers in the corpus of knowledge
– Watson: free text Project Gutenberg, wikipedia, dictionaries, encyclopedias, newspaper articles, etc.
– Clinical: EHR, PubMed, UpToDate, all of Internet? free text, images, numbers, video, data streams (eg GPS, ICU data)
• Score answers for likely “correctness”• Give best answer (or rank answers and be able to explain why)
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Example Jeopardy! Process
• http://blog.reddit.com/2011/02/ibm-watson-research-team-answers-your.html
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Similarities/Differences
• very large scope
• natural language full of puns, ambiguities
• corpus is free text only • all fact based• there exists one and only one
right answer• right answer is in the corpus
somewhere, requiring only syntactic (ie grammatical) processing to get at
• is “one shot”
• very large scope• clinical notes and literature highly
idiosyncratic natural language• corpus includes text, numbers,
images (MRIs), video (eg echo) • not only facts (should pt. be on
warfarin to prevent stroke?)• often no single right answer, best
answer requires semantic (I.e., meaning) processing
– e.g., “azithromycin,” critical appraisal of literature
• often requires back and forth (e.g., to clarify context, values, constraints)
Clin Decision SupportJeopardy!
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
“Data + Watson”
. .
Doc
Computer: Ms. Lee has had paroxysmal cough
for 2 weeks, with emesis.
Adult pertussis is a strong possibility.
Symptom inquiry, diagnosis using
neural network or rule-based system
. .
Doc
What is the current incidence of pertussis?
17.8 cases / 100,000 in S.F. county Jan thru December
2010
Question answering: public health
reports, data, culture results, etc.
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
“Data + Watson”
Your patient is 4 months post-partum. I suggest treating presumptively for
pertussis.*
Rule-based checking of EHR
. .
Doc
I agree. Don’t macrolides treat pertussis?
Yes, erytho, clarithro and azithromycin are the preferred antibiotics. Bactrim is second-
line.
Question answering: reference sources, literature
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
“Data + Watson”I would suggest azithromycin 500 mg on Day 1 and
250 mg on Days 2 to 5.**CDC guidelines 2005, local cultures uniformly sensitive to azithro,
pt not allergic, azithro covered by insurance
Question answering and rule-based checking of allergies, insurance, local sensitivities
. .
Doc
Make it so!
CPOEAPEX
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Dr. Watson
• Went to medical school in 2011– ingested textbooks, PubMed. Took board exam questions,
solved NEJM cases
• Went to “residency” in 2012 at Memorial Sloan Kettering’s cancer patient records– has now analyzed 605,000 pieces of medical evidence, 2 m
pages of text, 25,000 training cases, assisted by 14,700 clinician hours
• Is now for sale through exclusive reseller Wellpoint– for doctors: reads the medical record and makes recommendations in
decreasing order of confidence– for oncologists: states which treatment is most likely to succeed– for insurers: what treatment should be authorized for payment (“90%
accurate”)
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Other Clinical AI Systems
• Question Answering– askHermes http://www.askhermes.org/
• Diagnostic support– http://dxplain.org/dxp/dxp.pl
– http://www.isabelhealthcare.com/home/default
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Fundamental CDSS Conundrum
• Better quality care <-- better decision support• Better decision support <-- “smarter” systems
– “know” more about the patient, evidence, context• “Smarter” systems <-- more richly coded D-I-K
– for EHR: SNOMED, standard EMR structure– for knowledge: coding, structures for guidelines,
RCTs…• Coded data <-- Coding of data entry• Coding of data entry <-- Greater physician time• Greater physician time --> no play --> no gain
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Implications
• Clear trade-off between physician coding effort and “smarter” decision support– can NLP help? do we really need to “code” if we have Watson-like
abilities to understand (un)natural language
• For now, don’t expect more decision support than coding allows
– generally successful decision support• preventive care: age, last mammogram, etc.
• allergies: Yes/No on specific drugs
• drug dosing: weight, height, creatinine, age
– generally unsuccessful decision support• diagnostic assistance
• complicated therapies (e.g., management of hypertension, treatment of depression)
Brave New World
“fully expects Watson to be widely deployed wherever the Clinic does business by 2020.”1
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Chris Coburn, executive director for innovations, Cleveland Clinic
http://www.forbes.com/sites/bruceupbin/2013/02/08/ibms-watson-gets-its-first-piece-of-business-in-healthcare/
February 28, 2012: I. Sim Decision Support SystemsMedical Informatics – Epi 206
Summary on Decision Support
• Most CDSSs are rule-based• Equivocal evidence of benefit
– workflow/organizational inputs underappreciated
• Fundamental trade-off between – effort of coding data and quality of decision support
• Greater decision support adoption will require– wider EHR use, better interoperability, more coding or far
more powerful NLP
• Need to be realistic on what decisions most computers can support