Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose...

47
Explainable Prediction of Medical Codes from Clinical Text James Mullenbach Sarah Wiegreffe Jon Duke Jimeng Sun Jacob Eisenstein

Transcript of Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose...

Page 1: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Explainable Prediction of Medical Codes from Clinical Text

James Mullenbach

Sarah Wiegreffe

Jon Duke

Jimeng Sun

Jacob Eisenstein

Page 2: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem● Electronic Health Records have seen increasing

adoption in the last 5 years

2

Page 3: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem● Electronic Health Records have seen increasing

adoption in the last 5 years● They abound with written physician notes

3

Page 4: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem● Electronic Health Records have seen increasing

adoption in the last 5 years● They abound with written physician notes● ICD: taxonomy of diagnoses and procedures

4

Page 5: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem● Electronic Health Records have seen increasing

adoption in the last 5 years● They abound with written physician notes● ICD: taxonomy of diagnoses and procedures● Human coding laborious, error-prone [Birman-Deych et

al., 2005]

5

Page 6: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem, as an NLPer● Highly multi-label classification

○ Of nearly 9,000 labels, predict a subset

6

Page 7: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem, as an NLPer● Highly multi-label classification

○ Of nearly 9,000 labels, predict a subset● Testbed for document representations

7

Page 8: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The Clinical Coding Problem, as an NLPer● Highly multi-label classification

○ Of nearly 9,000 labels, predict a subset● Testbed for document representations● Documents are long and loosely structured

8

Page 9: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The dataset

● Open-access, de-identified

9

Page 10: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The dataset

● Open-access, de-identified● 47k admissions -> 47k documents for training

10

Page 11: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The dataset

● Open-access, de-identified● 47k admissions -> 47k documents for training● Loosely structured:

11

Admission Date: [**2118-6-2**] Discharge Date: [**2118-6-14**]

Date of Birth: Sex: F

Service: MICU and then to [**Doctor Last Name **] Medicine

HISTORY OF PRESENT ILLNESS: This is an 81-year-old femalewith a history of emphysema (not on home O2), who presents...

Page 12: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The dataset

● Open-access, de-identified● 47k admissions -> 47k documents for training● Loosely structured:

12

Admission Date: [**2118-6-2**] Discharge Date: [**2118-6-14**]

Date of Birth: Sex: F

Service: MICU and then to [**Doctor Last Name **] Medicine

HISTORY OF PRESENT ILLNESS: This is an 81-year-old femalewith a history of emphysema (not on home O2), who presents...

Long: Median post-processed document length: 1,341

Page 13: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

The dataset

● Open-access, de-identified● 47k admissions -> 47k documents for training● Loosely structured:

13

Admission Date: [**2118-6-2**] Discharge Date: [**2118-6-14**]

Date of Birth: Sex: F

Service: MICU and then to [**Doctor Last Name **] Medicine

HISTORY OF PRESENT ILLNESS: This is an 81-year-old femalewith a history of emphysema (not on home O2), who presents...

Long: Median post-processed document length: 1,341

Many labels: 519.1: ‘Other disease…’491.21: ‘Obstructive …’518.81: ‘Acute respir…’486: ‘Pneumonia, orga…’276.1: ‘Hyposmolality…’244.9: ‘Unspecified h…’31.99: ‘Other operati…’...

Median # labels: 14

Page 14: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Prior approaches● Predict from a subset of labels

14

This work● Predict from all labels

Page 15: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Prior approaches● Predict from a subset of labels● Sub-domain focus, e.g.

radiology

15

This work● Predict from all labels● General ICU setting

Page 16: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Prior approaches● Predict from a subset of labels● Sub-domain focus, e.g.

radiology● Private dataset

16

This work● Predict from all labels● General ICU setting● Open-access data

Page 17: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Modeling desiderata

...who sustained a fall at home she was found to have a large acute on chronic subdural hematoma with extensive midline shift...

17

● Focus on the parts that matter

Page 18: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Modeling desiderata

...who sustained a fall at home she was found to have a large acute on chronic subdural hematoma with extensive midline shift...

18

E849.0: Home accidents

801.26: ...subdural, and extradural hemorrhage...

● Focus on the parts that matter● Treat labels individually

Page 19: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Convolution

19

h1

h2

h3

patient

was

initially

placed

on

embed19

convolve

Page 20: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Convolution + Attention

20

U

α11

α21

α31

vh1

h2

h3

patient

was

initially

placed

on

parameter

variable

embed convolve attend (separately for each label)20

Page 21: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Convolution + Attention = the CAML model

21

U

α11

α21

α31

v

lossh1

h2

h3

patient

was

initially

placed

on

parameter

variable

classifyembed convolve attend21

β

Page 22: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Dealing with the long tail● Huge label space (nearly 9,000 total)

2222

Page 23: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Dealing with the long tail● Huge label space (nearly 9,000 total)● Many labels are similar250.00: “Diabetes mellitus without mention of complication, type II or unspecified type, not stated as uncontrolled”

250.02: “Diabetes mellitus without mention of complication, type II or unspecified type, uncontrolled”

2323

Page 24: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Dealing with the long tail: DR-CAML

24

U

α11

α21

α31

lossh1

h2

h3

patient

was

initially

placed

on

parameter

variable

“Atrial fibrillation” Max-pool CNN Label embedding

L2 similarity

24

v

β

Page 25: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

MIMIC results● Few works use the whole

label space

2525

*[Scheurwegs et al., 2017]

Page 26: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

MIMIC results● Few works use the whole

label space● 20%, 8% relative

improvement over prior SotA

26

*[Scheurwegs et al., 2017]

26

Page 27: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

More MIMIC metrics

27

● Enable future comparison● Precision @ k: decision support use-case

27

Page 28: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

But, is the model focusing on reasonable areas?● We want interpretability, especially in the clinical domain

2828

Page 29: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

But, is the model focusing on reasonable areas?● We want interpretability, especially in the clinical domain● Physician evaluation: select all text snippet(s) that explain the code

2929

Page 30: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

But, is the model focusing on reasonable areas?● We want interpretability, especially in the clinical domain● Physician evaluation: select all text snippet(s) that explain the code● Baselines: LogReg, CNN, cosine similarity

3030

Page 31: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

But, is the model focusing on reasonable areas?● We want interpretability, especially in the clinical domain● Physician evaluation: select all text snippet(s) that explain the code● Baselines: LogReg, CNN, cosine similarity● 100 random label-document samples

3131

Page 32: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

But, is the model focusing on reasonable areas?● We want interpretability, especially in the clinical domain● Physician evaluation: select all text snippet(s) that explain the code● Baselines: LogReg, CNN, cosine similarity● 100 random label-document samples● “Informative” and “Highly Informative”

3232

Page 33: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation example*Code: 575.4

Full descriptions: Perforation of gallbladder

“. . in the setting of gallbladder perforation secondary to acute acalculous cholecystitis after . . . . . . . . inhalation hospital1 times a day metronidazole mg tablet sig one tablet po tid times . . . . . . . to have an infection in your gallbladder requiring iv antibiotics and tube placement for . . .”

33

*not exact format used33

Page 34: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation example*Code: 575.4

Full descriptions: Perforation of gallbladder

“. . in the setting of gallbladder perforation secondary to acute acalculous cholecystitis after . . . . . . . . inhalation hospital1 times a day metronidazole mg tablet sig one tablet po tid times . . . . . . . to have an infection in your gallbladder requiring iv antibiotics and tube placement for . . .”

34

*not exact format used

attention

34

Page 35: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation example*Code: 575.4

Full descriptions: Perforation of gallbladder

“. . in the setting of gallbladder perforation secondary to acute acalculous cholecystitis after . . . . . . . . inhalation hospital1 times a day metronidazole mg tablet sig one tablet po tid times . . . . . . . to have an infection in your gallbladder requiring iv antibiotics and tube placement for . . .”

35

*not exact format used

attentionCNN

Page 36: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation example*Code: 575.4

Full descriptions: Perforation of gallbladder

“. . in the setting of gallbladder perforation secondary to acute acalculous cholecystitis after . . . . . . . . inhalation hospital1 times a day metronidazole mg tablet sig one tablet po tid times . . . . . . . to have an infection in your gallbladder requiring iv antibiotics and tube placement for . . .”

36

*not exact format used

attention cosine sim.CNN

Page 37: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation example*Code: 575.4

Full descriptions: Perforation of gallbladder

“. . in the setting of gallbladder perforation secondary to acute acalculous cholecystitis after . . . . . . . . inhalation hospital1 times a day metronidazole mg tablet sig one tablet po tid times . . . . . . . to have an infection in your gallbladder requiring iv antibiotics and tube placement for . . .”

37

*not exact format used

attention cosine sim.CNN

LogReg

Page 38: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation results● Improves upon CNN,

LogReg

3838

Page 39: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Physician evaluation results● Improves upon CNN,

LogReg● More experts needed!

3939

Page 40: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Future work● Exploit the loose structure of discharge summaries

○ Some have already done this [Shi et al., 2017]

4040

Page 41: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Future work● Exploit the loose structure of discharge summaries

○ Some have already done this [Shi et al., 2017]● Sparsify or structure attention to improve interpretability, accuracy

○ “fusedmax” [Niculae & Blondel, 2017]

4141

Page 42: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Future work● Exploit the loose structure of discharge summaries

○ Some have already done this [Shi et al., 2017]● Sparsify or structure attention to improve interpretability, accuracy

○ “fusedmax” [Niculae & Blondel, 2017]● Develop methods for few-shot or zero-shot classification

4242

Page 43: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Future work● Exploit the loose structure of discharge summaries

○ Some have already done this [Shi et al., 2017]● Sparsify or structure attention to improve interpretability, accuracy

○ “fusedmax” [Niculae & Blondel, 2017]● Develop methods for few-shot or zero-shot classification● Exploit hierarchy of codes

○ GRAM [Choi et al., 2017]

43

250.0 / \250.00 250.01

DM type 2 DM type 1

43

Page 44: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Summary● ICD coding is valuable and challenging

4444

Page 45: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Summary● ICD coding is valuable and challenging● Convolution + attention works well

4545

Page 46: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Summary● ICD coding is valuable and challenging● Convolution + attention works well● Attention can explain the predictions

4646

Page 47: Explainable Prediction of Medical Codes from Clinical Text · Future work Exploit the loose structure of discharge summaries Some have already done this [Shi et al., 2017] Sparsify

Summary● ICD coding is valuable and challenging● Convolution + attention works well● Attention can explain the predictions

4747

Thank you!Questions?

Code, data splits, pretrained models: github.com/jamesmullenbach/caml-mimic


Explainable Artificial Intelligence: Objectives ...

Explainable Artificial Intelligence: Objectives ...

Explainable and Interpretable Machine Learning

Explainable and Interpretable Machine Learning

Towards Explainable Requirements Engineering Tools · Towards Explainable Requirements Engineering Tools @andivogelsang andreas.vogelsang@tu-berlin.de 07.01.2019 Technische Universität

Towards Explainable Requirements Engineering Tools · Towards Explainable Requirements Engineering Tools @andivogelsang [email protected] 07.01.2019 Technische Universität

Exploit Kit - IWSEC · Exploit Kit Drive-by Download Exploit Kit Exploit Kit MWS2015 Improving Cyber Attack Detection System To Adopt The Changing Of Exploit Kit ... Angler Exploit

Exploit Kit - IWSEC · Exploit Kit Drive-by Download Exploit Kit Exploit Kit MWS2015 Improving Cyber Attack Detection System To Adopt The Changing Of Exploit Kit ... Angler Exploit

Fairness-Aware Explainable Recommendationover Knowledge …

Fairness-Aware Explainable Recommendationover Knowledge …

Explainable Artificial Intelligencefsvg.math.ntua.gr/RulesLogApps/2019/NickBassiliades-XAI.pdf · 2019-12-27 · Explainable Artificial Intelligence Nick Bassiliades, Professor School

Explainable Artificial Intelligencefsvg.math.ntua.gr/RulesLogApps/2019/NickBassiliades-XAI.pdf · 2019-12-27 · Explainable Artificial Intelligence Nick Bassiliades, Professor School

Explainable Artificial Intelligence (XAI)alanwags/DLAI2016/(Gunning) IJCAI-16... · Explainable Artificial Intelligence (XAI) ... (Approved for Public Release, ... • The system

Explainable Artificial Intelligence (XAI)alanwags/DLAI2016/(Gunning) IJCAI-16... · Explainable Artificial Intelligence (XAI) ... (Approved for Public Release, ... • The system

Dynamic Explainable Recommendation based on Neural Attentive …yongfeng.me/attach/dynamic-explainable-recommendation.pdf · 2018-12-10 · namic preference in the context of explainable

Dynamic Explainable Recommendation based on Neural Attentive …yongfeng.me/attach/dynamic-explainable-recommendation.pdf · 2018-12-10 · namic preference in the context of explainable

Explainable Neural Computation via Stack Neural Module …openaccess.thecvf.com/content_ECCV_2018/papers/Ronghang_Hu_Explainable... · Explainable Neural Computationvia Stack Neural

Explainable Neural Computation via Stack Neural Module …openaccess.thecvf.com/content_ECCV_2018/papers/Ronghang_Hu_Explainable... · Explainable Neural Computationvia Stack Neural

Adaptive Explainable Continual Learning Framework for ...

Adaptive Explainable Continual Learning Framework for ...

Explainable AI: the basics - European Commission

Explainable AI: the basics - European Commission

Explainable Machinery Faults Prediction Using Ensemble ...

Explainable Machinery Faults Prediction Using Ensemble ...

Explainable Deep Reinforcement Learning for Portfolio ...

Explainable Deep Reinforcement Learning for Portfolio ...

XTE: Explainable Text Entailment - arXiv

XTE: Explainable Text Entailment - arXiv

Explainable and Explicit Visual Reasoning Over Scene Graphsopenaccess.thecvf.com/content_CVPR_2019/papers/Shi... · ing this, we can insulate the “low-level” visual perception

Explainable and Explicit Visual Reasoning Over Scene Graphsopenaccess.thecvf.com/content_CVPR_2019/papers/Shi... · ing this, we can insulate the “low-level” visual perception

SUPER DRAGONBALL HEROES SHI -CPB SHI -09 [R] SHI-S9[C] SHI ... · super dragonball heroes shi -cpb shi -09 [r] shi-s9[c] shi-io[r] shi -so shi -cpa shi -gcpi shi -04 [r] sh i -14

SUPER DRAGONBALL HEROES SHI -CPB SHI -09 [R] SHI-S9[C] SHI ... · super dragonball heroes shi -cpb shi -09 [r] shi-s9[c] shi-io[r] shi -so shi -cpa shi -gcpi shi -04 [r] sh i -14

MOLECULE OPTIMIZATION BY EXPLAINABLE EVOLUTION

MOLECULE OPTIMIZATION BY EXPLAINABLE EVOLUTION

Explainable AI for Tailored Electricity Consumption ...

Explainable AI for Tailored Electricity Consumption ...

Combinatorial Methods for Explainable AI

Combinatorial Methods for Explainable AI

Explainable AI: the new 42?

Explainable AI: the new 42?