Image Labeling for Deep Learning: Human vs. Machine · 2018-09-26 · Image Labeling for Deep...
Transcript of Image Labeling for Deep Learning: Human vs. Machine · 2018-09-26 · Image Labeling for Deep...
Image Labeling for Deep Learning:Human vs. MachineSeptember 10, 2018
Curtis P. Langlotz, MD, PhDProfessor of Radiology and Biomedical InformaticsDirector, Center for Artificial Intelligence in Medicine & Imaging (AIMI)Associate Chair, Information Systems, Department of RadiologyMedical Informatics Director for Radiology, Stanford Health Care
Annotation:an explanation or comment added to a text or diagram
Labeling:a descriptive or identifying word or phrase
http://www.radiologyassistant.nl/ https://arxiv.org/abs/1603.08486
Radiologist Labels1: No Significant Abnormality
4: Possible Significant Abnormality, May Need Action
9: Critical, Clinical Notified
1.5 million studies
Leslie Zatz, MDhttps://whatsnext.nuance.com/healthcare/radiologists-role-in-patient-centered-care/
Penn “Code Abdomen”Solid Organ Masses
0: Incompletely evaluated. See RECOMMENDATION.1: No mass.2: Benign. No further evaluation needed.3: Indeterminate. Future imaging follow up may
be needed. See RECOMMENDATION.4: Suspicious. May represent malignancy.
5: Highly suspicious. Clear imaging evidence of malignancy.
6: Known cancer.7: Completely treated cancer.
Zafar, H et al. JACR 2015; 12(9):947-50
Reverse-Index Radiology Report Search
5
Reverse-Index Radiology Report Search
6
https://www.youtube.com/watch?v=iSQHelJ1xxUhttps://github.com/HazyResearch/snorkel
Snorkel: Data Programming for Weakly-Supervised Machine Learning
“Crowd” Labeling
• Part of speech• Porter stemmer• Word shape • NegEx negation• RadLex ontology class
Information Extraction Results
Hassanpour, S & Langlotz, CP. Artif Intell Med 23(1):84-9, 2016.
• Part of speech• Porter stemmer• Word shape• NegEx negation• RadLex class
Word Representations
“You shall know a word by the company it keeps”(Firth, J. R. 1957:11)
Adapted from https://www.slideshare.net/BhaskarMitra3/a-simple-introduction-to-word-embeddings
0
(proximal, -2)
0
2
0
(proximal, -1)
1
1
0
0
(sclerotic, -2)
(sclerotic, -1)
(fibular, -1) (fracture, +1)
(metastasis, +1)
0 1 0 10 1
(humeral, -1)
0 1 0 10 1
0 0 1 01 0
2 0 1 01 0
similar
similar
“proximal fibular fracture”“proximal humeral fracture”“sclerotic fibular metastasis”“sclerotic humeral metastasis”
fibular
fracture
humeral
metastasis
Wor
dsContexts
Empiric Utility of Word Embeddings
http://nlp.stanford.edu/
https://doi.org/10.1148/radiol.2017171115
Information Extraction Results
Zhang, Y & Langlotz, CP. Artif Intell Med 23(1):84-9, 2016.
Effect of Noisy Labels on Training Data Requirements
0%
100%
200%
300%
400%
500%
600%
700%
0% 5% 10% 15% 20% 25% 30%
V Agarwal et al. Learning Statistical Models of Phenotypes Using Noisy Labeled Training Data. JAMIA 23 (6): 1166–73, 2016.
HU Simon. General Bounds on the Number of Examples Needed for Learning Probabilistic Concepts. J Comput System Sci 52 (2): 239–54, 1996.
NLP Accuracy
Dat
a Se
t Siz
e
Noise
Noisy data
Clean data
Effect of Noisy Labels on Accuracy
https://arxiv.org/abs/1805.00932
17
EDEMAINFILTRATECONSOLIDATIONPNEUMONIAATELECTASISMASSNODULEEMPHYSEMAPLEURAL THICKENINGEFFUSIONFIBROSISPNEUMOTHORAXCARDIOMEGALYHERNIA
Label Hierarchy
CheXpert: A Large-Scale Uncertainty-Labeled Dataset for Multi-LabelClassification of Observations in Chest Radiographs
Your Ontology May Vary
• Alveolar opacity
• Edema
• Consolidation
• Pneumonia
• Atelectasis
• Infiltrate
• Alveolar opacity
• Edema
• Pneumonia
• Consolidation
• Interstitial
• Atelectasis
EDEMA
INFILTRATE
CONSOLIDATION
PNEUMONIA
ATELECTASIS
• Alveolar opacity
• Edema
• Pneumonia
• Consolidation
• Interstitial
• Atelectasis
• Opacity
• Pneumonia
• Consolidation
• Interstitial
• Edema
• Atelectasis
Your Ontology May Vary
• Alveolar opacity
• Edema
• Consolidation
• Pneumonia
• Atelectasis
• Infiltrate
EDEMA
CONSOLIDATION
PNEUMONIA
ATELECTASIS
INTERSTITIAL
Image Labeling Best Practices• Large training set with automated noisy labels
• Accurately labeled test set
• Multiple expert observers
• User training with examples and hierarchy
• Validate that users are following hierarchy
• Method to adjudicate observers
Conclusions
MEDICAL IMAGING DATA
IS MESSY
LABELS HAVE HIERARCHICAL RELATIONSHIPS
DATA VOLUME CAN OVERCOME DATA
NOISE
ACCURATE TEST SET LABELS ARE
IMPORTANT
Courtesy of Matt Lungren
Thank YouCurtis P. Langlotz, MD, PhD
Professor of Radiology and Biomedical InformaticsDirector, Center for Artificial Intelligence for Medicine & Imaging
Associate Chair for Information SystemsDepartment of Radiology, Stanford University
Informatics Director for RadiologyStanford Health Care
@curtlanglotz