Highlighting Challenges for Machine Learning in the Pathology...
Transcript of Highlighting Challenges for Machine Learning in the Pathology...
Asmaa Aljuhani, PhD Candidate
Dr. James Cronin, DVM, PhD Candidate
Dr. Jany Chan, BSCS, PhD
Carly Vroom, BSCS, MS Candidate
Raghu Machiraju, PhD
Highlighting Challenges for Machine
Learning in the Pathology Clinic through
Specific Use Cases
Anil Parwani
https://cancer.osu.edu/news-and-
media/news/digital-pathology-could-improve-
accuracy-timeliness-of-cancer-diagnosis
doi: 10.1097/PAP.0000000000000264
Typical clinical workflows
Domain-specific Search Task
Human
H&E Processing - circa 2000s
Placenta
H+E Slides Alignment
Segmentation
Visualization/Surface Extraction
Aperio
H&E Reveal A Lot About Phenotypes
3D Placenta
OSU/Stanford/GaTech - Circa 2010s
1Whole
Slide
Images2
Representative
Patches3 Image
Preprocessing4 SuperPixel
Segmentation
5 LBP FeaturesTissue
Classification6
Stromal
tissue
Epithelial
tissue
Cell
Segmentation7Feature
Extraction8
Epithelial
Features
Stromal
Features
Pooya Mobadersany et al. PNAS 2018;115:13:E2970-E2979
Deep Learning In Cancer PathologyBreast
DL algorithms reach comparable or better performance than human pathologists:
- Breast cancer micrometastasis (lymph nodes)
- Detect tubule formation (corresponding to Oncotype DX and tumor grade in ER+ breast cancer)
Discrimination of benign vs malignant based on stromal compartment
Discrimination of normal tissue, atypia, DCIS, and invasive carcinoma
doi:10.1111/joim.13030
Prostate
DL algorithms reach comparable or better performance than human pathologists:
- Gleason grading (better risk stratification)
Slide level cancer detection
SPOP mutation status
Lung
Classify normal, adenocarcinoma, squamous cell carcinoma
- Prediction of recurrence in early-stage non-small cell lung cancer (using nuclear orientation, nuclear shape, and tumor
architecture)
- Predict mutation status of six genes in lung adenocarcinoma - KRAS, FAT1, TP53, SETBP1, EGFR, STK11
Brain
- Prognosis in gliomas (incorporated genomic data too)
Skin
- Prognosis in early stage melanoma - lymphocyte content most important factor to predict outcome
GI
- Predict microsatellite instability from gastric and colorectal cancer
Pancancer
- Local patterns and overall structural patterns of TILs are differentially represented among tumor types and tumor
molecular subtypes (the patterns are differentially related to survival amongst different tumor types)
doi:10.1111/joim.13030
Attaining the Gold Standard of Diagnosis
- Also called Criterion Standard!
- Diagnostic test or criteria best available under reasonable conditions
- Ideally has sensitivity and specificity of 100% with respect to disease
From: The Gold Standard Paradox in Digital Image Analysis: Manual Versus
Automated Scoring as Ground Truth
Arch Pathol Lab Med. 2017;141(9):1267-1275. doi:10.5858/arpa.2016-0386-RA
Us & Machines To Attain The Gold Standard
doi: 10.1097/PAP.0000000000000264
Machines @ Work for the Gold Standard
doi: 10.1097/PAP.0000000000000264
Machines @ Work
Machine
Human
This Talk – Unusual Suspects
- Case Studies
- Image as a Proxy? – ER+ Breast Cancer
- Need for Clear Labels! - Tall Cell Variant of Pap. Thyroid Cancer
- Wild West of Subtyping - Sarcomas
- In-situ Imaging of Omics!- Prostate Cancer
- Closing Arguments
Whole Slide Images
Sheer Size
Sheer Variety - Variability
Variability within & across WSIs & over all subjects
Variation in Features and Organization!
Sheer Variety - Types
Sheer Variety – State & Tissue Type
Tissue compartments within WSI of different types & in different states
Histologic (Molecular/Outcome) Heterogeneity
BIAS IN
PATHOLOGY
SCORING
BIAS IN
PATHOLOGY
SCORING WITH
WSIs?
Use Case I – Image as a proxy for molecular
markers
ROI-1
ROI-2
ROI-3
Multiple tasks
Biomarkers
Clinical
Biopsy
Biomarkers
Patient Stratification
Survival?
Oncotype DX® Genes
PROLIFERATIONKi-67STK15Survivin
Cyclin B1MYBL2
HER2
GRB7
HER2
ESTROGEN
ER
PGR
BCL2
SCUBE2
INVASION
Stromelysin 3
Cathepsin L2
REFERENCE
Beta-actin
GAPDH
RPLPO
GUS
TFRC
BAG1
GSTM1
CD68
Paik et al. N Engl J Med. 2004;351:2817-2826
Recurrence Score
RS (unscaled) =
+ 0.47 x HER2 group score
– 0.34 x ER group score
+ 1.04 x proliferation group score
+ 0.1 x invasion group score
+ 0.05 x CD68
– 0.08 x GSTM1
– 0.07 x BAG1
Representative images of two classical invasive lobular
carcinoma cases with Oncotype DX RS > 30
A. Parwani Z. Li, OSU
Recurrence Score & Images
Which image features best correlate with different different ranges?
Creating a Morphological Proxy
Recurrence Score
Genomic Input
Using grading criteria used in the clinic
AJCC - Nottingham Score
Representative images of two classical invasive lobular
carcinoma cases with Oncotype DX RS > 30
Nottingham grading score
AJCC - Nottingham Score DOI: 10.1186/bcr2607
Grade 1
Grade 2
Grade 3
Attention based techniques - Relevance
Identify the disease “relevant” regions capturing state
Mitosis Datasets
- 73 breast cancer cases
- ×40 magnification
- Annotated
TUPAC16
MITOS
ATYPiA_14
- Breast Cancer: Mitosis: #749
- Different magnification level:
x40,x20,x10
- Breast Cancer: Atypia
BreCaHAD19
- Mitosis: #115
- Apoptosis: #271
- Tumor nuclei: #20155
- Non-tumor nuclei: #1905
The Framework
36
1. Pre-select regions of the image that are “relevant” to the disease
2. Process the regions for recognition of state
3. Predict the regions and label tumor/normal and subtypes
4. Confirm that these correspond with correct RS range and pick features
that predict in RS-range
5. Repeat for better prediction/prognosis
Deep learning pipelines that importance samples input and interprets output
The Framework
Enhanced Model Prediction
0
5
10
15
20
25
30
35
40
45
Accuracy Precision Recall F-score
Node Status
All tiles
High Mitotic
Activity Tiles
Interpreted CNN predictions
Whole Slide
Image
tile
Stage
prediction
ROI mask
Non CNN-ROI
Dominant Color
CNN-ROI
Dominant Color
Non CNN-ROI
Cell shape and size
CNN-ROI
Cell shape and size
Non CNN-ROI
Region Texture
CNN-ROI
Region Texture
a...2
.
a...1
.
test
123
a..
.2.
a..
.1.
test
123
54% of
sampled
data
Dominant colors in
image
Texture_AngularSecondMoment_OrigRed_2_03
Texture_AngularSecondMoment_OrigRed_2_01
Texture_AngularSecondMoment_OrigRed_4_00
Texture_AngularSecondMoment_OrigRed_3_02
Texture_AngularSecondMoment_OrigRed_3_00
Texture_AngularSecondMoment_OrigRed_2_02
Texture_AngularSecondMoment_OrigRed_2_00
Texture_AngularSecondMoment_OrigRed_4_02
Texture_AngularSecondMoment_OrigRed_3_01
Texture_AngularSecondMoment_OrigRed_3_03
Texture_AngularSecondMoment_OrigRed_4_01
Texture_AngularSecondMoment_OrigRed_4_03
Texture_AngularSecondMoment_OrigRed_10_00
Texture_AngularSecondMoment_OrigRed_10_02
Texture_AngularSecondMoment_OrigRed_20_02
Texture_AngularSecondMoment_OrigRed_10_01
Texture_AngularSecondMoment_OrigRed_10_03
Texture_AngularSecondMoment_OrigRed_20_00
Texture_AngularSecondMoment_OrigRed_30_00
Texture_AngularSecondMoment_OrigRed_30_01
Texture_AngularSecondMoment_OrigRed_30_03
Texture_AngularSecondMoment_OrigRed_30_02
Texture_AngularSecondMoment_OrigRed_20_01
Texture_AngularSecondMoment_OrigRed_20_03
Granularity_11_OrigRed
Granularity_15_OrigRed
Granularity_12_OrigRed
Granularity_13_OrigRed
Granularity_14_OrigRed
Granularity_16_OrigRed
Granularity_2_OrigRed
Granularity_3_OrigRed
Granularity_1_OrigRed
Granularity_7_OrigRed
Granularity_6_OrigRed
Granularity_5_OrigRed
Granularity_4_OrigRed
Granularity_9_OrigRed
Granularity_10_OrigRed
Granularity_8_OrigRed
Intensity_PercentMaximal_OrigRed
Normalized_Texture_Features
−2
−1
0
1
2
Angular
Second
Moment
Features
Granularity
(11-16px)
Granularity
(1-10px)
Pixels at maximum
intensity
StDev_IdentifyPrimaryObjects_AreaShape_Compactness
StDev_IdentifyPrimaryObjects_AreaShape_MeanRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaximumRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaxFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MajorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_MinFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MinorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_Perimeter
Mean_IdentifyPrimaryObjects_AreaShape_Area
StDev_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_4_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
StDev_IdentifyPrimaryObjects_AreaShape_Orientation
AreaOccupied_AreaOccupied_IdentifyPrimaryObjects
AreaOccupied_Perimeter_IdentifyPrimaryObjects
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Orientation
Median_IdentifyPrimaryObjects_AreaShape_Orientation
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_0_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_8_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_4
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Normalized_Size_Shape_Features
−2
−1
0
1
2 Scaled
Texture
Measures
StDev_IdentifyPrimaryObjects_AreaShape_Compactness
StDev_IdentifyPrimaryObjects_AreaShape_MeanRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaximumRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaxFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MajorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_MinFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MinorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_Perimeter
Mean_IdentifyPrimaryObjects_AreaShape_Area
StDev_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_4_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
StDev_IdentifyPrimaryObjects_AreaShape_Orientation
AreaOccupied_AreaOccupied_IdentifyPrimaryObjects
AreaOccupied_Perimeter_IdentifyPrimaryObjects
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Orientation
Median_IdentifyPrimaryObjects_AreaShape_Orientation
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_0_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_8_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_4
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Normalized_Size_Shape_Features
−2
−1
0
1
2 Scaled
Size/Shape
Measures StDev_IdentifyPrimaryObjects_AreaShape_Compactness
StDev_IdentifyPrimaryObjects_AreaShape_MeanRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaximumRadius
StDev_IdentifyPrimaryObjects_AreaShape_MaxFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MajorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_MinFeretDiameter
StDev_IdentifyPrimaryObjects_AreaShape_MinorAxisLength
StDev_IdentifyPrimaryObjects_AreaShape_Perimeter
Mean_IdentifyPrimaryObjects_AreaShape_Area
StDev_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Area
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Median_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_4_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_2
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_9
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_1
Median_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_7
Median_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_3
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_9_5
Median_IdentifyPrimaryObjects_AreaShape_Zernike_9_1
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_4_2
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_2_2
Median_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
StDev_IdentifyPrimaryObjects_AreaShape_Orientation
AreaOccupied_AreaOccupied_IdentifyPrimaryObjects
AreaOccupied_Perimeter_IdentifyPrimaryObjects
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_5
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_5_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_7_3
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_3_3
Mean_IdentifyPrimaryObjects_AreaShape_Orientation
Median_IdentifyPrimaryObjects_AreaShape_Orientation
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_0_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_1_1
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_8_0
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_6_4
StDev_IdentifyPrimaryObjects_AreaShape_Zernike_2_0
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_6_6
Mean_IdentifyPrimaryObjects_AreaShape_Zernike_8_6
Normalized_Size_Shape_Features
−2
−1
0
1
2
Standard Deviation of Shape
Mean, Standard Deviation and Median of Area
Mean, Standard
Deviation and
Median of
Zernike Features
Total Area & Perimeter of cells
Standard
deviation of
cell
orientation
Standard deviation of Zernike features Mean and Median of cell orientation
Mean and Standard Deviation of Zernike features
Dominant
colors in
image
a..
.2.
a..
.1.
test
123
- Collect training data of all parameters
- More attention-based learning
- Process patch and images at different resolutions
- Refine multi-task learning
- Leverage Un/Semi-supervised learning
- Connect with Recurrent Score
Work in Progress
Per Becich Survey
A: Very High; P: Medium; D: High
Use Case II – Few and Fuzzy labels
Task well not defined
B. Finding Tall Cell Variants in Papillary Thyroid Cancer
Juan Prera, USF Paul Wakely
“Tall cell” Variant
● “Tall cell” variant diagnosis depends exclusively on histopathology
● Associated with worse prognosis among papillary thyroid cancer
● Should not definition of “tall cell” and “tall cell variant” be well
established?
● Well, no!
1976
Tall cell variant first
described
“Tall cell” H:W > 2:1
1976-2017
Obfuscation of “tall
cell” and “tall cell
variant” definition
“Tall cell” H:W ranged
from > 2:1 to > 3:1 and a
tumor percentage criterion
introduced, which ranged
from 30% to 70%
2017
4th Edition WHO
classification
H:W between 2:1 and 3:1
and minimum tumor
percentage criterion = 30%
2017-present
Continued discord
Tumor percentage criterion
ranging from >10% to >50%
“Tall cell” determination
highly prone to
interobserver variability
Annotation PlanWSI
Tumor vs Not Tumor
10x?Definitely not Tall Cells
Papillary Cancer
PTC
Maybe Tall Cells
Tall CellsPTC
3 Cohorts
Tumor
Not Tall Cell
Tall Cell
Finding Tall Cells Training phase
Testing phase%TCV
Tiling CNN
P(Tall)Tall
Not Tall P(Not Tall)
Extract Tiles
P(Non Tumor)Non Tumor
Tumor P(Tumor)10x
10x
40x
40x
P(Tall)Tall
Not Tall
Extract Image-based
features
Tall
NotTall
High probability predicted
tiles
Detection phase
Going Forward – Tissue-specific Patterns
- Attention on features!
- Like Tall?
- Tram like patterns
- Domain knowledge helps!
- Data is a problem
- Not too many tall cells
Per Becich Survey
A: Medium; P: Very High; D: Very High
49
Use case III – Too many subtypes and little
data
The Goal: Genomic & Histopathologic Composite Grading System
Dr. David Liebner Dr. Xiaoyin Cui
Use Case III: Few Definitions & Workflow
An approach to pleomorphic sarcomas: can we subclassify, and does it matter?
(A) Mitotic activity (B)
Variation in the size, shape,
and chromatin texture of
tumor nuclei often. (C)
Necrosis (left) of neoplastic
cells.
Consider This ...
osteosarcomaperipheral nerve sheath
rhabdomyosarcomaleiomyosarcomaliposarcoma
pleomorphic neoplasm
https://www.nature.com/articles/modpathol2013174
Working w/ Grading
The Classification Task
CPTAC
Tumor
Normal
Tiles
P(Tumor)P(Normal)
TCGA
ROI(Tumor Tiles)P(UPS)P(DDLPS)P(ULMS)P(STLMS)P(MFS)P(SS)P(MPNST)
De
ep
Ne
ura
l Ne
two
rk f
or
sub
typ
e
clas
sifi
cati
on
Mitosis Datasets
De
ep
Ne
ura
l Ne
two
rk f
or
Gra
de
Pre
dic
tio
n
Collect Annotation for
Atypia and Necrosis
Collect OSU Sarcoma dataset
Mitotic Count
Atypia Cells
Necrosis%
Subtype
Grade
WSI
Mitosis Datasets
- 73 breast cancer cases- ×40 magnification- Annotated
TUPAC16
MITOS
ATYPiA_14
- Breast Cancer: Mitosis: #749- Different magnification level:
x40,x20,x10- Breast Cancer: Atypia
BreCaHAD19
- Mitosis: 115- Apoptosis: 271- Tumor nuclei: 20155- Non-tumor nuclei: 1905
Unsupervised Learning
Tiles Feature Extraction
LBP , SIFT, ...
Cluster Features
...
....
....
.
WSI
Variational Autoencoders
Use Case IV- Multiplexed Single Cell Resolution
P. Mallick
Stanford
Pathomics
Summary & Closing
The Becich Survey
The survey
- Algorithms are needed when workflows are MIA
- Pathologists always needed when etiology is least understood
- Data is always needed
- A: Very High; P: Medium; D: Very High : Prostate
- A: Very High; P: Very High; D: Very High: Sarcoma
- A: Medium; P: Very High; D: Very High: Tall Cell Variant of Thyroid Cancer
- A: Very High; P: Medium; D: High: Breast Cancer
From: The Gold Standard Paradox in Digital Image Analysis: Manual Versus
Automated Scoring as Ground Truth
Arch Pathol Lab Med. 2017;141(9):1267-1275. doi:10.5858/arpa.2016-0386-RA
Attaining Gold Standard
doi: 10.1097/PAP.0000000000000264
Thank You for Listening!