Artificial Intelligence for Medical Imaging and Treatment...
Transcript of Artificial Intelligence for Medical Imaging and Treatment...
![Page 1: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/1.jpg)
Stanford UniversityDepartment of Radiation Oncology
School of Medicine
Artificial Intelligence for Medical Imaging and Treatment Planning
Lei Xing, PhD, DABR, Jacob Haimson Professor
Departments of Radiation Oncology & Electrical Engineering (by courtesy), Human-Centered Artificial Intelligence (HAI)
Stanford University
![Page 2: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/2.jpg)
Machine learning
NLP
Expert system
Robotics
Vision
![Page 3: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/3.jpg)
Artificial Intelligence and its Applications in MedicineArtificial Intelligence and its Applications in Medicine
Machine Learning/ Deep Learning
Computer Vision Natural Language Processing (NLP)
Expert System Robotics & Control
FUNDAMENTALS ‐ Data science & mathematical framework ‐High performance computing (GPU/TPU/multi‐core CPU, cloud computing, quantum computing)‐ Analytics tools & algorithms (data dimensionality reduction, visualization, compression, various machine learning/deep learning algorithms)‐ Basic machine learning software platforms
APPLICATIONS‐ AI augmented medical devices & wearables.‐ Analysis of biological, imaging, EMR, and therapeutic data for clinical decision‐making.‐ Robotic interventions.‐ Biomarker discovery& drug design.
OTHER RELATED ISSUES‐ Training of future physicians, healthcare professionals, & next generation of AI workforce.
‐ Economic, politic, social, ethic and legal issues.
‐Workflow and clinical implementation.
DATA & DATABASE‐ Data curation & augmentation‐ Data harmonization & mining‐ Data sharing & security‐ Federated learning‐ Search engine (data, text, audio, video, image, etc.)
TECHNOLOGY‐ New algorithms for improved classification, detection, segmentation & other image analysis tasks.‐ NLP tools for medical semantics & search
‐ Enhancement & expansion of existing AIM techniques
![Page 4: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/4.jpg)
![Page 5: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/5.jpg)
Mapping between the same data domains
‐ Superresolutionimaging‐ Image search‐ Image inpaiting
‐ Image reconstruction‐ sparse data problem‐Modeling physical/ mathematical relation
Machine Learning/Deep LearningInput and Output
Data for AI Modeling Applications & Examples
Data domains related by known law(s)
Data domains related by empirical evidence or
measurement(s)
‐‐Modeling of therapeutic response‐ Drug design & biomarker discovery ‐ Translation, semantic analysis ‐ Auto‐annotation‐ Modeling correlative relationship
![Page 6: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/6.jpg)
Types of learning
![Page 7: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/7.jpg)
ImagingModeling Treatment planning Image-guided patient
setup & deliveryFollow up
Stanford UniversityDepartment of Radiation Oncology
School of Medicine
Image reconstruction
Image reconstruction
Different modalities
Artifacts removal Sparse data …....
High dimensional imaging
High dimensional imaging
Real‐timeDynamic contrast imaging
Functional imaging …..
SparsificationcompressionSparsificationcompression
Sparse representation
Sparse learning
Image compression ….....
OAR segmentation
OAR segmentation supervised unsupervised Network
structure …....
Tumor detection & segmentation
Tumor detection & segmentation
Detection Muli‐modalities …..
Image registration
Image registration Intra‐modality Cross‐
dimensionInter‐
modalitie ….....
Supervised planningSupervised planning …....
UnsupervisedUnsupervised …..Reimforcement
learningReimforcement
learning ….
Monoscopicimaging
Monoscopicimaging …....
Stereoscopic imaging
Stereoscopic imaging …..
Cone beam CT
Cone beam CT ….....
Therapeutic response
Therapeutic response
Survival Survival
ToxityToxity …
![Page 8: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/8.jpg)
X‐Ray
1895 1942 1972 1986 1993 1995 2020
Information technology
X‐CT
MRI
fMRI
PET
US
Anatomy imaging
Molecularimaging
Histology
Images reconstruction – low dose CT, fast MRI Imaging is one of the first choices for clinical diagnosis 70% clinical decisions depend on medical images
ML for Medical Image Analysis
Endoscopy
![Page 9: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/9.jpg)
����������� � � ��������� � ��
We can directly visualize the network’s attention when processing an input video.
The discriminative regions of tumor are highlighted, suggesting the model works as expected and is able to identify tumors from artifacts and background.
E. Shkolyar, X. Jia, T.C. Chang, D. Trivedi, K. E. Mach, M. Meng, L. Xing, J. Liao, European Urology 76, 714-718, 2019
![Page 10: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/10.jpg)
Importance
Different cancer levels (Gleason score) lead to different therapyReduce the core needle biopsy
Modality for diagnosis
Magnetic Resonance Imaging (MRI)
Image‐based prostate cancer classification & virtual biopsy
T2-weighted images (transaxial)
T2-weighted images (sagittal)
Apparent Diffusion Coefficient images
T1-weighted Contrast images
Y. Yuan, W. Qin, B. Han, et al, Medical Physics, 2019
Stanford UniversityDepartment of Radiation Oncology
School of Medicine
![Page 11: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/11.jpg)
CT/CBCT artifacts removal
![Page 12: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/12.jpg)
Stanford UniversityDepartment of Radiation Oncology
School of Medicine
The HU difference between the predicted and original high-energy CT images are 3.47 HU, 2.95 HU, 2.38 HU and 2.40 HU for ROIs on spine, aorta, liver and stomach, respectively.
Dual-energy CT imaging using deep learning (Full 3D Meeting, 2019)
![Page 13: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/13.jpg)
From super‐resolution imaging to super resolution dose calculation
![Page 14: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/14.jpg)
Super‐resolution dose transformation and machine learning‐based dose calculation
P. Dong & L. Xing, Deep DoseNet: a deep neural network for accurate dosimetric transformation between different spatial resolutions and/or different dose calculation algorithms for precision radiation therapy, Phys. Med. Biol., 2019
low resolution dose
high resolution dose
low cost algorithm &
low resolution
high resolution dose, highly
accurate algorithm
ultra‐low cost algorithm &
low resolution
high resolution dose, highly
accurate algorithm
DL model
DL model
DL model
1. Nomura Y, Wang J, Shirato H, Shimizu S, Xing L, Fast spot-scanningproton dose calculation method with uncertainty quantification using athree-dimensional convolutional neural network, PMB Jun. 2020
![Page 15: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/15.jpg)
Machine learning provides a new way for small field dosimetry and plan QA
J. Fan, L. Xing, Y. Yang, under review E. Schueler, W. Zhao, et al, in preparation
Output factor prediction
![Page 16: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/16.jpg)
![Page 17: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/17.jpg)
Stanford UniversityDepartment of Radiation Oncology
School of MedicineShen L, Zhao W, Xing L, Nature Biomedical Engineering (in press), 2019
Pushing the sparsity to the limit ‐
Shen L, Zhao W, Xing L, Nature Biomedical Engineering 3, 880-808, 2019
![Page 18: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/18.jpg)
Shen L, Zhao W, Xing L, Nature Biomedical Engineering 3, 880-808, 2019
![Page 19: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/19.jpg)
Sparse Data MR Image Reconstruction
Data SamplingRaw data are sampled point by point in Fourier domain (k-space)
Image ReconstructionInverse Fourier transform is applied on the raw data to generate output in the
image domain
Inverse Fourier Transform
M. Mardani,…, L Xing, J Pauly, TMI, 2019Y. Wu, et al, Mag. Res. Imag., 2019
![Page 20: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/20.jpg)
Integrated MRI‐Radiotherapy Systems:MRI Guided Localization & Delivery
![Page 21: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/21.jpg)
#1 in the Liver Tumor Segmentation Challenge (LiTS‐ISBI2017)
- H. Seo, R. Xiao, L. Xing
![Page 22: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/22.jpg)
Autonomous treatment planning for RT
M. Ma, N. Kovalchuk, M. Buyyounouski, L. Xing, Y. Yang, Med Phys, 2019
Clinical plan Plan predicted by deep learning
•CT•SegmentationINPUTINPUT DL
MODELDL
MODEL OutputOutput
![Page 23: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/23.jpg)
Beam trajectory selection using reinforcement learning
Kahn, Fahimian et al
![Page 24: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/24.jpg)
![Page 25: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/25.jpg)
Beam level imaging
![Page 26: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/26.jpg)
Landmarkdetectionincephalometricanalysis
Stanford UniversityDepartment of Radiation Oncology
School of Medicine
![Page 27: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/27.jpg)
Visualizing the invisible –Deep learning‐augmented IGRT
Zhao et al ,IJROBP, 2019
![Page 28: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/28.jpg)
Example of prostate motion tracking in AP direction The predict prostate position match the ground truth quite well.
Target tracking
![Page 29: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/29.jpg)
• Pancreas, lung, etc.
Tracking on PTV for pancreas radiotherapy
Ground truth
Predicted
Zhao et al ,IJROBP, 2019
![Page 30: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/30.jpg)
![Page 31: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/31.jpg)
From population‐average nomogram to deep learning‐based toxicity prediction
31
- B. Ibrambrov, D. Toesca, D. Chang, A Koong, L Xing
Current approach:
(i) NTCP/TCP types of modeling
Problems: biological heterogeneity, spatial information
Predictive model
![Page 32: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/32.jpg)
Multi-path network: 1) 3D CNN for dose plan; 2) fully-connected path for features
Deep dose‐toxicity prediction
![Page 33: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/33.jpg)
survival & toxicity results
Magical deep learning box
3D dose plan+
Image analysisPrediction
Problem:
![Page 34: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/34.jpg)
Data Dimension‐Reduction
Islam T & Xing L, Nature Biomedical Engineering, under revision, 2020
![Page 35: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/35.jpg)
On‐going research
Interpretable and trustworthy AI.
General instead of task‐specific AI.
Data & annotation.
Clinical implementation and workflow related issues.
Better AI models.
![Page 36: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/36.jpg)
ImagingModeling Treatment planning Image-guided patient
setup & deliveryFollow up
![Page 37: Artificial Intelligence for Medical Imaging and Treatment …ionimaging.org/assets/talks/ws2020llu-lei-xing.pdfImage‐based prostate cancer classification & virtual biopsy T2-weighted](https://reader033.fdocuments.us/reader033/viewer/2022052804/604c6bf24bd45a25dc6cb499/html5/thumbnails/37.jpg)
Acknowledgements M. Bassenne, J.‐E. Bibault, Y. Chen, D. P. Capaldi, J. Fan, C. Huang, T. Islam, M. Jia,
L. Shen, H. Ren, M. Ma, H. Seo, X. Li, L.Yu, T. Liu, S. Gennatas, M. Khuzani, E. Schueler, E. Simiele, K., Sivasubramanian, H. Zhang, V. Vasudevan, Y. Wu, W. Zhao, Z. Zhang, D. P.I. Capaldi
P. Dong, B. Han, Y. Yang, N. Kovalchuk, D. Hristov, L. Skinner, C. Chuang, L. Wang, J. Lewis, D. Chang, D. Toesca, Q. Le, S. Soltys, M. Buyounouski, H. Bagshaw, S. Hancock, G. Pratx, R. Li, J. Pauly, S. Boyd, ....
Funding: NIH/NCI/NIBIB, DOD, NSF, ACS, Varian, & Google.