AAPM Special Symposium in Honor of Charles E....
Transcript of AAPM Special Symposium in Honor of Charles E....
![Page 1: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/1.jpg)
AAPM Special Symposium in Honor of Charles E. Metz
“Past, Present and Future Roles of ROC
Analysis in Medical Imaging and Quantitative Image Analysis”
Kyle J. Myers Division of Imaging and Applied Mathematics
CDRH/FDA
August 2013
1
![Page 2: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/2.jpg)
Charles E. Metz: The Story of a Decades-long Relationship with the FDA
2
Charles E. Metz Robert F. Wagner and David G. Brown
![Page 3: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/3.jpg)
1981 Coded aperture paper
Question: which gamma-ray aperture is best?
![Page 4: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/4.jpg)
Barrett and Swindell, Academic Press,1981
Image of point source is scaled, shifted version of aperture
For ANY aperture:
image of general object
(“collection of point sources”)
is convolution with aperture
Film image (dark=exposed)
![Page 5: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/5.jpg)
URA = Uniformly Redundant Array Square
Grand Array of Pinhole Apertures
▪▪▪ ▪▪▪
![Page 6: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/6.jpg)
Grand array gives set N of nonoverlapping images
Pinholes spaced far enough apart to ensure images of object don’t overlap.
Grand array
of pinholes
Barrett and Swindell, Academic Press,1981
![Page 7: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/7.jpg)
There is a class of imaging applications for which an exact solution exists
for the question of which aperture to use to collect photons via signal detection
theory
-- detection/discrimination tasks
-- observer is given raw, undecoded data (no display)
-- accounting for Poisson noise
-- optimal decision variable is the likelihood ratio
TASK SNR = ability of ideal observer to discriminate:
images of double Gaussian objects (class “a”)
from
images of single Gaussian objects (class “b”)
Signal detection theory
![Page 8: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/8.jpg)
Example results
Square outperforms URA
for this SKE/BKE task!
![Page 9: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/9.jpg)
MANY important lessons learned, follow-on studies and changes to assessment culture
• Aperture OPTIMIZATION for task would find a different Coded Aperture that beats the Great Gaping Hole
H
Noise
n
f
Data
g
T Decision
t
Performance
evaluation
Performance
Measure
Object Imaging
System
Observer
Inference Figure of Merit
Feedback/optimization loop
![Page 10: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/10.jpg)
MANY important follow-on studies and changes to
assessment culture – cont’d • Comparison of SKE tasks to those with
randomness in signal or background – Randomness in signal parameters (location)
– Structured backgrounds (lumpy BGs)
Clustered Lumpy Background Lumpy Background
![Page 11: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/11.jpg)
• Predrag R. Bakic, U. of Pennsylvania: Real Time Simulation of Breast Anatomy
• Ingrid Reiser, U. of Chicago: Simulation of Small Scale Breast Tissue Structures for X-ray Imaging
• Nooshin Kiarashi, Duke U.: Generation of X-ray Relevant Software Breast Phantoms from Clinical Datasets
• Rongping Zeng, FDA: Task Based Assessment of X-Ray Breast Imaging Systems using In Silico Modeling Tools
State of the Art: AAPM TG234 • Virtual Clinical Trials in 3D X-ray Breast
Imaging – Special Session on Monday morning
![Page 12: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/12.jpg)
DBT virtual imaging chain (Zeng, AAPM’13)
Simulated DBT image chain
X-ray source
Detector
Object
θ
SART
ML
TV-LS-mild
TV-LS-strong
FBP
Object DBT data
acquisition
Reconstruction
Numerical breast phantom*:
500 m, cupsize B, 25% glandular density,
6 mm lesions (6 in each LP phantom)
*BakicEtAl-MedPhys2011-v38(6)
DBT image slices
Point and mono-energetic x-ray
source; detector pixel size 500 m;
Separable footprints forward
projector**; Poisson noise model,
fixed total exposure.
**Long&FesslerEtAl-IEEE-
TMI2010-v29p1839
Lesion-absent (40)
Lesion-present (40)
Step size was tuned to obtain
relatively fast convergence;
Number of iterations was decided
to have optimal lesion detectability
in a small ensemble of breast
phantoms.
Recon. Voxel size: 500
m in-plane, 2 mm slice
interval
![Page 13: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/13.jpg)
Need for new phantom designs for OAIQ* studies
Objects at same radial location with randomly placed ROIs
search task
Uncertainty in signal location (or size, shape) allows for more
“dynamic range” in task SNRs available for a given image set
Courtesy L. Popescu
*Objective Assessment of Image Quality
Myers AAPM 2012 presentation on Evaluation of Iterative Recon in CT
![Page 14: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/14.jpg)
ICRU report Ideal observer
Task SNR and
Imaging Chain!
Peter Sharp, Chris Taylor, David Barber, Charles Metz, Kyle Myers
![Page 15: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/15.jpg)
![Page 16: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/16.jpg)
![Page 17: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/17.jpg)
![Page 18: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/18.jpg)
Landmark approval of ATL device – April 1996.
![Page 19: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/19.jpg)
FDA
![Page 20: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/20.jpg)
FDA Advisory Panel meeting for 1st 3D automated breast
ultrasound system (April 2012)
![Page 21: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/21.jpg)
0.0
0.0
0.0
0.0
0.1
0.1
0.1
0.1
0.2
0.2
0.2
0.2
0.3
0.3
0.3
0.3
0.4
0.4
0.4
0.4
0.5
0.5
0.5 0.5
0.6
0.6
0.6
0.6
0.7
0.7
0.7
0.7
0.8
0.8
0.8
0.8
0.9
0.9
0.9
0.9
1.0
1.01.0
1.0
False Positive Fraction
True Negative FractionT
rue
Po
siti
ve
Fra
ctio
n
Fals
e N
egati
ve
Fra
ctio
n TPF vs FPF for 108 mammographers in study by Beam et al.
The Complication of Reader Variability:
TPF vs FPF for
108 U.S.
radiologists all
reading same
images
No unique ROC
curve
No unique
(TPF, FPF) or
(Se, Sp) point
Arch Intern Med 1996
![Page 22: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/22.jpg)
![Page 23: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/23.jpg)
FDA Advisory Panel meeting for 1st FFDM system (Dec. 1999)
Sponsor’s data
analysis done using
Metz LABMRMC
software to account for
reader variability
![Page 24: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/24.jpg)
![Page 25: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/25.jpg)
MIPS 2001: The CAD meeting
![Page 26: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/26.jpg)
![Page 27: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/27.jpg)
![Page 28: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/28.jpg)
2010 FDA/MIPS workshop
• “Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA”
– Followed by questionnaire to speakers and
panelists probing consensus statements for a workshop summary paper
28
• “This is an *excellent* questionaire overall, in my opinion.”
• “This document is very, VERY good, in my opinion.”
• “This sentence strikes me as not only awkward, but impenetrably opaque.”
![Page 29: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/29.jpg)
![Page 30: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/30.jpg)
Software
• http://metz-roc.uchicago.edu/
•
• http://js.cx/~xin/index.html
•
30
iMRMC Analyzing and Sizing Multi-reader Multi-case ROC Trials
Xin He, Brandon Gallas
![Page 31: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/31.jpg)
iMRMC: Webpage and Software for Sizing an MRMC Clinical Trial
A resource for investigators designing a trial to compare two imaging modalities.
Uses datasets from previous imaging trials to estimate power of new trial designs.
Over time, database growth will benefit wide community of clinical trialists.
http://js.cx/~xin/index.html
Incl. Database of Components
of Variance
Send us your reader data!
![Page 32: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/32.jpg)
iMRMC 2.0 in development
• Allow for arbitrary study design
• Roe and Metz App in development – Simulate MRMC experiments – Allow variance to differ across truth and
modality – Numerically calculate components of variance
32
iMRMC Analyzing and Sizing Multi-reader Multi-case ROC Trials
Xin He, Brandon Gallas
![Page 33: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/33.jpg)
Charlie’s Impact
• Major influence on the culture of imaging system assessment at the FDA
• Facilitated bringing significant innovations to patients in Breast Imaging (FFDM, US, and CAD) and beyond!
![Page 34: AAPM Special Symposium in Honor of Charles E. Metzamos3.aapm.org/abstracts/pdf/77-22628-310436-101883.pdf · 2013. 8. 27. · AAPM Special Symposium in Honor of Charles E. Metz “Past,](https://reader036.fdocuments.us/reader036/viewer/2022071407/60fd7ff4c4662d369f1f0d2d/html5/thumbnails/34.jpg)
Heang-Ping Chan’s Research updates: U Michigan, Charlie, FDA
34
Charlie Berkman Sahiner
Bob
Nick Petrick
Heang-Ping Chan
Brandon Gallas
Lubomir Hadjiyski