PowerPoint Presentation

LIDC Education Exhibit - RSNA 2003LIDC Education Exhibit - RSNA 2003 11/99/99End/Return to Home

The Lung Image Database Consortium The Lung Image Database Consortium (LIDC):(LIDC):

Fundamental Issues for the Creation of Fundamental Issues for the Creation of a Resource for the Image Processing a Resource for the Image Processing

Research CommunityResearch Community


Exhibit Learning Objectives:Exhibit Learning Objectives:

Learn about the LIDC’s goals and Learn about the LIDC’s goals and methods for creating: methods for creating: a publicly available database,a publicly available database,

for the development, training, and for the development, training, and evaluation of Computer-Aided Diagnosis evaluation of Computer-Aided Diagnosis (CAD) methods, (CAD) methods,

for lung cancer detection and diagnosis for lung cancer detection and diagnosis using helical CT.using helical CT.


Learning Objectives:Learning Objectives:

Learn about challenges in Learn about challenges in interpreting CT image data sets for interpreting CT image data sets for the detection and diagnosis of lung the detection and diagnosis of lung cancercancer


Learning Objectives:Learning Objectives:

Learn about the intricacies of Learn about the intricacies of establishing spatial “truth” for lesion establishing spatial “truth” for lesion location and boundary.location and boundary.


The Challenge: Lung CancerThe Challenge: Lung Cancer

Cancer of the lung and bronchus is the Cancer of the lung and bronchus is the leading fatal malignancy in the United leading fatal malignancy in the United States. States. Five-year survival is low, but treatment Five-year survival is low, but treatment of early-stage disease improves chances of early-stage disease improves chances of survival considerably.of survival considerably.


The Challenge: Lung CancerThe Challenge: Lung Cancer

Given:Given: Promising results from recent studies involving Promising results from recent studies involving

the use of helical computed tomography (CT) for the use of helical computed tomography (CT) for the early detection of lung cancer the early detection of lung cancer

As well as rapid developments in Multi-detector As well as rapid developments in Multi-detector CT (MDCT) technology which provide for the CT (MDCT) technology which provide for the possibility of the detection of smaller lung possibility of the detection of smaller lung nodules and offers a potentially effective tool for nodules and offers a potentially effective tool for earlier detection. earlier detection.

There has been an increased interest in computer-There has been an increased interest in computer-aided diagnosis (CAD) techniques applied to CT aided diagnosis (CAD) techniques applied to CT imaging for lung cancer to assist radiologists’ with imaging for lung cancer to assist radiologists’ with their decision-making.their decision-making.


The NCI Response: Forming the The NCI Response: Forming the LIDCLIDC

To stimulate research in the area of To stimulate research in the area of CAD, the National Cancer Institute CAD, the National Cancer Institute (NCI) formed a consortium of (NCI) formed a consortium of institutions to develop the standards institutions to develop the standards and consensus necessary for and consensus necessary for constructing an image database constructing an image database resource of thoracic helical CT resource of thoracic helical CT images. images.


MotivatiMotivationon

• The development of CAD methods The development of CAD methods by the imaging research community by the imaging research community would be facilitated and enhanced would be facilitated and enhanced through access to a repository of CT through access to a repository of CT image dataimage data




(1) It would provide data to researchers (1) It would provide data to researchers without access to clinical imageswithout access to clinical images




(1) It would provide data to researchers (1) It would provide data to researchers without access to clinical images without access to clinical images

(2) It would also allow for meaningful (2) It would also allow for meaningful comparisons of different CAD comparisons of different CAD methodsmethods


The LIDCThe LIDC

This consortium - called the Lung This consortium - called the Lung Image Database Consortium (LIDC) - Image Database Consortium (LIDC) - seeks to establish standard formats seeks to establish standard formats and processes by which to manage and processes by which to manage lung images and the related technical lung images and the related technical and clinical data that will be used by and clinical data that will be used by researchers to develop, train and researchers to develop, train and evaluate CAD algorithms for lung evaluate CAD algorithms for lung cancer detection and diagnosis.cancer detection and diagnosis.


Member Member InstitutionsInstitutions

• Five institutions were selected to form Five institutions were selected to form the the Lung Image Database Consortium (LIDC)Lung Image Database Consortium (LIDC)


Member Member InstitutionsInstitutionsCornell University

UCLA

University of Chicago

University of Iowa

University of Michigan


Steering Steering CommitteeCommitteeCornell University David Yankelevitz

Anthony P. Reeves

UCLA Michael F. McNitt-GrayDenise R. Aberle

University of Chicago Samuel G. Armato IIIHeber MacMahon

University of Iowa Geoffrey McLennanEric A. Hoffman

University of Michigan Charles R. MeyerElla Kazerooni

NCI Laurence P. ClarkeBarbara Y. Croft


Contributing Contributing ParticipantsParticipantsClaudia Henschke, Cornell

David Gur, U. of PittsburghRobert Wagner, FDANicholas Petrick, FDALori Dodd, NCIEd Staab, NCIDaniel Sullivan, NCIHouston Baker, NCICarey Floyd, DukeAliya Husain, U. of ChicagoMatthew Brown, UCLAChristopher Piker, U. of IowaPeyton Bland, U. of MichiganAndinet Asmamaw, Cornell

Richie Pais, UCLAAntoni Chan, CornellGary Laderach, U. of MichiganJunfeng Guo, U. of IowaCharles Metz, U. of ChicagoRoger Engelmann, U. of ChicagoAdam Starkey, U. of ChicagoJim Sayre, UCLAMike Fishbein, UCLAAndy Flint, U. of MichiganBarry DeYoung, U. of IowaBrian Mullan, U. of IowaMadeline Vazquez, Cornell


MissioMissionn

The mission of the LIDC is the The mission of the LIDC is the sharing of lung images, especially low-sharing of lung images, especially low-dose helical CT scans of adults dose helical CT scans of adults screened for lung cancer, and related screened for lung cancer, and related technical and clinical data for the technical and clinical data for the development and testing of computer-development and testing of computer-aided detection and diagnosis aided detection and diagnosis technologytechnology


Principal Principal GoalsGoalsTo establish standard formats and To establish standard formats and

processes for managing thoracic CT scans processes for managing thoracic CT scans and related technical and clinical data for and related technical and clinical data for use in the development and testing of use in the development and testing of computer-aided diagnostic algorithms.computer-aided diagnostic algorithms.


Principal Principal GoalsGoalsTo establish standard formats and To establish standard formats and

processes for managing thoracic CT scans processes for managing thoracic CT scans and related technical and clinical data for and related technical and clinical data for use in the development and testing of use in the development and testing of computer-aided diagnostic algorithms.computer-aided diagnostic algorithms.

To develop an image database as a web-To develop an image database as a web-accessible international research resource accessible international research resource for the development, training, and for the development, training, and evaluation of computer-aided diagnostic evaluation of computer-aided diagnostic (CAD) methods for lung cancer detection and (CAD) methods for lung cancer detection and diagnosis using helical CT.diagnosis using helical CT.


The The DatabaseDatabase

•The database will contain:The database will contain:

1)1) a collection of CT scan imagesa collection of CT scan images

2)2) a searchable relational databasea searchable relational database


Fundamental Issues for the LIDCFundamental Issues for the LIDC


LIDC Challenge #1 - LIDC Challenge #1 - Define a NoduleDefine a Nodule

Though at first this seems trivial, the Though at first this seems trivial, the LIDC had significant discussion about LIDC had significant discussion about what to include and what not to what to include and what not to include as a noduleinclude as a nodule

A Nodule is part of a spectrum of focal abnormalities.

This spectrum includes scars, cancers, benign lesions, calcified lesions, etc.


What is a Nodule?What is a Nodule?

“nodule: any pulmonary or pleural lesion represented in a radiograph by a sharply defined, discrete, nearly circular opacity 2-30 mm in diameter”

from the Fleischner Society's Glossary of Terms for Thoracic Radiology (AJR 1984)


“nodule: round opacity, at least moderately well marginated and no greater than 3 cm in maximum diameter”

from the Fleishner Society's Glossary of Terms for CT of the Lungs (Radiology 1996)



Nodule


Is this a Nodule?


NOTE: This is the slice which was shown earlier



Focal AbnormalitiesFocal Abnormalities

( )NodulesNodules

• a spectrum of abnormalities

ScarSpiculated Nodule

Calcified Nodule


LIDC Challenge #1 - LIDC Challenge #1 - Define a NoduleDefine a Nodule

LIDC Response is to develop a Nodule LIDC Response is to develop a Nodule Visual Library using:Visual Library using: Cases that ARE in Nodule portion of Cases that ARE in Nodule portion of

spectrumspectrum Cases that are OUTSIDE Nodule portion of Cases that are OUTSIDE Nodule portion of

spectrumspectrum Classification by Thoracic RadiologistsClassification by Thoracic Radiologists

In Development NowIn Development Now Expected Completion Feb 2004.Expected Completion Feb 2004.


Truth AssessmentTruth Assessment

Investigators will require “truth” Investigators will require “truth” informationinformation

•



Investigators will require “truth” Investigators will require “truth” informationinformation location of noduleslocation of nodules

•



Investigators will require “truth” Investigators will require “truth” informationinformation location of noduleslocation of nodules spatial extent of nodulesspatial extent of nodules

•



Investigators will require “truth” Investigators will require “truth” informationinformation location of noduleslocation of nodules spatial extent of nodulesspatial extent of nodules

Spatial “Truth” will be estimated by Spatial “Truth” will be estimated by “Radiologic Truth” “Radiologic Truth”

•


LIDC Challenge #2LIDC Challenge #2Define the Boundary of a NoduleDefine the Boundary of a Nodule

Though it seems that the boundary of a Though it seems that the boundary of a nodule should be easy to define, we (and nodule should be easy to define, we (and others) have found that there is others) have found that there is considerable inter-reader variability in considerable inter-reader variability in defining the boundary of a nodule. defining the boundary of a nodule.

This is difficult enough with a solid nodule, This is difficult enough with a solid nodule, but even more difficult with spiculated but even more difficult with spiculated nodules, ground glass nodules or non-solid nodules, ground glass nodules or non-solid nodules.nodules.


SPICULATED NODULEInstructions to Thoracic Radiologists were

“Draw the Boundary of the Nodule”


SPICULATED NODULEExpert Number 1 Contour


SPICULATED NODULEExpert Number 2 Contour


SPICULATED NODULEComparison of Contours


Five radiologists using 3 drawing Five radiologists using 3 drawing methods:methods: One manual 3-panel (3D) drawing One manual 3-panel (3D) drawing

methodmethod Two different semiautomatic 3D Two different semiautomatic 3D

methodsmethods

Reader and Method Variability Reader and Method Variability in Drawing Boundariesin Drawing Boundaries


Case 5, Slice 19Case 5, Slice 19


Radiologist 1 - Method Radiologist 1 - Method 11


For each voxel, sum the number of For each voxel, sum the number of occurrences (across reader and method occurrences (across reader and method combinations) that it was included as combinations) that it was included as part of the nodulepart of the nodule

Create a probabilistic map of nodule Create a probabilistic map of nodule voxelsvoxels

Higher probability voxels are shown as Higher probability voxels are shown as brighter; lower probability are darkerbrighter; lower probability are darker

Can use apply a threshold and show only Can use apply a threshold and show only voxels > some prob. Value if desired.voxels > some prob. Value if desired.

Create a Probabilistic Description Create a Probabilistic Description of Nodule Boundaryof Nodule Boundary


Probabilistic Description of Probabilistic Description of BoundaryBoundary


Apply Threshold if Apply Threshold if DesiredDesired


LIDC Challenge #2LIDC Challenge #2Define the Boundary of a NoduleDefine the Boundary of a Nodule

Do we need to reconcile these Boundaries?Do we need to reconcile these Boundaries? LIDC’s answer is no.LIDC’s answer is no.

LIDC Approach will be to:LIDC Approach will be to: Come to a consistent definition of the desired Come to a consistent definition of the desired

boundary (include just solid portion? non-solid boundary (include just solid portion? non-solid portion?)portion?)

Assess reader variability of contours Assess reader variability of contours Construct a probabilistic description of Construct a probabilistic description of

boundaries to capture reader variabilityboundaries to capture reader variability


LIDC Challenge #3- LIDC Challenge #3- Data Collection Process Data Collection Process

Recent research has demonstrated Recent research has demonstrated that Single reads are not sufficient – that Single reads are not sufficient – At least two and perhaps four readers At least two and perhaps four readers may be required.may be required.

Not practical to do joint readings Not practical to do joint readings across five institutionsacross five institutions

LIDC Will NOT do a forced consensus LIDC Will NOT do a forced consensus read.read.


LIDC Challenge #3- LIDC Challenge #3- Data Collection Process Data Collection Process

Will do a Two-Staged Process: Will do a Two-Staged Process: Perform independent (Blinded) readings of cases Perform independent (Blinded) readings of cases

by multiple radiologistsby multiple radiologists Compile readings and redistribute composite readingsCompile readings and redistribute composite readings

Perform a Second, Unblinded read by same set Perform a Second, Unblinded read by same set of radiologistsof radiologists

Each reader can see readings of every other reader.Each reader can see readings of every other reader.

No forced consensusNo forced consensus Capture probabilistic detection (e.g. a Capture probabilistic detection (e.g. a

nodule can be identified by 3 of 4 readers) nodule can be identified by 3 of 4 readers) and probabilistic contours.and probabilistic contours.

8080

LIDC Process Model 2.4 LIDC Process Model 2.4 October 2003October 2003

Overview Overview PrerequisitesPrerequisitesmajor data collection steps, major data collection steps, and and data collected at each step.data collected at each step.


Prerequisites

Actions

Data Collected

IRB approvals

Participants Scanned as part of Study/Clinical Program

Non-Image DataDemographic DataLabelingImage Quality scoreScan ClassificationPatient Classification

Image Data/Dicom Data

LIDC Activities

CT Scan Criteria

Apply Inclusion Criteria

-Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible]

Apply Inclusion Criteria:

-IF Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible]

- AND IF Meets Patient Inclusion Criteria

Patient/Nodule Criteria

Apply Inclusion Criteria:

-IF Meets Scan Parameter Criteria [NOTE: All NLST & ELCAP eligible]

- AND IF Meets Patient Inclusion Criteria

-THEN Include in Db, Label Nodule Characteristics and Score Image Quality

Labeling VocabularyImage Quality Criteria


Prerequisites

Radiologist Review Process (described next)

Identified lesions for each condition: Each reader, blinded and unblinded read• Location • Outline• Label

•Definition of Nodules to be included in Db•Agreement on Marking /Contouring process

LIDC Activities

#2 Case 1#3 Case 1 #4 Case 1 #1 Case 1

Actions

Data CollectedReader 1 Reader 2 Reader 3 Reader 4


Blinded Reads – Each Reader Reads Independently (Blinded to Results of Other Readers)


Reader 1

Blinded Read for Reader 1 – Marks Only One Nodule


Reader 2

Blinded Read for Reader 2 – Marks Two Nodules(Note: One nodule is same as Reader 1)


Reader 3

Blinded Read for Reader 3 – Marks Two Nodules(Note: Again, One nodule is same as for Reader 1)


Reader 4

Blinded Read for Reader 4 – Did Not Mark Any Nodules


UnBlinded Reads – Readings in Which Readers AreShown Results of Other Readers

Each Reader Marks Nodules After Being Shown Results From Other Readers’ Blinded Reads (Each Reader Decides to Include or Ignore).


Reader 1

Unblinded Read for Reader 1 – Now Marks Two Nodules(Originally only marked one)


Reader 2

Unblinded Read for Reader 2 – Still Marks Two Nodules(No Change)


Reader 3

Unblinded Read for Reader 3 – Now Marks Three Nodules(Originally only marked two)


Reader 4

Unblinded Read for Reader 4 – Now Marks Three Nodules(Originally did not mark any)


4 Markings

2 Markings 2 Markings

Composite on Unblinded Reads for All Four Readers


Database ImplementationDatabase Implementation

TASKS COMPLETED (see reports on website):TASKS COMPLETED (see reports on website): Specification of Inclusion Criteria:Specification of Inclusion Criteria:

CT scanning technical parametersCT scanning technical parameters Patient inclusion criteriaPatient inclusion criteria

Process Model for Data collectionProcess Model for Data collection Determination of Spatial "truth" Using Blinded Determination of Spatial "truth" Using Blinded

and Unblinded readsand Unblinded reads Development of Boundary Development of Boundary

Drawing/Contouring ToolsDrawing/Contouring Tools


Database ImplementationDatabase Implementation

TASKS ONGOING (expected completion date)TASKS ONGOING (expected completion date) Definition of Nodule - Nodule Visual Definition of Nodule - Nodule Visual

Library (Feb 04)Library (Feb 04)

Evaluation of Boundary Variability (Feb Evaluation of Boundary Variability (Feb 04):04): Inter-Reader VariabilityInter-Reader Variability Boundary Drawing Tool VariabilityBoundary Drawing Tool Variability


Implementation TimelineImplementation TimelineTaskTask Date ExpectedDate Expected Specify Complete Data ModelSpecify Complete Data Model Jan 04 Jan 04 Specify LIDC internal workflowSpecify LIDC internal workflow Jan 04 Jan 04

Data passing, Performing reviewsData passing, Performing reviews Initial implementation, testing workflow Jan/Feb 04Initial implementation, testing workflow Jan/Feb 04 Database Implementation- Start Database Implementation- Start Jan 04 Jan 04 Database Implementation- Completion Mar/Apr 04Database Implementation- Completion Mar/Apr 04 Implement Public Interface to Database Apr/May 04Implement Public Interface to Database Apr/May 04

PUBLIC ACCESS TO CASES – EXPECTED MAY/JUN 04PUBLIC ACCESS TO CASES – EXPECTED MAY/JUN 04


Publications/PresentationsPublications/Presentations

LIDC Overview manuscriptLIDC Overview manuscript In Preparation, submission in 1In Preparation, submission in 1stst Quarter 04 Quarter 04

Assessment Methodologies manuscriptAssessment Methodologies manuscript In Preparation, submission in 1In Preparation, submission in 1stst Quarter 04 Quarter 04

Special Session Special Session SPIE Medical Imaging SPIE Medical Imaging Sunday evening Feb 15, 2004Sunday evening Feb 15, 2004


To learn more about the LIDC:To learn more about the LIDC:

Return for CME Category 1 Credit:Return for CME Category 1 Credit: Monday – Thursday Monday – Thursday 12:15 pm to 1:15 pm12:15 pm to 1:15 pm

At these times, LIDC members will At these times, LIDC members will be here to describe the efforts of the be here to describe the efforts of the consortium, this exhibit and any consortium, this exhibit and any other questions you might haveother questions you might have


Return to Exhibit Home PageReturn to Exhibit Home Page

Click HereClick Here

PowerPoint Presentation

Health & Medicine

Transcript of PowerPoint Presentation