Slide - 1 Confidential

Slide - 2 Confidential

Slide - 3 Confidential

Interalgorithm Study using CT Images of synthetic

nodules……….

Slide - 4 Confidential

An inter-algorithm study, in the same way QIBA has been working on inter-reader,inter-scanner, and inter-site.  We will also connect it to the analysis section of QIBA Profile. 

The aim of the study is to estimate inter- and intra-algorithm variability by the volume estimation of synthetic nodules from CT scans of an anthropomorphic phantom (according to the work of the QIBA 1A Group (see Dr. Petrick‘s paper , SPIE 2011)

For the impelmentation of the obejctives a challenge could be organized (?)

Objectives of the QIBA 3A Group

Slide - 5 Confidential

Motivation for the Study

Study Design

Dataset

Algorithms

Analysis Protocol

Analysis method

Result evaluation

Result presentation

Result evaluation for the QIBA protocol

Study Motivation

Slide - 6 Confidential

Motivation

Slide - 7 Confidential

Changes in nodules volume is important for diagnosis, therapy planning, therapy response evaluation

Measuring volume changes requires high accuracy in measurement of absolute volume

Ground true has to be exact measured. This is not the case by data annotation (inter- and intra-observer variability)

Volumes of synthetic nodules are measured (high accuracy)

Therefore it make sense to use such data (as ground truth) in order to calculate accuracy measurement of algorithms

The study results could be combined with the QIBA 1A Group work. This combination will improve the QIBA volumetric CT Profile development.

The study has to be complementary to the study of the 1A Group:

1A study: Radiologists and synthetic nodules (inter- intra-observer

variability)

3A Study: Algorithms and synthetic nodules: intra-algorithms variability)

Using of the same technological basis (scanner type, Philips (1A Group data, synthetic nodules)) to secure independence for inter-scanner variability (except BIAS between different serial numbers)

Motivation

Slide - 8 Confidential

Describe the overall analysis procedure (use as example the Bio-change Challenge Protocol http://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm: How to Participate in Biochange Challenge (Dr Fenimore)

Download and read the Bio-change Challenge Protocol and email or fax Statement of Interest to NIST.

Download the Bio-changeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as described in the Protocol.

Run your change analysis algorithm or CAD tool in your lab on the validation data.

Report your change results in one of the required formats and send a Participation Agreement signed by the your team leader to NIST by January 18, 2011.

NIST will analyze the reported results, comparing them to the limited available ground truth as described in the Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without publicly identifying individual scores by Participant.

Analysis procedure (analysis Protocol according Bio-change or Volcano Challenge?)

Slide - 9 Confidential

Has to be discussed during the meeting March 17th 2011

The aim of the study is not a challenge to know:

Who is the best image analysis algorithm

The aim of the study is to gain knowledge for the QIBA profile (paragraph 9 concerns image analysis)

Avoid competition

Support Cooperation, conjoint approach

OUR CONTEXT is QIBA

Slide - 10 Confidential

Phantom data as used for the 1A Group Study (see Dr. Patrick’s publication, SPIE 2011)

1C clinical data (Dr Fenimore/ Phantom –synthetic data)

Training dataset

Test dataset

Or only test dataset (analyze without trainings dataset)

data pool specification (simple geometries, up to anatomical representations)

(if clear than we could propose an experimental design)

Nodule classification concerning position. Shape and margins of the nodule … (according to the QIBA Profile)

Algorithm classification

Analysis protocol:

each algorithm is applied to the dataset separately

each algorithms is applied to the “training” data set

each algorithms is applied to the “test” data set

Statistical analysis of the analysis results of each algorithm separately

Data/Nodules/Algorithms

Slide - 11 Confidential

Academia and non profit organizations

Industrial vendors

(for example possible vendors, according to the Volcano 2009 challenge could be: Siemens, Phipps, MeVis, Kitware, Definiens, VIA CAD etc…

Description/Classification of the algorithms: according to the grade of user intervention is needed (for example Volcano’09, A. P. Reeves et al):

Totally automatic using seed points

Limited parameter adjustment (on less than 15% of the cases)

Moderate parameter adjustment (on less than 50% of the cases)

Extensive parameter adjustment (more than 50% of the cases)

Limited image/boundary modification (on less than 15% of the cases)

Moderate image/boundary modification (on less than 50% of the cases)

Extensive image/boundary modification (more than 15% of the cases)

Or:

M. Gavrielide et al., Noncalcified Lung Nodules, Volumetric assessment with thoracic CT (RSNA 2009):

Semi automated

Manually derived

Semi automated algorithms are typically initiated by defining a region of interest around a nodule or by a user-provided point inside

the nodule area. Depending on the application, segmentation algorithms are then employed to delineate nodules from the surrounding lung parenchyma and neighboring structures such as attached vasculature and pleural surfaces.

Manually derived methods require users to interactively delineate nodule boundaries, typically in a section-by-section fashion; this is followed by an application of three-dimensional software to merge the two-dimensional boundaries into a volume. The estimate of nodule volume is then based on the total number of voxels within the segmented region.

The majority of volume measurement methods use voxel counting

Algorithm vendors and Algorithm classification examples from the literature

Slide - 12 Confidential

IEEE Transactions on medical imaging, vol. 28 MICCAI

Algorithm Classification:

An automatic segmentation algorithm does not require any user intervention.

A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation.

An interactive algorithm requires manual editing of the final results.

Abstract: In this paper, we present the organization of a competition of liver tumor segmentation techniques. The liver tumor segmentation competition is part of the workshop "3D Segmentation in the Clinic:

A Grand Challenge II" at Medical Image Computing and Computer Assisted Intervention 2008 conference. The goal of this contest is to compare the he performance of different algorithms for segmenting liver tumor from contrast enhanced CT images. Several organizing topics are described, contrast including the motivation for organizing this competition, training and testing dataset evaluation methods

Algorithm Classification according 3D Segmentation in the Clinic:

A Grand Challenge II - Liver Tumor Segmentation, Xiang Deng and Guangwei Du, Center for Medical Imaging Validation, Siemens Corporate Technology

(citation)

Slide - 13 Confidential

MICCAI Liver Challenge proposed metrics (citation)

Slide - 14 Confidential

Slide - 15 Confidential

Sensitivity

Sens = TP/TP+FN

Specificity

Spec = TN/TN+FP

Accuracy = (TN +TP)/all

Lucila Ohno-Machado

An introduction to calibration and discrimination methods, HST951 Medical Decision SupportHarvard Medical School, Massachusetts Institute of Technology

Slide - 16 Confidential

Box plot (for example: for each data set, nodule volume vs. method, relative Volume)

Volrel= 100%* (Volest – Voltrue)/ Voltrue

Volrel= 100%* (Volalg – Volsynthetic nodule)/ Volsynthetic nodule

We propose: A simple but well defined and reliable approach

Other

Slide - 17 Confidential

We propose: keep it simple

Algorithm Classification:

•An automatic segmentation algorithm does not require any user intervention.

•A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation.

•An interactive algorithm requires manual editing of the final results.

Slide - 18 Confidential

Known: Vsn Measurement: Valg

StdvValg (1) <- same Phantom, multiple measurements, same algorithm, multiple seed points (two groups of algorithms: semiautomatic, manually)

StdvValg (2)<- different phantoms (according QIBA profile: nodule classification), same algorithm: • StdValg (nodule class)• StdValg(all nodule classes)

Bias out of digitalization in the area of lesion margins (Voxels/Pixels on the lesion margins, mixed pixels) (Group 1A, Group 1C, Group 1B, QIBA VIA Profile, Definitions and how should be calculated?)

Achievements:

• Transparent and Objective comparability between different algorithms

• One independent Error value for further investigations of clinical data

Algorithm Evaluation Method

Slide - 19 Confidential

Definition of the Study Data/Measurements/Metrics /

Which data:

1A (Phantom/synthetic data) (defined experimental conditions, experimental design

Phantom Data for dV?)

1B Clinical data

according to that: do we have we to separate the study design as following?

Study design for Phantom/Synthetic data (focus)

Study design for Clinical data

Metrics

Slide - 20 Confidential

Decision

Slide - 21 Confidential

Each participant has to be informed (only their own algorithm results)

Publication of the results (all participants)

Using the results for the QIBA protocol: knowledge exploitation for the QIBA protocol

Resulting recommendations for FDA (Is this possible?)

Result information for the participants