Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20051

Brisbane, 02. Nov 2007 Intra-Modality Registration EvaluationMartin Urschler

A Framework for Comparison and Evaluation of Nonlinear Intra-Subject Image Registration

Algorithms

Martin Urschler, Stefan Kluckner, Horst Bischof

Institute for Computer Graphics and Vision,

Graz, University of Technology, Austria

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20052

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Motivation

Nonlinear Image RegistrationValidation very difficult!

- Lack of direct ground truth

- Lack of gold standard methods

- Highly ill-posed problem

- Large space of possible solutions

We claim: Standardizing Evaluation Protocols at least as important as Developing Novel Methods!

-> Put framework to discussion:- Open-source community effort- Instantiate framework by showing sample evaluation

Presented algorithm at last years MICCAI.

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20053

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Contents

• Problem Definition

• Related Work & Similar Efforts

• Open Science Nonlinear Registration Evaluation Framework

• Sample Evaluation – Intra-subject Thorax CT

• Conclusion & Outlook

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20054

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

h(x)

The Nonlinear Registration Problem

„Find a deformable mapping h(x) aligning moving and fixed image such that a defined similarity criterium is minimized.“

Base Illustration taken from ITK Software Guide.

SSD, NCC, Mutual Information, …

B-Spline, Thin-Plate Spline, Def Field, …

Gradient Descent, BFGS, …

Intensity-Based, Feature-Based, …

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20055

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Medical Applications of Nonlinear Image Registration

• Angiography Studies• Anatomy & Function

– Perfusion/Ventilation

• Correction of Motion Artifacts• Studies of Shape Variation• Segmentation by Atlas Registration• Surgical Planning• …

Lung Perfusion for Pulmonary Embolism Detection[Wildberger et al.]

Validation Study: Atlas-based Brain VolumeSegmentation from MRI Images[Ng02] – taken from ITK Documentation

Inter- & Intra-Modality Inter- & Intra-Subject

No Ground Truth or Gold StandardsHighly Ill-Posed

How should we reachclinical acceptance?

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20056

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Related Work on Evaluation (Frameworks)

There exist some good ideasfor evaluation

in (Medical) Computer Vision!

Multi-View 3DReconstruction

[Seitz et al]Stereo

Reconstruction[Scharstein et al]

Middlesbury

VALMET Segmentation Evaluation [Gerig et al]

Retrospective Evaluation of Intersubject Brain Registration [Hellier et al]

Retrospective Rigid Registration Evaluation Project [West et al]

NIREP (Inter-subject Brain-RegistrationEvaluation) [Christensen et al]

Validation of Nonrigid Image

Registration Using FEM [Schnabel et al]

Segmentations are compared!

Community would benefitfrom open-sourceimplementation!

MICCAI 2007 SegmentationChallenge Workshop

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20057

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Open Science Evaluation Framework

• Algorithm Evaluation vs. Validation [Hellier et al]– Specific Problems: Lacking ground truth, ill-posedness, Lack of gold standard – General Problems: Noise, Partial Volume Effect, Interpolation, Numerics

• Modular Evaluation Framework– Open-source, open-access, open-data, open protocols

• Building Blocks:

RegistrationAlgorithms

Public domaindata sets

Synthetic Deformations

Similarity MeasuresPython Framework

as Glue

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20058

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Open Science Evaluation Framework

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.20059

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Synthetic Deformations Used

• Simple synthetic transformation models– Regular grid with random deformations & TPS– Uniform periodic cosine transformation

– Tailored to breathing difference registration:• Simulated Breathing Model

• Synthetic Airway Tree Movement from Manual Correspondences

• …

• Increase number of models– Pool of synthetic transformations defined

by pool of algorithms („bronze standard“ Glatard et al – MICCAI 2006)

– Community effort needed

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200510

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Data Sets and Algorithms

• Public domain data sets from – National Library of Medicine Dataset Collection (currently offline)

– MIDAS data collection project

– National Lung Cancer Archives (NCIA)

• ITK Algorithms– Demons

– Symmetric Demons

– Level Set Motion

– Curvature

– Fast Block Matching (Workshop Contribution)

– Diffeomorphic Demons (Workshop Contribution)

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200511

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Quantitative Measures

• Measures on Displacement Fields– RMS of displacement field differences– MAD of displacement field differences– MAX of displacement field differences– Jacobian determinant of displacement field

• Measures on fixed and warped moving image– Clamped RMS intensity differences– MAD intensity differences– MAX intensity differences– Normalized Mutual Information– Edge Overlap

• IMHO only first group says something about registration performance!

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200512

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

A Sample Evaluation• Purpose: Evaluate intra-modality thoracic CT registration

subject to breathing differences

• 2 data sets 256^3– NLM NormalChestCTNoContrast– NCIA LIDC 30047

• Single choice of algorithm parameters• 64 bit Opteron with 2.4GHz and 8GB RAM

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200513

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

A Sample Evaluation

Simulated Data Difference to Original Standard Demons

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200514

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

• Visual Results (Problem Cases)– Level Set motion -> artifacts– Fast Block Matching ->

Implementation Issues (Test Framework)

– Symmetric Demons

A Sample Evaluation - Results

• Quantitative measures– Large number (see paper)– Quantities are single numbers

• Problematic?

– Standard Demons & Diffeomorphic Demons very well

– Algorithm Parameter Studies

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200515

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Conclusion

• Open Science Evaluation Framwork presented• Intra-Modality Sample Evaluation shown

– Results should not be seen as final algorithm quality statements• Framework has to grow…

– Useful for comparing & testing algorithms

– Useful for parameter studies

• A small step towards establishing standardized protocol to gain clinical acceptance…

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200516

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Further Work

• Common Web Repository (Hosted by Kitware?)• Upgrade by community effort

– More algorithms

– More quality measures • Segmentation based -> Problems for general applicability

– More synthetic deformations, including noise models

• Extension to inter-modality, inter-subject problems• Cooperate with NIREP?• How to prevent „evaluation framework tuning“?

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200517

Brisbane, 02. Nov 2007 Intra-Modality Registration EvaluationMartin Urschler

Thank you for your attention!

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200518

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200519

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler

Graz, University of Technology, Institute for Computer Graphics and Vision

Professor Horst Cerjak, 19.12.200520

Brisbane, 02. Nov 2007 Nonlinear Registration EvaluationMartin Urschler