Medical ImageRegistration

Dept. of Biomedical Engineering

Biomedical Image Analysis

www.bmia.bmt.tue.nl

Image registration definition

‘‘ Image registration is about determining a spatial transformation –

or mapping – that relates positions in one image, to corresponding

positions in one or more other images’’

• 3D - 3D• 3D - 2D• 3D/2D - patient

Source image Target image

Example from our group

Medtronic Polestar N20Intra-operative MRI

Pre-Operative Intra-Operative

Student Project Wenxin Wang: REGISTRATION

Many more examples of imaging modalities

X-rays CTAngiographyMRI

Ultrasound SPECT PET

Application of image registration

Same modality, same patient

- monitoring and quantifying disease progression over time,

- evaluation of intra-operative brain deformation, etc…

Different modalities, same patient

- correction for different patient position between scans,

- linking between structural and functional images, etc…

Same modality, different patients

- atlas construction

- studies of variability between subjects, etc…

Temporalregistration

PET

Fusion of images

MRI CT

Colored overlay

PET - CT

Region of interest (ROI) selection & color display

Fusion of images

CT scan of a thyroid gland Fusion of SPECT and CT

Fusion of images

Protein localization

Different spectral bandsfor optical biomarkers

Fusion of images

Mapping of calculatedprobability maps

Fusion of images

Functional MRI maps onAnatomical MRI

fMRI

Weighted intensitycombination

Fusion of images

CT MRI Also possiblewith intermittendpresentation(flicker)

Fusion of images

Checkerboard fusion

Fusion of images

Linkedcursor

Fusion of images

Radiotherapy planning

Iso-dosis contours on CT

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

• Matching with pointbased methods• Matching with surface based methods• Matching with intensity based methods

CTimages

Dynamicseries

Workstation Perfusion images

o

o

o

o

CT Perfusion: matching over time

Marcel QuistPhilips Medical Systems

Medical IT – Advanced Development

• infarct• tumor properties• blood perfusion

o

o

o

o

CT PerfusionMarcel Quist

Philips Medical SystemsMedical IT – Advanced Development

CTimages

Dynamicseries

Workstation Perfusion images

• infarct• tumor properties• blood perfusion

Blood volume Blood current Time to maximumAver. passage time

Courtesy: Charité, Berlin

Functionalperfusionimages

Registration

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

• Matching with pointbased methods• Matching with surface based methods• Matching with intensity based methods

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Image markers

Point-Based Registration

Coordinates for the fiducials can be found on multiple images

One set of fiducials can be lined up with another.

Fiducials

Devicepositiontracking

2 cameras

Finding the Fiducials

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

2D Affine Transforms

Translations by tx and ty

x1 = a x0 + b y0 + tx

y1 = c x0 + d y0 + ty

Rotation around the origin by radians

x1 = cos() x0 + sin() y0

y1 = -sin() x0 + cos() y0

Zooms by sx and sy

x1 = sx x0

y1 = sy y0

Shearx1 = x0 + h y0

y1 = y0

http://www.dt.org/html/meshwarp.html

3D Rigid-body Transformations

A 3D rigid body transform is defined by:

3 translations - in X, Y & Z directions

3 rotations - about X, Y & Z axes

The order of the operations matters

1000

0100

00cossin

00sincos

1000

0cos0sin

0010

0sin0cos

1000

0cossin0

0sincos0

0001

1000

Zt100

Y010

X001

rans

trans

trans

ΩΩ

ΩΩ

ΘΘ

ΘΘ

ΦΦ

ΦΦ

Translations Pitchabout x axis

Rollabout y axis

Yawabout z axis

Geometrical transformations

• Rigid• preserves straightness of lines• intra-patient, rigid anatomy• rotation, translation, zoom, skew

• Curved• inter-patient• atlas• tissue deformation

Image Metrics

FixedImage

MovingImage

Metric

Transform

Interpolator

Value

Parameters

Distance measures

link to pdf

Image Metrics – similarity measures

1. Subtraction:

2. Mean squared differences:

3. Correlation coefficient:

if the intensities are linearly related.

Demo

Entropy

A measure of dispersion or disorder.

High entropy high disorder.

Mutual information

A measure of how well one random variable

(image intensities) “explains” another.

High mutual information high similarity

Similarity Based on Information Theory

Mutual Information

Correct registration Large mis-registration

Wachowiak et al., Proc. SPIE Medical Imaging, 2003

Entropy

Mutual information

Normalized mutual information

H X i 1

np iln p i H X , Y

i 1

nj 1

mp i jln p i j

IX , Y H X H Y H X , Y IX , Y

H X H Y H X , Y

MR – MR (identical images)Translation 2 and 5 mm.

Mutual Information

Mutual Information

MR – CTTranslation 2 and 5 mm.

Demo

Two images are similar if changes of intensity occur at the same locations.

Gradient Field

Normalized Gradient Field:

Regularized Normalized Gradient Field:

Registration Distance Measure (1): Normalized Gradient

Field

I I n

2 2

I

I

n

I

Distance measure of NGF:22

NGF 2 2D [ , ] ( ) ( ) ( ) ( ) sin( )R T R T R T n n n n

Normalized Gradient Field

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Optimization

Optimization involves finding some “best”

parameters according to an “objective function”,

which is either minimised or maximised

The “objective function” is often related to a

probability based on some model

Value of parameter

Objective function

Most probable solution (global

optimum)Local optimumLocal optimum

Plotting the Metric

Mean Squared Differences

Transform Parametric Space

Sensitivity analysis

The Best Transform Parameters

Evaluation of thefull parameter space

is equivalent to performingoptimization by exhaustive searchVery Safe

but

Very SlowBetter Optimization Methods: for example: Gradient Descent

Optimization in Image Registration

Main goal: To determine the transformation

parameters that result in the minimum value of a

‘distance measure’.

Transformation parameters:

Translations

Rotations

Scaling

Find the “best”, or optimum value

of an objective (cost) function.

Very large research area.

Multitude of applications.

Image Registration Framework

FixedImage

MovingImage

Metric

Transform

InterpolatorOptimize

r

Parameters

Applications of Optimization

Engineering designBusiness and

industry

Radiotherapyplanning

Biology and medicine

Economics

Systems biologyManagement

Design ofmaterials

Manufacturing design

BioinformaticsProteomics

Image registration

Finance

Simulation and modeling

Global and local optimization

Local Optimization

Start

End

Local Optimization

Start

Global Optimization

End

Global Optimization

Gradient Descent Optimizer f( x , y )

S = L ∙ G( x , y )f( x , y )

∆

G( x , y ) =

S = Step

L = LearningRate

Gradient Descent Optimizer f( x , y )

S = L ∙ G( x , y )f( x , y )

∆

G( x , y ) =

Registration Framework

ReferenceImage

TemplateImage

CalculateDistanceMeasure

ConditionMet?

TransformedTemplateImage

OptimizeTransformationParameters

TransformTemplateImage

Yes

NO

Multi-Resolution Registration Framework

Registration

Registration

Registration

Fixed Image Moving Image

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Multi-Modality Registration

Fixed Image Moving Image

Registered Moving Image

Classification of registration algorithms:

• Image dimensionality 2D, 3D, time, ...• Registration basis point sets, markers, surfaces, ...• Geometrical transformations affine, perspective, ...• Degree of interaction user initialization, automatic• Optimization procedure max distance, gradient descent• Modalities multi-modal, intra-modal, ...• Subject inter-patient, atlas, ...• Object head, vertebra, liver, ...

Visual Integration Platform for Enhanced Reality (VIPER)

Collaboration withDr. Wieslaw Nowinski,Cerefy Atlas,A*Star, Singapore

Substantia Nigra

NucleusSubthalami

Motor Tract

Atlas

Substantia Nigra

NucleusSubthalami

Motor Tract

Atlas

Cerefy Anat.Brain Atlas

Wieslaw Nowinski, Singapore

Anatomy atlas vs. function atlas (fMRI)

Manual marking of recognizable landmarks in both atlas and high resolution data.

D30L D32L

D28L

D31L

D29L

D30L D32L

D28L

D31L

D29L

TT88s / L+5mm

D30LD30L D32LD32L

D28LD28L

D31LD31L

D29LD29L

D30LD30L D32LD32L

D28LD28L

D31LD31L

D29LD29L

TT88s / L+5mm

Example of slice TT88s / L+5mm

Registration ofreference databy landmarks

Select points on conditions:

Clearly visible in both atlas and reference

data;

Distribution in whole brain volume;

Number of landmarks is unlimited.E. BenninkJ. Korbeeck