A Highly Deformable Object Model for Tracking

Nilanjan Ray

Department of Computing Science

University of Alberta

Overview of Presentation

• Tracking deformable objects– Motivations: desirable properties of a deformable object model– An example application (mouse heart tracking)

• Some technical background– Level set function and its application in image processing– Non-parametric probability density function (pdf) estimation– Similarity/dissimilarity measures for pdfs

• Proposed deformable object model and tracking technique

• Results, comparisons and demos

• Ongoing investigations– Incorporating color cues, and other features– Adding constraints on object shape– Application in morphing (?)

• Summary

• Acknowledgements

Tracking Deformable Objects

• Motivations for a new deformable model– desirable properties:– Adapt with deformations (sometimes drastic

deformations, depending on applications)– Ability to learn object and background:

• Ability to separate foreground and background• Ability to recognize object from one image frame to the next,

in an image sequence

Motivations: Some Existing Deformable Models

• Deformable models: – Highly deformable

• Examples: snake or active contour, B-spline snakes, …• Good deformation, but poor recognition (learning) ability

– Not-so-deformable• Examples

– Active shape and appearance models

– G-snake

– …

• Good recognition (learning) capability, but of course poor deformation ability

So, how about good deformation and good recognition capabilities?

An Example Application

Cardiac video tracking

Technical Background: Level Set Function

• A level set function represents a contour or a front geometrically• Consider a single-valued function (x, y) over the image domain;

intersection of the x-y plane and represents a contour:

(X(x, y), Y(x, y)) is the point on the curve that is closest to the (x, y) point

• Matlab demo (lev_demo.m)

otherwise,,)),(()),((

objecttheinsideis),(if,)),(()),((),(

22

22

yxYyyxXx

yxyxYyyxXxyx

Applications of Level Set: Image Segmentation

• Matlab segmentation demo (yezzi.m)

• Vessel segmentation

• Brain reconstruction

• Virtual endoscopy

• Trachea fly through

• …tons out there

Show videos

Level Set Applications: Image Denoising

• Two example videos

Show video

Level Set Applications: Robotics

• Finding shortest path

Show video

Level Set Applications: Computer Graphics

• Morphing• Simulation• Animation• ….

http://www.sci.utah.edu/stories/2004/fall_levelset.html

Go to http://graphics.stanford.edu/~fedkiw/for amazing videos

More Applications of Level Set Methods

• Go to http://math.berkeley.edu/~sethian/2006/Applications/Menu_Expanded_Applications.html

Technical Background: Non-Parametric Density Estimation

dxdyiyxI

CiH

i

)2

)),((exp(

1)(

2

2

I(x, y) is the image intensity at (x, y)i is the standard deviation of the Gaussian kernelC is a normalization factor that forces H(i) to integrate to unity

Normalized image intensity histogram:

Technical Background: Similarity and Dissimilarity Measures for PDFs

dzzP

zQzQQPKL )

)(

)(log()(),(

dzzPzQQPBC )()(),(

Kullback-Leibler (KL) divergence (a dissimilarity measure):

Bhattacharya coefficient (a similarity measure):

P(z) and Q(z) are two PDFs being compared

Proposed Method: Deformable Object Model and Tracking

• Deformable Object model:

– From the first frame learn the joint pdf of level set function and image intensity (image feature)

• Tracking:

– From second frame onward search for similar joint pdf

Proposed Deformable Object Model

• Joint probability density estimation with Gaussian kernels:

dxdyiyxJlyx

CilQ

il

)2

)),((exp()

2

)),((exp(

1),(

2

2

2

2

J(x, y) is the image intensity at (x, y) point on the first image frame(x, y) is the value of level set function at (x, y) on the first image frameC is a normalization factor

We learn Q on the first video frame given the object contour (representedby the level set function)

Level set function value: lImage intensity: i

Proposed Object Tracking

• On the second (or subsequent) frame compute the density:

• Match the densities P and Q by KL-divergence:

• Minimize KL-divergence by varying the level set function (x, y)

dxdyiyxIlyx

CilP

il

)2

)),((exp()

2

)),((exp(

1),(

2

2

2

2

I(x, y) is the image intensity at (x, y) on the second/subsequent frame(x, y) is the level set function at on the second/subsequent frame

dldiilP

ilQilQKL

),(

),(log),(

Note that here only P is a function of (x, y)

Minimizing KL-divergence

• In order to minimize KL-divergence we use Calculus of variations

• After applying Calculus of variations the rule of update (gradient descent rule) for the level set function becomes:

dldiiyxIlyxlyx

ilP

ilQtyxyx

il

t

l

ttt )

2

)),((exp()

2

)),((exp(

)),((

),(

),()(),(),(

2

2

2

2

21

t : iteration numbert : timestep size

Minimizing KL-divergence: Implementation

• There is a compact way of expressing the update rule:

)),(),,()()((),(),( 11 yxIyxg

P

Qtyxyx ttt

Where g1 is a convolution kernel:

)2

exp()2

exp()(2

2

2

2

21ill

illlg

is a function defined simply as:

),(

),(

il

ilQ

P

Q

P

Q

convolution

dldiiyxIlyxlyx

ilP

ilQtyxyx

il

t

l

ttt )

2

)),((exp()

2

)),((exp(

)),((

),(

),()(),(),(

2

2

2

2

21

Minimizing KL-divergence: A Stable Implementation

• The previous implementation is called explicit scheme and is unstable for large time steps; if small time step is used then the convergence will be extremely slow

• One remedy is a semi-implicit scheme of numerical implementation:

)),(),,()()((1

)),(),,()()((),(),(1

yxIyxgPQ

t

yxIyxgPlQ

tyxyx

t

t

Where g is a convolution kernel:

)2

exp()2

exp()(2

2

2

2

il

illg

In this numerical scheme t can be large and still the solution willbe convergent; So very quick convergence is achieved in this scheme

is a function defined simply as:

),(

),(

il

ilQl

P

lQ

P

lQ

Results: Tracking Cardiac Motion

A few cine MRI frames and delineated boundaries on them

Show videos

Numerical Results and Comparison

10 20 30 40 50 60 70 80 900

0.5

1

Frame number

Pra

tt FO

M

GVF snake method

Proposed method10 20 30 40 50 60 70 80 90

0

0.5

1

Frame number

Seg

men

tatio

n S

core

GVF snake method

Proposed method

20 40 60 80 100 120 140 160 1800

0.5

1

Frame number

Prat

t FOM

GVF snake method

Proposed method 20 40 60 80 100 120 140 160 1800

0.5

1

Seg

men

tatio

n S

core

Frame number

GVF snake method

Proposed method

Pratt’s FOM Segmentation Score

Slow Seq. Rapid Seq. Slow Seq. Rapid Seq.

GVF Snake Method 0.51 0.62 0.51 0.44

Proposed Method 0.88 0.77 0.74 0.79

Comparison of mean performance measures

Sequence with slow heart motion

Sequence with rapid heart motion

Extensions: Tracking Objects in Color Video

• If we want to learn joint distribution of level set function and color channels (say, r, g, b), then non-parametric density estimation suffers from:– Slowness– Curse of dimensionality

• Another important theme is to combine edge information and region information for objects

• One remedy sometimes is to take a linear combination of r, g, and b channels– Fisher’s linear discriminant can be used to learn the

coefficients of linear combination

Extensions: Adding Object Shape Constraint

• Can we constrain the object shape in this computational framework?

),(),()( pqKLPQKLE

dxdylyx

Klq

l

)2

)),((exp(

1)(

2

2

dxdylyx

Klp

l

)2

)),((exp(

1)(

2

2

Frame number: 1

50 100 150 200

20

40

60

80

100

120

140

160

180

Frame number: 2

50 100 150 200

20

40

60

80

100

120

140

160

180

Minimize:

where

Application in Computer Graphics: Morphing

(J1, 1) (I2, 2)

Initial object Shape and intensity/texture

Final object Shape and intensity/texture

Morphing: generate realistic intermediate tuples (It, t)

(I1, 1), (I2, 2), …..

Morphing: Formulation

• Generate intermediate shapes, i.e., level set function t (say, via interpolation):

• Next, generate intermediate intensity It by maximizing:

• Once again we get a similar PDE for It

11

21 ,

1

t

tt

)),(),,((

)),(),,(()(

22

11

tt

ttt

IPDFJPDFBC

IPDFJPDFBCIE

Morphing: Preliminary Results

Source

20 40 60 80 100 120

20

40

60

80

100

120

Target

20 40 60 80 100 120

20

40

60

80

100

120

Show videos

Summary

• Contributions– A novel deformable object model for tracking– Related efficient and stable computation

• Several exciting extensions

• Application in morphing

Acknowledgements

• Baidyanath Saha• CIMS lab and Prof. Hong Zhang• Prof. Dipti P. Mukherjee, Indian Statistical Institute• Department of Computing Science, UofA