Geometric Active ContoursRon Kimmel

www.cs.technion.ac.il/~ron

Computer Science Department Technion-Israel Institute of Technology

Geometric Image Processing Lab

Edge Detection Edge Detection:

The process of labeling the locations in the image where the gray level’s “rate of change” is high. OUTPUT: “edgels” locations,

direction, strength

Edge Integration: The process of combining “local” and perhaps sparse and

non-contiguous “edgel”-data into meaningful, long edge curves (or closed contours) for segmentation OUTPUT: edges/curves consistent with the local data

The Classics Edge detection:

Sobel, Prewitt, Other gradient estimators Marr Hildreth

zero crossings of Haralick/Canny/Deriche et al.

“optimal” directional local max of derivative

Edge Integration: tensor voting (Rom, Medioni, Williams, …) dynamic programming (Shashua & Ullman) generalized “grouping” processes (Lindenbaum et al.)

IG *

The “New-Wave” Snakes Geodesic Active Contours Model Driven Edge Detection

Edge Curves

“nice” curves that optimize a functional of g( ), i.e.

nice: “regularized”,

smooth, fit some prior information

curve

dsg ) (

Image

),( yxg

Edge Indicator Function

2

1

1 | ( * ) |G I

Geodesic Active Contours

Snakes Terzopoulos-Witkin-Kass 88 Linear functional efficient implementation non-geometric depends on parameterization

Open geometric scaling invariant, Fua-Leclerc 90 Non-variational geometric flow

Caselles et al. 93, Malladi et al. 93 Geometric, yet does not minimize any functional

Geodesic active contours Caselles-Kimmel-Sapiro 95 derived from geometric functional non-linear inefficient implementations:

Explicit Euler schemes limit numerical step for stability Level set method Ohta-Jansow-Karasaki 82, Osher-Sethian 88

automatically handles contour topology Fast geodesic active contours Goldenberg-Kimmel-Rivlin-

Rudzsky 99 no limitation on the time step efficient computations in a narrow band

Laplacian Active Contours Closed contours on vector fields

Non-variational models Xu-Prince 98, Paragios et al. 01 A variational model Vasilevskiy-Siddiqi 01

Laplacian active contours open/closed/robust Kimmel-Bruckstein 01

Most recent:variational measures

for good old operatorsKimmel-Bruckstein 03

Segmentation

Segmentation

Ultrasound images

Caselles,Kimmel, Sapiro ICCV’95

Segmentation

Pintos

Woodland Encounter Bev Doolittle 1985

With a good prior who needs the data…

Segmentation

Caselles,Kimmel, Sapiro ICCV’95

Prior knowledge…

Prior knowledge…

Segmentation

Segmentation

Segmentation

Caselles,Kimmel, Sapiro ICCV’95

Segmentation

With a good prior who needs the data…

Wrong Prior???

Wrong Prior???

Wrong Prior???

Curves in the Plane C(p)={x(p),y(p)}, p [0,1]

y

x

C(0)

C(0.1) C(0.2)

C(0.4)

C(0.7)

C(0.95)

C(0.9)

C(0.8)

pC =tangent

Arc-length and Curvature

s(p)= | |dp 0

p

| | 1,sC pC | |p

sp

CC

C

ssC N

1

ssC N

C

Calculus of Variations

Find C for which is an extremum

Euler-Lagrange:

1

0

( , )pE L C C dp( ) ( , , , )p p pE L x y x y dp

0

( )px x x

E dE L L dp

dp

0

0

p

p

x x

y y

dL L

dp

dL L

dp

Calculus of Variations

Important Example Euler-Lagrange: , setting Curvature flow

( )t ss ssC C C N

1

0

| |pE C dp| | 0pN C

QQQQQQQQQQQQQQ| |pds C dp

0N

Potential Functions (g)

x

I(x,y) I(x)

x

g(x)

xx

g(x,y)

Image

Edges2

1

1 | ( * ) |G I

Snakes & Geodesic Active Contours

Snake modelTerzopoulos-Witkin-Kass 88

Euler Lagrange as a gradient descent

Geodesic active contour modelCaselles-Kimmel-Sapiro 95

Euler Lagrange gradient descent

CgCCdt

dCpppppp

0arg min

L C

Cg C ds

NΝg(C),κCgdt

dC

1

0

22 )(2minarg dpCgCC ppp

C

Maupertuis Principle of Least Action

Snake = Geodesic active contourup to some , i.e Snakes depend on parameterization. Different initial parameterizations yield solutions for different geometric functionals

0g g E 0E

12

0

0

0

arg min ( ( ( )) | | )

arg min ( ( ))

pC

L

C

g C p C dp

g C s E ds

x

y

p

1

0

Caselles Kimmel Sapiro, IJCV 97

Geodesic Active Contours in 1D

Geodesic active contours are reparameterization invariant

I(x)

x

g(x)

x

NΝ

),(CgCgdt

dC

Geodesic Active Contours in 2D

2

1

1 | ( * ) |G I g(x)=

G *I

NNCgCgdt

dC ),(

Controlling -max

0

( )Lg C ds

I g

Smoothness

Cohen Kimmel, IJCV 97

Fermat’s PrincipleIn an isotropic medium, the paths taken by

light rays are extremal geodesics w.r.t.

i.e.,

Cohen Kimmel, IJCV 97

),(),( yxIyxg

CC

dssCg ))((minarg

Experiments - Color Segmentation

Goldenberg, Kimmel, Rivlin, Rudzsky,

IEEE T-IP 2001

NNCgCgdt

dC

dsCg

),(

)(

Tumor in 3D MRI

Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

NNSgHSgdt

dS

daSg

),(

)(

Segmentation in 4D

NNMgHMgdt

dM ),(

Malladi, Kimmel, Adalsteinsson,  Caselles, Sapiro, Sethian

SIAM Biomedical workshop 96

NNMgHMgdt

dM

dvMg

),(

)(

Tracking in Color Movies

Goldenberg, Kimmel, Rivlin, Rudzsky,

IEEE T-IP 2001

Tracking in Color Movies

Goldenberg, Kimmel, Rivlin, Rudzsky,

IEEE T-IP 2001

Edge Gradient Estimators

)],(),,([),( yxvyxuIyxI

Xu-Prince 98, Paragios et al. 01, Vasilevskiy-Siddiqi 01, Kimmel-Bruckstein 01

Edge Gradient Estimators

We want a curve with large points and small ‘s so:

Consider the functional

Where is a scalar function, e.g. .

C

dsINCE ),()(

)],(),,([ yxvyxuI

)(sC

)(sN

cos|)(|)(, CICIN

)(s

|| I

) ( ) (

The Classic ConnectionSuppose and we consider a closed contour for C(s).We have

and by Green’s Theorem we have

dsxyvudsNICEsC

ss

sC

)()(

),(),,(,)(

dxdyyxI

dxdyII

Ivu

dxdyuvCE

sC

sC

yyxx

sC

xy

)( nArea withi

)( nArea withi

)( nArea withi

),( of Laplacian

yields this,),(for But,

)(

)(

Therefore:

Hence curves that maximize are curves that enclose all regions where is positive!

We have that the optimal curves in this case are The Zero Crossings of the Laplacian

isn’t this familiar?

The Classic Connection

dxdyIGCEC

)*()(

),(* yxIG

C )(sC

IG *)(CE

IG *

It is pedagogically nice, but the MARR-HILDRETH edge detector is a bit too sensitive.

So we do not propose a grand return to MH but a rethinking of the functionals used in active contours in view of this.

INDEED, why should we ignore the gradient directions (estimates) and have every edge integrator controlled by the local gradient intensity alone?

The Classic Connection

Our Proposal

Consider functional of the form

These functionals yield “regularized” curves that combine the good properties of LZC’s where precise border following is needed, with the good properties of the GAC over noisy regions!

s...other term ||1

1)( where

))((),()(

ICg

dssCgdsINCECC

Implementation Details

We implement curve evolution that do gradient descent w.r.t. the functional

Here the Euler Lagrange Equations provide the explicit formulae.

For closed contours we compute the evolved curve via the Osher-Sethian “miracle” numeric level set formulation.

C

tCE

dt

dC

))((

)(CE

Closed contours

EL eq.

0,)div(),sign( NgNgVVN

GAC

GAC

LZC

LZC

Kimmel-Bruckstein IVCNZ01

gLL

Closed contours

EL eq.

0,)div(),sign( NgNgVVN

GAC

LZC

LZC+GAC

Kimmel-Bruckstein IVCNZ01

gLL

Along the curveb.c. at C(0) and C(L)

Open contours 0)div(),sign( NVVN

0,),sign(|),|1( NVTVNTVN

Kimmel-Bruckstein IVCNZ01

LL

Open contours

Kimmel-Bruckstein IVCNZ01

LL

Geometric Measures

Weighted arc-length

Weighted area

Alignment

Robustalignment e.g.

RR 2:, yx

0),(

Ngdayxg

0),())(( NNggdssCgC

0)div(, NVdsVNC

0)div(),sign(|,| NVVNdsVNC

0, NIdsINC

Variational meaning for Marr-Hildreth edge detector Kimmel-Bruckstein IVCNZ01

Geometric Measures

Minimal varianceChan-Vese, Mumford-Shah, Max-Lloyd, Threshold,…

0))((

)()(

221

\

22

21

21

NIcc

dacIdacI

cc

CC

C

0))(( 212t

2

1

21

\

\

NIccC

c

c

cc

da

Ida

da

Ida

C

C

C

C

C

C \

Geometric Measures

Robust minimal deviation

0- 21

\

21

NcIcI

dacIdacICC

C

NcIcIC

yxIc

yxIc

C

C

12t

\2

1

),(median

),(median

C

C \

Haralick/Canny-like Edge Detector

Haralick suggested as edge detector0I

IIIII yyxx

III

Laplace

Alignment Topological Homogeneity

Haralick/Canny Edge Detector

III

II I

div

III

dIdsdIdxdyIdxdyICCC

II 2

0I

Haralick

co-area

h hdxdyIC

2

x

y

I

Thus, indicates optimal alignment + topological homogeneity

0I

Closed Contours & Level Set Method

implicit representation of CThen,

Geodesic active contour level set formulation

Including weighted (by g) area minimization

RR 2:, yx 0),(:, yxyxC

dC d

VN Vdt dt

yxgdt

d,div

divd

g gdt

y

x

C(t)

C(t) level set 0

, ,x y t

x

y

Operator Splitting Schemes

Additive operator splitting (AOS) Lu et al. 90, Weickert, et al. 98 unconditionally stable for non-linear diffusion

Given the evolution write

Consider the operator Explicit scheme

, the time step, is upper bounded for stability

gt div 00 uu

ll xxl yxgA ,

k

ll

k A

2

1

1 Ι

2

1

divl

xx llgg

LOD:

Operator Splitting Schemes

Implicit scheme

inverting large bandwidth matrix First order, semi-implicit, additive operator splitting (AOS),

or locally one-dimensional (LOD) multiplicative schemes are

stable and efficient given by linear tridiagonal systems of equations

that can be solved for by Thomas algorithm

k

ll

k A 12

1

1

I

k

ll

k A

2

1

11 22

1I

1k

kl

l

k A 12

1

1

I AOS:

Operator Splitting Schemes

We used the following relation (AOS)

Locally One-Dimensional scheme (LOD)

Decoupling the axes and the implicit formulation leads to computational efficiencyThe 1st order `splitting’ idea is based on the operator expansion

)(21212

1 )1( 21

21

11

211 OAAAA kkkk

)(11 )1( 212

11

121

1 OAAAA kkkk

)(11 21 OAA

The geodesic active contour model

Where I is the image and the implicit representation of the curve

If is a distance, then , and the short time evolution is

Note that and thus can be computed once for the whole image

1

Igt div

IAl IAlI

, ,x y t

Example: Geodesic Active Contour

y

x

C(t)

x

y

Goldenberg, Kimmel, Rivlin, Rudzsky,

IEEE T-IP 2001

),(div yxIgt

Example: Geodesic Active Contour

is restricted to be a distance map:Re-initialization by Sethian’s fast marching method every iteration in O(n).

Computations are performed in a narrow band around the zero set

Multi-scale approach:process a Gaussian pyramid of the image

NNO log

tyxz ,,

y

x

C(t)

x

y

Tracking Objects in Movies

Movie volume as a spatial-temporal 3D hybrid space The AOS scheme is

Edge function derived by the Beltrami framework Sochen Kimmel Malladi 98

Contour in frame n is the initial condition for frame n+1.

k

ll

k IA 11 33

1 I

222

222

222

1

1

1

BGRBBGGRRBBGGRR

BBGGRRBGRBBGGRR

BBGGRRBBGGRRBGR

g

yyyxxx

yyyyyyyxyxyx

xxxyxyxyxxxx

ij

x

x

y

y

t

t

Experiments - Curvature Flow

20 50

Experiments - Curvature Flow CPU Time

400

600

800

1000

1200

0 1 2 3 4 5 6 7 8 9 10 11

time step

CP

U t

ime

(sec

) [1

]

0

50

100

150

200

250

300

CP

U t

ime

(sec

) [2

,3]

1. explicit (w hole image) 2. explicit (narrow ) 3. AOS (narrow )

Tracking

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Information extraction

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Holzman-Gazit, Goldshier, Kimmel 2003

=

I I I I

I H I

Thin Structures

Segmentation in 3D

Change in topology

Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

Gray Matter Segmentation

Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Coupled surfaces

EL equations

dxdydzhg

dxdydzhg

||)(

||)(

212

121

),(||

div

),(||

div

1222

222

2111

111

Fg

Fg

t

t

Gray Matter Segmentation

Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Gray Matter Segmentation

Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Classification (dogs & cats)

walk run gallop cat...

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Classification (people)

walk run run45

Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Conclusions Geometric-Variational method for segmentation

and tracking in finite dimensions based on prior knowledge (more accurately, good initial conditions).

Using the directional information for edge integration.

Geometric-variational meaning for the Marr-Hildreth and the Haralick (Canny) edge detectors, leads to ways to design improved ones.

Efficient numerical implementation for active contours.

Various medical and more general applications.www.cs.technion.ac.il/

~ron

Edge Indicator Function for Color

Beltrami framework: Color image = 2D surface in space The induced metric tensor for the image surface

Edge indicator = largest eigenvalue of the structure tensor metric. It represents the direction of maximal change in

, , , , , , ,GRx y x y y Bx x y , , , ,Rx Gy B

i i j

ji

i

ii uuuu2

222

2

1

2

1

2

11

2 2 2

2 2 2

1

1x x x x y x y x y

ijx y x y x y y y y

R R R

R R

G G GB B B

B B BG GRGg

2 2 2Gd d BR d

X

I

Y

AOS

Proof:

The whole low order splitting idea is based on the operator expansion

)(1 )()(11

)()(221

11

)(221

)()(1

)(221

)(1

)(221

)(22

2

1

)(221

2121

2

1

21

1

21

1

2

1

221

221

22

2121

221

221

221

21

221

212

21

21

21

OAAOOAA

OOAA

AA

OAA

OAA

OAA

AA

OAA

AA

OAA

AA

AA

)(21

1

21

1

2

11 2

2121

O

AAAA

)(11

1 2

OAA