Pyramidal Implementation of Lucas Kanade Feature Tracker

Jia HuangXiaoyan Liu

Han XinYizhen Tan

Abstract

IntroductionTracking algorithm

Lucas-Kanade algorithm Iterative implementation

Tracking features analysis Feature lost Feature selection

Objective For a given point u in image A, find its

corresponding location v = u + d in image B.

Image A Image B

d

Residual function and Window size

2( ) ( , ) ( ( , ) ( , ))y yx x

x x x x

u wu w

x y x yx u w y u w

d d d A x y B x d y d

To find the location Minimize residual function:

: Integration window size

Small integration window Higher accuracy

Larger integration window Higher robustness

Nature tradeoff:

,x yw w

Pyramid Implementation of LK algorithm Calculate a set of pyramid representations of original image Apply traditional tracking algorithm for each level Results of current iteration is propagated to next iteration Key point: the same window size is used for each level

Top View Side View

Lucas-Kanade algorithm(1)

At the level L, we define images A and B: ( , ) [ 1, 1] [ 1, 1]x x x x y y y yx y p w p w p w p w

( , ) [ , ] [ , ]x x x x y y y yx y p w p w p w p w

( , ) ( , )LA x y I x y

( , ) ( , )L L Lx yB x y J x g y g

2( ) ( , ) ( ( , ) ( , ))y yx x

x x y y

p wp w

x y x yx p w y p w

v v v A x y B x v y v

Lucas-Kanade algorithm(2) At the optimum, the first derivative of

After first order Taylor expansion

Components in the equation above

( )| [0 0]

optv v

v

v

，

( )( ) ( ( , ) ( , ) )

y yx x

x x y y

p wp w

x p w y p w

v B B B Bv A x y B x y v

v x y x y

( , ) ( , ) ( , )I x y A x y B x y T

x

y

I B BI

I x y

2

2

1 ( )

2

y yx x

x x y y

T p wp wxx x y

x p w y p w yx y y

I II I Ivv

I II I Iv

Lucas-Kanade algorithm(3) Two derivative images are expressed:

With these notation, we can get:

The optimum optical flow vector is ( , ) ( 1, ) ( 1. )( , )

2x

A x y A x y A x yI x y

x

( , ) ( , 1) ( . 1)

( , )2y

A x y A x y A x yI x y

y

1optv G b

bG

Pyramidal diagram

Inner loop: K-level K initialized to 1, assume that the previous

computations from iterations 1,2,...,k-1 provide an initial guess

The new translated image according to

Iterative scheme of LK algorithm(1)

. , 3me g L

0mL

1 1 1[ ]k k k Tx yv v v

1kv

( , ) [ , ] [ , ]x x x x y y y yx y p w p w p w p w 1 1( , ) ( , )k k

k x yB x y B x v y v

Iterative scheme of LK algorithm(2) The goal: to compute the residual pixel motion vector

, that minimizes the error function

Image mismatch vector , where the image difference delta I k defined as:

New pixel displacement guess is computed for the next iteration step k+1:

[ ]k k kx y

2( ) ( , ) ( ( , ) ( , ))y yx x

x x y y

p wp wk k k kx y k x y

x p w y p w

A x y B x y

( , ) ( , )

( , ) ( , )

y yx x

x x y y

p wp wk x

k

x p w y p w k y

I x y I x yb

I x y I x y

( , ) ( , ) ( , )k kI x y A x y B x y

kbthk

1k k kv v

Iterative scheme of LK algorithm(3)

On average, 5 iterations are enough At the 1st iteration (k=1), the initial guess is set

to zero

The final solution for the optical flow vector is

Outer loop: L-levelThe vector d is propagated to the next level

L-1 and overall procedure is repeated L-1, L-2, …, 0

1

KL K k

k

v d v

0 [0 0]T

Declaring a Feature Lost

Several cases of lost feature the point falls outside of the image image patch around the tracked point varies

too much between image A and image B too large displacement

How to solve it combine a traditional tracking approach with an affine image matching

Feature Lost Example(1)

Image A Image B

Feature Lost Example(2)

Image A Image B

Feature Selection

Intuitive To select the point u on image A good to track.

Process steps: Compute the G matrix and λm

Call λmax the maximum value of λm Retain the pixels that have a λm value larger than a percentage of λmax Retain the local max. pixels Keep the subset of those pixels so that the minimum distance between pixels is larger than a threshold

Example of LK Feature Tracking

Image A Image B

More Examples

Image BImage A

Summary

Lucas-Kanade Feature Tracker is one of the most popular versions of two-frame differential methods for motion estimation

Iterative implementation of the Lucas-Kanade optical flow computation provides sufficient local tracking accuracy.

Thanks for your attention

Any question?