Lucas-Kanade Optical FlowComputer Vision 16-385

Carnegie Mellon University (Kris Kitani)

spatial derivative optical flow temporal derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y u =dx

dt

v =dy

dtIt =

@I

@t

How can we use the brightness constancy equation to estimate the optical flow?

Ix

u+ Iy

v + It

= 0

known

unknown

We need at least ____ equations to solve for 2 unknowns.

Ix

u+ Iy

v + It

= 0

known

unknown

Where do we get more equations (constraints)?

Ix

u+ Iy

v + It

= 0

Where do we get more equations (constraints)?

Assume that the surrounding patch (say 5x5) has constant flow

Flow is locally smooth

Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

Assumptions:

Flow is locally smooth

Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

Assumptions:

Ix

(p1)u+ Iy

(p1)v = �It

(p1)

Ix

(p2)u+ Iy

(p2)v = �It

(p2)

...

Ix

(p25)u+ Iy

(p25)v = �It

(p25)

Flow is locally smooth

Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations2

6664

Ix

(p1) Iy

(p1)Ix

(p2) Iy

(p2)...

...Ix

(p25) Iy

(p25)

3

7775

uv

�= �

2

6664

It

(p1)It

(p2)...

It

(p25)

3

7775

Assumptions:

Matrix form

Flow is locally smooth

Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations2

6664

Ix

(p1) Iy

(p1)Ix

(p2) Iy

(p2)...

...Ix

(p25) Iy

(p25)

3

7775

uv

�= �

2

6664

It

(p1)It

(p2)...

It

(p25)

3

7775

Ax

b25⇥ 2 2⇥ 1 25⇥ 1

How many equations? How many unknowns? How do we solve this?

Assumptions:

Least squares approximation

x = argminx

||Ax� b||2A

>Ax = A

>b

is equivalent to solving

To obtain the least squares solution solve:

Least squares approximation

x = argminx

||Ax� b||2A

>Ax = A

>b

is equivalent to solving

2

4

Pp2P

Ix

Ix

Pp2P

Ix

Iy

Pp2P

Iy

Ix

Pp2P

Iy

Iy

3

5

uv

�= �

2

4

Pp2P

Ix

It

Pp2P

Iy

It

3

5

where the summation is over each pixel p in patch P

A>A A>bx

x = (A>A)�1

A

>b

To obtain the least squares solution solve:

Least squares approximation

x = argminx

||Ax� b||2A

>Ax = A

>b

is equivalent to solving

2

4

Pp2P

Ix

Ix

Pp2P

Ix

Iy

Pp2P

Iy

Ix

Pp2P

Iy

Iy

3

5

uv

�= �

2

4

Pp2P

Ix

It

Pp2P

Iy

It

3

5

where the summation is over each pixel p in patch P

A>A A>bx

Sometimes called ‘Lucas-Kanade Optical Flow’(special case of the LK method with a translational warp model)

A

>Ax = A

>b

When is this solvable?

ATA should be invertible

ATA should not be too small

ATA should be well conditioned

λ1 and λ2 should not be too small

λ1/λ2 should not be too large (λ1=larger eigenvalue)

2

4

Pp2P

Ix

Ix

Pp2P

Ix

Iy

Pp2P

Iy

Ix

Pp2P

Iy

Iy

3

5

Where have you seen this before?

A>A =

2

4

Pp2P

Ix

Ix

Pp2P

Ix

Iy

Pp2P

Iy

Ix

Pp2P

Iy

Iy

3

5

Where have you seen this before?

Harris Corner Detector!

Implications• Corners are when λ1, λ2 are big; this is also when

Lucas-Kanade optical flow works best

• Corners are regions with two different directions of gradient (at least)

• Corners are good places to compute flow!

What happens when you have no ‘corners’?

You want to compute optical flow. What happens if the image patch contains only a line?

������������

In which direction is the line moving?

small visible image patch

������������

In which direction is the line moving?

small visible image patch

������������

������������

������������

������������

Want patches with different gradients to the avoid aperture problem

Want patches with different gradients to the avoid aperture problem

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 5

- - - -- 1 1 1- 2 2 2- 3 3 3- 4 4 4

y

x

y

x

optical flow: (1,1)

H(x,y) = y I(x,y)

It(3, 3) = I(3, 3)�H(3, 3) = �1

Ix

u+ Iy

v + It

= 0

Iy(3, 3) = 1

Ix

(3, 3) = 0v = 1

We recover the v of the optical flow but not the u. This is the aperture problem.

Compute gradients Solution:

Applied to optical flow: Assume that the flow over a small image patch is constant

Note: The Lucas-Kanade method is a general

framework for image matching (not only optical flow)

Horn-Shunck!Optical Flow

(1981)

Lucas-Kanade!Optical Flow!

(1981)

‘constant’ flow‘smooth’ flow

Constraint: Flow of neighboring pixels

should be smooth

Contraint: Flow over a small patch

should be the same

Horn-Schunck16-385 Computer Vision

Carnegie Mellon University (Kris Kitani)

Horn-Schunck !Optical Flow (1981)

Lucas-Kanade!Optical Flow (1981)

‘constant’ flow!(flow is constant for all pixels)

‘smooth’ flow!(flow can vary from pixel to pixel)

brightness constancymethod of differences

global method (dense)

local method (sparse)

small motion

Optical Flow (Problem definition)

Estimate the motion (flow) between these

two consecutive images

I(x, y, t)I(x, y, t0)

How is this different from estimating a 2D transform?

Key Assumptions (unique to optical flow)

Color Constancy!(Brightness constancy for intensity images)

Small Motion!(pixels only move a little bit)

Implication: allows for pixel to pixel comparison (not image features)

Implication: linearization of the brightness constancy constraint

Approach

I(x, y, t)I(x, y, t0)

Look for nearby pixels with the same color(small motion) (color constancy)

Brightness constancy

I(x, y, t)

(x, y)

(x+ u�t, y + v�t)

(x, y)

(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

I(x, y, t+ �t)

For a really small time step…

Optical Flow Constraint equationI(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

I(x, y, t) +@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

Ix

u+ Iy

v + It

= 0

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

Ix

u+ Iy

v + It

= 0

u

v

Solution lies on a straight line

The solution cannot be determined uniquely with a single constraint (a single pixel)

Where can we get an additional constraint?

Horn-Schunck !Optical Flow (1981)

Lucas-Kanade!Optical Flow (1981)

‘constant’ flow!(flow is constant for all pixels)

‘smooth’ flow!(flow can vary from pixel to pixel)

brightness constancymethod of differences

global method local method

small motion

Smoothness

most objects in the world are rigid or deform elastically !

moving together coherently

we expect optical flow fields to be smooth

Key idea (of Horn-Schunck optical flow)

Enforce brightness constancy

Enforce smooth flow field

to compute optical flow

Enforce brightness constancy

Ix

u+ Iy

v + It

= 0

minu,v

Ix

uij

+ Iy

vij

+ It

�2For every pixel,

Enforce brightness constancy

Ix

u+ Iy

v + It

= 0

minu,v

Ix

uij

+ Iy

vij

+ It

�2For every pixel,

lazy notation for Ix

(i, j)

Ix

Iy

changes these

displacement of object

u v changes these

It

+ =

Find the optical flow such that it satisfies:

Enforce smooth flow field

uij ui+1,jui�1,j

ui,j+1

ui,j�1

minu

(ui,j � ui+1,j)2

u-component of flow

Which flow field is more likely?

X

ij

(uij � ui+1,j)2

X

ij

(uij � ui+1,j)2?

Which flow field is more likely?

more likelyless likely

optical flow objective function

Optical flow constraint

Smoothness constraint

(uij � ui+1,j)

ij

i, j + 1

i+ 1, j ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

i� 1, j

i, j � 1

(uij � ui,j+1)

ij

i, j + 1

i+ 1, j ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

i� 1, j

i, j � 1

(vij � vi+1,j) (vij � vi,j+1)

Ed

(i, j) =

Ix

uij

+ Iy

vij

+ It

�2

Es(i, j) =1

4

(uij � ui+1,j)

2 + (uij � ui,j+1)2 + (vij � vi+1,j)

2 + (vij � vi,j+1)2

�

Horn-Schunck optical flow

minu, v

X

ij

⇢Ed(i, j) + �Es(i, j)

�

Ed

(i, j) =

Ix

uij

+ Iy

vij

+ It

�2

where

and

Es(i, j) =1

4

(uij � ui+1,j)

2 + (uij � ui,j+1)2 + (vij � vi+1,j)

2 + (vij � vi,j+1)2

�

How do we solve this minimization problem?

minu, v

X

ij

⇢Ed(i, j) + �Es(i, j)

�

How do we solve this minimization problem?

minu, v

X

ij

⇢Ed(i, j) + �Es(i, j)

�

Compute partial derivative, derive update equations (gradient decent)

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

it’s not so bad…

how many terms depend on k and l?

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

it’s not so bad…

how many terms depend on k and l?

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

(uij � ui+1,j)

ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

uij =1

4

⇢ui+1,j + ui�1,j + ui,j+1 + ui,j�1

�local average

(u2ij � 2uijui+1,j + u2

i+1,j) (u2ij � 2uijui,j+1 + u2

i,j+1)

(variable will appear four times in sum)

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

Ax = b

how do you solve this?

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

this is a linear system

ok, take a step back, why are we doing all this math?

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

We are solving for the optical flow (u,v) given two constraints

smoothness brightness constancy

We need the math to minimize this (back to the math)

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

Ax = b

how do you solve this?

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

Partial derivatives of Horn-Schunck objective function E:

x = A�1b =

adjA

detAb

Recall

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

x = A�1b =

adjA

detAb

Recall

(determinant of A)

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

(determinant of A)

Same as the linear system:{1 + �(I2

x

+ I2y

)}ukl

= (1 + �I2x

)ukl

� �Ix

Iy

vkl

� �Ix

It

{1 + �(I2x

+ I2y

)}vkl

= (1 + �I2y

)vkl

� �Ix

Iy

ukl

� �Iy

It

(determinant of A)

(determinant of A)

Rearrange to get update equations:

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

1 + �(I2x

+ I2y

)Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

1 + �(I2x

+ I2y

)Iy

new value

old average

{1 + �(I2x

+ I2y

)}ukl

= (1 + �I2x

)ukl

� �Ix

Iy

vkl

� �Ix

It

{1 + �(I2x

+ I2y

)}vkl

= (1 + �I2y

)vkl

� �Ix

Iy

ukl

� �Iy

It

check for missing lambda

ok, take a step back, why did we do all this math?

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

We are solving for the optical flow (u,v) given two constraints

smoothness brightness constancy

We needed the math to minimize this (now to the algorithm)

Horn-Schunck Optical Flow Algorithm

1. Precompute image gradients!

2. Precompute temporal gradients!

3. Initialize flow field!

4. While not converged Compute flow field updates for each pixel:

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

1 + �(I2x

+ I2y

)Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

1 + �(I2x

+ I2y

)Iy

Ix

Iy

It

u = 0

v = 0