Optical Flow: Horn-Schunck16-385 Computer Vision (Kris Kitani)

Carnegie Mellon University

Horn-Schunck Optical Flow (1981)

Lucas-KanadeOptical Flow (1981)

‘constant’ flow(flow is constant for all pixels)

‘smooth’ flow(flow can vary from pixel to pixel)

brightness constancymethod of differences

global method (dense)

local method (sparse)

small motion

Optical Flow (Problem definition)

Estimate the motion (flow) between these

two consecutive images

I(x, y, t)I(x, y, t0)

How is this different from estimating a 2D transform?

Key Assumptions (unique to optical flow)

Color Constancy(Brightness constancy for intensity images)

Small Motion(pixels only move a little bit)

Implication: allows for pixel to pixel comparison (not image features)

Implication: linearization of the brightness constancy constraint

Approach

I(x, y, t)I(x, y, t0)

Look for nearby pixels with the same color(small motion) (color constancy)

Brightness constancy

I(x, y, t)

(x, y)

(x+ u�t, y + v�t)

(x, y)

(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

I(x, y, t+ �t)

For a really small time step…

Optical Flow Constraint equationI(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

I(x, y, t) +@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

Ix

u+ Iy

v + It

= 0

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

Ix

u+ Iy

v + It

= 0

u

v

Solution lies on a straight line

The solution cannot be determined uniquely with a single constraint (a single pixel)

Where can we get an additional constraint?

Horn-Schunck Optical Flow (1981)

Lucas-KanadeOptical Flow (1981)

‘constant’ flow(flow is constant for all pixels)

‘smooth’ flow(flow is smooth from pixel to pixel)

brightness constancymethod of differences

global method local method

small motion

Smoothness

most objects in the world are rigid or deform elastically

moving together coherently

we expect optical flow fields to be smooth

Key idea (of Horn-Schunck optical flow)

Enforce brightness constancy

Enforce smooth flow field

to compute optical flow

Key idea (of Horn-Schunck optical flow)

Enforce brightness constancy

Enforce smooth flow field

to compute optical flow

Enforce brightness constancy

Ix

u+ Iy

v + It

= 0

minu,v

Ix

uij

+ Iy

vij

+ It

�2For every pixel,

Ix

u+ Iy

v + It

= 0

minu,v

Ix

uij

+ Iy

vij

+ It

�2For every pixel,

lazy notation for Ix

(i, j)

Enforce brightness constancy

Ix

Iy

changes these

displacement of object

u v changes these

It

+ =

Find the optical flow such that it satisfies:

Key idea (of Horn-Schunck optical flow)

Enforce brightness constancy

Enforce smooth flow field

to compute optical flow

Enforce smooth flow field

uij ui+1,jui�1,j

ui,j+1

ui,j�1

minu

(ui,j � ui+1,j)2

u-component of flow

Which flow field optimizes the objective?

X

ij

(uij � ui+1,j)2

X

ij

(uij � ui+1,j)2?

minu

(ui,j � ui+1,j)2

smallbig

Which flow field optimizes the objective? minu

(ui,j � ui+1,j)2

Key idea (of Horn-Schunck optical flow)

Enforce brightness constancy

Enforce smooth flow field

to compute optical flow

bringing it all together…

Horn-Schunck optical flow

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�smoothness brightness constancy

weight

HS optical flow objective function

Brightness constancy

Smoothness

(uij � ui+1,j)

ij

i, j + 1

i+ 1, j ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

i� 1, j

i, j � 1

(uij � ui,j+1)

ij

i, j + 1

i+ 1, j ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

i� 1, j

i, j � 1

(vij � vi+1,j) (vij � vi,j+1)

Ed

(i, j) =

Ix

uij

+ Iy

vij

+ It

�2

Es(i, j) =1

4

(uij � ui+1,j)

2 + (uij � ui,j+1)2 + (vij � vi+1,j)

2 + (vij � vi,j+1)2

�

why not all four neighbors?

How do we solve this minimization problem?

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�

Compute partial derivative, derive update equations (gradient decent!)

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�

How do we solve this minimization problem?

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

smoothness term brightness constancy

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

it’s not so bad…

how many u terms depend on k and l?

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

it’s not so bad…

how many u terms depend on k and l?ONE from brightness constancyFOUR from smoothness

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

it’s not so bad…

how many u terms depend on k and l?ONE from brightness constancyFOUR from smoothness

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

(uij � ui+1,j)

ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

(u2ij � 2uijui+1,j + u2

i+1,j) (u2ij � 2uijui,j+1 + u2

i,j+1)

(variable will appear four times in sum)

Compute the partial derivatives of this huge sum!

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

(uij � ui+1,j)

ij

i, j + 1

i+ 1, ji� 1, j

i, j � 1

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

uij =1

4

⇢ui+1,j + ui�1,j + ui,j+1 + ui,j�1

�short hand for local average

(u2ij � 2uijui+1,j + u2

i+1,j) (u2ij � 2uijui,j+1 + u2

i,j+1)

(variable will appear four times in sum)

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

Ax = b

how do you solve this?

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

this is a linear system

ok, take a step back, why are we doing all this math?

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

We are solving for the optical flow (u,v) given two constraints

smoothness brightness constancy

We need the math to minimize this (back to the math)

@E

@vkl

= 2(vkl

� vkl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Iy

@E

@ukl

= 2(ukl

� ukl

) + 2�(Ix

ukl

+ Iy

vkl

+ It

)Ix

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

Ax = b

how do you solve this?

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

Partial derivatives of Horn-Schunck objective function E:

x = A�1b =

adjA

detAb

Recall

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

x = A�1b =

adjA

detAb

Recall

(det A)

�Ix

Iy

ukl

+ (1 + �I2y

)vkl

= vkl

� �Iy

It

(1 + �I2x

)ukl

+ �Ix

Iy

vkl

= ukl

� �Ix

It

Same as the linear system:{1 + �(I2

x

+ I2y

)}ukl

= (1 + �I2x

)ukl

� �Ix

Iy

vkl

� �Ix

It

{1 + �(I2x

+ I2y

)}vkl

= (1 + �I2y

)vkl

� �Ix

Iy

ukl

� �Iy

It

(det A)

Rearrange to get update equations:

new value

old average

{1 + �(I2x

+ I2y

)}ukl

= (1 + �I2x

)ukl

� �Ix

Iy

vkl

� �Ix

It

{1 + �(I2x

+ I2y

)}vkl

= (1 + �I2y

)vkl

� �Ix

Iy

ukl

� �Iy

It

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Iy

new value

old average

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Iy

When lambda is small (lambda inverse is big)…

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�Recall:

new value

old average

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Iy

When lambda is small (lambda inverse is big)…goes to

zero

goes to zero

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�Recall:

new value

old average

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Iy

When lambda is small (lambda inverse is big)…goes to

zero

goes to zero

…we only care about smoothness.

minu,v

X

i,j

⇢Es(i, j) + �Ed(i, j)

�Recall:

ok, take a step back, why did we do all this math?

X

ij

(1

4

(u

ij

� ui+1,j)

2 + (uij

� ui,j+1)

2 + (vij

� vi+1,j)

2 + (vij

� vi,j+1)

2

�+ �

Ix

uij

+ Iy

vij

+ It

�2)

We are solving for the optical flow (u,v) given two constraints

smoothness brightness constancy

We needed the math to minimize this (now to the algorithm)

Horn-Schunck Optical Flow Algorithm

1. Precompute image gradients

2. Precompute temporal gradients

3. Initialize flow field

4. While not converged Compute flow field updates for each pixel:

Ix

Iy

It

u = 0

v = 0

ukl

= ukl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Ix

vkl

= vkl

� Ix

ukl

+ Iy

vkl

+ It

��1 + I2x

+ I2y

Iy

Just 8 lines of code!