Optical Flow

Optical Flow

• Distribution of apparent velocities of movement of brightness pattern in an image

Optical Flow

• Perspective projection (pinhole camera)

• Instantaneous 2D velocity

4

Definitions

The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ]

The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ]

+ =

frame #1 frame #2flow field

5

Ambiguity of optical flow

Frame 1

flow (1): true motionflow (2)

Determining Optical Flow

Berthold K.P. Horn and Brian G. RhunckArtificial Intelligence Laboratory, MIT.

Optical Flow• Optical flow field: Estimate the 2D motion field, which are

the 2D velocities for all visible points

• Two key problems– Determine what image property to track– Determine how to track it

8

Brightness Constancy

0),,()1,,( tyxItvyuxI

frame t

frame t+1

)1,,( tvyux

),,( tyx

v

u

Brightness Constancy

• Track points of constant brightness, assuming that surface radiance is constant over time

• Brightness constancy is often violated by Mother Nature, so the resulting optical flow field is sometimes a very poor approximation to the 2D motion field

1 2( , , 1) ( , , )I x y t I x u y u t

Optical Flow

• The optical flow cannot be computed at a point independently of neighboring points without introducing additional constraints

Brightness Constancy Constraint

• The brightness of a particular point in the pattern is constant

• Cannot determine the flow velocity locally without additional constraints

d0

d

E

t

d d0

d d

E x E y E

x t y t t

d

d

xu

t

d

d

yv

t 0x y tE u E v E

u

v

( , )x yE EtE

( , )u vConstraint line

Brightness Constancy Constraint

• Normal Velocity• No constraint when the gradient magnitude is

zero

13

Optical Flow Assumptions:

* Slide from Michael Black, CS143 2003

14

Optical Flow Assumptions:

* Slide from Michael Black, CS143 2003

Smoothness Constraint

• Constraint Assumption– Neighboring points on the objects have similar

velocities and – The velocity field of the brightness patterns in the

image varies smoothly almost everywhere– Discontinuities in flow can be expected where one

object occludes another

• Algorithms based on smoothness constraint are likely to have difficulties with occluding edges

Smoothness Constraint

• Minimize the square of the gradient magnitude of the optical flow velocity

• Sum of the squares of Laplacians of x- and y- components of the flow

22u u

x y

22v v

x y

2 22

2 2

u uu

x y

2 22

2 2

v vv

x y

Estimate the Partial Derivatives

Estimate the Laplacian of Flow Velocities

Minimization

• Minimize the sum of errors in rate of change of image brightness

• And measure of departure from smoothness

• Total error to be minimized

Minimization

• Good (u,v) occurs at

Difference of Flow from Local Average

• (u,v) lies in the direction towards the constraint line along a line that perpendicular to it

• a^2 plays significant role when Ex, Ey is small: roughly equal to the expected noise in the estimate of

0x y tE u E v E

Iterative Solution

24

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf

0)(

txt t

I

t

x

x

I

Ix v It

x

t

I

Iv

{0

)(

t

xfBecause no change in brightness with time

25

v

?

Tracking in the 1D case:

x

),( txI )1,( txI

p

26

v

xI

Spatial derivative

Temporal derivativetI

Tracking in the 1D case:

x

),( txI )1,( txI

p

tx x

II

px

t t

II

x

t

I

Iv

Assumptions:• Brightness constancy• Small motion

27

Tracking in the 1D case:

x

),( txI )1,( txI

p

xI

tI

Temporal derivative at 2nd iteration

Iterating helps refining the velocity vector

Can keep the same estimate for spatial derivative

x

tprevious I

Ivv

Converges in about 5 iterations

28

Algorithm for 1D tracking:

x

tI

Ivv

Compute local image derivative at p: Initialize velocity vector: Repeat until convergence:

Compensate for current velocity vector: Compute temporal derivative:Update velocity vector:

For all pixel of interest p:

xI

0v

)1,()1,(' tvxItxI

),()1,(' tpItpII t

Need access to neighborhood pixels round p to compute Need access to the second image patch, for velocity compensation:

The pixel data to be accessed in next image depends on current velocity estimate (bad?)

Compensation stage requires a bilinear interpolation (because v is not integer) The image derivative needs to be kept in memory throughout the iteration

process

Requirements:

xI

xI

Iterative Solution

• Very costly to solve these equations simultaneously • Iterative algorithms, such as Gauss-Seidel method

It is interesting to note that the new estimates at a particular point do not depend directly on the previous estimate at the same point

Fill in Uniform Region

• Uniform region: brightness gradient is zero

• No local information to constrain the vecocity• Fill in uniform region– Given the values on region boudnary– Solve the Laplace equation under Dirichlet

boundary condition

0x y tE u E v E

2 *min | |f

f with f f

Tightness of Constraint

• In general, the solution is most accurately determined in the region where the brightness is not too small and varies in direction from point to point– Constraint information is available in a relative

small neighborhood

Choice of Iterative Scheme

• How to interlace the iterations with the time step? – Get stable values before going to next frame– Only one iteration per time-step gives good initial

guess • Ability to deal with more images per unit time• Better estimation in uniform regions because of the

noise in measurements of images will be independent and tend to cancel out

Experiment

• Very low resolution video, say, 32*32

• A rotating ball with smoothly varying patterns, but no shading

Results I

• Simple linear translation of the entire pattern• Estimate between two images• Showing the velocity • Few changes occur after 32 iterations when

errors ~ 10%

Results I

1 time step 4 time steps

Results I

16 time steps 64 time steps

Results II

• Only one iteration per time-step• More rapid • Few changes after 16 iteration when errors ~

7%

Results II

1 time steps 4 time steps

Results II

16 time steps 64 time steps

Results III

• Simple rotation and contraction of the brightness pattern

Result III

initial 32 time steps

Result IV

• Rigid body motion– Laplacian for one of the velocity components

becomes infinite on the occluding bound– Reasonable results are still obtained with finite

velocities

Results IV

Rotating cylinder Rotating sphere

Results IV

Exact flow pattern for a rotating cylinder and rotating sphere

Summary

• A method for computing optical flow from a sequence of images

• Two components of flow velocity• Additional constraint: smoothness • Iterative method

46

Horn&Schunck

Optical Flow Research: Timeline

Lucas&Kanade

1981 1992 1998 now

Benchmark:Barron et.al.

Benchmark:Galvin et.al.

many methods

more methods

A slow and not very consistent improvement in results,

but a lot of useful ingredients were developed

Seminal papers

http://people.csail.mit.edu/celiu/ECCV2008/z`z`