3D Motion Estimation

3D model construction

3D model construction

Video Manipulation

Visual Motion

• Allows us to compute useful properties of the 3D world, with very little knowledge.

• Example: Time to collision

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

An object of heightAn object of height L L moves with moves with

constant velocity constant velocity v:v:•At time At time t=0t=0 the object is at: the object is at:

• D(0) = DD(0) = Doo

•At time At time tt it is at it is at

•D(t) = DD(t) = Doo – vt – vt

•It will crash with the camera at It will crash with the camera at

time:time:

• D(D() = D) = Doo – v – v = 0 = 0

• = D= Doo/v/v

t=0t=0tt

D(t)D(t)

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

The image of the object has size The image of the object has size

l(t):l(t):

Taking derivative wrt time:Taking derivative wrt time:

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And their ratio is:And their ratio is:

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And And time to collisiontime to collision::

Can be Can be directly directly measured measured from from imageimage

Can be found, without knowing Can be found, without knowing LL or or DDoo or or vv !! !!

Structure from Motion

rtzu ˆ

1tr Z

rrzu ωˆ1

rot Ft,

r

Zrottr uuu

u

Passive Navigation and Structure from Motion

. velocity rotational and ,

velocity ionalh translatmotion wit rigid a with moves system The

,T

zyxT

zyx ωωωttt t

fyxZYX T ,, points image ontoproject ,, points Scene rR

image in the observed ispoint scene a of ,, velocity 3D theand zyx VVVR

.0., velocity as vur

Consider a 3D point P and its image:

ff

PP

pp

ZZ zz

Using pinhole camera equation:Using pinhole camera equation:

Let things move:

ff

PPVV

ppvv

ZZ zz

The The relative velocityrelative velocity of P wrt camera: of P wrt camera:

TranslatioTranslation velocityn velocity Rotation Rotation

angular angular velocityvelocity

3D Relative Velocity:

ff

PPVV

ppvv

ZZ zz

The relative velocity of P wrt camera:The relative velocity of P wrt camera:

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Taking derivative wrt time:Taking derivative wrt time:

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

TranslationaTranslational componentl component

Scaling ambiguity(t and Z can onlybe derived up to a scale Factor)

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Rotational Rotational componentcomponent

NOTE: The rotational component is independent of depth Z !NOTE: The rotational component is independent of depth Z !

Translational flow field

(FOC).n contractio of focusor

(FOE)expansion of focus theis ,, p where 00 o

f

t

tf

t

tyx

z

y

z

x

Rotational flow field

(AOR). plane image thepierces axisrotation theepoint wher theis ,

ff

z

y

z

x

Pure Translation

What if tWhat if tzz 0 ? 0 ?

All motion field vectors are parallel to each other and All motion field vectors are parallel to each other and inversely proportional to depth !inversely proportional to depth !

Pure Translation: Properties of the MF

• If tz 0 the MF is RADIAL with all vectors pointing towards (or away from) a single point po. If tz = 0 the MF is PARALLEL.

• The length of the MF vectors is inversely proportional to depth Z. If tz 0 it is also directly proportional to the distance between p and po.

Pure Translation: Properties of the MF

• po is the vanishing point of the direction of translation.

• po is the intersection of the ray parallel to the translation vector and the image plane.

Special Case: Moving Plane

ff

PPVV

ppvv

ZZ zz

nnPlanar surfaces are Planar surfaces are common in man-made common in man-made environmentsenvironments

Question: How does the MF of a moving plane look like?Question: How does the MF of a moving plane look like?

Special Case: Moving PlanePoints on the plane must satisfy Points on the plane must satisfy the equation describing the plane. the equation describing the plane.

Let Let •nn be the unit vector normal to the be the unit vector normal to the plane.plane.•dd be the distance from the plane to be the distance from the plane to the origin.the origin.•NOTE:NOTE: If the plane is moving wrt If the plane is moving wrt camera, camera, nn and and dd are functions of timeare functions of time..

Then:Then:

dd PP

ZZ

nn

OOYY

XX

wherewhere

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

Using the pinhole Using the pinhole projection projection equation:equation:pp

LetLet be the image of Pbe the image of P

Using the plane equation:Using the plane equation:

Solving for Z:Solving for Z:

Special Case: Moving Plane

And Plug in:And Plug in:

dd PP

ZZ

nn

OOYY

XX

pp

Now consider the MF equations:Now consider the MF equations:

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

pp

The MF equations become:The MF equations become:

wherewhere

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

pp

MF equations:MF equations:

Q: What is the significance of this?Q: What is the significance of this?

A: The MF vectors are given by low order (second) A: The MF vectors are given by low order (second) polynomials. polynomials.

•Their coeffs. aTheir coeffs. a11 to a to a8 8 (only 8 !) are functions of n, d, t and (only 8 !) are functions of n, d, t and ..•The same coeffs. can be obtained with a different plane The same coeffs. can be obtained with a different plane and relative velocity.and relative velocity.

Moving Plane: Properties of the MF

• The MF of a planar surface is at any time a quadratic function in the image coordinates.

• A plane nTP=d moving with velocity V=-t-£ P has the same MF than a plane with normal n’=t/|t| , distance d and moving with velocity

V=|t|n –( + n£ t/d)£ P

Classical Structure from Motion

• Established approach is the epipolar minimization: The “derotated flow” should be parallel to the translational flow.

uu

-urot

-urot

EE

utr

t

utr

The Translational Case(a least squares formulation)

Substitute back

min

Substitute

)),(( ||),(||

min

22

2

22

Zv

Zu

yttxtt

dydxyxfyxf

dydxZ

tytv

Z

txtu

zyzx

yzxz

min22

2

dydxvu

)minimized. be would

for which oflength the(Find . respect to with Minimize :1 Step tr dZ u

vu

ZZZ

vZZ

u

22

220

2 minimize We d

Step 2: Differentiate with respect to tx, ty, tz, set expression to zero.

222

Let

vuvuK

0III

0II

0I

dydxKxtyt

dydxKxtt

dydxKytt

yx

zx

zy

0IIIIII equationsdependent linearly 3 zyx ttt

Equations are nonlinear in tx, ty, tz

Using a different norm

dydxyxfyxf

dydxZ

tytv

Z

txtu yzxz

)(),(||),(||

min )(

222,

22

22

vu

ZZZ

vZZ

au

22

220

First, differentiate integrand with respect to Z and set to zero

),,(

min2

zyx tttg

dydxvu

zxzyyxzyxzyx tfttettdtctbtattttg 222),,( 222

where

, )(

, )(

,

, )(

,

,

2

2

dydxyuxvvf

dydxyuxvue

dydxuvd

dydxyuxvc

dydxub

dydxva

Differentiate g( tx, ty, tz ) with respect to tx, ty, tz and set to zero

cef

ebd

fda

G

||t||Gt 1 with 0

Solution is singular vector corresponding to smallest singular value

The Rotational Case min2

rot2

rot vvuu

01

012

2

xxyyv

yxxyu

zyx

zyx

v

u

xxyfy

yfxxy

z

y

x

2

2

In matrix form

u

u

TT AAA

A1

Let f=1

The General Case

Minimization of epipolar distance

or, in vector notation

min2

tr

tr

rot

rot

dydxu

v

vv

uu

min2 rrrrt d

Motion ParallaxThe difference in motion between two very close points does not depend on rotation.Can be used at depth discontinuities to obtain the direction of translation.

FOE

Motion Parallax

0

0

21,2,1

21,2,1

21

2 1

)11

)((

)11

)((

, ,

have we),(p andp pointsAt

),(

xx

yy

u

v

ZZyyvvv

ZZxxuuu

yx

vu

otrtrtr

otrtrtr

rotrot

rottr

uu

uuu

Vectors perpendicular to translational component

)|| ,||() ,(

00

00

xxyyxxyy

tr u

rotrotxxyytr vxxuyy )()( 00)|| ,||(1

00

uu

Vector component perpendicular to translational componentis only due to rotation rotation can be estimated from it.

Motion Estimation Techniques• Prazdny (1981), Burger Bhanu (1990), Nelson Aloimonos (1988), Heeger

Jepson (1992):

Decomposition of flow field into translational and rotational components. Translational flow field is radial (all vectors are emanating from (or pouring into) one point), rotational flow field is quadratic in image coordinates.

Either search in the space of rotations: remainig flow field

should be translational. Translational flow field

is evaluated by minimizing deviation from radial field:

.

Or search in the space of directions of translation: Vectors

perpendicular to translation are due to rotation only

0),(),( 00 yyxxuv

Motion Estimation Techniques

• Longuet-Higgins Prazdny (1980), Waxman (1987): Parametric model for local surface patches (planes or quadrics) solve locally for motion parameters and structure, because flow is linear in the motion parameters

(quadratic or higher order in the image coordinates)