Lecture 22: Camera Motionrtc12/CSE486/lecture22.pdf · Lecture 22: Camera Motion Readings: ......
Transcript of Lecture 22: Camera Motionrtc12/CSE486/lecture22.pdf · Lecture 22: Camera Motion Readings: ......
CSE486, Penn StateRobert Collins
Lecture 22:
Camera Motion
Readings: T&V Sec 8.1 and 8.2
CSE486, Penn StateRobert Collins
Moving Camera
yaw
pitch
roll
Tx
TyTz
From one time to the next, the camera undergoesrotation (roll, pitch, yaw) and translation (tx,ty,tz)
t t+1
t+2t-1
t-2t-3
t-4
Camera takes a sequence of images (frames)indexed by time t
CSE486, Penn StateRobert Collins
Motion (Displacement) Fields
Time t
Time t+1
Displacement field
CSE486, Penn StateRobert Collins
Motion Field vs Optic Flow
Motion Field: projection of 3D relative velocity vectors onto the 2D image plane.
Optic Flow: observed 2D displacements of brightness patterns in the image.
Motion field is what we want to know.Optic flow is what we can estimate.
CSE486, Penn StateRobert Collins
Motion Field vs Optic FlowSometimes optic flow is a good approximationto the unknown motion flow.
optic flow field
We can then infer relative motion between the camera and objects in the world.
CSE486, Penn StateRobert Collins
Warning: Optic Flow ≠ Motion Field
Consider a moving light source:
MF = 0 since the points on the scene are not movingOF ≠ 0 since there is a moving pattern in the images
CSE486, Penn StateRobert Collins
Motion Field
We are going to derive an equation relating 3D scene structure and velocity to the 2D motion flow field.
CSE486, Penn StateRobert Collins
Motion Field
What is a Field anyways?
Image a vector at each point in space. This is avector field. In 3D space, we will look at the fieldof 3D velocity vectors induced by camera motion.
In 2D, we will be looking at the projections of those3D vectors in the image. There will be a 2D flow vector at each point in the image. This is the 2Dmotion flow field.
CSE486, Penn StateRobert Collins
Recall : General Projection Equation
Relative R,TProjectionInternalparams
Strategy:1) Assume internal params known (set to identity)2) At time t, set R=I, T=03) At time t+1, movement generates a relative R and T simplifying assumption: small motion --> small rotation
4) Compute 3D velocity vector 5) Compute 2D velocity vector a function of X,Y,Z,R,T,f
CSE486, Penn StateRobert Collins
Displacement of 3D World Point
Time t: 3D position PTime t+1: 3D position RP+T
3D Displacement = RP+T - P
Now consider short time period (like time betweentwo video frames = 1/30 sec). Can assume a smallrotation angle in that amount of time. Make a smallangle approximation and rewrite displacement. In thelimit (infinitesimal time period), we will get a velocity.
CSE486, Penn StateRobert Collins Write Rotation matrix in terms of Euler Angles
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
Small Angle Approx
for small angle x
xx ≈sin
1cos ≈x
x
0sinsin ≈yx
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
3D Velocity
Under small angle approx,displacement = RP+T - P = (I+S)P+T-P = SP+T
Note: SP = [θx, θy, θz]T X P
In limit, displacement becomes a velocity
V = T + ω X P
3D velocity
linear velocityangular velocity
where
Caution: Abuse of notation ==> we are re-using T as a velocity
CSE486, Penn StateRobert Collins
3D Relative Velocity
ff
PPVV
ppvv
ZZ zz
The relative velocity of P The relative velocity of P wrt wrt camera:camera:
Camps, PSU
Note: we change signs of the velocity to beconsistent with the book. This should notcause concern. It just depends on whetheryou want to think of the motion as being dueto the camera or the scene.
CSE486, Penn StateRobert Collins
3D Relative Velocity
ff
PPVV
ppvv
ZZ zz
The relative velocity of P The relative velocity of P wrt wrt camera:camera:
Camps, PSU
CSE486, Penn StateRobert Collins
3D Relative Velocity
ff
PPVV
ppvv
ZZ zz
Where are we in this lecture?
• We just derived an equationrelating R,T and to the 3D velocity vector at each scenepoint.
• We can think of that velocity as a little vector inthe scene.
• Now ask, what does the projection of that vectorlook like in the image? It is a 2D vector. It is oneof the vectors that make up the Motion Field!
CSE486, Penn StateRobert Collins
Motion Field: the 2D velocity of p
ff
PPVV
ppvv
ZZ zz
Taking derivative Taking derivative wrt wrt time:time:
Camps, PSU
Perspectiveprojection
CSE486, Penn StateRobert Collins
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Camps, PSU
CSE486, Penn StateRobert Collins
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Camps, PSU
CSE486, Penn StateRobert Collins
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
Camps, PSUTHIS IS THE EQUATION WE WANT!!!!
CSE486, Penn StateRobert Collins
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
TranslationalTranslationalcomponentcomponent
Camps, PSU
CSE486, Penn StateRobert Collins
Motion Field: the velocity of p
ff
PPVV
ppvv
ZZ zz
RotationalRotationalcomponentcomponent
NOTE: The rotational component is independent of depth Z !NOTE: The rotational component is independent of depth Z !
Camps, PSU
CSE486, Penn StateRobert Collins
Special Case I: Pure Translation
Assume Assume TTzz ≠≠ 0 0
Define:Define:
Camps, PSU
CSE486, Penn StateRobert Collins
Special Case I: Pure Translation
What if What if TTzz == 0 ? 0 ?
All motion field vectors are parallel to each other andAll motion field vectors are parallel to each other andinversely proportional to depth !inversely proportional to depth !
Camps, PSUTIE IN WITH SIMPLE STEREO!
CSE486, Penn StateRobert Collins
Special Case I: Pure Translation
The motion field in this case is RADIAL:The motion field in this case is RADIAL:••It consists of vectors passing through pIt consists of vectors passing through poo = (x = (xoo,,yyoo))••If:If:
•• TTzz > 0, > 0, (camera moving towards object)(camera moving towards object)
•• the vectors point away from p the vectors point away from poo
••ppoo is the is the POINT OF EXPANSIONPOINT OF EXPANSION•• TTzz < 0, < 0, (camera moving away from object)(camera moving away from object)
••the vectors point towards pthe vectors point towards poo
••ppoo is the is the POINT OF CONTRACTIONPOINT OF CONTRACTION
TTzz > 0> 0 TTzz < 0< 0
Camps, PSU
CSE486, Penn StateRobert Collins Pure Translation:
Properties of the MF
• If Tz≠ 0 the MF is RADIAL with all vectorspointing towards (or away from) a singlepoint po. If Tz = 0 the MF is PARALLEL.
• The length of the MF vectors is inverselyproportional to depth Z. If Tz ≠ 0 it is alsodirectly proportional to the distancebetween p and po.
Camps, PSU
CSE486, Penn StateRobert Collins Pure Translation:
Properties of the MF
• po is the vanishing point of the direction oftranslation.
• po is the intersection of the ray parallel tothe translation vector and the image plane.
Camps, PSU
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
ff
PPVV
ppvv
ZZ zz
nnPlanar surfaces are commonPlanar surfaces are commonin man-made environmentsin man-made environments
Question: How does the MF of a moving plane look like?Question: How does the MF of a moving plane look like?
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
Points on the plane must satisfy thePoints on the plane must satisfy theequation describing the plane.equation describing the plane.
LetLet••nn be the unit vector normal to the plane. be the unit vector normal to the plane.••dd be the distance from the plane to the be the distance from the plane to theorigin.origin.••NOTE:NOTE: If the plane is moving If the plane is moving wrtwrtcamera, camera, nn and and dd are functions of timeare functions of time..
Then:Then:
dd PP
ZZ
nn
OOYY
XX
wherewhere
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
Camps, PSU
dd PP
ZZ
nn
OOYY
XX
Using the pinholeUsing the pinholeprojection equation:projection equation:pp
LetLet be the image of Pbe the image of P
Using the plane equation:Using the plane equation:
Solving for Z:Solving for Z:
Self-study
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
And Plug in:And Plug in:
dd PP
ZZ
nn
OOYY
XX
pp
Now consider the MF equations:Now consider the MF equations:
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
dd PP
ZZ
nn
OOYY
XX
pp
The MF equations become:The MF equations become:
wherewhere
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Special Case II: Moving Plane
dd PP
ZZ
nn
OOYY
XX
pp
MF equations: MF equations:
Q: What is the significance of this?Q: What is the significance of this?
A: The MF vectors are given by low order (second)A: The MF vectors are given by low order (second)polynomials.polynomials.
••Their Their coeffscoeffs. a. a11 to a to a8 8 (only 8 !) are functions of n, d, T and (only 8 !) are functions of n, d, T and ωω..••That is, can estimate 8That is, can estimate 8 param param global flow rather than 2global flow rather than 2 params params per pixel! Huge savings in time and robustness.per pixel! Huge savings in time and robustness.
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Application: Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
An object of heightAn object of height L L moves with moves withconstant velocity constant velocity v:v:••At time At time t=0t=0 the object is at: the object is at:
•• D(0) = DD(0) = Doo
••At time At time tt it is at it is at••D(t) = DD(t) = Doo –– vt vt
••It will crash into the camera atIt will crash into the camera attime:time:
•• D(D(ττ) = D) = Doo –– v vττ = 0 = 0•• ττ = D = Doo/v/v
t=0t=0tt
D(t)D(t)
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
The image of the object has size l(t):The image of the object has size l(t):
Taking derivative Taking derivative wrt wrt time:time:
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Time to Collision
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Time to Collision
Camps, PSU
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
And their ratio is:And their ratio is:
Self-study
CSE486, Penn StateRobert Collins
Time to Collision
Can beCan bedirectlydirectlymeasuredmeasuredfrom imagefrom image
ff
LL
vv
LL
DDoo
l(t)l(t)
t=0t=0tt
D(t)D(t)
And And time to collisiontime to collision::
Can be found, without knowing Can be found, without knowing LL or or DDoo or or vv !! !!
Camps, PSU
Self-study
CSE486, Penn StateRobert Collins
Application: Decomposition of Vehicle Flow Fieldinto Rotational and Translational Components
Motivation•Estimate steering angles•Computation of scene structure simplifies when rotational motion is removed
CSE486, Penn StateRobert Collins
Review: Flow Due to Self-Motion
Rotation Translation
Flow:
Rotation component Translation component
Note: I’ve changed sign again!
CSE486, Penn StateRobert Collins
Approach
Use prior knowledge of car’s likely motion to simplify the problem.
Assume we know approx direction of translation (Tx,Ty,Tz)
[ e.g. if driving forward, we choose (0,0,1) ]
Assume roll angle of motion (wz) is 0
Problem reduces to solving for pitch and yaw angles (wx and wy)
0known
CSE486, Penn StateRobert Collins
Approach (continued)
For each observed flow vector, form a linear constraint on wx and wy
by taking the dot product of the flow equation with a new vector that
is constructed to “annihilate” the translation component of flow.
This annihilation vector is
Verify:
0
Note: these are allknown values!
CSE486, Penn StateRobert Collins
Approach (continued)
For each observed flow vector, we annihilate the translation component
to form one linear constraint of the form (ai wx + bi wy + ci ) = 0.
Given the whole set of n observed flow vectors, we seek wx and wy
that simultaneously satisfy the set of n linear equations
Geometric intuition:
image space solution space
wx
wy
CSE486, Penn StateRobert Collins
Use Robust Estimator
robust solution
least-squares solution
red: outliers blue: inliers
Solving using least-squares will be sensitive to outliers (grossly incorrect data).Instead, we use a robust estimator (Random Sample Consensus – RANSAC).
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins
CSE486, Penn StateRobert Collins