lec15 - motion field optical flow · 2019. 5. 22. · 1 CSE152, Spring 2019 Intro Computer Vision...
Transcript of lec15 - motion field optical flow · 2019. 5. 22. · 1 CSE152, Spring 2019 Intro Computer Vision...
-
1
CSE152, Spring 2019 Intro Computer Vision
Motion Field and Optical Flow
Introduction to Computer Vision CSE 152
Lecture 15
CSE152, Spring 2019 Intro Computer Vision
• Reading on optical flow from Trucco & Verri is on ereserves
• HW3 assigned • Midterm graded
-
2
CSE152, Spring 2019 Intro Computer Vision
Midterm • Maximum78 (93%) • Median 56.2 (67%) • Minimum 29.6 (35%) • Score Distribution
• Recap of answers.
CSE152, Spring 2019 Intro Computer Vision
Lambertian Photometric Stereo Videos this time
Reconsstruction with Albedo Map
-
3
CSE152, Spring 2019 Intro Computer Vision
Constant albedo map
CSE152, Spring 2019 Intro Computer Vision
Another person
-
4
CSE152, Spring 2019 Intro Computer Vision
Constant Albedo map
CSE152, Spring 2019 Intro Computer Vision
Structure-from-Motion (SFM) Given two or more images or video w/o any
information on camera position/motion as input, estimate camera motion and 3-D structure of a scene.
Two Approaches
1. Discrete motion (wide baseline) 2. Continuous (Infinitesimal) motion usually
from video
-
5
CSE152, Spring 2019 Intro Computer Vision
Motion field • The motion field is the projection of the
3D scene motion into the image
CSE152, Spring 2019 Intro Computer Vision
Rigid Motion: General Case
Position and orientation of a rigid body Rotation Matrix: R Translation vector: t
Rigid Motion: Velocity vector: T
Angular velocity vector: ω
P
pTp ×+= ω!
p!ω
||ω||
I know, using T for velocity is poor notation, but it’s consistent with Trucco and Verri textbook
-
6
CSE152, Spring 2019 Intro Computer Vision
General Motion
Let (x,y,z) be functions of time (x(t), y(t), z(t)):
Substitute where p=(x,y,z)T
CSE152, Spring 2019 Intro Computer Vision
Motion Field Equation
• Image • (u,v): Image point coordinates • : Image point velocity
• Camera • T: Components of 3-D linear motion • ω: Angular velocity vector • f: focal length
• Scene • z: depth
!u =Tzu−Tx f
z−ω y f +ω zv +
ωxuvf
−ω yu
2
f
!v =Tzv −Ty f
z+ωx f −ω zu−
ω yuv
f−ωxv
2
f
( !u, !v)
-
7
CSE152, Spring 2019 Intro Computer Vision
fv
fuv
ufZ
fTvTv
fu
fuvvf
ZfTuTu
xyzx
yz
yxzy
xz
2
2
ωωωω
ωωωω
−−−+−
=
−++−−
=
!
!
Pure Translation
ω = 0
• is inversely proportional to Z (remember Euclid) • Focus of expansion is located at (u,v) where !u, !v
!u = !v = 0
If camera is just translating with velocity (Tx, Ty, Tz), there’s no rotation:
CSE152, Spring 2019 Intro Computer Vision
Forward Translation & Focus of Expansion [Gibson, 1950]
-
8
CSE152, Spring 2019 Intro Computer Vision
Focus of Expansion (FOE)
• Focus of expansion is located at (u,v) where
• Solve for u,v
Tzu−Tx f = 0Tzv −Ty f = 0
!u = !v = 0
!u =Tzu−Tx fZ
!v =Tzv −Ty f
Z
u = fTxT z
v = fTyT z
Insight: The FOE is the perspective projection of the linear velocity vector (Tx, Ty, Tz).
CSE152, Spring 2019 Intro Computer Vision
Pure Translation
Radial about FOE
Parallel ( FOE point at infinity)
TZ = 0 Motion parallel to image plane
-
9
CSE152, Spring 2019 Intro Computer Vision
Sideways Translation [Gibson, 1950]
Parallel ( FOE point at infinity)
TZ = 0 Motion parallel to image plane
CSE152, Spring 2019 Intro Computer Vision
Pure Rotation: T=0
• Independent of Tx Ty Tz • Independent of Z • Only function of image plane position (u,v), f and ω
fv
fuv
ufZ
fTvTv
fu
fuvvf
ZfTuTu
xyzx
yz
yxzy
xz
2
2
ωωωω
ωωωω
−−−+−
=
−++−−
=
!
!
-
10
CSE152, Spring 2019 Intro Computer Vision
Rotational MOTION FIELD The “instantaneous” velocity of points in an image
PURE ROTATION
ω = (0,0,1)T
CSE152, Spring 2019 Intro Computer Vision
Motion Field Equation: Depth Estimation
If T, ω, and f are known or measured, then for each image point (u,v), one can solve for the depth Z given measured motion at (u,v).
€
˙ u =Tzu −Tx f
Z−ω y f +ω zv +
ω xuvf
−ω yu
2
f
˙ v =Tzv −Ty f
Z+ω x f −ω zu −
ω yuvf
−ω xv
2
f
€
Z =Tzu −Tx f
˙ u +ω y f −ω zv −ω xuv
f+ω yu
2
f
!u, !v
Inversely proportional to image velocity .
-
11
CSE152, Spring 2019 Intro Computer Vision
Measuring Motion
CSE152, Spring 2019 Intro Computer Vision
Estimating the motion field from images
1. Feature-based (Sect. 8.4.2 of Trucco & Verri)
1. Detect Features (corners) in an image 2. Search for the same features nearby (Feature
tracking).
2. Differential techniques (Sect. 8.4.1)
-
12
CSE152, Spring 2019 Intro Computer Vision
Optical Flow • Optical flow is the pattern of apparent
motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene
• Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image. i.e., the motion field.
• As we’ll see, it’s not but more on that later.
CSE152, Spring 2019 Intro Computer Vision
Optical Flow
• How to estimate pixel motion from image H to image I? – Find pixel correspondences
• Given a pixel in H, look for nearby pixels of the same color in I
• Key assumptions – color constancy: a point in H looks “the same” in image I
• For grayscale images, this is brightness constancy – small motion: points do not move very far
-
13
CSE152, Spring 2019 Intro Computer Vision
Warning • Notation shift: • (x,y) will be image positions • (u,v) will be image velocities
• Why? Notation in Trucco & Verri
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Constraint Equation
– Assume brightness of patch remains same in both images:
– Assume small motion: (Taylor series expansion of LHS about (x,y,t) and retain first order terms)
),( yx
(x +u δt, y + v δt)
ttime tttime δ+),( yx
Optical Flow: Velocities ),( vuDisplacement:
€
I(x + u δt,y + v δt, t +δt) = I(x,y,t)
€
I(x,y,t) +δx ∂I∂x
+δy∂I∂y
+δt∂I∂t
= I(x,y,t)
(δx,δ y) = (u δt,v δt)
-
14
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Constraint Equation
– Assume small interval, this becomes:
),( yx
),( tvytux δδ ++
ttime tttime δ+),( yx
Optical Flow: Velocities ),( vuDisplacement: (δx,δ y) = (u δt,v δt)
€
δxδt
∂I∂x
+δyδt
∂I∂y
+∂I∂t
= 0
– Subtracting I(x,y,t) from both sides and dividing by
€
δt
€
dxdt
∂I∂x
+dydt
∂I∂y
+∂I∂t
= 0
CSE152, Spring 2019 Intro Computer Vision
Solving for flow
• We can measure
§ Convolve image with [-1, 0, 1]
§ Convolve image with [-1, 0, 1]T
§ Consider stacking 3 images at (t-1, t, t+1). Convolve with [-1,0,1] over time
• We want to solve for • One equation, two unknowns à Can’t solve it
Optical flow constraint equation :
dtdy
dtdx,
∂I∂x,∂I∂y,∂I∂t
∂I∂t
∂I∂y
∂I∂x€
dxdt
∂I∂x
+dydt
∂I∂y
+∂I∂t
= 0
-
15
CSE152, Spring 2019 Intro Computer Vision
Ix =∂I∂x
,
, ∂∂
=yII y
tIIt ∂∂
=
Measurements
u = dxdt
,
dtdyv =
Flow vector
The component of the optical flow in the direction of the image gradient.
CSE152, Spring 2019 Intro Computer Vision
Barber Pole Illusion
http://www.opticalillusion.net/optical-illusions/the-barber-pole-illusion/
Optical Flow
Motion Field
Optical flow field isn’t always the same as the motion field
-
16
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Motion Field ≠
Motion field exists but no optical flow No motion field but shading changes
CSE152, Spring 2019 Intro Computer Vision
Apparently an aperture problem
-
17
CSE152, Spring 2019 Intro Computer Vision
Assume a single velocity (u,v) for pixels within an image patch Ω
dE(u,v)du
= 2Ix Ixu+ I yv + It( ) = 0∑dE(u,v)dv
= 2I y Ixu+ I yv + It( ) = 0∑
E(u,v) is minimized when partial derivatives equal zero.
Rewrite as
CSE152, Spring 2019 Intro Computer Vision
Lukas-Kanade (cont.)
• So, the optical flow U = (u,v) can be written as
MU=b • And optical is just U=M-1b