Differential Motion Analysis (I)
Transcript of Differential Motion Analysis (I)
Differential Motion Analysis (I)
Ying Wu
Electrical Engineering and Computer ScienceNorthwestern University, Evanston, IL 60208
http://www.eecs.northwestern.edu/~yingwu
1 / 23
Outline
Optical FlowBrightness Constancy and Optical Flow ConstraintLucas-Kanade’s MethodHorn-Schunck’s MethodParametric FlowRobust Flow and Multi-frame Flow
Beyond Basic Optical FlowConsidering Lighting VariationsConsidering Appearance VariationsConsidering Spatial-Appearance Variations
2 / 23
Brightness Constancy and Optical Flow Optical flow: the apparent motion of the brightness pattern Optical flow 6= motion field Denote an image by I (x , y , t), and the velocity of a pixel
m = [x , y ]T is
vm = m = [vx , vy ]T =
[
dx/dt
dy/dt
]
Brightness constancy: the intensity of m keeps the sameduring dt, i.e.,
I (x + vxdt, y + vydt, t + dt) = I (x , y , t)
Optical flow constraint:
∂I
∂xvx +
∂I
∂yvy +
∂I
∂t= 0
i.e.
∇I · vm +∂I
∂t= 0
3 / 23
The Aperture Problem
For each pixel, one constraint equation, but two unknowns.
Normal flow
normalflow
optical constraint line
v
v
I
∆
x
y
Aperture problem: the motion along the directionperpendicular to the image gradient cannot be determined
p’
pC
C’
t
t
Other constraints are needed.
4 / 23
Lucas-Kanade’s Method Assume: a constant motion for a small image patch Ω. Define a weight function W (m),m ∈ Ω, for the pixels. Weighted LS formulation
minv
E =∑
m∈Ω
W 2(m)
(
∇I · v +∂I
∂t
)2
WLS solution:
v = (ATW2A)−1ATW2b
A =
∂I1∂x1
∂I1∂y1
......
∂IN∂xN
∂IN∂yN
, W = diag(W (m1), . . . ,W (mN))
v =
[
∂x
∂t,∂y
∂t
]T
= [vx , vy ]T , b = −[
∂I1∂t
, . . . , ∂IN∂t
]T
the intersection of all the flow constraint lines correspondingto the pixels in Ω.
5 / 23
Horn-Schunck’s Method
Assume: flow varies smoothly ←− global regularization
The measure of departure from smoothness can be written by:
es =
∫ ∫
(||∇vx ||2 + ||∇vy ||
2)dxdy
=
∫ ∫
(∂vx
∂x)2 + (
∂vx
∂y)2 + (
∂vy
∂x)2 + (
∂vy
∂y)2dxdy
The error of optical flow is:
ec =
∫ ∫
(∇I · vm +∂I
∂t)2dxdy
Objective function:
e = ec + λes
=
∫ ∫
(∇I · vm +∂I
∂t)2 + λ(||∇vx ||
2 + ||∇vy ||2)dxdy
6 / 23
Horn-Schunck’s Method
Fixed-point iteration:
vk+1x = vk
x −
(
∂I∂x
)
vkx +
(
∂I∂y
)
vky + ∂I
∂t
λ +(
∂I∂x
)2+(
∂I∂y
)2
∂I
∂x
vk+1y = vk
y −
(
∂I∂x
)
vkx +
(
∂I∂y
)
vky + ∂I
∂t
λ +(
∂I∂x
)2+(
∂I∂y
)2
∂I
∂y
Concisely, it is:vk+1 = vk − α(∇I )
In each iteration, the new optical flow field is constrained byits local average and the optical flow constraints.
7 / 23
Parametric Flow: Affine flow
Affine model is under two assumptions: planar surface orthographic projection
We can write a 3D plane by Z = AX + BY + C . Then wehave the 6-parameter affine flow model:
[
vx
vy
]
=
[
a1 a2
a3 a4
] [
x
y
]
+
[
a5
a6
]
In this case, the flow can be determined by at least 3 points.
8 / 23
Parametric Flow: Quadratic flow
Quadratic model is under two assumptions: planar surface perspective projection
Under perspective projection, a plane can be written as
1
Z=
1
C−
A
CX −
B
CY
So, we have
vx = a1 + a2x + a3y + a7x2 + a8xy
vy = a4 + a5x + a6y + a7xy + a8y2
In this case, if we know at least 4 points on a planar object,we can also a1, . . . , a8.
9 / 23
Parametric Flow: Parametric flow fitting LS formulation
minθ
∑
Ω
||I (x + vx(Θ)dt, y + vy (Θ)dt, t + dt)− I (x , y , t)||2
or
minθ
∑
Ω
[
∇ITv(Θ) + It
]2
denote by ∇Iθ = ∇IT∇v(Θ),
minθ
∑
Ω
[
∇ITθ Θ + It
]2
Easy to figure out the LS solution.
10 / 23
Exercises
Exercise 1: Recovering rotationAssume the motion is a pure rotation, i.e.,
[
x2
y2
]
= R(θ)
[
x1
y1
]
=
[
cos θ − sin θsin θ cos θ
] [
x1
y1
]
minθ
∑
[
I (R(θ)
[
x1
y1
]
, t + dt)− I (x , y , t)
]2
Exercise 2: Recovering 2D affine motion
[
vx
vy
]
=
[
a1 a2
a3 a4
] [
x
y
]
+
[
a5
a6
]
minA
∑
||∇ITv(A) + It ||2
11 / 23
Robust Flow Computation1
Motivation Brightness constancy
violation←− specular reflection
Spatial smoothnessviolation←− motion discontinuities
Outliers ruin LS estimation One solution: influence function ρ(x , σ)
Applying influence function to flow estimation
minv
∑
Ω
ρ(I (x , y , t)− I (x + vxdt, y + vydt, t + dt), σ)
minv
∑
Ω
ρc(∇ITv(θ) + It , σc) + λ[ρs(vx , σs) + ρs(vy , σs)]
1Michael Black and P. Anandan, “The Robust Estimation of Multiple Motions: Parametric and
Piecewise-Smooth Flow Fields”, CVIU, vol.63. no.1, pp.75-104, 199612 / 23
Multi-frame Optical Flow2
For a static scene, the flow induced by camera motion inmultiple frames lie in a low-dimensional subspace
Example: 3D planar scene under orthographic projection
[
vx
vy
]
=
[
a5 a1 a2
a6 a3 a4
]
1x
y
We have F frames and all have the same N points. Denote by[v ij
x vijy ]T the flow for the i-th point at the j-th frame, w.r.t. a
reference frame. Collect all the flows:
U =
v11x v21
x . . . vN1x
...... . . .
...v1Fx v2F
x . . . vNFx
=
a15 a1
1 a12
......
...aF5 aF
1 aF2
1 1 . . . 1x1 x2 . . . xN
y1 y2 . . . yN
2Michal Irani, “Multi-Frame Optical Flow Estimation Using Subspace Constraints”, ICCV’99
13 / 23
Multi-frame Optical Flow
And
V =
v11y v21
y . . . vN1y
...... . . .
...v1Fy v2F
y . . . vNFy
=
a16 a1
3 a14
......
...aF6 aF
3 aF4
1 1 . . . 1x1 x2 . . . xN
y1 y2 . . . yN
It is clear that
rank
[
UV
]
≤ 3
rank
[
U V]
≤ 6
14 / 23
Outline
Optical FlowBrightness Constancy and Optical Flow ConstraintLucas-Kanade’s MethodHorn-Schunck’s MethodParametric FlowRobust Flow and Multi-frame Flow
Beyond Basic Optical FlowConsidering Lighting VariationsConsidering Appearance VariationsConsidering Spatial-Appearance Variations
15 / 23
Considering lighting models
Brightness constancy assumption is too restrictive
Are there constraints for lighting?
For a pure Lambertain surface if no shadowing, then all images under varying illumination lie
in a 3-D subspace in RN
with shadowing, the dimension will be higher, but we maylearn it
The subspace can be learnt from a set of training images byPCA, so we have the basis B = [B1, B2, . . . ,Bm] (note:BTB = I).
Then the appearance of the template at t is modeled by
I (x , y , t) + BΛ, where Λ =
λ1...
λm
16 / 23
Considering lighting models3
therefore, we have
E (Θ,Λ) =∑
Ω
||I (x + vx(Θ)dt, y + vy (Θ)dt, t + dt)− I (x , y , t)− BΛ||2
or E (Θ,Λ) =∑
Ω
||∇Tv(Θ) + It − BΛ||2
or E (Θ,Λ) =∑
Ω
||∇Tθ Θ + It − BΛ||2
denote by ∇ITθ = M, we have
[
M −B]
[
ΘΛ
]
= −It
3Gregory Hager and Peter Belhumeur, “Real-Time Tracking of Image Regions with Changes in Geometry and
Illumination”, CVPR’96
17 / 23
Considering lighting models
So, we have
[
ΘΛ
]
=[
M −B]†
It
= −
[
MTM −MTB−BTM BTB
]−1 [MT
−BT
]
It
easy to see
Θ = −[
MT (I− BBT )M]−1
MT (I + BBT )It
18 / 23
Considering appearance variations
In-class appearance variations
Low-level matching −→ high-level matching
If we know the target, we may learn its appearance variations
We may build a classifier for matching
minv
S (I (u + vxdt, v + vydt, t + dt) : Λ) ,
where Λ are parameters of the classifier
E.g., using an SVM classifier
n∑
j=1
yjαjk(I, xj) + b
Let’s maximize the SVM matching score
maxu,v
n∑
j=1
yjαjk(I + uIx + vIy , xj)
19 / 23
Considering appearance variations4
Let use a 2nd order polynomial kernel k(x, xj) = (xTxj)2
so, we have
E (u, v) =n∑
j=1
yjαj
[
(I + uIx + vIy )Txj
]2
∂E
∂u=
∑
yjαj ITx xj(I + uIx + vIy )Txj = 0
∂E
∂v=
∑
yjαj ITy xj(I + uIx + vIy )Txj = 0
the solution is:[ ∑
αjyj(xTj Ix)
2∑
αjyj(xTj Ix)(xj Iy )
∑
αjyj(xTj Ix)(x
Tj Iy )
∑
αjyj(xTj Iy )2
] [
u
v
]
=
[
−∑
αjyj(xTj Ix)(x
Tj I )
−∑
αjyj(xTj Iy )(xT
j I )
]
4Shai Avidan, “Subset Selection for Efficient SVM Tracking”, CVPR’03
20 / 23
Spatial-appearance model (SAM)5
Denote by y = [x c(x)], where x is the location and c color Assume a Gaussian component be factorized
g(y; µk , Σk) = g(x; µks , Σks)g(c(x); µkc , Σkc)
For a pixel, the likelihood is a mixture
p(y|Θ) =K∑
k=1
pkg(y; µk , Σk)
Let’s use an affine motion here
T (x; at) =
[
a1t a2t
a3t a4t
]
x +
[
a5t
a6t
]
then, we have
p(T (y; at)|Θ) = p(T (x; at), c(T (x; at))|Θ)
=K∑
k=1
pkg(x;µks ,Σks)g(c(T (x; at));µkc ,Σkc)=
K∑
k=1
q(k, yi ; at)
5Ting Yu and Ying Wu, “Differential Tracking based on Spatial-Appearance Model (SAM)”, CVPR’06
21 / 23
Spatial-appearance model (SAM)
For an image region
E (at ; Θ) =∑
xi∈Ω
log p(T (yi ; at)|Θ) =∑
xi∈Ω
log
(
K∑
k=1
q(k , yi ; at)
)
our task is tomax
at
E (at ; Θ)
Solution: similar to the general EM algorithm
22 / 23
Spatial-appearance model (SAM)
23 / 23