Post on 21-Dec-2015
),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):
Brightness Constancy Equation:
The Brightness Constraint
),(),( ),(),( yxyx vyuxIyxJ
Where: ),(),( yxJyxIIt
0 tyx IvIuI
Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info.
Another constraint: Global Motion Model Constraint
Global Motion Models2D Models:• Affine• Quadratic• Planar projective transform (Homography)
3D Models:• Rotation, Translation, 1/Depth • Instantaneous camera motion models• Plane+Parallax
0)()( 654321 tyx IyaxaaIyaxaaI
Example: Affine Motion
Substituting into the B.C. Equation:
yaxaayxv
yaxaayxu
654
321
),(
),(
Each pixel provides 1 linear constraint in 6 global unknowns
0 tyx IvIuI
(minimum 6 pixels necessary)
2 tyx IyaxaaIyaxaaIaErr )()()( 654321
Least Square Minimization (over all pixels):
image Iimage J
aJwwarp refine
a
aΔ+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
0 tyx IvIuI ==> small u and v ...
Quadratic – instantaneous approximation to planar motion
Other 2D Motion Models
287654
82
7321
yqxyqyqxqqv
xyqxqyqxqqu
yyvxxu
yhxhh
yhxhhy
yhxhh
yhxhhx
','
and
'
'
987
654
987
321
Projective – exact planar motion
(Homography)
Panoramic Mosaic ImageOriginal video clip
Generated Mosaic image
Alignment accuracy (between a pair of frames): error < 0.1 pixel
Original
Outliers
Original
Synthesized
Video Removal
ORIGINAL ENHANCED
Video Enhancement
Direct Methods:
Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image.
Minimization step:
Direct methods:
Error measure based on dense measurable image quantities(Confidence-weighted regression; Exploits all available information)
Feature-based methods:
Error measure based on distances of a sparse set of distinct features.
Benefits of Direct Methods
• Subpixel accuracy.• Does not need distinct features.
•Locking property.
Limitations
• Limited search range (up to 10% of the image size).
• Brightness constancy.
Handling Varying Brightness
Preprocessing:• Mean and contrast normalization.• Laplacian pyramids.
Measurable image quantities:• brightness values• correlation surfaces [Irani-Anandan:iccv98, Mandelbaum-et-
al:iccv99]
• mutual information [Viola-et-al]
Video Indexing and Editing
The 2D/3D Dichotomy
Image motion =
Camera induced motion =
+ Independent motions =
Camera motion
+Scene structure
+Independent motions
2D techniques
3D techniques Singularities in
“2D scenes”
Do not model
“3D scenes”
The Plane+Parallax Decomposition
Original Sequence Plane-Stabilized Sequence
The residual parallax lies on a radial (epipolar) field: wp'
p
epipole
p'
Benefits of the P+P Decomposition
• Eliminates effects of rotation
• Eliminates changes in camera parameters / zoom
• Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole)
• Image displacements: Constrained to lie on radial lines (1-D search problem)
A result of aligning an existing structure in the image.
1. Reduces the search space:
Remove global component which dilutes information !
Translation or pure rotation ???
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Focus on relevant portion of info
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene
STAB_RUG SEQ
- fewer bits, progressive encoding
Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene• Height vs. Depth (e.g., obstacle avoidance)
• A compact representation
global (100)component
local [-3..+3]component
total distance [97..103]
camera center
scene
• Appropriate units for shape
• Start with 2D estimation (homography).
• 3D info builds on top of 2D info.
3. Stratified 2D-3D Representation:
Avoids a-priori model selection.
Benefits of the P+P Decomposition
Original sequence Plane-aligned sequence Recovered shape
Dense 3D Reconstruction(Plane+Parallax)
Dense 3D Reconstruction(Plane+Parallax)
Original sequence
Plane-aligned sequence
Recovered shape
Results
Original sequence Plane-aligned sequence
Recovered shape
Brightness Constancy constraint
Multi-Frame vs. 2-Frame Estimation
The intersection of the two line constraints uniquely defines the displacement.
1. Eliminating Aperture Problem
Epipolar line
epipole
p
0 TYX IvIuI
other epipolar line
Epipolar line
Multi-Frame vs. 2-Frame Estimation
The two line constraints are parallel ==> do NOT intersect
1. Eliminating Aperture Problem
p
0 TYX IvIuI
anotherepipole
epipole
Brightness C
onstancy constra
int
The other epipole resolves the ambiguity !
3D Motion Models
ZxTTxxyyv
ZxTTyxxyu
ZYZYX
ZXZYX
)()1(
)()1(2
2
)(1
)(1
233
133
tytt
xyv
txtt
xxu
w
w
Local Parameter:
ZYXZYX TTT ,,,,,
),( yxZ
Instantaneous camera motion:
Global parameters:
Residual Planar Parallax Motion
Global parameters: 321 ,, ttt
),( yxLocal Parameter: