Www.dbta.tu-berlin.de [email protected] Separation column in MOSAIC.
Marc Pollefeys - math.tu-berlin.de
Transcript of Marc Pollefeys - math.tu-berlin.de
Algebraic Vision 2015 Slide 1
(radial) multi-focal tensors concept, internal constraints and geometric insight
Marc Pollefeys
Algebraic Vision 2015 Slide 2
Backprojection
Represent point as intersection of row and column
Useful presentation for deriving and understanding multiple view geometry (notice 3D planes are linear in 2D point coordinates)
Condition for solution?
Algebraic Vision 2015 Slide 3
Multi-view geometry
(intersection constraint)
(multi-linearity of determinants)
(= epipolar constraint!) (counting argument: 11x2-15=7)
Algebraic Vision 2015 Slide 4
Epipolar geometry
(courtesy Hartley and Zisserman)!
x’TFx=0 for all x!x’ !
Algebraic Vision 2015 Slide 5
Internal constraint for fundamental matrix
Geometric interpretation of rank-2 constraint !
(courtesy Hartley and Zisserman)!
e’TFx=0, !x " e’TF=0 similarly Fe=0!
Algebraic Vision 2015 Slide 6
Multi-view geometry
(multi-linearity of determinants)
(= trifocal constraint!)
(3x3x3=27 coefficients)
Algebraic Vision 2015 Slide 7
T has 27 coefficients Q has 18DOF (i.e. 3x(4x3-1)-(4x4-1)=33-15) Q has 8 internal consistency constraints (i.e. 27-1-18)
T internal consistency constraint
Notice T31k is a point in view 3 corresponding to the projection of the intersection of reprojected (x,y)=(0,0) from view 1 and x=0 from view 2. There are 9 such points in view 3.
T31k, T322k, T333k have to be collinear (as projections of points on same 3D line), i.e. det(T11kl)=0. There are 3 such cubic constraints in view 3. Also, the above 3 lines need to intersect in epipole e13 (degree 6) (four additional constraints more complicated, see Resl PhD 2003)
Algebraic Vision 2015 Slide 8
Multi-view geometry
(multi-linearity of determinants)
(= quadrifocal constraint!)
(3x3x3x3=81 coefficients)
Algebraic Vision 2015 Slide 9
Q has 81 coefficients Q has 29 DOF (i.e. 4x(4x3-1)-(4x4-1)=44-15) Q has 51 internal consistency constraints (i.e. 81-1-29)
Q internal consistency constraint
Notice Q111l is a point in image 4 corresponding to the projection of the intersection of reprojected x=0 from image 1; x=0 from image 2; and x=0 from image 3. There are 27 such points in image 4.
Obviously Q111l, Q112l, Q113l have to be collinear (as projections of points on same 3D line), i.e. det(Q11kl)=0, same for all 9 combinations of view 1 & 2 in this view Similarly all 6 view-pairs yield 9 constraints (i.e. 1-2, 1-3, 1-4, 2-3, 2-4, 3-4) We verified that in general these 54 constraints yield 51 independent ones
Algebraic Vision 2015 Slide 10
perspective camera (2 constraints / feature)
radial camera (uncalibrated) (1 constraints / feature)
3 constraints allow to reconstruct 3D point
more constraints also tell something about cameras
multilinear constraints known as epipolar, trifocal and quadrifocal constraints
(0,0)
l=(y,-x)
(x,y)
Multiple view geometry
Algebraic Vision 2015 Slide 11
ℓ1!
ℓ4!ℓ
3!ℓ2!
ℓ5!
"1!
"2!
"3!
"4!"5!
crad! Optical Axis!
Radial 1D camera
Algebraic Vision 2015 Slide 12 12
Quadrifocal constraint
with!
Algebraic Vision 2015 Slide 13
QRAD has 16 coefficients QRAD has 13 DOF (i.e. 4x(4x2-1)-(4x4-1)=28-15) QRAD has 2 internal consistency constraints (i.e. 16-1-13) 4 optical axes have 2 lines that they all intersect with
QRAD internal consistency constraint
Pick projection of one of these lines in 3 views, then 4th line is arbitrary => requires liljlkQijk1,2=0, same for other views (compare to F.e=0) Feasible QRAD needs existence of 2 special lines => 2 internal constraints
(Thirthala and Pollefeys IJCV 2012)
12 degree polynomials characterized in (Lin and Sturmfels, 2009)!
Algebraic Vision 2015 Slide 14
•" Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
•" Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
(x,y)
Radial quadrifocal tensor (Thirthala and Pollefeys IJCV 2012)
Algebraic Vision 2015 Slide 15 15
Synthetic quadrifocal tensor example
•" Perspective •" Fish-eye •" Spherical mirror •" Hyperbolic mirror
Algebraic Vision 2015 Slide 16 16
Perspective! Fish-eye!
Algebraic Vision 2015 Slide 17 17
Spherical mirror! Hyperbolic mirror!
Algebraic Vision 2015 Slide 18
•" Radial trifocal tensor Tijk from 7 points in 3 views
•" Reconstruct 2D panorama and use it for calibration
•" Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
•" Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
(2x2x2 tensor)
Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view
algorithm for pure rotation
(x,y)
Radial quadrifocal tensor
Algebraic Vision 2015 Slide 19
•" Radial trifocal tensor Tijk from 7 points in 3 views
•" Reconstruct 2D panorama and use it for calibration (2x2x2 tensor)
(x,y)
Radial trifocal tensor
Internal consistency constraints? TRAD has 8 coefficients TRAD has 7 DOF (i.e. 3x(3x2-1)-(3x3-1)=15-8) TRAD has NO internal consistency constraints (i.e. 8-1-7) All set of 8 numbers represent valid radial trifocal tensors
Same as (Quan and Kanade‘97)
Algebraic Vision 2015 Slide 20 20
•" Two-step linear approach to compute radial distortion
•" Estimates distortion polynomial of arbitrary degree
undistorted image
estimated distortion (4-8 coefficients)
Dealing with Wide FOV cameras (Thirthala and Pollefeys, ICCV’05/IJCV‘12)
Algebraic Vision 2015 Slide 21 21
unfolded cubemap estimated distortion
(4-8 coefficients)
Dealing with Wide FOV cameras
•" Two-step linear approach to compute radial distortion
•" Estimates distortion polynomial of arbitrary degree
(Thirthala and Pollefeys, ICCV’05/IJCV‘12)
Algebraic Vision 2015 Slide 22 22
Non-parametric distortion calibration
•" Models fish-eye lenses, cata-dioptric systems, etc.
Algebraic Vision 2015
(Thirthala and Pollefeys, ICCV’05/IJCV‘12)
normalized radius
angl
e
Algebraic Vision 2015 Slide 23 23 Algebraic Vision 2015
normalized radius
angl
e
90o
Non-parametric distortion calibration
•" Models fish-eye lenses, cata-dioptric systems, etc.
(Thirthala and Pollefeys, ICCV’05/IJCV‘12)
Algebraic Vision 2015 Slide 24
Non-Parametric Structure-Based Calibration of Radially Symmetric Cameras
24!
Camposeco et al. ICCV 2015!
Algebraic Vision 2015 Slide 25
Minimal relative pose with know vertical
25
-g
Fraundorfer, Tanskanen and Pollefeys, ECCV2010!
5 linear unknowns # linear 5 point algorithm 3 unknowns # quartic 3 point algorithm
Vertical direction can often be estimated!•" inertial sensor!•" vanishing point!
Algebraic Vision 2015 Slide 26
Multi-camera systems
•" Light rays do not meet at a single center of projection. xj!
x2!26!
Algebraic Vision 2015 Slide 27
Multi-camera systems
•" 6-vector Plücker line to represent the light rays. xj!
uij!
tCi!V : Reference frame!
uij : Unit direction of ray!Tci : Translation from Ci to V!
27!
Algebraic Vision 2015 Slide 28
Generalized Epipolar Constraint
•" Generalized Epipolar constraint:
•" E = [R]xt : conventional Essential matrix.
•" [R, t] is the relative transformation.
•" Absolute scale can be obtained!
Using many cameras as one #Robert Pless!IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2003 !
28!
Algebraic Vision 2015 Slide 29
Generalized Epipolar Constraint
•"Problems:
–" Linear solution requires 17-point correspondences. ! ~600k RANSAC loops needed.
–" Minimal problem needs 6-point correspondences but gives 64 solutions.
29!
Algebraic Vision 2015 Slide 30
Motion Estimation: Ackermann Constraint
•" Enforce Ackermann motion constraint:
•" 2 degree-of-freedom
! 2-point minimal problem.
17 RANSAC Loops!!
Motion Estimation for a Self-Driving Car with a Generalized Camera##Gim Hee Lee, Friedrich Fraundorfer, and Marc Pollefeys$#IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2013$!
30!
Algebraic Vision 2015 Slide 31
Motion Estimation: Ackermann Constraint
•" Generalized Essential matrix with Ackermann constraint:
•" Solve for " and # from the Epipolar constraint.
! 2 polynomial equations with 2 unknowns.
02
cossincos
02
cossincos
2222
1111
=+++
=+++
ecba
ecba
!"!!
!"!!
31!
Algebraic Vision 2015 Slide 32
•" Can be solved in closed-form with .
•" Up to 6 real solutions.
Motion Estimation: Ackermann Constraint
32!
Algebraic Vision 2015 Slide 33
Motion estimation with generalized camera
33!
Algebraic Vision 2015 Slide 34
Structure from Sound
sound source location si !microphone location mj!dij!
measure time difference of arrival (TDOA) at microphones tij !
dij =v.(tij - ti )!
time of emission ti !
We want to solve for:!
similar problem formulation in sensor networks, range-only SLAM, ultra-wide band localization, … !
Algebraic Vision 2015 Slide 35
Sound and microphone factorization
with!
rank 5 matrix!
(Pollefeys and Nister, ICASSP’08)
rank 5 matrix (which contains [1 1… 1]T )!
Given time-difference-of-arrival , !compute position of microphones!and time of emission and position of sound sources ,!
Algebraic Vision 2015 Slide 36
Subspace intersection problem
36!
(from Teller and Hohmeier 99)!
Find subspaces that intersect a set of subspaces!e.g. given 4 lines in 3D, find lines that intersect them!
e.g. sound factorization: given 5 lines and 1 point in 10D, !! !find the 5D subspace that intersects all of them!
(more details in Roland Angst PhD)!
Algebraic Vision 2015 Slide 37
Conclusion
•" Polynomial constraints in computer vision often have intuitive geometric interpretation
•" Multi-view geometry of radial cameras allow to handle much more general camera models
•" Additional constraints can greatly simplify the problem
•" Also interesting geometric problems with sound (and vision)
•" Subspace intersection problem
Thank you!
37!