Epipolar Geometry - Silvio Savarese · Chapter: 9 “Epipolar Geometry and the Fundamental Matrix...
Transcript of Epipolar Geometry - Silvio Savarese · Chapter: 9 “Epipolar Geometry and the Fundamental Matrix...
Lecture 5 -Silvio Savarese 5-Feb-14
Professor Silvio Savarese
Computational Vision and Geometry Lab
Lecture 5Epipolar Geometry
Lecture 5 -Silvio Savarese 5-Feb-14
• Why is stereo useful?• Epipolar constraints• Essential and fundamental matrix• Estimating F• Examples
Lecture 5Epipolar Geometry
Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformationsChapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”Chapter: 11 “Computation of the Fundamental Matrix F”
[FP] Chapters: 10 “The geometry of multiple views”
Pinhole perspective projectionRecovering structure from a single view
C
Ow
Pp
Calibration rig
Scene
Camera K
From calibration rig
From points and lines at infinity
+ orthogonal lines and planes structure of the scene, K
location/pose of the rig, K
Knowledge about scene (point correspondences, geometry of lines & planes, etc…
Pinhole perspective projectionRecovering structure from a single view
C
Ow
Pp
Calibration rig
Scene
Camera K
Why is it so difficult?
Intrinsic ambiguity of the mapping from 3D to image (2D)
Recovering structure from a single view
Intrinsic ambiguity of the mapping from 3D to image (2D)
Courtesy slide S. Lazebnik
Two eyes help!
O2O1
x2
x1
?
Two eyes help!
This is called triangulation
K =knownK =known
R, T
llX X
l'l
• Find X that minimizes
),(),( 22
2
11
2XMxdXMxd
O1O2
x1
x2
X
Triangulation
Stereo-view geometry
• Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?
• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.
• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.
• Epipolar Plane • Epipoles e1, e2
• Epipolar Lines
• Baseline
Epipolar geometry
O1O2
x2
X
x1
e1 e2
= intersections of baseline with image planes
= projections of the other camera center
For details see CS131A
Lecture 9
Example: Converging image planes
e
e’
O1O2
X
e2
x1 x2
e1
Example: Parallel image planes
• Baseline intersects the image plane at infinity
• Epipoles are at infinity
• Epipolar lines are parallel to x axis
Example: Parallel Image Planes
e’ at
infinity
e at
infinity
Example: Forward translation
• The epipoles have same position in both images
• Epipole called FOE (focus of expansion)
O2
e1
e2
O2
- Two views of the same object
- Suppose I know the camera positions and camera matrices
- Given a point on left image, how can I find the corresponding point on right image?
Epipolar Constraint
Epipolar geometry
O1O2
X
- Two views of the same object
- Suppose I know the camera positions and camera matrices
- Given a point on left image, how can I find the corresponding point on right image?
Epipolar Constraint
1
v
u
PMP
0IKM
OO’
pp’
P
R, T
Epipolar Constraint
TRRKMTT
'
1
v
u
PMP
T
O
O’
T = O’ in the camera 1 reference systemR is such that a vector p’ in camera 2 is equal to R p’ in camera 1
The cameras are related by R, T
p’
Camera 1
Camera 2
0IKM
OO’
pp’
P
R, T
Epipolar Constraint
K is known
(canonical cameras)
0IM
100
010
001
K
TRRKMTT
'
TRRMTT
'
OO’
pp’
P
R, T
0)( pRTpT
Epipolar Constraint
is perpendicular to epipolar plane)( pRT
pR p’ in first camera reference system is
baba ][
0
0
0
z
y
x
xy
xz
yz
b
b
b
aa
aa
aa
Cross product as matrix multiplication
OO’
pp’
P
R, T
0)pR(TpT
0pRTpT
E = essential matrix(Longuet-Higgins, 1981)
Epipolar Constraint
• E p2 is the epipolar line associated with p2 (l1 = E p2)
• ET p1 is the epipolar line associated with p1 (l2 = ET p1)
• E e2 = 0 and ET e1 = 0
• E is 3x3 matrix; 5 DOF
• E is singular (rank two)
Epipolar Constraint
O1O2
p2
P
p1
e1e2
l1 l2
021 pEpT
Epipolar Constraint
OO’
pp’
P
0IKM K is unknown
R, T
TRRKMTT
'
OO’
pp’
P
Epipolar Constraint
0pFpT
F = Fundamental Matrix(Faugeras and Luong, 1992)
1
KRTKFT
Epipolar Constraint
O1O2
p2
P
p1
e1e2
• F p2 is the epipolar line associated with p2 (l1 = F p2)
• FT p1 is the epipolar line associated with x1 (l2 = FT p1)
• F e2 = 0 and FT e1 = 0
• F is 3x3 matrix; 7 DOF
• F is singular (rank two)
021 pFpT
Why F is useful?
- Suppose F is known
- No additional information about the scene and camera is given
- Given a point on left image, how can I find the corresponding point on right image?
l’ = FT xx
Why F is useful?
• F captures information about the epipolar geometry of 2 views + camera parameters
• MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!)
• Powerful tool in:• 3D reconstruction• Multi-view object/scene matching
OO’
pp’
P
Estimating F
The Eight-Point Algorithm
1
v
u
pP
1
v
u
pP 0pFpT
(Hartley, 1995)
(Longuet-Higgins, 1981)
0pFpT
Estimating F
Let’s take 8 corresponding points
11 12 13
21 22 23
31 32 33
'
, ,1 ' 0
1
F F F u
u v F F F v
F F F
11
12
13
21
22
23
31
32
33
', ', , ', ', , ', ',1 0
F
F
F
F
uu uv u vu vv v u v F
F
F
F
F
Estimating F
Estimating F
Lsq. solution by SVD!
• Rank 8 A non-zero solution exists (unique)
• Homogeneous system
• If N>8 F̂
f
1f
W 0f
W
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' '
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v
11
12
13
21
22
23
31
7 7 7 7 7 7
32
8 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1
' ' ' ' ' ' 1
F
F
F
F
F
F
Fu v v v u v
Fu u u v u v u v v v u v
F
0pF̂pT
and estimated F may have full rank (det(F) ≠0)
0F̂F Find F that minimizes
Subject to det(F)=0
But remember: fundamental matrix is Rank2
Frobenius norm (*)
SVD (again!) can be used to solve this problem
^ ^
(*) Sqrt root of the sum of squares of all entries
satisfies:F̂
[HZ] pag 281, chapter 11, “Computation of F”
Data courtesy of R. Mohr and B. Boufama.
Mean errors:
10.0pixel
9.1pixel
Problems with the 8-Point Algorithm
- Recall the structure of W:- do we see any potential
(numerical) issue?
F1f
,0fWLsq solution
by SVD
Problems with the 8-Point Algorithm
• Highly un-balanced (not well conditioned)
• Values of W must have similar magnitude
• This creates problems during the SVD decomposition
f
W
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' ' ' ' ' ' 1
' '
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v u v v v u v
u u u v u v
11
12
13
21
22
23
31
7 7 7 7 7 7
32
8 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1
' ' ' ' ' ' 1
F
F
F
F
F
F
Fu v v v u v
Fu u u v u v u v v v u v
F
0Wf
HZ
pag
10
8
Normalization
IDEA: Transform image coordinate
such that the matrix W become better
conditioned (pre-conditioning)
Apply following transformation T:
(translation and scaling)
• Origin = centroid of image points
• Mean square distance of the data
points from origin is 2 pixels
ii pTq ii pTq (normalization)
1. Normalize coordinates:
2. Use the eight-point algorithm to compute F’q from the
points q and q’ .
1. Enforce the rank-2 constraint.
2. De-normalize Fq:
i i
ii pTq ii pTq
0qFq q
T
The Normalized Eight-Point Algorithm
qF
'TFTF q
T
0)Fdet( q
0. Compute T and T’
With
tra
nsfo
rma
tio
nW
ith
ou
t tr
an
sfo
rmatio
n Mean errors:
10.0pixel
9.1pixel
Mean errors:
1.0pixel
0.9pixel
Same issue for the DLT algorithm
ixix
H
ii xHx [Section 4.4 in AZ]
Lecture 5 -Silvio Savarese 5-Feb-14
• Why is stereo useful?• Epipolar constraints• Essential and fundamental matrix• Estimating F• Examples
Lecture 5Epipolar Geometry
Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformationsChapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”Chapter: 11 “Computation of the Fundamental Matrix F”
[FP] Chapters: 10 “The geometry of multiple views”
O1O2
X
e2
x1 x2
e1
Example: Parallel image planes
E=?
K K
K1=K2 = known
Hint :
x
y
z
x parallel to O1O2
R = I t = (T, 0, 0)
00
00
000
0
0
0
T
T
TT
TT
TT
xy
xz
yz
IE
Essential matrix for parallel images
T = [ T 0 0 ]
R = I
RTE
O1
X
e2
x1 x2
e1
Example: Parallel image planes
K K
x
y
z
horizontal!What are the
directions of
epipolar lines?
2
3221
0
00
00
000
E
xT
xT
T
Tl xx
O1
X
e2
x1 x2
e1
Example: Parallel image planes
K K
x
y
z
How are
x1 and x2
related?021 xEx
T
O1
X
e2
x1 x2
e1
Example: Parallel image planes
K K
x
y
z
T
0 0 0 0
p E p 0 1 0 0 0 1 0
0 0 1
u
u v T v u v T Tv Tv
T Tv
How are
x1 and x2
related?
O1O2
X
e2
x1 x2
e1
Example: Parallel image planes
K K
x
y
z
Rectification: making two images “parallel”
Why it is useful? • Epipolar constraint y = y’• New views can be synthesized by linear interpolation
For details on how to rectify
two views see CS131A
Lecture 9
Next lecture:
Stereo systems and affine SFM