The fundamental matrix F
description
Transcript of The fundamental matrix F
![Page 1: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/1.jpg)
The fundamental matrix F
• It can be used for – Simplifies matching– Allows to detect wrong matches
![Page 2: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/2.jpg)
Estimation of F — 8-point algorithm
• The fundamental matrix F is defined by
0Fxx'for any pair of matches x and x’ in two images.
• Let x=(u,v,1)T and x’=(u’,v’,1)T,
333231
232221
131211
fff
fff
fff
F
each match gives a linear equation
0'''''' 333231232221131211 fvfuffvfvvfuvfufvufuu
![Page 3: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/3.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
vuvvvvuuuvuu
nnnnnnnnnnnn
0Af
![Page 4: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/4.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
vuvvvvuuuvuu
nnnnnnnnnnnn
• In reality, instead of solving , we seek f to minimize , least eigenvector of .
0AfAf AA
![Page 5: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/5.jpg)
8-point algorithm
• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to . 'FF 0'det F
![Page 6: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/6.jpg)
8-point algorithm
• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to . 'FF 0'det F
• It is achieved by SVD. Let , where
, let
then is the solution.
VUF Σ
3
2
1
00
00
00
Σ
000
00
00
Σ' 2
1
VUF Σ''
![Page 7: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/7.jpg)
8-point algorithm% Build the constraint matrix A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ... x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ... x1(1,:)' x1(2,:)' ones(npts,1) ]; [U,D,V] = svd(A); % Extract fundamental matrix from the column of V % corresponding to the smallest singular value. F = reshape(V(:,9),3,3)'; % Enforce rank2 constraint [U,D,V] = svd(F); F = U*diag([D(1,1) D(2,2) 0])*V';
![Page 8: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/8.jpg)
8-point algorithm
• Pros: it is linear, easy to implement and fast• Cons: susceptible to noise
![Page 9: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/9.jpg)
Results (ground truth)
![Page 10: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/10.jpg)
Results (8-point algorithm)
![Page 11: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/11.jpg)
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
vuvvvvuuuvuu
nnnnnnnnnnnn
Problem with 8-point algorithm
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differencebetween column of data matrix least-squares yields poor results
![Page 12: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/12.jpg)
Normalized 8-point algorithm
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
normalized least squares yields good results
Transform image to ~[-1,1]x[-1,1]
![Page 13: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/13.jpg)
Normalized 8-point algorithm
1. Transform input by ,
2. Call 8-point on to obtain
3.
ii Txx ˆ 'i
'i Txx ˆ
'ii xx ˆ,ˆ
TFTF ˆΤ'F̂
0Fxx'
0ˆ'ˆ 1 xFTTx'
F̂
![Page 14: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/14.jpg)
Normalized 8-point algorithm
A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ... x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ... x1(1,:)' x1(2,:)' ones(npts,1) ]; [U,D,V] = svd(A); F = reshape(V(:,9),3,3)'; [U,D,V] = svd(F); F = U*diag([D(1,1) D(2,2) 0])*V';
% Denormalise F = T2'*F*T1;
[x1, T1] = normalise2dpts(x1);[x2, T2] = normalise2dpts(x2);
![Page 15: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/15.jpg)
Normalizationfunction [newpts, T] = normalise2dpts(pts)
c = mean(pts(1:2,:)')'; % Centroid newp(1,:) = pts(1,:)-c(1); % Shift origin to centroid. newp(2,:) = pts(2,:)-c(2); meandist = mean(sqrt(newp(1,:).^2 + newp(2,:).^2)); scale = sqrt(2)/meandist; T = [scale 0 -scale*c(1) 0 scale -scale*c(2) 0 0 1 ]; newpts = T*pts;
![Page 16: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/16.jpg)
RANSAC
repeatselect minimal sample (8 matches)
compute solution(s) for F
determine inliers
until (#inliers,#samples)>95% or too many times
compute F based on all inliers
![Page 17: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/17.jpg)
Results (ground truth)
![Page 18: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/18.jpg)
Results (8-point algorithm)
![Page 19: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/19.jpg)
Results (normalized 8-point algorithm)
![Page 20: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/20.jpg)
From F to R, T
0' Fxx
If we know camera parameters
][TRE
Hartley and Zisserman, Multiple View Geometry, 2nd edition, pp 259
![Page 21: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/21.jpg)
Application: View morphing
![Page 22: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/22.jpg)
Application: View morphing
![Page 23: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/23.jpg)
Problem with morphing
• Without rectification
![Page 24: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/24.jpg)
![Page 25: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/25.jpg)
Main trick
• Prewarp with a homography to rectify images
• So that the two views are parallel• Because linear
interpolation works when views are parallel
![Page 26: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/26.jpg)
prewarp prewarp
morph morph
homographies
input inputoutput
![Page 27: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/27.jpg)
![Page 28: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/28.jpg)
Video demo
![Page 29: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/29.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
29
Triangulation
• Problem: Given some points in correspondence across two or more images (taken from calibrated cameras), {(uj,vj)}, compute the 3D location X
![Page 30: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/30.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
30
Triangulation• Method I: intersect viewing rays in 3D, minimize:
• X is the unknown 3D point
• Cj is the optical center of camera j
• Vj is the viewing ray for pixel (uj,vj)
• sj is unknown distance along Vj
• Advantage: geometrically intuitive
Cj
Vj
X
![Page 31: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/31.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
31
Triangulation• Method II: solve linear equations in X
• advantage: very simple
• Method III: non-linear minimization• advantage: most accurate (image plane error)
![Page 32: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/32.jpg)
Structure from motion
![Page 33: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/33.jpg)
Structure from motion
structure from motion: automatic recovery of camera motion and scene structure from two or more images. It is a self calibration technique and called automatic camera tracking or matchmoving.
UnknownUnknowncameracameraviewpointsviewpoints
![Page 34: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/34.jpg)
Applications
• For computer vision, multiple-view environment reconstruction, novel view synthesis and autonomous vehicle navigation.
• For film production, seamless insertion of CGI into live-action backgrounds
![Page 35: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/35.jpg)
Structure from motion
2D featurematching
3D estimation optimization(bundle adjust)
geometry fitting
SFM pipeline
![Page 36: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/36.jpg)
Structure from motion
• Step 1: Track Features• Detect good features, Shi & Tomasi, SIFT• Find correspondences between frames
– Lucas & Kanade-style motion estimation– window-based correlation– SIFT matching
![Page 37: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/37.jpg)
Structure from Motion
• Step 2: Estimate Motion and Structure• Simplified projection model, e.g., [Tomasi 92]• 2 or 3 views at a time [Hartley 00]
![Page 38: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/38.jpg)
Structure from Motion
• Step 3: Refine estimates• “Bundle adjustment” in photogrammetry• Other iterative methods
![Page 39: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/39.jpg)
Structure from Motion
• Step 4: Recover surfaces (image-based triangulation, silhouettes, stereo…)
Good mesh
![Page 40: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/40.jpg)
Example : Photo Tourism
![Page 41: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/41.jpg)
Factorization methods
![Page 42: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/42.jpg)
Problem statement
![Page 43: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/43.jpg)
Other projection models
![Page 44: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/44.jpg)
SFM under orthographic projection
2D image point
orthographicprojectionmatrix
3D scenepoint
imageoffset
tΠpq 12 32 13 12
• Trick• Choose scene origin to be centroid of 3D points• Choose image origins to be centroid of 2D points• Allows us to drop the camera translation:
Πpq
![Page 45: The fundamental matrix F](https://reader036.fdocuments.us/reader036/viewer/2022062305/5681489d550346895db5b28c/html5/thumbnails/45.jpg)
factorization (Tomasi & Kanade)
n332n2
n21n21 pppqqq
projection of n features in one image:
n3
32mn2m
212
1
21
22221
11211
n
mmnmm
n
n
ppp
Π
Π
Π
qqq
qqq
qqq
projection of n features in m images
W measurement M motion S shape
Key Observation: rank(W) <= 3