Last lecture
-
Upload
laurel-jacobson -
Category
Documents
-
view
41 -
download
0
description
Transcript of Last lecture
![Page 1: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/1.jpg)
Last lecture• Passive Stereo• Spacetime Stereo
![Page 2: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/2.jpg)
Today• Structure from Motion:
Given pixel correspondences, how to compute 3D structure and camera motion?
Slides stolen from Prof Yungyu Chuang
![Page 3: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/3.jpg)
Epipolar geometry & fundamental matrix
![Page 4: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/4.jpg)
The epipolar geometry
What if only C,C’,x are known?
![Page 5: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/5.jpg)
The epipolar geometry
C,C’,x,x’ and X are coplanar
epipolar geometry demo
![Page 6: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/6.jpg)
The epipolar geometry
All points on project on l and l’
![Page 7: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/7.jpg)
The epipolar geometry
Family of planes and lines l and l’ intersect at e and e’
![Page 8: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/8.jpg)
The epipolar geometry
epipolar plane = plane containing baselineepipolar line = intersection of epipolar plane with image
epipolar pole= intersection of baseline with image plane = projection of projection center in other image
epipolar geometry demo
![Page 9: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/9.jpg)
The fundamental matrix F
C C’T=C’-C
Rp p’
0)()( pTTXThe equation of the epipolar plane through X is
0)()'( pTpR
KXx
XxKp -1
T)-R(XK'x'T)-X(Rx'K'p' -1
![Page 10: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/10.jpg)
The fundamental matrix F
0)()'( pTpRSppT
00
0
xy
xz
yz
TTTT
TTS
0)()'( SppR0))('( SpRp
0' Epp essential matrix
![Page 11: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/11.jpg)
The fundamental matrix F
C C’T=C’-C
Rp p’
0' Epp
![Page 12: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/12.jpg)
The fundamental matrix F
0' Epp
Let M and M’ be the intrinsic matrices, then
xKp 1 ''' 1 xKp
0)()'( 11 xKExK'0' 1 xEKK'x
0' Fxx fundamental matrix
![Page 13: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/13.jpg)
The fundamental matrix F
C C’T=C’-C
Rp p’
0' Epp0' Fxx
![Page 14: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/14.jpg)
The fundamental matrix F
• The fundamental matrix is the algebraic representation of epipolar geometry
• The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images
0Fxx'T 0l'x'T
![Page 15: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/15.jpg)
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
1. Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)
2. Epipolar lines: l’=Fx & l=FTx’3. Epipoles: on all epipolar lines, thus e’TFx=0, x
e’TF=0, similarly Fe=04. F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)5. F maps from a point x to a line l’=Fx (not
invertible)
The fundamental matrix F
![Page 16: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/16.jpg)
The fundamental matrix F
• It can be used for – Simplifies matching– Allows to detect wrong matches
![Page 17: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/17.jpg)
Estimation of F — 8-point algorithm• The fundamental matrix F is defined by
0Fxx'for any pair of matches x and x’ in two images.
• Let x=(u,v,1)T and x’=(u’,v’,1)T,
333231
232221
131211
fffffffff
F
each match gives a linear equation
0'''''' 333231232221131211 fvfuffvfvvfuvfufvufuu
![Page 18: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/18.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
fffffffff
vuvvvvuuuvuu
vuvvvvuuuvuuvuvvvvuuuvuu
nnnnnnnnnnnn
• In reality, instead of solving , we seek f to minimize , least eigenvector of .
0AfAf AA
![Page 19: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/19.jpg)
8-point algorithm
• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to . 'FF 0'det F
• It is achieved by SVD. Let , where
, let
then is the solution.
VUF Σ
3
2
1
000000
Σ
0000000
Σ' 2
1
VUF Σ''
![Page 20: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/20.jpg)
8-point algorithm% Build the constraint matrix A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ... x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ... x1(1,:)' x1(2,:)' ones(npts,1) ]; [U,D,V] = svd(A); % Extract fundamental matrix from the column of V % corresponding to the smallest singular value. F = reshape(V(:,9),3,3)'; % Enforce rank2 constraint [U,D,V] = svd(F); F = U*diag([D(1,1) D(2,2) 0])*V';
![Page 21: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/21.jpg)
8-point algorithm• Pros: it is linear, easy to implement and fast• Cons: susceptible to noise
![Page 22: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/22.jpg)
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
fffffffff
vuvvvvuuuvuu
vuvvvvuuuvuuvuvvvvuuuvuu
nnnnnnnnnnnn
Problem with 8-point algorithm
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differencebetween column of data matrix least-squares yields poor results
![Page 23: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/23.jpg)
Normalized 8-point algorithm
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
normalized least squares yields good resultsTransform image to ~[-1,1]x[-1,1]
![Page 24: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/24.jpg)
Normalized 8-point algorithm1. Transform input by ,2. Call 8-point on to obtain3.
ii Txx ˆ 'i
'i Txx ˆ
'ii xx ˆ,ˆ
TFTF ˆΤ'F̂
0Fxx'
0ˆ'ˆ 1 xFTTx'
F̂
![Page 25: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/25.jpg)
Normalized 8-point algorithm
A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ... x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ... x1(1,:)' x1(2,:)' ones(npts,1) ]; [U,D,V] = svd(A); F = reshape(V(:,9),3,3)'; [U,D,V] = svd(F); F = U*diag([D(1,1) D(2,2) 0])*V'; % Denormalise F = T2'*F*T1;
[x1, T1] = normalise2dpts(x1);[x2, T2] = normalise2dpts(x2);
![Page 26: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/26.jpg)
Normalizationfunction [newpts, T] = normalise2dpts(pts)
c = mean(pts(1:2,:)')'; % Centroid newp(1,:) = pts(1,:)-c(1); % Shift origin to centroid. newp(2,:) = pts(2,:)-c(2); meandist = mean(sqrt(newp(1,:).^2 + newp(2,:).^2)); scale = sqrt(2)/meandist; T = [scale 0 -scale*c(1) 0 scale -scale*c(2) 0 0 1 ]; newpts = T*pts;
![Page 27: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/27.jpg)
RANSAC
repeatselect minimal sample (8 matches)compute solution(s) for Fdetermine inliers
until (#inliers,#samples)>95% or too many times
compute F based on all inliers
![Page 28: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/28.jpg)
Results (ground truth)
![Page 29: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/29.jpg)
Results (8-point algorithm)
![Page 30: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/30.jpg)
Results (normalized 8-point algorithm)
![Page 31: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/31.jpg)
From F to R, T
0' 1 xEMM'x0' Fxx
FMM'E If we know camera parameters
][TREHartley and Zisserman, Multiple View Geometry, 2nd edition, pp 259
![Page 32: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/32.jpg)
Application: View morphing
![Page 33: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/33.jpg)
Application: View morphing
![Page 34: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/34.jpg)
Main trick
• Prewarp with a homography to rectify images
• So that the two views are parallel• Because linear
interpolation works when views are parallel
![Page 35: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/35.jpg)
Problem with morphing
• Without rectification
![Page 36: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/36.jpg)
![Page 37: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/37.jpg)
prewarp prewarp
morph morph
homographies
input inputoutput
![Page 38: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/38.jpg)
![Page 39: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/39.jpg)
Video demo
![Page 40: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/40.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
40
Triangulation
• Problem: Given some points in correspondence across two or more images (taken from calibrated cameras), {(uj,vj)}, compute the 3D location X
![Page 41: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/41.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
41
Triangulation• Method I: intersect viewing rays in 3D, minimize:
• X is the unknown 3D point• Cj is the optical center of camera j• Vj is the viewing ray for pixel (uj,vj)
• sj is unknown distance along Vj
• Advantage: geometrically intuitive
Cj
Vj
X
![Page 42: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/42.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
42
Triangulation• Method II: solve linear equations in X
• advantage: very simple
• Method III: non-linear minimization• advantage: most accurate (image plane error)
![Page 43: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/43.jpg)
Structure from motion
![Page 44: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/44.jpg)
Structure from motion
structure from motion: automatic recovery of camera motion and scene structure from two or more images. It is a self calibration technique and called automatic camera tracking or matchmoving.
UnknownUnknowncameracameraviewpointsviewpoints
![Page 45: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/45.jpg)
Applications• For computer vision, multiple-view shape
reconstruction, novel view synthesis and autonomous vehicle navigation.
• For film production, seamless insertion of CGI into live-action backgrounds
![Page 46: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/46.jpg)
Structure from motion
2D featuretracking 3D estimation optimization
(bundle adjust)geometry fitting
SFM pipeline
![Page 47: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/47.jpg)
Structure from motion
• Step 1: Track Features• Detect good features, Shi & Tomasi, SIFT• Find correspondences between frames
– Lucas & Kanade-style motion estimation– window-based correlation– SIFT matching
![Page 48: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/48.jpg)
Structure from Motion
• Step 2: Estimate Motion and Structure• Simplified projection model, e.g., [Tomasi 92]• 2 or 3 views at a time [Hartley 00]
![Page 49: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/49.jpg)
Structure from Motion
• Step 3: Refine estimates• “Bundle adjustment” in photogrammetry• Other iterative methods
![Page 50: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/50.jpg)
Structure from Motion
• Step 4: Recover surfaces (image-based triangulation, silhouettes, stereo…)
Good mesh
![Page 51: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/51.jpg)
Example : Photo Tourism
![Page 52: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/52.jpg)
Factorization methods
![Page 53: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/53.jpg)
Problem statement
![Page 54: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/54.jpg)
Other projection models
![Page 55: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/55.jpg)
SFM under orthographic projection
2D image point
orthographicprojectionmatrix
3D scenepoint
imageoffset
tΠpq 12 32 13 12• Trick
• Choose scene origin to be centroid of 3D points• Choose image origins to be centroid of 2D points• Allows us to drop the camera translation:
Πpq
![Page 56: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/56.jpg)
factorization (Tomasi & Kanade)
n332n2
n21n21 pppqqq
projection of n features in one image:
n3
32mn2m
212
1
21
22221
11211
n
mmnmm
n
n
ppp
Π
ΠΠ
qqq
qqqqqq
projection of n features in m images
W measurement M motion S shape
Key Observation: rank(W) <= 3
![Page 57: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/57.jpg)
n33m2n2m''
SMW
• Factorization Technique– W is at most rank 3 (assuming no noise)– We can use singular value decomposition to
factor W:
Factorization
– S’ differs from S by a linear transformation A:
– Solve for A by enforcing metric constraints on M
))(('' ASMASMW 1
n33m2n2m SMWknown solve for
![Page 58: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/58.jpg)
Metric constraints
• Orthographic Camera• Rows of are orthonormal:
• Enforcing “Metric” Constraints• Compute A such that rows of M have these properties
MAM '
10
01Tii
Trick (not in original Tomasi/Kanade paper, but in followup work)
• Constraints are linear in AAT :
• Solve for G first by writing equations for every i in M• Then G = AAT by SVD
Tii
Tiiii where AAGGAA
TTT ''''1001
![Page 59: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/59.jpg)
Results
![Page 60: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/60.jpg)
Extensions to factorization methods• Paraperspective [Poelman & Kanade, PAMI 97]• Sequential Factorization [Morita & Kanade, PAMI 97]• Factorization under perspective [Christy & Horaud,
PAMI 96] [Sturm & Triggs, ECCV 96]• Factorization with Uncertainty [Anandan & Irani, IJCV
2002]
![Page 61: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/61.jpg)
Bundle adjustment
![Page 62: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/62.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
62
Structure from motion
• How many points do we need to match?• 2 frames:
(R,t): 5 dof + 3n point locations 4n point measurements n 5
• k frames:6(k–1)-1 + 3n 2kn
• always want to use many more
![Page 63: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/63.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
63
Bundle Adjustment
• What makes this non-linear minimization hard?• many more parameters: potentially slow• poorer conditioning (high correlation)• potentially lots of outliers
![Page 64: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/64.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
64
Lots of parameters: sparsity
• Only a few entries in Jacobian are non-zero
![Page 65: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/65.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
65
Robust error models• Outlier rejection
• use robust penalty appliedto each set of jointmeasurements
• for extremely bad data, use random sampling [RANSAC, Fischler & Bolles, CACM’81]
![Page 66: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/66.jpg)
Richard Szeliski CSE 576 (Spring 2005): Computer Vision
67
Structure from motion: limitations• Very difficult to reliably estimate metric
structure and motion unless:• large (x or y) rotation or• large field of view and depth variation
• Camera calibration important for Euclidean reconstructions
• Need good feature tracker• Lens distortion
![Page 67: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/67.jpg)
Issues in SFM• Track lifetime• Nonlinear lens distortion• Prior knowledge and scene constraints• Multiple motions
![Page 68: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/68.jpg)
Track lifetime
every 50th frame of a 800-frame sequence
![Page 69: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/69.jpg)
Track lifetime
lifetime of 3192 tracks from the previous sequence
![Page 70: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/70.jpg)
Track lifetime
track length histogram
![Page 71: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/71.jpg)
Nonlinear lens distortion
![Page 72: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/72.jpg)
Nonlinear lens distortion
effect of lens distortion
![Page 73: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/73.jpg)
Prior knowledge and scene constraints
add a constraint that several lines are parallel
![Page 74: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/74.jpg)
Prior knowledge and scene constraints
add a constraint that it is a turntable sequence
![Page 75: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/75.jpg)
Applications of Structure from Motion
![Page 76: Last lecture](https://reader034.fdocuments.us/reader034/viewer/2022051518/56813886550346895da0391a/html5/thumbnails/76.jpg)
Jurassic park