More on single-view geometry class 10 Multiple View Geometry Comp 290-089 Marc Pollefeys.
Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.
-
Upload
lauren-chriswell -
Category
Documents
-
view
229 -
download
0
Transcript of Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.
![Page 1: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/1.jpg)
ComputerVision
Multiple View Geometry
Marc PollefeysCOMP 256
![Page 2: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/2.jpg)
ComputerVision
2
Last class
Gaussian pyramid
Laplacian pyramid
Gaborfilters
Fouriertransform
![Page 3: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/3.jpg)
ComputerVision
3
Histograms : co-occurrence matrix Not last class …
![Page 4: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/4.jpg)
ComputerVision
4
Texture synthesis [Zalesny & Van Gool 2000]
2 analysis iterations
6 analysis iterations
9 analysis iterations
![Page 5: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/5.jpg)
ComputerVision
5
View-dependent texture synthesis[Zalesny & Van Gool 2000]
![Page 6: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/6.jpg)
ComputerVision
6
Efros & Leung ’99
• Assuming Markov property, compute P(p|N(p))– Building explicit probability tables infeasible
– Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p
• To synthesize p, just pick one match at random
pp
non-parametricsampling
Input image Synthesizing a pixel
![Page 7: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/7.jpg)
ComputerVision
7
pp
Efros & Leung ’99 extended
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
![Page 8: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/8.jpg)
ComputerVision
8
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
![Page 9: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/9.jpg)
ComputerVision
9min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
![Page 10: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/10.jpg)
ComputerVision
10
![Page 11: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/11.jpg)
ComputerVision
11
![Page 12: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/12.jpg)
ComputerVision
12
![Page 13: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/13.jpg)
ComputerVision
13
Why do we see more flowers in the distance?
Perpendicular textures
[Leung & Malik CVPR97]
![Page 14: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/14.jpg)
ComputerVision
14
Shape-from-texture
![Page 15: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/15.jpg)
ComputerVision
15
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
![Page 16: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/16.jpg)
ComputerVision
16
THE GEOMETRY OF MULTIPLE VIEWS
Reading: Chapter 10.
• Epipolar Geometry
• The Essential Matrix
• The Fundamental Matrix
• The Trifocal Tensor
• The Quadrifocal Tensor
![Page 17: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/17.jpg)
ComputerVision
17
Epipolar Geometry
• Epipolar Plane
• Epipoles
• Epipolar Lines
• Baseline
![Page 18: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/18.jpg)
ComputerVision
18
Epipolar Constraint
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
![Page 19: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/19.jpg)
ComputerVision
19
Epipolar Constraint: Calibrated Case
Essential Matrix(Longuet-Higgins, 1981)
![Page 20: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/20.jpg)
ComputerVision
20
Properties of the Essential Matrix
• E p’ is the epipolar line associated with p’.
• ETp is the epipolar line associated with p.
• E e’=0 and ETe=0.
• E is singular.
• E has two equal non-zero singular values (Huang and Faugeras, 1989).
T
T
![Page 21: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/21.jpg)
ComputerVision
21
Epipolar Constraint: Small MotionsTo First-Order:
Pure translation:Focus of Expansion
![Page 22: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/22.jpg)
ComputerVision
22
Epipolar Constraint: Uncalibrated Case
Fundamental Matrix(Faugeras and Luong, 1992)
![Page 23: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/23.jpg)
ComputerVision
23
Properties of the Fundamental Matrix
• F p’ is the epipolar line associated with p’.
• FT p is the epipolar line associated with p.
• F e’=0 and FT e=0.
• F is singular.
T
T
![Page 24: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/24.jpg)
ComputerVision
24
The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F| =1.
Minimize:
under the constraint
2
![Page 25: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/25.jpg)
ComputerVision
25
Non-Linear Least-Squares Approach (Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an appropriate rank-2 parameterization.
![Page 26: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/26.jpg)
ComputerVision
26
Problem with eight-point algorithm
linear least-squares: unit norm vector F yielding smallest residual
What happens when there is noise?
![Page 27: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/27.jpg)
ComputerVision
27
The Normalized Eight-Point Algorithm (Hartley, 1995)
• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’.
• Use the eight-point algorithm to compute F from thepoints q and q’ .
• Enforce the rank-2 constraint.
• Output T F T’.T
i i i i
i i
![Page 28: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/28.jpg)
ComputerVision
28
Epipolar geometry example
![Page 29: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/29.jpg)
ComputerVision
29
Example: converging cameras
courtesy of Andrew Zisserman
![Page 30: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/30.jpg)
ComputerVision
30
Example: motion parallel with image plane
(simple for stereo rectification)courtesy of Andrew Zisserman
![Page 31: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/31.jpg)
ComputerVision
31
Example: forward motion
e
e’
courtesy of Andrew Zisserman
![Page 32: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/32.jpg)
ComputerVision
32
Fundamental matrix for pure translation
courtesy of Andrew Zisserman
auto-epipolar
![Page 33: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/33.jpg)
ComputerVision
33
Fundamental matrix for pure translation
courtesy of Andrew Zisserman
![Page 34: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/34.jpg)
ComputerVision
34
Trinocular Epipolar Constraints
These constraints are not independent!
![Page 35: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/35.jpg)
ComputerVision
35
Trinocular Epipolar Constraints: Transfer
Given p and p , p can be computed
as the solution of linear equations.
1 2 3
![Page 36: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/36.jpg)
ComputerVision
36
• problem for epipolar transfer in trifocal plane!
image from Hartley and Zisserman
Trinocular Epipolar Constraints: Transfer
There must be more to trifocal geometry…
![Page 37: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/37.jpg)
ComputerVision
37
Trifocal Constraints
![Page 38: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/38.jpg)
ComputerVision
38
Trifocal Constraints
All 3x3 minorsmust be zero!
Calibrated Case
Trifocal Tensor
![Page 39: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/39.jpg)
ComputerVision
39
Trifocal Constraints
Uncalibrated Case
Trifocal Tensor
![Page 40: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/40.jpg)
ComputerVision
40
Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p .
Do it again.2 3 2 3
T( p , p , p )=01 2 3
![Page 41: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/41.jpg)
ComputerVision
41
Properties of the Trifocal Tensor
Estimating the Trifocal Tensor
• Ignore the non-linear constraints and use linear least-squares
• Impose the constraints a posteriori.
• For any matching epipolar lines, l G l = 0.
• The matrices G are singular.
• They satisfy 8 independent constraints in theuncalibrated case (Faugeras and Mourrain, 1995).
2 1 3T i
1i
![Page 42: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/42.jpg)
ComputerVision
42
For any matching epipolar lines, l G l = 0. 2 1 3T i
The backprojections of the two lines do not define a line!
![Page 43: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/43.jpg)
ComputerVision
43
108 putative matches 18 outliers
88 inliers 95 final inliers
(26 samples)
(0.43)(0.23)
(0.19)
TrifocalTensor
Example
courtesy of Andrew Zisserman
![Page 44: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/44.jpg)
ComputerVision
44
additional line matches
TrifocalTensor
Example
images courtesy of Andrew Zisserman
![Page 45: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/45.jpg)
ComputerVision
45
Transfer: trifocal transfer
doesn’t work if l’=epipolar line
image courtesy of Hartley and Zisserman
(using tensor notation)
![Page 46: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/46.jpg)
ComputerVision
46
Image warping using T(1,2,N)(Avidan and Shashua `97)
![Page 47: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/47.jpg)
ComputerVision
47
Multiple Views (Faugeras and Mourrain, 1995)
![Page 48: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/48.jpg)
ComputerVision
48
Two Views
Epipolar Constraint
![Page 49: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/49.jpg)
ComputerVision
49
Three Views
Trifocal Constraint
![Page 50: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/50.jpg)
ComputerVision
50
Four Views
Quadrifocal Constraint(Triggs, 1995)
![Page 51: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/51.jpg)
ComputerVision
51
Geometrically, the four rays must intersect in P..
![Page 52: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/52.jpg)
ComputerVision
52
Quadrifocal Tensorand Lines
![Page 53: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/53.jpg)
ComputerVision
53
Quadrifocal tensor
• determinant is multilinear
thus linear in coefficients of lines !
• There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor
wi
wi
yi
yi
xi
xii
Ti MlMlMl Ml
Twi
yi
xi lll ][
s
r
q
p
pqrs
MMMM
Q
4
3
2
1
det
![Page 54: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/54.jpg)
ComputerVision
55
from perspective to omnidirectional cameras
perspective camera(2 constraints / feature)
radial camera (uncalibrated)(1 constraints / feature)
3 constraints allow to reconstruct 3D point
more constraints also tell something about cameras
multilinear constraints known as epipolar, trifocal and quadrifocal constraints
(0,0)
l=(y,-x)
(x,y)
![Page 55: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/55.jpg)
ComputerVision
56
Radial quadrifocal tensor
• Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
• Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
(2x2x2 tensor)
Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation• Radial trifocal tensor Tijk from 7 points in 3 views
• Reconstruct 2D panorama and use it for calibration
(x,y)
![Page 56: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/56.jpg)
ComputerVision
57
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radius
angle
• Models fish-eye lenses, cata-dioptric systems, etc.
![Page 57: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/57.jpg)
ComputerVision
58
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radiusangle
90o
• Models fish-eye lenses, cata-dioptric systems, etc.
![Page 58: Computer Vision Multiple View Geometry Marc Pollefeys COMP 256.](https://reader036.fdocuments.us/reader036/viewer/2022081511/551a529655034643688b464c/html5/thumbnails/58.jpg)
ComputerVision
59
Next class:Stereo
F&PChapter 11
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)