Computer Vision and Applications Prof. Trevor. Darrell
Transcript of Computer Vision and Applications Prof. Trevor. Darrell
![Page 1: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/1.jpg)
1
6.891
Computer Vision and Applications
Prof. Trevor. Darrell
Lecture 9: Affine SFM– Geometric Approach– Algebraic Approach– Tomasi/Kanade Factorization
Readings: F&P Ch. 12
![Page 2: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/2.jpg)
2
“Affine geometry is, roughly speaking, what is left after all ability to measure lengths, areas, angles, etc. has been removed from Euclidean geometry. The concept of parallelism remains, however, as well as the ability to measure the ratio of distances between collinear points.”
[Snapper and Troyer, 1989]
![Page 3: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/3.jpg)
3
![Page 4: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/4.jpg)
4
Affine camera
• DEFINE A
Tracked feature j in camera i:
![Page 5: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/5.jpg)
5
We define affine structure from motion as the problem of estimating the m2 × 4 matrices
and the npositions Pj of the points Pj in some fixed coordinate system from the mnimage correspondences pij
![Page 6: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/6.jpg)
6
titleThis equation provides 2mnconstraints on the 8m+3nunknown coefficients defining the matrices Mi and the point positions Pj.
Fortunately, 2mnis greater than 8m+3nfor large enough values of mand n…
But, the solution is ambiguous…
![Page 7: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/7.jpg)
7
and Q is an arbitrary affine transformation matrix, that is,
If Mi and Pj are solutions to
then so are M’i and P’j, where
where C is a non-singular 3 × 3 matrix and d is a vector in R3. In other words, any solution of the affine structure-from-motion problem can only defined up to an affine transformation ambiguity.
![Page 8: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/8.jpg)
8
Affine Structure from Motion
• Two views– Geometric Approach: infer affine shape (then
recover affine projection matricies if needed)– Algebraic Approach: estimate projection
matricies (then determine position of scene points)
• Sequence– Factorization Approach
![Page 9: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/9.jpg)
9
Affine Structure from Motion Theorem
Two affine views of four non co-planar points are sufficient to compute the affine coordinate of any other point P.
[Koenderink and Van Doorn, 1990]
![Page 10: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/10.jpg)
10
![Page 11: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/11.jpg)
11
Two affine views of four points are sufficient to compute the affine coordinate of any other point P…
![Page 12: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/12.jpg)
12
Given Affine Basis (A,B,C,D)(e.g., A=(0,0,0), B=(0,0,1), C=(0,1,0),
D=(1,0,0))
![Page 13: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/13.jpg)
13
Given Affine Basis (A,B,C,D)(e.g., A=(0,0,0), B=(0,0,1), C=(0,1,0),
D=(1,0,0))
And correspondencesa’,b’,c’,d’ anda’’,b’’,c’’,d’’…
![Page 14: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/14.jpg)
14
Given Affine Basis (A,B,C,D)(e.g., A=(0,0,0), B=(0,0,1), C=(0,1,0),
D=(1,0,0))
And correspondencesa’,b’,c’,d’ anda’’,b’’,c’’,d’’…
p’,p’’ uniquelydetermine the location of P in thebasis (A,B,C,D)!
?
![Page 15: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/15.jpg)
15
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…?
P=αAB+βAC+λAD
Find α,β,λ ?
![Page 16: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/16.jpg)
16
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using basis A,B,C
?
Find α,β,λ ?P=αAB+βAC+λAD
![Page 17: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/17.jpg)
17
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using (A,B,C)
?
Find α,β,λ ?P=αAB+βAC+λAD
compute λ=QP/ED=q’’p’’/e’’d’’
λ
![Page 18: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/18.jpg)
18
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using (A,B,C)
?
Find α,β,λ ?P=αAB+βAC+λAD
use coordinates of D, P in ABC…
λ
Compute λ=QP/ED=q’’p’’/e’’d’’Use coordinates of D, P in ABC…
![Page 19: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/19.jpg)
19
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using (A,B,C)
?
Find α,β,λ ?P=αAB+βAC+λAD
λ
Compute λ=QP/ED=q’’p’’/e’’d’’Use coordinates of D, P in ABC…
![Page 20: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/20.jpg)
20
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using (A,B,C)
?
Find α,β,λ ?P=αAB+βAC+λAD
λ
Compute λ=QP/ED=q’’p’’/e’’d’’Use coordinates of D, P in ABC…
![Page 21: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/21.jpg)
21
p’,p’’ uniquely determine the location of P in the
basis (A,B,C,D)…
Find e’’ and q’’ using (A,B,C)Find α,β,λ ?P=αAB+βAC+λAD
λ
Compute λ=QP/ED=q’’p’’/e’’d’’Use coordinates of D, P in ABC…
![Page 22: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/22.jpg)
22
Geometric Approach
p’,p’’ uniquely determined the location of P in thebasis (A,B,C,D)
AP was expressed using weighted combination of AB, AC, AD
Weights were determined by a’,a’’,b’,b’’,c’,c’’,d’,d’’,p’,p’’.
![Page 23: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/23.jpg)
23
Affine Structure from Motion
• Two views– Geometric Approach: infer affine shape (then
recover affine projection matricies if needed)– Algebraic Approach: estimate projection
matricies (then determine position of scene points)
• Sequence– Factorization Approach
![Page 24: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/24.jpg)
24
Algebraic approach
3-d P satisfies two affine views:
![Page 25: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/25.jpg)
25
But any affine transform of A is equally good…
for any affine transform
![Page 26: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/26.jpg)
26
Let’s pick a special C, d…
which is equivalent to choosing cannonicalaffine projection matrices
![Page 27: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/27.jpg)
27
and our determinant becomes very simple:
a,b,c,d can be estimated using least squares with a sufficient number of points. Then P can be recovered with:
![Page 28: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/28.jpg)
28
Affine Structure from Motion
• Two views– Geometric Approach: infer affine shape (then
recover affine projection matricies if needed)– Algebraic Approach: estimate projection
matricies (then determine position of scene points)
• Sequence– Factorization Approach
![Page 29: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/29.jpg)
29
Factorization Approach
Consider a sequence of affine cameras….
Stack affine projection equations:
![Page 30: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/30.jpg)
30
Form the (2m+1)n data matrix where each column is the observed data from one point:
![Page 31: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/31.jpg)
31
Form the (2m+1)n data matrix where each column is the observed data from one point:
Since
then
Rank(D) ¡= 4
![Page 32: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/32.jpg)
32
With an appropriate choice of origin (e.g., first point, centriod),
and the data matrix becomes:
![Page 33: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/33.jpg)
33
Rank of Object-relative Data Matrix
D = A P Data-Matrix = Affine-Motions x 3-d-
Points(2m x n) = (2m x 3) x (3 x n)
2m
n
= 2m
3
3 n
D is now rank 3
![Page 34: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/34.jpg)
34
Factorization algorithm
Given a data matrix,find Motion (A) and Shape (P) matrices that
generate that data…
Tomasi and Kanade Factorization algorithm (1992):
Use Singular Value Decomposition to factor D into appropriately sized A and P.
![Page 35: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/35.jpg)
35
SVD
![Page 36: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/36.jpg)
36
Factorization algorithm
![Page 37: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/37.jpg)
37
Factorization algorithm
Input
result
comparision
![Page 38: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/38.jpg)
38
Factorization algorithm
Can perform Euclidean upgrade to estimate metric quantities...– Of all the family of affine solutions, find the one that
obeys calibration constraints.
Extensions to basic algorithm:– sparse data– multiple motions– projective cameras (later)
![Page 39: Computer Vision and Applications Prof. Trevor. Darrell](https://reader036.fdocuments.us/reader036/viewer/2022070815/62c660ee90d76768192b85ef/html5/thumbnails/39.jpg)
39
Affine Structure from Motion
• Two views– Geometric Approach: infer affine shape (then
recover affine projection matricies if needed)– Algebraic Approach: estimate projection
matricies (then determine position of scene points)
• Sequence– Factorization Approach
[Most Figures from Forsythe and Ponce]