Class 41 Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics...
-
Upload
robyn-webster -
Category
Documents
-
view
214 -
download
0
Transcript of Class 41 Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics...
Class 4 1
Multi-linear Systems and Invariant Theory
in the Context of Computer Vision and Graphics
Class 4: Mutli-View 3D-from-2D
CS329Stanford University
Amnon Shashua
Class 4 2
Material We Will Cover Today
• Epipolar Geometry and Fundamental Matrix
• The plane+parallax model and relative affine structure
• Why 3 views?
• Trifocal Tensor
Class 4 3
PeneHp T ';''
'' epHp
TneHH '
PIp ]0;[
PeHp ]';['
'p
P
1
y
x
p
p
P
H Stands for the family of 2D projective transformations
between two fixed images induced by a plane in space
Reminder (from class 1):
Class 4 4
PIp ]0,[
p
p’
),1,,( yxP
e’
PeHp '' '' epHp
pH
• what does stand for?
• what would we obtain after eliminating
Plane + Parallax
Class 4 5
tKZ
pRKKp '1
'' 1
Reminder (from class 1):
1' RKKH
,'' tKe
1)1
(' Ktnd
RKH T
PKp ]0;[
PtRKp ][''
1
Z
Y
X
P
Class 4 6
tKZ
pRKKp '1
'' 1
'1
'1
' 1 eZ
pKned
pHp T
1)1
(' Ktnd
RKH T
')( 1
eZd
pZKndpH
T
Z
Y
X
KZ
p1
Recall:
)( 1 pZKndd T Let:
'' eZd
dpHp
'' epHp
Class 4 7
Note that He ,' are determined (each) up to a scale.
'00 , ppLet Be any “reference” point not arising from
'00'0 epHp
'1
00
'0 epHp
Let H
0
1 be the homography we will use
Class 4 8
'1
'00
epHp
0
0
0 d
d
Z
Z
Zd
d
Recall:
P0P
Z0Z
0d d
Class 4 9
Plane + Parallax
'' epHp
P0P
Z0Z
0d d
We have used 4 space points for a basis:3 for the reference plane1 for the reference point (scaling)
Since 4 points determinean affine basis:
is called “relative affine structure”
Note: we need 5 points for a projective basis. The 5th point is thefirst camera center.
0
0
d
d
Z
Z
Class 4 10
Note: A projective invariant
'' epHp
P0P
Z0Z
0d d
0
0
d
d
Z
Zd̂
0d̂
'ˆ' ˆ epHp
0
0
ˆ
ˆ
ˆ d
d
d
d
This invariant (“projective depth”) is independent of both camera positions, therefore is projective.
5 basis points: 4 non-coplanar defines two planes, andA 5th point for scaling.
Class 4 11
Note: An Affine Invariant
P0P
Z0Z
0d d
What happens when camera center is at infinity? (parallel projection)
0,
0
0
0 d
d
d
d
Z
ZZZ
This invariant is independent ofboth camera positions, and is Affine.
Class 4 12
p
p’
),1,,( yxP
e’
'' epHp
pH
Fundamental Matrix
2'' epHprank
0)'(' pHep T
0)]'([' pHep T 0' Fpp T
Class 4 13
Fundamental Matrix
0)]'([' pHep T
0' Fpp TDefines a bilinear matching constraint whose coefficientsdepend only on the camera geometry (shape was eliminated)
• F does not depend on the choice of the reference plane
HeneHeHe T ]'[)'(]'[]'[
Class 4 14
Epipoles from F
Note: any homography matrix maps between epipoles:
c
e
'c
'e'eeH
Class 4 15
Epipoles from F
0Fe 0']'[]'[ eeeHe
0'eF T 0']'[ eeH T
Class 4 16
Estimating F from matching points
0' iT
i Fpp 8,...,1i Linear solution
0' iT
i Fpp 7,...,1i
0)det( F
N on-linear solution
0)det( F is cubic in the elements of F, thus we should expect3 solutions.
Class 4 17
Estimating F from Homographies
FH T is skew-symmetric (i.e. provides 6 constraints on F)
HeHHeneHFH TTTT ]'[]'[)'(
HeHneHeHHF TTTT ]'[)'(]'[
HFFH TT
2 homography matrices are required for a solution for F
Class 4 18
F Induces a Homography
p
Fp
F][ is a homography matrix induced by the plane definedby the join of the image line and the camera center
Class 4 19
Projective Reconstruction
1. Solve for F via the system 0' iT
i Fpp (8 points or 7 points)
2. Solve for e’ via the system 0'eF T
3. Select an arbitrary vector 0'eT
4. 0I '][ eFand are a pair of camera matrices.
'][' eFpp
Class 4 20
Trifocal Geometry
The three fundamental matrices completely describe the trifocalgeometry (as long as the three camera centers are not collinear)
1
3
212e
32e
13e23e
21e
31e
32123112 eeeF
0311232 eFeT
Likewise: 0122313 eFeT
0211323 eFeT
Each constraint is non-linear in the entries of the fundamental matrices (because the epipoles are the respective null spaces)
Class 4 21
Trifocal Geometry
0311232 eFeT
0122313 eFeT
0211323 eFeT
3 fundamental matrices provide 21 parameters. Subtract 3 constraints,Thus we have that the trifocal geometry is determined by 18 parameters.
This is consistent with the straight-forward counting:
3x11 – 15 = 18
(3 camera matrices provide 33 parameters, minus the projective basis)
Class 4 22
What Goes Wrong with 3 views?
13 212e32e
13e
23e21e31e
2131 ee
3212 ee
2313 ee
2 constraints each, thus we have21-6=15 parameters
Class 4 23
What Goes Wrong with 3 views?
13 212e32e
13e
23e21e31e
3t2t 1t
213 ttt
Thus, to represent 3t we need only 1 parameter
(instead of 3).
18-2=16 parameters are needed to represent the trifocal geometry in this case.
but the pairwise fundamental matrices can account for only 15!
Class 4 24
What Else Goes Wrong: Reprojection
1
3
2p'p
''p
'23 pFpF13
''' 2313 pFpFp
Given p,p’ and the pairwise F-matsone can directly determine the positionof the matching point p’’
This fails when the 3 camera centers are collinearbecause all three line of sights are coplanarthus there is only one epipolar line!
Class 4 25
The Trifocal Constraints
PIp 0 PeAp '' PeBp ''''
'
0
1
1
x
s
'
1
0
2
y
s0'1 psT
0'2 psT 'p
1s
2s
''
0
1
1
x
r
''
1
0
2
y
r0''1 prT
0''2 prT
Class 4 26
PIp 0
PeAp ''
PeBp ''''
0'1 psT
0'2 psT
0''1 prT
0''2 prT
0'1 PeAsT
0'2 PeAsT
0''1 PeBrT
0''2 PeBrT
0001 Px
0010 Py
The Trifocal Constraints
Class 4 27
0
''
''
'
'
010
001
4622
11
22
11
P
erBr
erBr
esAs
esAs
y
x
TT
TT
TT
TT
Every 4x4 minor must vanish!
12 of those involve all 3 views, they are arranged in 3 groupsDepending on which view is the reference view.
The Trifocal Constraints
Class 4 28
''
''
'
'
010
001
22
11
22
11
erBr
erBr
esAs
esAs
y
x
TT
TT
TT
TT
The reference view
Choose 1 row from here
Choose 1 row from here
We should expect to have 4 matching constraints 0)'',',( pppfi
The Trifocal Constraints
Class 4 29
Expanding the determinants:
'' eApp 0' esAps Ti
Ti 2,1i
'''' eBpp 0'' erBpr Tj
Tj 2,1j
eliminate
''' er
Bpr
es
ApsTj
Tj
Ti
Ti
))('())(''( BpresApser Tj
Ti
Ti
Tj 2,1, ji
The Trifocal Constraints
Class 4 30
P
r
sC
C’
C’’
p
x
0
1
y
1
0
4 planes intersect at P !
What is going on geometrically:
)',( esAps TT 0' PeAsT is a plane
The Trifocal Constraints
Class 4 31
The Trifocal Tensor
))('())(''( BpresApser Tj
Ti
Ti
Tj
New index notations: i-image 1, j-image 2, k-image 3
0' esAps TT 0' jj
ijij espas
js is a line in image 2
ip is a point in image 1
je' is a point in image 2
Class 4 32
The Trifocal Tensor
0' jlj
iji
lj espas
ljs 2,1l are the two lines coincident with p’, i.e. 0' jl
j ps
mkr 2,1m are the two lines coincident with p’’, i.e. 0'' km
k pr
0'' kmk
iki
mk erpbr
Eliminate
0))(''())('( iji
lj
kmk
iki
mk
jlj paserpbres
Class 4 33
The Trifocal Tensor
0))(''())('( iji
lj
kmk
iki
mk
jlj paserpbres
Rearrange terms:
0)'''( ji
kki
jmk
lj
i aebersp
The trifocal tensor is:
ji
kki
jjki aebeT '''
2,1, ml
Class 4 34
x” Ti13pi - x”x’ Ti
33pi + x’ Ti31pi- Ti
11pi = 0y” Ti
13pi - y”x’ Ti33pi + x’ Ti
32pi- Ti12pi = 0
x” Ti23pi - x”y’ Ti
33pi + y’ Ti31pi- Ti
21pi = 0y” Ti
23pi - y”y’ Ti33pi + x’ Ti
32pi- Ti22pi = 0
0jki
mk
lj
i Trsp
'10
'01
y
xs j
l
''10
''01
y
xr k
m
The Trifocal Tensor
The four “trilinearities”:
Class 4 35
The Trifocal Tensor
21jjj sss
'p
2s
1ss
21kkk rrr
0))(( 2121 jkikkjj
ijkikj
i TrrsspTrsp
A trilinearity is a contraction with a point-line-line where the linesare coincident with the respective matching points.
Class 4 36
Slices of the Trifocal Tensor
Now that we have an explicit form of the tensor, what can we do with it?
?jkij
i TspThe result must be a contravariant vector (a point). This pointis coincident with r for all lines coincident with ''p
kjkij
i pTsp ''
The point reprojection equation (will work when camera centersare collinear as well).
'' pes
Note: reprojection is possible after observing 7 matching points,(because one needs 7 matching triplets to solve for the tensor).This is in contrast to reprojection using pairwise fundamental matricesWhich requires 8 matching points (in order to solve for the F-mats).
Class 4 37
Slices of the Trifocal Tensor
p3
21
s
''p
'p
kjkij
i pTsp ''
Class 4 38
Slices of the Trifocal Tensor
?jkikj Trs
The result must be a line.
ijk
ikj qTrs
rk
sj
O
O’
O’’
qi13 matching linesare necessary forsolving for the tensor(compared to 7 matching points)
Line reprojection equation
Class 4 39
Slices of the Trifocal Tensor
?jkikT
The result must be a matrix.
3
21
p
jkik
ji TH
HpH is a homography matrix
jkikT
is a family of homography matrices (from 1 to 2) induced by the family of planes coincidant with the 3rd camera center.
jkik
i Tp is the reprojection equation
Class 4 40
Slices of the Trifocal Tensor
jkijT is the homography matrix from 1 to 3 induced by the plane
defined by the image line and the second camera center.
?jki
iT
3
21
s
13F
jkij
i Ts is the reprojection equation
The result is a point on theepipolar line of on
image 3
Class 4 41
Slices of the Trifocal Tensor
jkjki
i GT
3
21
s
13F
Gs Is a point on the epipolar line 13F
2)( Grank
(because it maps the dual planeonto collinear points)
13)( FGnull
12)( FGnull T
Class 4 42
18 Parameters for the Trifocal Tensor
ji
kki
jjki aebeT '''
)'('')''(' ji
ji
kki
ki
j enaeenbe kj
ikj
ijk
i eeneenT '''''' jk
iTjk
iT Has 24 parameters (9+9+3+3)minus 1 for global scaleminus 2 for scaling e’,e’’ to be unit vectorsminus 3 for setting insuch that B has a vanishing column
n
= 18 independent parameters
We should expect to find 9 non-linear constraints among the27 entries of the tensor (admissibility constraints).
Class 4 43
18 Parameters for the Trifocal Tensor
What happens when the 3 camera centers are collinear?
(we saw that pairwise F-mats account for 15 parameters).
''' 11 eBeA 13 2
'e''e''1eB
'1eA
This provides two additional (non-linear) constraints, thus18-2=16.
Class 4 44
Items not Covered in Class
• Degenerate configurations (Linear Line Complex, Quartic Curve)
• The source of the 9 admissibility constraints (come from the homography slices).
• Concatenation of trifocal tensors along a sequence
• Quadrifocal tensor (and its relation to the homography tensor)