Post on 07-Oct-2020
Lecture 3 -Silvio Savarese & Jeanette Bohg 20-January-2021
Professor Silvio SavareseStanford Vision and Learning Lab
Lecture 3Camera Models 2 &Camera Calibration
Lecture 3 -Silvio Savarese & Jeanette Bohg 20-January-2021
• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example
Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li
Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”
Lecture 3Camera Models 2 &Camera Calibration
P '3×1 =M Pw = K3×3 R T"#
$% 3×4
Pw4×1
=
m1
m2
m3
!
"
####
$
%
&&&&
PW =m1PWm2PWm3PW
!
"
####
$
%
&&&&
→ P 'E = (m1Pwm3Pw
, m2Pwm3Pw
)E
f
Projective camera
Ow
kw
jwR,T
úúú
û
ù
êêê
ë
é=
3
2
1
mmm
M
Exercise!
Ow
iw
kw
jwR,T
M = K R T!"
#$
= (m1Pwm3Pw
, m2Pwm3Pw
)
P
P’
f
P 'E = ( f xwzw, f yw
zw)
=
f 0 0 00 f 0 00 0 1 0
!
"
###
$
%
&&&
Pw =
xwywzw1
!
"
#####
$
%
&&&&&
à
= K I 0!"
#$
Canonical Projective Transformation
P ' =xyz
!
"
####
$
%
&&&&
=1 0 0 00 1 0 00 0 1 0
!
"
###
$
%
&&&
xyz1
!
"
####
$
%
&&&&
xzyz
!
"
####
$
%
&&&&
Pi ' =
P ' =M P
M3
H4 ®Â
Projective camera
p’q’r’ O
f’R
Q
P
Weak perspective projection
R
Q
P
O
f’
When the relative scene depth is small compared to its distance from the camera
zo
p’q’r’ Q_
P_
R_
π
Weak perspective projection
O
f’ zo
p’q’r’ Q_
P_
R_
x ' = f 'zx
y ' = f 'zy
!
"
###
$
###
→
x ' = f 'z0x
y ' = f 'z0y
!
"
##
$
##
Magnification m
R
Q
Pπ
úû
ùêë
é=
10bA
M[ ]TRKM = úû
ùêë
é=
1vbA
O
f’ zo
p’q’r’ Q_
P_
R_R
Q
P
π
à
Projective (perspective) Weak perspective
Weak perspective projection
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é=
10002
1
3
2
1
mm
mmm
=
m1
m2
m3
!
"
####
$
%
&&&&
PW
)P,P( 21 ww mm®E
wPMP =¢úû
ùêë
é=
10bA
M
magnification
úúú
û
ù
êêê
ë
é=
3
2
1
mmm
wPMP =¢ úû
ùêë
é=
1vbA
Múúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é=
W3
W2
W1
W
3
2
1
PPP
Pmmm
mmm
)PP,
PP(
3
2
3
1
w
w
w
w
mm
mm
®E
Perspective
Weak perspective
=
m1 PWm2 PW1
!
"
####
$
%
&&&&
Orthographic (affine) projectionDistance from center of projection to image plane is infinite
x ' = f 'zx
y ' = f 'zy
!
"
###
$
###
→x ' = xy ' = y
!"#
$#
Pros and Cons of These Models
• Weak perspective results in much simpler math.– Accurate when object is small and distant.– Most useful for recognition.
• Pinhole perspective is much more accurate for modeling the 3D-to-2D mapping.– Used in structure from motion or SLAM.
One-point perspective• Masaccio, Trinity,
Santa Maria Novella, Florence, 1425-28
Credit slide S. Lazebnik
The Kangxi Emperor's Southern Inspection Tour (1691-1698)By Wang Hui
Weak perspective projection
Weak perspective projection
The Kangxi Emperor's Southern Inspection Tour (1691-1698)By Wang Hui
Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21
• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example
Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li
Lecture 3Camera Calibration
Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”
Projective camera
úúú
û
ù
êêê
ë
é -
=
100v0
ucot
K o
o
sinqb
qaa
úúú
û
ù
êêê
ë
é
=T3
T2
T1
Rrrr
úúú
û
ù
êêê
ë
é
=
z
y
x
ttt
T
43´
wPMP =¢ [ ] wPTRK=Internal parameters
External parameters
Goal of calibration
Estimate intrinsic and extrinsic parameters from 1 or multiple images
Change notation:P = Pw
p = P’
wPMP =¢ [ ] wPTRK=Internal parameters
External parameters
Calibration Problem
•P1… Pn with known positions in [Ow,iw,jw,kw]
jC
Calibration rig
Calibration Problem
•P1… Pn with known positions in [Ow,iw,jw,kw]•p1, … pn known positions in the image
jC
Calibration rig
pi
Goal: compute intrinsic and extrinsic parameters
pi
image
image
Calibration Problem
jC
Calibration rig
How many correspondences do we need?• M has 11 unknowns• We need 11 equations • 6 correspondences would do it
pi
jCpi
pi
image
Calibration Problem
jC
Calibration rig
In practice, using more than 6 correspondences enables more robust results
pi
jCpi
pi
image
Calibration Problem
jC
Calibration rig
pi
jCpi
pi
image
=M Pipi =uivi
!
"##
$
%&&
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
mmmm
in pixels[Eq. 1]
úúú
û
ù
êêê
ë
é=
3
2
1
Mmmm
Calibration Problem
i3
i1i P
Pumm
=
i2i3i P)P(v mm =®
i1i3i P)P(u mm =®
i3
i2i P
Pvmm
=
úû
ùêë
é
i
i
vu
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
mmmm
0P)P(v 23 =-® iii mm
→ ui (m3 Pi )−m1 Pi = 0
[Eq. 1]
[Eqs. 2]
Calibration Problem
……
0)( 12131 =- PPv mm
0)( 11131 =- PPu mm
0)( 23 =- iii PPv mm
0)( 13 =- iii PPu mm
0)( 23 =- nnn PPv mm
0)( 13 =- nnn PPu mm
[Eqs. 3]
Block Matrix Multiplication
úû
ùêë
é=ú
û
ùêë
é=
2221
1211
2221
1211
BBBB
BAAAA
A
What is AB ?
úû
ùêë
é++++
=2222122121221121
2212121121121111
BABABABABABABABA
AB
÷÷÷
ø
ö
ççç
è
æ
=T3
T2
T1def
mmm
mP = def
P1T
0T0T
P1T
−u1P1T
−v1P1T
!PnT
0T0T
PnT
−unPnT
−vnPnT
"
#
$$$$$$$$
%
&
''''''''
Calibration Problem
2n x 1212x1
1x44x1
…
Homogenous linear system
knownunknown
0)( 12131 =+- PPv mm
0)( 11131 =+- PPu mm
0)( 23 =+- nnn PPv mm
0)( 13 =+- nnn PPu mm
0=P m [Eq. 4]
Homogeneous M x N Linear Systems
P m 0=
Rectangular system (M>N)
• 0 is always a solution
Minimize |P m|2
under the constraint |m|2 =1
M=number of equations = 2nN=number of unknown = 11
• To find non-zero solution
N
M
• How do we solve this homogenous linear system?
Calibration Problem
• Via SVD decomposition!
0=P m
Calibration Problem
1212T
121212n2 VDU ´´´
Last column of V gives m
M
SVD decomposition of P
Why? See pag 592 of HZ
÷÷÷
ø
ö
ççç
è
æ
=T3
T2
T1def
mmm
m
0=P m
Extracting camera parameters
M = ρ
Extracting camera parameters
A
úúú
û
ù
êêê
ë
é
=T3
T2
T1
Aaaa
[ ]TRK=
3
1a±
=r
úúú
û
ù
êêê
ë
é
=
3
2
1
bbb
b
Estimated values
)( 312 aa ×= rou
)(v 322
o aa ×= r
( ) ( )3231
3231cosaaaaaaaa
´×´´×´
=q
Intrinsic
b úúú
û
ù
êêê
ë
é -
=
100v0
ucot
K o
o
sinqb
qaarM =
Box 1
See [FP], Sec. 1.3.1
Extracting camera parameters
úúú
û
ù
êêê
ë
é
=T3
T2
T1
Aaaa
[ ]TRK=
úúú
û
ù
êêê
ë
é
=
3
2
1
bbb
b
Estimated values
Intrinsic
qra sin312 aa ´=
qrb sin322 aa ´=
A brM =
úúú
û
ù
êêê
ë
é -
=
100v0
ucot
K o
o
sinqb
qaa
Box 1
See [FP], Sec. 1.3.1
Extracting camera parameters
úúú
û
ù
êêê
ë
é
=T3
T2
T1
Aaaa
[ ]TRK=
úúú
û
ù
êêê
ë
é
=
3
2
1
bbb
b
Estimated values
A brM =
úúú
û
ù
êêê
ë
é -
=
100v0
ucot
K o
o
sinqb
qaa
Box 1
See [FP], Sec. 1.3.1
Extrinsic( )
32
321 aa
aar´´
=3
33 a
ar ±=
132 rrr ´= b1KT -= r
Degenerate cases
•Pi’s cannot lie on the same plane!• Points cannot lie on the intersection curve of two
quadric surfaces [FP] section 1.3
Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21
• Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• Example
Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li
Lecture 3Camera Calibration
Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”
No distortion
Pin cushion
Barrel
Radial Distortion– Image magnification (in)decreases with distance from the optical axis– Caused by imperfect lenses– Deviations are most noticeable for rays that pass through the edge of
the lens
Radial Distortion
ii
ii p
vu
PM
100
00
00
1
1
=úû
ùêë
é®
úúúúúú
û
ù
êêêêêê
ë
é
l
ld
v
vucvbuad 222 ++=
u
To model radial behavior
å±==
3
1p
2ppdκ1λ
Polynomial function
Distortion coefficient
[Eq. 5] [Eq. 6]
Sλ
Image magnification decreases with distance from the optical center
How do we model that?
úû
ùêë
é=
i
ii v
up
úúú
û
ù
êêê
ë
é=
3
2
1
Qqqq
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
qqqq
ii
ii p
vu
PM
100
00
00
1
1
=úû
ùêë
é®
úúúúúú
û
ù
êêêêêê
ë
é
l
l
Q
uiq3 Pi = q1 Piviq3 Pi = q2 Pi
!"#
$#
Is this a linear system of equations?
Radial Distortion
No! why? [Eqs.7]
X = f (Q)
measurements parameters
f( ) is the nonlinear mapping
-Newton Method-Levenberg-Marquardt Algorithm
• Iterative, starts from initial solution • May be slow if initial solution far from real solution • Estimated solution may be function of the initial solution (because of local minima)
• Newton requires the computation of J, H• Levenberg-Marquardt doesn’t require the computation of H
General Calibration Problem
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
qqqq
úû
ùêë
é
i
i
vu
i=1…n
[Eq .8]
A possible algorithm
1. Solve linear part of the system to find approximated solution 2. Use this solution as initial condition for the full system3. Solve full system using Newton or L.M.
General Calibration Problem
X = f (Q)
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
qqqq
úû
ùêë
é
i
i
vu
i=1…n
measurements parameters
[Eq .8]
f( ) is the nonlinear mapping
Typical assumptions:
- zero-skew, square pixel- uo, vo = known center of the image
General Calibration Problem
measurements parameters
f( ) is the nonlinear mapping
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
qqqq
úû
ùêë
é
i
i
vu X = f (Q) [Eq .8]
i=1…n
Can we estimate m1 and m2 and ignore the radial distortion?
Radial Distortion
úû
ùêë
é=
i
ii v
up
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
1
mmmm
l
d
v
uHint: slopevu
i
i =
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
qqqq
Estimating m1 and m2…
Radial Distortion
úû
ùêë
é=
i
ii v
up
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
1
mmmm
l
0)()( 121111 =- PuPv mm0)()( 21 =- iiii PuPv mm
0)()( 21 =- nnnn PuPv mm…
L n = 0
n =m1
T
m2T
!
"
##
$
%
&&
Tsai [87]
i
i
i
i
i
i
i
i
PP
PPPP
vu
2
1
3
2
3
1
)()()()(
mm
mmmm
==
Get m1 and m2 by SVD
[Eq .9]
[Eq .10][Eq .11]
L = def
v1P1T −u1P1
T
v2P2T −u2P2
T
! !vnPn
T −unPnT
"
#
$$$$$$
%
&
''''''
Once that m1 and m2 are estimated…
Radial Distortion
úû
ùêë
é=
i
ii v
up
úúúú
û
ù
êêêê
ë
é
=
i3
i2
i3
i1
PPPP
1
mmmm
l
3m is non linear function of 1m 2m l
There are some degenerate configurations for which m1 and m2 cannot be computed
, ,
Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21
Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li
Lecture 3Camera Calibration
• Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• ExampleReading: [FP] Chapter 1 “Geometric Camera Calibration”
[HZ] Chapter 7 “Computation of Camera Matrix P”
Calibration ProcedureCamera Calibration Toolbox for MatlabJ. Bouguet – [1998-2000]
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples
Calibration Procedure
Calibration Procedure
Calibration Procedure
Calibration Procedure
Calibration Procedure
Calibration Procedure
Calibration Procedure
Next lecture
•Single view reconstruction