Ch. 3: Geometric Camera Calibration
description
Transcript of Ch. 3: Geometric Camera Calibration
1
Ch. 3: Geometric Camera Calibration Objective: Estimates the intrinsic and extrinsic parameters of a camera.Idea: Formulate camera calibration as an optimization process, in which the discrepancy between the theoretical and observed image features is minimized w.r.t. the camera’s parameters.Steps: (1) Evaluate the perspective projection matrix M of the camera, (2) Estimate the intrinsic and extrinsic parameters of the camera from M.
2
,Mp P
。 Perspective Projection (Imaging Process)
practically,
where
ideally,
3
p PM11 12 13 14
21 22 23 24
31 32 33 34 3 4
m m m m
M m m m m
m m m m
○ Evaluate M
Let
Measure n pairs ( , )p Pi i of corresponding
image and scene points.
4
( , ),i ip P
11 12 13 14
21 22 23 24
31 32 33 3411
ii
ii
i
xu m m m m
yv m m m m
zm m m m
11 12 13 14
21 22 23 24
31 32 33 34
,
1
i i i i
i i i i
i i i
m x m y m z m u
m x m y m z m v
m x m y m z m
1, ,i n
For each pair
we obtain
For all pairs,
5
11 1 12 1 13 1 14 1
21 1 22 1 23 1 24 1
31 1 32 1 33 1 34
11 12 13 14
21 22 23 24
31 32 33
1
n n n n
n n n n
n n
m x m y m z m u
m x m y m z m v
m x m y m z m
m x m y m z m u
m x m y m z m v
m x m y m
34 1nz m
,U x yIn matrix form, where
11 12 13 14 31 32 33 34( )x Tm m m m m m m m
1 1 2 2( 1 1 1)Tn nu v u v u v y
6
1 1 1
1 1 1
1 1 1
2 2 2
2 2 2
2 2 2
1 0 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 1
x y z
x y z
x y z
x y z
x y zU
x y z
1 0 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 1
n n n
n n n
n n n
x y z
x y z
x y z
Solve for x using optimization techniques
7
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
q q
q q
p p pq q p
u x u x u x y
u x u x u x y
u x u x u x y
3.1 Least-Squares Parameter Estimation
3.1.1 Linear Least-Squares Methods ○ Consider a system of p linear equations in q unknowns:
U x yIdea: Find the solution x by minimizing the squared deviation ( ) from theoretical (Ux) to observed (y) image features
2U x y
8
11 12 1
21 22 2
1 2
q
q
p p pq
u u u
u u uU
u u u
1
2 x
q
x
x
x
1
2 y
p
y
y
y
,
In vector-matrix form, , wherex yU
9
2x yE U
2x y
x x
dE dU
d d
( )2 ( ) 2 ( )
TTd U
U U Ud
0
x yx y x y
x
( ) , ,x y x y x y T T T T TU U U U U U U U 0 01( )x y yT TU U U U
1( )T TU U U U
Let
○ Consider the over-constrained case (p > q) Find x that minimizes the error
The normal equations
: the pseudoinverse of U.
10
xU 0
U x yx 0
1x
U 0xTU U
○ Homogenous systems:
Two issues: (i) By equation , we obtain trivial
(ii) If x is a solution, x is also a
solution.
。 The least squares error solution of
is the eigenvector of
to the smallest eigenvalue.
solution
To resolve the issues, impose
corresponding
11
1x 2
( ) ( ) ( 1)x x x xT TF U U
( )2( ) 2
xx x
xTdF
U Ud
0
( )TU U x xTU U
。 Find the least squares error solution by the method of Lagrange multipliers Error:
We obtain
The solution x is an eigenvector of with eigenvalue
: Lagrange multiplier
Let
2 2( )x x x xT TE U U U U 0
Constraint
Minimize
where
12
( )T T TE U U x x x x xU 0
TU U
The associated error
The least squares error solution to
is the eigenvector of
to the smallest eigenvalue.
corresponding
Example:
Fit a line to a set of data points in the 2D space
13
2 2 2 2 2 2
a b dx y
a b a b a b
a x b y d
2 2 2 2 2 2, ,
a b da b d
a b a b a b
Line equation:
Let
ax by d 2 2 1.a b
14
ax by d ( , )i ix y
| |i iax by d 2
1
( , , ) ( )n
i ii
E a b d ax by d
The perpendicular distance from point
to line
Error measure:
Minimize E w.r.t. (a, b, d)
1
2 ( ) 0n
i ii
Eax by d
d
1
( ) 0,n
i ii
ax by nd
Let
is
1
( )n
i ii
nd ax by
1 1 1
1 1( ) ( )
n n n
i i i ii i i
d ax by ax by ax byn n
15
2 2
1 1
22
1
( ) ( ( ))
[ ( ) ( )] n
n n
i i i ii i
n
i ii
E ax by d ax by ax by
a x x b y y U
,na
b
1 1
n n
x x y y
U
x x y y
where
xU 02
xE U2
Un
1n TU U
。 Recall , whose squared error
The solution of min w.r.t. n under
is the unit eigenvectorconstraint
with the minimum eigenvalue of
16
1 1 2
2 1 2
1 2
( , , , ) 0
( , , , ) 0( )
( , , , ) 0
f x
q
q
p q
f x x x
f x x x
f x x x
0
1 1( , , , ) ,f Tpf f f 1 2( , , , )T
qx x x x
where
3.1.2 Nonlinear Least-Squares Methods
(0,0, ,0)T 0
17
2( ) 3 4 2, ( ) 6 4f x x x f x x
( )( )
df xf x
dx f(x):
11
11
( ) ( ) ( )( )
( ) ( ) ( )
nn
nn
df f ff
d x x
f fx x
x x xx i i
x
i i x x
f(x) :
e.g.,
2( ) ( , ) 3 4 2 8 7,f f x y x y xy x y xe.g.,
22
( ) ( ) ( ) ( ) ( ) ( )
6 4 2 (6 4 2) (3 4 8)
3 4 8
Tf f f f
f fx y x y
xy yxy y x x
x x
x x x xx x i j
i j
18
1 ( )( )
( ) ( )
( )
T
Tp
fd
df
xf x
f x f xx
x
1 1
1
1
( ) ( )
( )
( ) ( )
q
p q
p p
q
f f
x x
f f
x x
f
x x
x
x x
( ) f x : the Jacobian of f where
( ) :f x
19
1 2( , , , )T
qx x x x
2
2
1 1( ) ( ) ( ) (| | )
0! 1!
( ) ( ) (| | )
f x x f x f x x O x
f x f x x O x
。 Taylor expansion of
( ) ( ) ( ) ff x x f x x x
1 2( )Tpf f f f
( ) ( ) ( ) (| )f f f O 2x x x x x x|
around point
f(x):
f(x) :
( ) :f x
20
x( )f x x 0
A. Newton’s Method (Gradient Descent) (i) Square Systems (p = q) Idea: Given an initial x, find s.t.
.
( ) ( ) ( )ff x x f x x x 0( ) ( ) f x x f x
Since
.
( ) f x -1( ) ( ) fx x f x
1 .n n x x xWhen : nonsingular,
Let
( ), f x
Repeat until f(x) stabilizes at some x
• Drawbacks: i) Square system, ii) Nonsingulariii) Locally optimal.
( ) 0 f x
21
( )xE 0Finding x s.t.Finding x s.t. F(x) = 0 (square system)
Since : p by q matrix, f(x): p by q, ( ) f x
( ) ( ) ( )T fF x x f x : q by q
22
where ( )F x : q by q matrix
23
24
25
26
1 2 0 3 0
2 0 3 0
3
cot cot
sin sin
T T Tx y z
T Ty z
Tz
KR K
u t t u t
v t v t
t
t
r r r
r r
r
27
1 2 0 31
2 2 0 3
33
cot
sin
T T TT
T T T
TT
u
v
r r ra
a r r
ar
28
29
30
31
(g) 1 3 2 3
1 3 2 3
( ) ( )cos
a a a a
a a a a
1 2 0 31
2 2 0 3
33
cot
sin
T T TT
T T T
TT
u
v
r r ra
a r r
ar
Proof: From
1 2 0 3
12 0 3
2
3
3
cot
sin
T T T
TT T
T
T
T
u
v
r r r
a r ra
ar
32
1 2 0 3 31 3
1 3 2 3 0 3 32
2 12
cot
( ) cot ( ) ( )
cot
T T T T
T T T T T T
T T
u
u
r r r ra a
r r r r r r
r r
2 0 33
2 3
2 3 0 3 31
2 2
sin
( ) ( )sin
sin
T TT
T T T TT
v r
v
r ra a
r r r r r
1 2 3 r r r
1 3 2r r r
33
2 1 11 3 2 3 2 2
2 1 1 14 4 2
cot( ) ( )
sin
( ) cot ( ) cos
sin sin
T T T
T T T T
r r ra a a a
r r r r
21 3 sin a a 2
2 3 sin a aFrom and
2 21 3 2 3
1 3 2 3 4 2
1 3 2 3
( sin )( sin )cos( ) ( )
sin
cos
a a a aa a a a
a a a a
1 3 2 3
1 3 2 3
( ) ( ) cos
a a a a
a a a a
34
35
○ Degenerated Point Configurations e.g., points lie on a line or a plane, may cause failure of camera calibration.3.3. Shape Distortions Types of distortions: (a) Tangential distortion (b) Radial distortion Barrel distortion, Pincushion distortion
36
Radial distortion: (a) Changes the distance between the image center and the image point (b) Does not affect the direction joining the image center and the image point
ˆ ( )d dd̂d: actual distance : distorted distance
( )d : distortion function
37
3 2
1
ˆ ( ) 1 ,pp
pd d k d
: coefficients
where
pk
• Polynomial model:
• FOV model:
• Logarithmic model, Fisheye model, Radial model, Rational function model
11ˆ tan (2 tan ),2
d d
ˆtan( ),
2 tan2
dd
2 2ˆ ˆ ˆ ,d x y 2 2d x y where
: distortion coefficient
38
Given an image point (u,v), determine its actual d
。 Consider Polynomial model
39
40
Determine the distortion function
3 2
1
ˆ ( ) 1 ,pp
pd d k d
pki.e., determine its coefficients
41
42
43
44
45
46
47
48
49
50
51
3.5 Application: Mobile Robot Localization -- Calibrate a static camera for monitoring a robot
52
20 images of the planar rectangular grid
Image resolution: 576 by 768Camera: height = 4m, focal length = 4.5mm, Skew = 0, precision = 0.1
pixel 3 radial distortion
coefficients.
Experimental results:Localization error: 2 cm in position
and 1 degree in
orientationMaximum error: 5 cm in position and 5 degrees in
orientation
53
3.4. A Nonlinear Approach
54
55
and2
( )g fdg gdf
df f
56
57
58