Post on 19-Dec-2015
3D Computer Vision
and Video Computing 3D Vision3D Vision
Topic 2 of Part IICalibration
CSc I6716Fall 2010
Zhigang Zhu, City College of New York zhu@cs.ccny.cuny.edu
3D Computer Vision
and Video Computing Lecture OutlineLecture Outline
n Calibration: Find the intrinsic and extrinsic parametersl Problem and assumptionsl Direct parameter estimation approachl Projection matrix approach
n Direct Parameter Estimation Approachl Basic equations (from Lecture 5)l Homogeneous Systeml Estimating the Image center using vanishing pointsl SVD (Singular Value Decomposition) l Focal length, Aspect ratio, and extrinsic parametersl Discussion: Why not do all the parameters together?
n Projection Matrix Approach (…after-class reading)l Estimating the projection matrix Ml Computing the camera parameters from Ml Discussion
n Comparison and Summaryl Any difference?
3D Computer Vision
and Video Computing Problem and AssumptionsProblem and Assumptionsn Given one or more images of a calibration pattern, n Estimate
l The intrinsic parametersl The extrinsic parameters, orl BOTH
n Issues: Accuracy of Calibrationl How to design and measure the calibration pattern
n Distribution of the control points to assure stability of solution – not coplanar
n Construction tolerance one or two order of magnitude smaller than the desired accuracy of calibration
n e.g. 0.01 mm tolerance versus 0.1mm desired accuracyl How to extract the image correspondences
n Corner detection?n Line fitting?
l Algorithms for camera calibration given both 3D-2D pairs
n Alternative approach: 3D from un-calibrated camera
3D Computer Vision
and Video Computing Camera ModelCamera Model
n Coordinate Systemsl Frame coordinates (xim, yim) pixelsl Image coordinates (x,y) in mml Camera coordinates (X,Y,Z) l World coordinates (Xw,Yw,Zw)
n Camera Parametersl Intrinsic Parameters (of the camera and the frame grabber): link the
frame coordinates of an image point with its corresponding camera coordinates
l Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system
Zw
Xw
Yw
Y
X
Zx
yO
Pw
P
p
xim
yim
(xim,yim)
Pose / Camera
Object / World
Image frame
Frame Grabber
3D Computer Vision
and Video ComputingLinear Version of Perspective ProjectionLinear Version of Perspective Projection
n World to Cameral Camera: P = (X,Y,Z)T
l World: Pw = (Xw,Yw,Zw)T
l Transform: R, T
n Camera to Imagel Camera: P = (X,Y,Z)T
l Image: p = (x,y)T
l Not linear equations
n Image to Framel Neglecting distortionl Frame (xim, yim)T
n World to Framel (Xw,Yw,Zw)T -> (xim, yim)T
l Effective focal lengthsn fx = f/sx, fy=f/sy
zT
yT
xT
zwww
ywww
xwww
T
T
T
TZrYrXr
TZrYrXr
TZrYrXr
w
w
w
w
PR
PR
PR
TRPP
3
2
1
333231
232221
131211
yyim
xxim
soyy
soxx
)(
)(
) ,(),(Z
Yf
Z
Xfyx
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXr
TZrYrXrfoy
TZrYrXr
TZrYrXrfox
333231
232221
333231
131211
3D Computer Vision
and Video Computing Direct Parameter MethodDirect Parameter Method
n Extrinsic Parametersl R, 3x3 rotation matrix
n Three angles a,b,gl T, 3-D translation vector
n Intrinsic Parametersl fx, fy :effective focal length in pixel
n a = fx/fy = sy/sx, and fxl (ox, oy): known Image center -> (x,y) knownl k1, radial distortion coefficient: neglect it in the basic algorithm
n Same Denominator in the two Equationsl Known : (Xw,Yw,Zw) and its (x,y)l Unknown: rpq, Tx, Ty, fx, fy
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXr
TZrYrXrfoyy
TZrYrXr
TZrYrXrfoxx
333231
232221
333231
131211
'
'
)(')(' 131211232221 xwwwxywwwy TZrYrXrfyTZrYrXrfx
'/)('/)( 131211232221 xTZrYrXrfyTZrYrXrf xwwwxywwwy
3D Computer Vision
and Video Computing Linear EquationsLinear Equations
n Linear Equation of 8 unknowns v = (v1,…,v8)
l Aspect ratio: a = fx/fyl Point pairs , {(Xi, Yi,, Zi) <-> (xi, yi) } drop the ‘ and subscript “w”
)(')(' 131211232221 xwwwywww TZrYrXryTZrYrXrx a
0)()()()( 131211232221 xiiiiiiiyiiiiiii TyrZyrYyrXyTxrZxrYxrXx aaaa
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
),,,,,,,(
),,,,,,,(
131211232221
87654321
xy TrrrTrrr
vvvvvvvv
aaaa
3D Computer Vision
and Video Computing Homogeneous SystemHomogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
0Av
NNNNNNNNNNNNNN yZyYyXyxZxYxXx
yZyYyXyxZxYxXx
yZyYyXyxZxYxXx
22222222222222
11111111111111
A
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of A
3D Computer Vision
and Video Computing Homogeneous SystemHomogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
0Av
NNNNNNNNNNNNNN yZyYyXyxZxYxXx
yZyYyXyxZxYxXx
yZyYyXyxZxYxXx
22222222222222
11111111111111
A
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of A
3D Computer Vision
and Video Computing SVD: definitionSVD: definition
n Singular Value Decomposition:l Any mxn matrix can be written as the product of three
matrices
TUDVA
n Singular values si are fully determined by An D is diagonal: dij =0 if ij; dii = si (i=1,2,…,n)
n s1 s2 … sN 0
n Both U and V are not uniquen Columns of each are mutual orthogonal vectors
nnnn
n
n
n
mmmm
m
m
mnmm
n
n
vvv
vvv
vvv
uuu
uuu
uuu
aaa
aaa
aaa
21
22212
121112
1
21
22221
11211
21
22221
11211
00
0
0
00
s
ss
V1U1
Appendix A.6
3D Computer Vision
and Video Computing SVD: propertiesSVD: properties
n 1. Singularity and Condition Numberl nxn A is nonsingular IFF all singular values are nonzerol Condition number : degree of singularity of A
n A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular
n 2. Rank of a square matrix Al Rank (A) = number of nonzero singular values
n 3. Inverse of a square Matrixl If A is nonsingularl In general, the pseudo-inverse of A
n 4. Eigenvalues and Eigenvectors (questions)l Eigenvalues of both ATA and AAT are si
2 (si > 0)l The columns of U are the eigenvectors of AAT (mxm)l The columns of V are the eigenvectors of ATA (nxn)
TUDVA
nC ss /1
TUVDA 11 TUVDA 1
0
iiT
iuuAA 2s
iiT
ivAvA 2s
3D Computer Vision
and Video Computing SVD: Application 1SVD: Application 1
n Least Squarel Solve a system of m equations for n unknowns x(m >= n)l A is a mxn matrix of the coefficients l b (0) is the m-D vector of the data l Solution:
l How to solve: compute the pseudo-inverse of ATA by SVDn (ATA)+ is more likely to coincide with (ATA)-1 given m > nn Always a good idea to look at the condition number of ATA
bAx
bAAxA TT bAAAx TT )(
nxn matrix Pseudo-inverse
3D Computer Vision
and Video Computing SVD: Application 2SVD: Application 2
n Homogeneous Systeml m equations for n unknowns x(m >= n-1)l Rank (A) = n-1 (by looking at the SVD of A)l A non-trivial solution (up to a arbitrary scale) by SVD:l Simply proportional to the eigenvector corresponding to the
only zero eigenvalue of ATA (nxn matrix)
n Note:l All the other eigenvalues are positive because
Rank (A)=n-1l For a proof, see Textbook p. 324-325l In practice, the eigenvector (i.e. vn) corresponding to
the minimum eigenvalue of ATA, i.e. sn2
0Ax
iiT
ivAvA 2s
3D Computer Vision
and Video Computing SVD: Application 3SVD: Application 3
n Problem Statementsl Numerical estimate of a matrix A whose entries are not
independentl Errors introduced by noise alter the estimate to Â
n Enforcing Constraints by SVDl Take orthogonal matrix A as an examplel Find the closest matrix to Â, which satisfies the constraints
exactlyn SVD of  n Observation: D = I (all the singular values are 1) if A is
orthogonaln Solution: changing the singular values to those expected
TUDVA ˆ
TUIVA
3D Computer Vision
and Video Computing Homogeneous SystemHomogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) },
i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of Al Rows of VT: eigenvectors {ei} of ATAl Solution: the 8th row e8 corresponding to the only zero
singular value l8=0
n Practical Considerationl The errors in localizing image and world points may make
the rank of A to be maximum (8)l In this case select the eigenvector corresponding to the
smallest eigenvalue.
0Av
TUDVA
8ev c
3D Computer Vision
and Video Computing Scale Factor and Aspect RatioScale Factor and Aspect Ratio
n Equations for scale factor g and aspect ratio a
n Knowledge: R is an orthogonal matrix
n Second row (i=j=2):
n First row (i=j=1)
),,,,,,,( 131211232221 xy TrrrTrrr aaaagv
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33R
R
R
R
ji if 0
ji if 1j
Ti RR
1222232221
rrr 23
22
21|| vvv g
1222131211
rrr 27
26
25|| vvv ga
a
||g
v1 v2 v3 v4 v5 v6 v7 v8
3D Computer Vision
and Video Computing Rotation R and Translation TRotation R and Translation T
n Equations for first 2 rows of R and T given a and |g|
n First 2 rows of R and T can be found up to a common sign s (+ or -)
n The third row of the rotation matrix by vector product
n Remaining Questions :l How to find the sign s?l Is R orthogonal?l How to find Tz and fx, fy?
),,,,,,,( || 131211232221 xy TrrrTrrrs aaaagv
yxTT sTsTss ,,, 21 RR
TTTTT ss 21213 RRRRR
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33R
R
R
R
3D Computer Vision
and Video Computing Find the sign sFind the sign s
n Facts:l fx > 0l Zc >0l x knownl Xw,Yw,Zw known
n SolutionÞ Check the sign of Xc
Þ Should be opposite
to x zwww
ywwwyy
zwww
xwwwxx
TZrYrXr
TZrYrXrf
Zc
Ycfy
TZrYrXr
TZrYrXrf
Zc
Xcfx
333231
232221
333231
131211
Zw
Xw
Yw
Y
X
Zx
yO
PwP
p
xim
yim
(xim,yim)
3D Computer Vision
and Video Computing Rotation R : OrthogonalityRotation R : Orthogonality
n Question: l First 2 rows of R are calculated
without using the mutual orthogonal constraint
n Solution: l Use SVD of estimate R
TTTTT ss 21213 RRRRR
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33R
R
R
R
?ˆˆ IRR T
TUDVR ˆ TUIVR
Replace the diagonal matrix D with the 3x3 identity matrix
3D Computer Vision
and Video Computing Find Tz, Fx and FyFind Tz, Fx and Fy
n Solutionl Solve the system of N linear
equations with two unknownn Tx, fx
l Least Square method
l SVD method to find inverse
zwww
xwwwx TZrYrXr
TZrYrXrfx
333231
131211
bA
x
z
f
T
)()( 333231131211 wwwxxwwwz ZrYrXrxfTZrYrXrxT
ai2ai1 bi
bAAA TT
x
z
f
T 1)(ˆ
ˆ
3D Computer Vision
and Video ComputingDirect parameter Calibration SummaryDirect parameter Calibration Summary
n Algorithm (p130-131)1. Measure N 3D coordinates (Xi, Yi,Zi)2. Locate their corresponding image
points (xi,yi) - Edge, Corner, Hough3. Build matrix A of a homogeneous
system Av = 0 4. Compute SVD of A , solution v5. Determine aspect ratio a and scale |g|6. Recover the first two rows of R and the
first two components of T up to a sign7. Determine sign s of g by checking the
projection equation8. Compute the 3rd row of R by vector
product, and enforce orthogonality constraint by SVD
9. Solve Tz and fx using Least Square and SVD, then fy = fx / a
Yw
Xw
Zw
3D Computer Vision
and Video Computing DiscussionsDiscussions
n Questions
l Can we select an arbitrary image center for solving other parameters?
l How to find the image center (ox,oy)?
l How about to include the radial distortion?
l Why not solve all the parameters once ?
n How many unknown with ox, oy? --- 20 ??? – projection matrix method
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXr
TZrYrXrfoyy
TZrYrXr
TZrYrXrfoxx
333231
232221
333231
131211
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
VP1
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in a point
on the image - the vanishing point, which is the common intersection of all the image lines
n Important property:l Vector OV (from the center of projection to the vanishing point)
is parallel to the parallel lines
O
VP1
Y
X
Z
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
VP1
VP2
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes VP1
VP2
VP3
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes
VP1
VP2
VP3
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes
n Orthocenter Theorem:l WHY?
VP1
VP2
VP3
h3
h1
h1
(ox,oy)
3D Computer Vision
and Video Computing Estimating the Image CenterEstimating the Image Center
n Assumptions:l Known aspect ratiol Without lens distortions
n Questions:l Can we solve both
aspect ratio and the image center?
l How about with lens distortions?
VP1
VP2
VP3
h3
h1
h1
(ox,oy) ?
3D Computer Vision
and Video ComputingDirect parameter Calibration SummaryDirect parameter Calibration Summaryn Algorithm (p130-131)
0. Estimate image center (and aspect ratio)
1. Measure N 3D coordinates (Xi, Yi,Zi)
2. Locate their corresponding image (xi,yi) - Edge, Corner, Hough
3. Build matrix A of a homogeneous system Av = 0
4. Compute SVD of A , solution v
5. Determine aspect ratio a and scale |g|
6. Recover the first two rows of R and the first two components of T up to a sign
7. Determine sign s of g by checking the projection equation
8. Compute the 3rd row of R by vector product, and enforce orthogonality constraint by SVD
9. Solve Tz and fx using Least Square and SVD , then fy = fx / a
Yw
Xw
Zw
3D Computer Vision
and Video ComputingRemaining Issues and Possible SolutionRemaining Issues and Possible Solution
n Original assumptions:l Without lens distortionsl Known aspect ratio when estimating image centerl Known image center when estimating others including aspect ratio
n New Assumptionsl Without lens distortion l Aspect ratio is approximately 1, or a = fx/fy = 4:3 ; image center about
(M/2, N/2) given a MxN image
n Solution (?)1. Using a = 1 to find image center (ox, oy)2. Using the estimated center to find others including a3. Refine image center using new a ; if change still significant, go to step
2; otherwise stop
l Projection Matrix Approach
3D Computer Vision
and Video Computing Linear Matrix Equation of perspective projection
Linear Matrix Equation of perspective projection
n Projective Spacel Add fourth coordinate
n Pw = (Xw,Yw,Zw, 1)T
l Define (u,v,w)T such thatn u/w =xim, v/w =yim
n 3x4 Matrix Eext
l Only extrinsic parametersl World to camera
n 3x3 Matrix Eint
l Only intrinsic parametersl Camera to frame
n Simple Matrix Product! Projective Matrix M= MintMext
l (Xw,Yw,Zw)T -> (xim, yim)T
l Linear Transform from projective space to projective planel M defined up to a scale factor – 11 independent entries
1w
w
w
ZY
X
w
v
u
extintMM
zT
yT
xT
z
y
x
ext
T
T
T
T
T
T
rrr
rrr
rrr
3
2
1
333231
232221
131211
R
R
R
M
100
0
0
int yy
xx
of
of
M
wv
wu
y
x
im
im
/
/
3D Computer Vision
and Video Computing Projection Matrix MProjection Matrix M
n World – Frame Transforml Drop “im” and “w” l N pairs (xi,yi) <-> (Xi,Yi,Zi)l Linear equations of m
n 3x4 Projection Matrix Ml Both intrinsic (4) and extrinsic (6) – 10 parameters
1i
i
i
i
i
i
ZY
X
w
v
u
M
34333231
24232221
34333231
14131211
mZmYmXm
mZmYmXm
w
uy
mZmYmXm
mZmYmXm
w
ux
iii
iii
i
ii
iii
iii
i
ii
z
zyyy
zxxx
yyyyyy
xxxxxx
T
ToTf
ToTf
rrr
rorfrorfrorf
rorfrorfrorf
333231
332332223121
331332123111
M
0Am
3D Computer Vision
and Video ComputingStep 1: Estimation of projection matrixStep 1: Estimation of projection matrix
n World – Frame Transforml Drop “im” and “w” l N pairs (xi,yi) <-> (Xi,Yi,Zi)
n Linear equations of m l 2N equations, 11 independent variablesl N >=6 , SVD => m up to a unknown scale
34333231
24232221
34333231
14131211
mZmYmXm
mZmYmXm
w
uy
mZmYmXm
mZmYmXm
w
ux
iii
iii
i
ii
iii
iii
i
ii
0Am
1111111111
1111111111
10000
00001
yYyYyXyZYX
xZxYxXxZYX
A
Tmmmmmmmmmmmm 343332312423222114131211m
3D Computer Vision
and Video ComputingStep 2: Computing camera parametersStep 2: Computing camera parameters
n 3x4 Projection Matrix Ml Both intrinsic and extrinsic
n From M^ to parameters (p134-135)l Find scale |g| by using unit vector R3
T
l Determine Tz and sign of g from m34 (i.e. q43)l Obtain R3
T
l Find (Ox, Oy) by dot products of Rows q1. q3, q2.q3, using the orthogonal constraints of R
l Determine fx and fy from q1 and q2 (Eq. 6.19) Wrong???)l All the rests: R1
T, R2T, Tx, Ty
l Enforce orthognoality on R?
z
zyyy
zxxx
yyyyyy
xxxxxx
T
ToTf
ToTf
rrr
rorfrorfrorf
rorfrorfrorf
333231
332332223121
331332123111
M
MM gˆ
43
42
41
3
2
1ˆ
q
q
q
q
q
q
M
3D Computer Vision
and Video Computing ComparisonsComparisons
n Direct parameter method and Projection Matrix method
n Properties in Common:l Linear system first, Parameter decomposition secondl Results should be exactly the same
n Differencesl Number of variables in homogeneous systems
n Matrix method: All parameters at once, 2N Equations of 12 variables
n Direct method in three steps: N Equations of 8 variables, N equations of 2 Variables, Image Center – maybe more stable
l Assumptionsn Matrix method: simpler, and more general; sometime projection
matrix is sufficient so no need for parameter decompostionn Direct method: Assume known image center in the first two steps,
and known aspect ratio in estimating image center
3D Computer Vision
and Video Computing Guidelines for CalibrationGuidelines for Calibration
n Pick up a well-known technique or a fewn Design and construct calibration patterns (with known 3D)n Make sure what parameters you want to find for your cameran Run algorithms on ideal simulated data
l You can either use the data of the real calibration pattern or using computer generated data
l Define a virtual camera with known intrinsic and extrinsic parametersl Generate 2D points from the 3D data using the virtual cameral Run algorithms on the 2D-3D data set
n Add noises in the simulated data to test the robustness n Run algorithms on the real data (images of calibration target)n If successful, you are all set n Otherwise:
l Check how you select the distribution of control pointsl Check the accuracy in 3D and 2D localizationl Check the robustness of your algorithms againl Develop your own algorithms NEW METHODS?