Today

Camera calibration

Conventional Camera Calibration

Input: pairs of 3D object points and 2D image points

General strategy:

• Take pictures from calibration patterns with known coordinates of 3D features (e.g., points, lines, or curves)

• Identify the corresponding image features

• obtain the projection matrix by minimizing projection error

• obtain intrinsic parameters from the projection matrix

11,vu

22 ,vu

33 ,vu

NN vu ,

111 ,, ZYX

222 ,, ZYX

333 ,, ZYX

NNN ZYX ,,

Camera Calibration from Projection Matrix P:

A Linear Method

0'' )(

34

)()(

314

)(

1 IIWtWt upuMpM pp

0'' )(

34

)()(

324

)(

2 IIWtWt vpvMpM pp

For every pair of 3D-2D points, we have a pair of linear equations:

11,vu

22 ,vu

33 ,vu

NN vu ,

111 ,, ZYX

222 ,, ZYX

333 ,, ZYX

NNN ZYX ,,

N pairs We can setup a linear equation systems for N

corresponding pairs with 2N equations:

• Known: 2D and 3D coordinates

• Unknown: 4x3 parameters of P

Av = 0

A is a 2𝑁 × 12 matrix containing the known

v is a 12 × 1 vector containing the unknown tttt ppp 343242141 pppv

Camera Calibration from Projection Matrix P:

A Linear Method

)()()()(

31

)()()(

31

)(

)(

1

)(

1

)(

1

)(

131

)(

1

)(

1

)(

131

)(

1

'1'00

'001'

'1'00

'001'

I

N

I

N

tW

N

tW

N

I

N

I

N

tW

N

tW

N

IItWtW

IItWtW

vvMM

uuMM

vvMM

uuMM

A

Require at least 6 pairs of non-coplanar points

Estimate Projection Matrix by SVD

The projection matrix can be estimated as a solution to

The solution lies in the null space of A – the set of all vectors x

that satisfies Ax =0.

𝑟𝑎𝑛𝑘 𝐴 + 𝑛𝑢𝑙𝑙𝑖𝑡𝑦 𝐴 = min(𝑟𝑜𝑤 𝐴 , 𝑐𝑜𝑙 𝐴 )

• Case 1: If rank(A) = 11, the solution to v is unique up

to a scaling factor

• Case 2: if rank(A) < 11, the solution to v is a linear

combination of all null vectors of A

Av = 0

Estimate v by performing SVD on A

What about rank(A)=12?

Brief Review on SVD

Singular Value Decomposition:

• Any mxn matrix can be written as the product of three matrices

T

nnnmmmnm VDUA

Singular values 𝜎𝑖 are fully determined by A

D: 𝑑𝑖𝑗 = 0 𝑖𝑓 𝑖𝑗; 𝑑𝑖𝑖 = 𝜎𝑖 (𝑖 = 1,2, … , 𝑛)

s1 s2 … sN 0; 𝑁 = 𝑟𝑎𝑛𝑘(𝐴)

Both U and V are not unique

Columns of each are mutual orthogonal vectors

nnnn

n

n

n

mmmm

m

m

mnmm

n

n

vvv

vvv

vvv

uuu

uuu

uuu

aaa

aaa

aaa

21

22212

12111

2

1

21

22221

11211

21

22221

11211

00

0

0

00

s

s

s

𝑽𝟏𝑻

U1

Brief Review on SVD

U contains the eigenvector of 𝐴𝐴𝑇

Columns of U called the left-singular vectors

V contains the eigenvector of 𝐴𝑇𝐴 Columns of V called the right-singular vectors

Both U and V are unitary matrices:

𝑈𝑈𝑇 = 𝐼𝑚×𝑚 and 𝑉𝑉𝑇 = 𝐼𝑛×𝑛

Estimate Projection Matrix by SVD

When rank(A) = 11, v can be estimated by performing SVD on A

v is the last column of the V matrix with an unknown scalar

How to solve the scalar 𝛼?tt

33 rp and tt

33 rp 13 r

2

11

2

10

2

9 '''

1

vvv

'vv The null vector of A

Projection matrix P is solved!

The sign of 𝛼 sign(𝛼) is

unknown!

tttt ppp 343242141 pppv

Discussion on the Rank of A

• Rank(A)=11 with a proper configuration of the 3D points

• In practice, rank(A)=12 (full rank) may happen due to the

image noise and error in location estimation

• Rank(A)<11

• Rank(A)=8 if the 3D points are coplanar

• Results in infinite solutions

Issues in the Linear Method

• Numerical issue

• Outliers in observations

Numerical Stability: Data Normalization

Motivation: the entries in A have large difference in unit and scale

)()()()(

31

)()()(

31

)(

)(

1

)(

1

)(

1

)(

131

)(

1

)(

1

)(

131

)(

1

'1'00

'001'

'1'00

'001'

I

N

I

N

tW

N

tW

N

I

N

I

N

tW

N

tW

N

IItWtW

IItWtW

vvMM

uuMM

rrMM

uuMM

A

Unit: mm

scale: vary

Unit: no

Scale: 0 or 1

Unit: mm*pixel

scale: vary

Unit: pixel

scale: the image size

Numerical instable: resulting problem when data

contains significant noise

Solution: data normalization

Numerical Stability: Data Normalization

• Replace A by A’ by centering and normalizing each point,

respectively.

• Centering: translate 2D/3D points wrt their centroid

• Normalization: scaling the coordinates by a scale factor

• E.g., the scale factor can be computed as the inverse of

average distance to the centroid

m′ = H𝟐𝑫 m

M′ = H𝟑𝑫 M

H𝟐𝑫 =

𝟏

𝒔𝟐𝑫𝟎 −

𝒎𝒙

𝒔𝟐𝑫

𝟎𝟏

𝒔𝟐𝑫−𝒎𝒚

𝒔𝟐𝑫𝟎 𝟎 𝟏

H𝟑𝑫 =

𝟏

𝒔𝟑𝑫𝟎 𝟎 −

𝑴𝑿

𝒔𝟑𝑫

𝟎𝟏

𝒔𝟑𝑫𝟎 −

𝑴𝒀

𝒔𝟑𝑫

𝟎 𝟎𝟏

𝒔𝟑𝑫−𝑴𝒁

𝒔𝟑𝑫𝟎 𝟎 𝟎 𝟏

Numerical Stability: Data Normalization

• Replace A by A’ by centering and normalizing each point,

respectively.

• Solve the problem of and recover P′ from v′

• Recover P=H𝟐𝑫

−𝟏P′H𝟑𝑫

A′v′ = 0

Note: this preprocessing step is recommended for all

methods involving image data

m′ = H𝟐𝑫 m

M′ = H𝟑𝑫 M

H𝟐𝑫 =

𝟏

𝒔𝟐𝑫𝟎 −

𝒎𝒙

𝒔𝟐𝑫

𝟎𝟏

𝒔𝟐𝑫−𝒎𝒚

𝒔𝟐𝑫𝟎 𝟎 𝟏

H𝟑𝑫 =

𝟏

𝒔𝟑𝑫𝟎 𝟎 −

𝑴𝑿

𝒔𝟑𝑫

𝟎𝟏

𝒔𝟑𝑫𝟎 −

𝑴𝒀

𝒔𝟑𝑫

𝟎 𝟎𝟏

𝒔𝟑𝑫−𝑴𝒁

𝒔𝟑𝑫𝟎 𝟎 𝟎 𝟏

Robust Linear Method with RANSAC

The linear method is sensitive to image noise and localization errors.

outliers

Estimated line

Ideal line

Robust Linear Method with RANSAC

A better solution can be obtained by using a robust method such as

the RANSAC (Random Sample Consensus) method.

Assumptions:

• The model can be estimated from K data items

• There are N (N>K) data items in total

• The whole data set contains a fraction 𝛼 of outliers

• A guess about the probability that the training set does not

contain any outliers

Robust Linear Method with RANSAC

• Use green points to estimate

the parameters

• Use red points to compute

errors

Robust Linear Method with RANSAC

Repeat: Step 1: randomly partition all N points into a training set (with K>6 points)

and a testing set (with N-K points).

Step 2: compute the projection matrix P using the training set

Step 3: for each point in the testing set, compute its projection error

if , increment a counter for “inliers”

Until run a sufficient number of times

Step 4: use the training set resulting the highest number of inliers plus all

the inlier points to recompute P

vAi

Threshold

How to determine?

Robust Linear Method with RANSAC

The probability that all K data in a subset are good is

The probability that all 𝑠 different subsets/iterations will contain

at least one or more ourliers is

The probability that at least one random subset has no outliers is

given by

The number of iterations/subsets needed is computed as

sKP ))1(1(1

K)1(

sK ))1(1(

))1(1ln(

)1ln(K

Ps

343

242

141

3

0302

0301

p

p

p

t

tvtfvf

tutfuf

Pt

t

t

z

t

zYv

tt

v

zXu

tt

u

p

p

p

r

rr

rr

Estimate Camera Parameters from P

310 pptu 320 pp

tv Why?

34ptz 33 pr

sign(𝛼)

is determined from

is negative if the object is before the camera Zt

Zt

How to determine

343

242

141

3

0302

0301

p

p

p

t

tvtfvf

tutfuf

Pt

t

t

z

t

zYv

tt

v

zXu

tt

u

p

p

p

r

rr

rr

Estimate Camera Parameters from P

2

011 uf t

u pp2

022 vf t

v pp

uX fpupt /34014 vY fpvpt /34024

ufu /3011 ppr vfv /3022 ppr