Lecture 3 -Silvio Savarese & Jeanette Bohg 20-January-2021

Professor Silvio SavareseStanford Vision and Learning Lab

Lecture 3Camera Models 2 &Camera Calibration

Lecture 3 -Silvio Savarese & Jeanette Bohg 20-January-2021

• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”

Lecture 3Camera Models 2 &Camera Calibration

P '3×1 =M Pw = K3×3 R T"#

$% 3×4

Pw4×1

=

m1

m2

m3

!

"

####

$

%

&&&&

PW =m1PWm2PWm3PW

!

"

####

$

%

&&&&

→ P 'E = (m1Pwm3Pw

, m2Pwm3Pw

)E

f

Projective camera

Ow

kw

jwR,T

úúú

û

ù

êêê

ë

é=

3

2

1

mmm

M

Exercise!

Ow

iw

kw

jwR,T

M = K R T!"

#$

= (m1Pwm3Pw

, m2Pwm3Pw

)

P

P’

f

P 'E = ( f xwzw, f yw

zw)

=

f 0 0 00 f 0 00 0 1 0

!

"

###

$

%

&&&

Pw =

xwywzw1

!

"

#####

$

%

&&&&&

à

= K I 0!"

#$

Canonical Projective Transformation

P ' =xyz

!

"

####

$

%

&&&&

=1 0 0 00 1 0 00 0 1 0

!

"

###

$

%

&&&

xyz1

!

"

####

$

%

&&&&

xzyz

!

"

####

$

%

&&&&

Pi ' =

P ' =M P

M3

H4 ®Â

Projective camera

p’q’r’ O

f’R

Q

P

Weak perspective projection

R

Q

P

O

f’

When the relative scene depth is small compared to its distance from the camera

zo

p’q’r’ Q_

P_

R_

π

Weak perspective projection

O

f’ zo

p’q’r’ Q_

P_

R_

x ' = f 'zx

y ' = f 'zy

!

"

###

$

###

→

x ' = f 'z0x

y ' = f 'z0y

!

"

##

$

##

Magnification m

R

Q

Pπ

úû

ùêë

é=

10bA

M[ ]TRKM = úû

ùêë

é=

1vbA

O

f’ zo

p’q’r’ Q_

P_

R_R

Q

P

π

à

Projective (perspective) Weak perspective

Weak perspective projection

úúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é=

10002

1

3

2

1

mm

mmm

=

m1

m2

m3

!

"

####

$

%

&&&&

PW

)P,P( 21 ww mm®E

wPMP =¢úû

ùêë

é=

10bA

M

magnification

úúú

û

ù

êêê

ë

é=

3

2

1

mmm

wPMP =¢ úû

ùêë

é=

1vbA

Múúú

û

ù

êêê

ë

é=

úúú

û

ù

êêê

ë

é=

W3

W2

W1

W

3

2

1

PPP

Pmmm

mmm

)PP,

PP(

3

2

3

1

w

w

w

w

mm

mm

®E

Perspective

Weak perspective

=

m1 PWm2 PW1

!

"

####

$

%

&&&&

Orthographic (affine) projectionDistance from center of projection to image plane is infinite

x ' = f 'zx

y ' = f 'zy

!

"

###

$

###

→x ' = xy ' = y

!"#

$#

Pros and Cons of These Models

• Weak perspective results in much simpler math.– Accurate when object is small and distant.– Most useful for recognition.

• Pinhole perspective is much more accurate for modeling the 3D-to-2D mapping.– Used in structure from motion or SLAM.

One-point perspective• Masaccio, Trinity,

Santa Maria Novella, Florence, 1425-28

Credit slide S. Lazebnik

The Kangxi Emperor's Southern Inspection Tour (1691-1698)By Wang Hui

Weak perspective projection

Weak perspective projection

The Kangxi Emperor's Southern Inspection Tour (1691-1698)By Wang Hui

Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21

• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3Camera Calibration

Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”

Projective camera

úúú

û

ù

êêê

ë

é -

=

100v0

ucot

K o

o

sinqb

qaa

úúú

û

ù

êêê

ë

é

=T3

T2

T1

Rrrr

úúú

û

ù

êêê

ë

é

=

z

y

x

ttt

T

43´

wPMP =¢ [ ] wPTRK=Internal parameters

External parameters

Goal of calibration

Estimate intrinsic and extrinsic parameters from 1 or multiple images

Change notation:P = Pw

p = P’

wPMP =¢ [ ] wPTRK=Internal parameters

External parameters

Calibration Problem

•P1… Pn with known positions in [Ow,iw,jw,kw]

jC

Calibration rig

Calibration Problem

•P1… Pn with known positions in [Ow,iw,jw,kw]•p1, … pn known positions in the image

jC

Calibration rig

pi

Goal: compute intrinsic and extrinsic parameters

pi

image

image

Calibration Problem

jC

Calibration rig

How many correspondences do we need?• M has 11 unknowns• We need 11 equations • 6 correspondences would do it

pi

jCpi

pi

image

Calibration Problem

jC

Calibration rig

In practice, using more than 6 correspondences enables more robust results

pi

jCpi

pi

image

Calibration Problem

jC

Calibration rig

pi

jCpi

pi

image

=M Pipi =uivi

!

"##

$

%&&

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

mmmm

in pixels[Eq. 1]

úúú

û

ù

êêê

ë

é=

3

2

1

Mmmm

Calibration Problem

i3

i1i P

Pumm

=

i2i3i P)P(v mm =®

i1i3i P)P(u mm =®

i3

i2i P

Pvmm

=

úû

ùêë

é

i

i

vu

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

mmmm

0P)P(v 23 =-® iii mm

→ ui (m3 Pi )−m1 Pi = 0

[Eq. 1]

[Eqs. 2]

Calibration Problem

……

0)( 12131 =- PPv mm

0)( 11131 =- PPu mm

0)( 23 =- iii PPv mm

0)( 13 =- iii PPu mm

0)( 23 =- nnn PPv mm

0)( 13 =- nnn PPu mm

[Eqs. 3]

Block Matrix Multiplication

úû

ùêë

é=ú

û

ùêë

é=

2221

1211

2221

1211

BBBB

BAAAA

A

What is AB ?

úû

ùêë

é++++

=2222122121221121

2212121121121111

BABABABABABABABA

AB

÷÷÷

ø

ö

ççç

è

æ

=T3

T2

T1def

mmm

mP = def

P1T

0T0T

P1T

−u1P1T

−v1P1T

!PnT

0T0T

PnT

−unPnT

−vnPnT

"

#

$$$$$$$$

%

&

''''''''

Calibration Problem

2n x 1212x1

1x44x1

…

Homogenous linear system

knownunknown

0)( 12131 =+- PPv mm

0)( 11131 =+- PPu mm

0)( 23 =+- nnn PPv mm

0)( 13 =+- nnn PPu mm

0=P m [Eq. 4]

Homogeneous M x N Linear Systems

P m 0=

Rectangular system (M>N)

• 0 is always a solution

Minimize |P m|2

under the constraint |m|2 =1

M=number of equations = 2nN=number of unknown = 11

• To find non-zero solution

N

M

• How do we solve this homogenous linear system?

Calibration Problem

• Via SVD decomposition!

0=P m

Calibration Problem

1212T

121212n2 VDU ´´´

Last column of V gives m

M

SVD decomposition of P

Why? See pag 592 of HZ

÷÷÷

ø

ö

ççç

è

æ

=T3

T2

T1def

mmm

m

0=P m

Extracting camera parameters

M = ρ

Extracting camera parameters

A

úúú

û

ù

êêê

ë

é

=T3

T2

T1

Aaaa

[ ]TRK=

3

1a±

=r

úúú

û

ù

êêê

ë

é

=

3

2

1

bbb

b

Estimated values

)( 312 aa ×= rou

)(v 322

o aa ×= r

( ) ( )3231

3231cosaaaaaaaa

´×´´×´

=q

Intrinsic

b úúú

û

ù

êêê

ë

é -

=

100v0

ucot

K o

o

sinqb

qaarM =

Box 1

See [FP], Sec. 1.3.1

Extracting camera parameters

úúú

û

ù

êêê

ë

é

=T3

T2

T1

Aaaa

[ ]TRK=

úúú

û

ù

êêê

ë

é

=

3

2

1

bbb

b

Estimated values

Intrinsic

qra sin312 aa ´=

qrb sin322 aa ´=

A brM =

úúú

û

ù

êêê

ë

é -

=

100v0

ucot

K o

o

sinqb

qaa

Box 1

See [FP], Sec. 1.3.1

Extracting camera parameters

úúú

û

ù

êêê

ë

é

=T3

T2

T1

Aaaa

[ ]TRK=

úúú

û

ù

êêê

ë

é

=

3

2

1

bbb

b

Estimated values

A brM =

úúú

û

ù

êêê

ë

é -

=

100v0

ucot

K o

o

sinqb

qaa

Box 1

See [FP], Sec. 1.3.1

Extrinsic( )

32

321 aa

aar´´

=3

33 a

ar ±=

132 rrr ´= b1KT -= r

Degenerate cases

•Pi’s cannot lie on the same plane!• Points cannot lie on the intersection curve of two

quadric surfaces [FP] section 1.3

Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21

• Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3Camera Calibration

Reading: [FP] Chapter 1 “Geometric Camera Calibration”[HZ] Chapter 7 “Computation of Camera Matrix P”

No distortion

Pin cushion

Barrel

Radial Distortion– Image magnification (in)decreases with distance from the optical axis– Caused by imperfect lenses– Deviations are most noticeable for rays that pass through the edge of

the lens

Radial Distortion

ii

ii p

vu

PM

100

00

00

1

1

=úû

ùêë

é®

úúúúúú

û

ù

êêêêêê

ë

é

l

ld

v

vucvbuad 222 ++=

u

To model radial behavior

å±==

3

1p

2ppdκ1λ

Polynomial function

Distortion coefficient

[Eq. 5] [Eq. 6]

Sλ

Image magnification decreases with distance from the optical center

How do we model that?

úû

ùêë

é=

i

ii v

up

úúú

û

ù

êêê

ë

é=

3

2

1

Qqqq

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

qqqq

ii

ii p

vu

PM

100

00

00

1

1

=úû

ùêë

é®

úúúúúú

û

ù

êêêêêê

ë

é

l

l

Q

uiq3 Pi = q1 Piviq3 Pi = q2 Pi

!"#

$#

Is this a linear system of equations?

Radial Distortion

No! why? [Eqs.7]

X = f (Q)

measurements parameters

f( ) is the nonlinear mapping

-Newton Method-Levenberg-Marquardt Algorithm

• Iterative, starts from initial solution • May be slow if initial solution far from real solution • Estimated solution may be function of the initial solution (because of local minima)

• Newton requires the computation of J, H• Levenberg-Marquardt doesn’t require the computation of H

General Calibration Problem

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

qqqq

úû

ùêë

é

i

i

vu

i=1…n

[Eq .8]

A possible algorithm

1. Solve linear part of the system to find approximated solution 2. Use this solution as initial condition for the full system3. Solve full system using Newton or L.M.

General Calibration Problem

X = f (Q)

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

qqqq

úû

ùêë

é

i

i

vu

i=1…n

measurements parameters

[Eq .8]

f( ) is the nonlinear mapping

Typical assumptions:

- zero-skew, square pixel- uo, vo = known center of the image

General Calibration Problem

measurements parameters

f( ) is the nonlinear mapping

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

qqqq

úû

ùêë

é

i

i

vu X = f (Q) [Eq .8]

i=1…n

Can we estimate m1 and m2 and ignore the radial distortion?

Radial Distortion

úû

ùêë

é=

i

ii v

up

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

1

mmmm

l

d

v

uHint: slopevu

i

i =

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

qqqq

Estimating m1 and m2…

Radial Distortion

úû

ùêë

é=

i

ii v

up

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

1

mmmm

l

0)()( 121111 =- PuPv mm0)()( 21 =- iiii PuPv mm

0)()( 21 =- nnnn PuPv mm…

L n = 0

n =m1

T

m2T

!

"

##

$

%

&&

Tsai [87]

i

i

i

i

i

i

i

i

PP

PPPP

vu

2

1

3

2

3

1

)()()()(

mm

mmmm

==

Get m1 and m2 by SVD

[Eq .9]

[Eq .10][Eq .11]

L = def

v1P1T −u1P1

T

v2P2T −u2P2

T

! !vnPn

T −unPnT

"

#

$$$$$$

%

&

''''''

Once that m1 and m2 are estimated…

Radial Distortion

úû

ùêë

é=

i

ii v

up

úúúú

û

ù

êêêê

ë

é

=

i3

i2

i3

i1

PPPP

1

mmmm

l

3m is non linear function of 1m 2m l

There are some degenerate configurations for which m1 and m2 cannot be computed

, ,

Lecture 3 -Silvio Savarese & Jeanette Bohg 19-Jan-21

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3Camera Calibration

• Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• ExampleReading: [FP] Chapter 1 “Geometric Camera Calibration”

[HZ] Chapter 7 “Computation of Camera Matrix P”

Calibration ProcedureCamera Calibration Toolbox for MatlabJ. Bouguet – [1998-2000]

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Next lecture

•Single view reconstruction