Lecture 8: Camera Calibration - Artificial...

Lecture 8 -Fei-Fei Li

Lecture 8:

Camera Calibration

Professor Fei-Fei Li

Stanford Vision Lab

19-Oct-111


What we will learn today?

• Review camera parameters

• Affine camera model (Problem Set 2 (Q4))

• Camera calibration

• Vanishing points and lines (Problem Set 2

(Q1))

19-Oct-112

Reading:

• [FP] Chapter 3

• [HZ] Chapter 7, 8.6




• Affine camera model


• Vanishing points and lines

19-Oct-113

Reading:

• [FP] Chapter 3


Lecture 8 -Fei-Fei Li 19-Oct-114

Projective cameraf

Oc

f = focal length


Projective camera

x

y

xc

yc

C=[uo, vo]

f

Oc

f = focal length

uo, vo = offset


Projective cameraf

Oc

Units: k,l [pixel/m]

f [m]

[pixel],αααα ββββNon-square pixels

f = focal length

uo, vo = offset

→ non-square pixels,αααα ββββ


Projective camera

=

10100

00

0

z

y

x

v

us

P' o

o

βα

f

Oc

K has 5 degrees of freedom!

Pc

P’

f = focal length

uo, vo = offset

→ non-square pixels,αααα ββββθ = skew angle


Projective cameraf

Oc

−

=′

1

z

y

x

0100

0v0

0ucot

P o

o

sinθθθθββββ

θθθθαααααααα

Pc

P’

f = focal length

uo, vo = offset


K has 5 degrees of freedom!


Projective cameraf

Oc

Pc

Ow

iw

kw

jwR,T

P’

f = focal length

uo, vo = offset


R,T = rotation, translation

wPTR

P44

10 ×

=

cORT~−=


Projective camera

f = focal length

uo, vo = offset

→ non-square pixels,αααα ββββ

f

Oc

P

Ow

iw

kw

jw

wPMP =′

[ ] wPTRK=Internal (intrinsic) parameters

External (extrinsic) parameters

θ = skew angle

R,T = rotation, translation

P’

R,T


Projective camera

wPMP =′ [ ] wPTRK=Internal (intrinsic) parameters

External (extrinsic) parameters


Projective camera

wPMP =′ [ ] wPTRK=

−

=100

v0

ucot

K o

o

sinθθθθββββ


=T3

T2

T1

R

r

r

r

=

z

y

x

t

t

t

T

43×


Goal of calibration

wPMP =′ [ ] wPTRK=

−

=100

v0

ucot

K o

o

sinθθθθββββ


=T3

T2

T1

R

r

r

r

=

z

y

x

t

t

t

T

43×

Estimate intrinsic and extrinsic parameters

from 1 or multiple images







19-Oct-1114

Reading:

• [FP] Chapter 3



Weak perspective projection

Relative scene depth is small compared to its distance from the camera

= magnification

−=−=

myy

mxx

'

'

0

'where

z

fm −=


Orthographic (affine) projection

Distance from center of projection to image plane is infinite

==

y'y

x'x


Affine cameras

[ ] PTRKP ='

=100

00

0s

K y

x

αααααααα

=10

TR

1000

0010

0001

KM

Affine case

Parallel projection matrix

=10

TR

0100

0010

0001

KM

=100

y0

xs

K oy

ox

αααααααα

Projective caseCompared to


Remember….

Projectivities:

=

=

1

y

x

H

1

y

x

bv

tA

1

'y

'x

p

Affinities:

=

=

1

y

x

H

1

y

x

10

tA

1

'y

'x

a


[ ] PTRKP ='

=100

00

00

y

x

K αα

=10

TR

1000

0010

0001

KM

=

=×

×=10

bA

1000

]affine44[

1000

0010

0001

]affine33[ 2232221

1131211

baaa

baaa

M

=+=

+

=

=

1'

2

1

232221

131211 PMP

b

b

Z

Y

X

aaa

aaa

y

xP EucbA

[ ]bAMM Euc ==

We can obtain a more compact formulation than:

Affine cameras


Affine cameras

PP’

P’

;1

'

=+=

=

PbAP M

v

uP [ ]bAM =

M = camera matrix

[non-homogeneous image coordinates]

To recap:

This notation is useful when we’ll discuss affine structure from motion


Affine cameras

• Weak perspective much simpler math.– Accurate when object is small and distant.

– Most useful for recognition.

• Pinhole perspective much more accurate for scenes.– Used in structure from motion.


The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang HuiYou tube video – click here

Weak perspective projection - examples


Weak perspective projection - examples

Qingming Festival by the Riverside Zhang Zeduan ~900 AD







19-Oct-1124

Reading:

• [FP] Chapter 3



Calibration Problem

•P1… Pn with known positions in [Ow, iw, jw, kw]

•p1, … pn known positions in the image

Goal: compute intrinsic and extrinsic parameters

jC

Calibration rig


Remember the “digital Michelangelo project”?

19-Oct-1126


Calibration Problem

jC

Calibration rig

How many correspondences do we need?

•M has 11 unknown • We need 11 equations • 6 correspondences would do it


Calibration Problem

imagejC

Calibration rig

In practice: user may need to look at the

image and select the n>=6 correspondences


Calibration Problem

jC

ii PMp →

=→

i

ii v

up

=

3

2

1

M

m

m

m

⋅⋅⋅⋅

=

i

i

i

i

P

PP

P

3

2

3

1

mmmm

in pixels


Calibration Problem

i3

i1i P

Pu

mm=

i2i3i P)P(v mm =→

i1i3i P)P(u mm =→

i3

i2i P

Pv

mm=

i

i

v

u

=

i3

i2

i3

i1

P

PP

P

mmmm

0)( 23 =−→ iii PPv mm

0)( 13 =−→ iii PPu mm


Calibration Problem

……

0)( 12131 =− PPv mm

0)( 11131 =− PPu mm

0)( 23 =− iii PPv mm

0)( 13 =− iii PPu mm

0)( 23 =− nnn PPv mm

0)( 13 =− nnn PPu mm


Block Matrix Multiplication

=

=

2221

1211

2221

1211

BB

BBB

AA

AAA

What is AB ?

++++

=2222122121221121

2212121121121111

BABABABA

BABABABAAB


Calibration Problem

2n x 12 12x1

1x4

=T3

T2

T1

def

m

m

m

m

4x1

…

Homogenous linear system

knownunknown

0)( 12131 =+− PPv mm

0)( 11131 =+− PPu mm

0)( 23 =+− nnn PPv mm

0)( 13 =+− nnn PPu mm


Homogeneous M x N Linear Systems

A x 0=

Rectangular system (M>N)

• 0 is always a solution

Minimize |Ax|2

under the constraint |x|2 =1

M=number of equations

N=number of unknown

• To find non-zero solution


How do we solve this homogenous linear system?

Calibration Problem

Singular Value Decomposition (SVD)


Calibration Problem

1212T

121212n2 VDU ×××

Last column of V gives m

MiPM ip→

Compute SVD

decomposition of P

Why? See page 593 of

Hartley & Zisserman


Extracting camera parameters

A

=T3

T2

T1

A

a

a

a

[ ]TRK=

3

1

a±=ρρρρ

=

3

2

1

b

b

b

b

Estimated values

)(u 212

o aa ⋅= ρρρρ)(v 32

2o aa ⋅= ρρρρ

( ) ( )3231

3231cosaaaaaaaa

×⋅××⋅×=θθθθ

Intrinsic

b

−

=100

v0

ucot

K o

o

sinθθθθββββ

θθθθααααααααρ


Theorem (Faugeras, 1993)

[ ] [ ] ][ bATKRKTRKM ===

=

3

2

1

a

a

a

A

=100

0 y

x

c

cs

K βα

lf

;kf

==

ββββαααα



A

=T3

T2

T1

A

a

a

a

[ ]TRK=

=

3

2

1

b

b

b

b

Estimated values

Intrinsic

θθθθρρρραααα sin312 aa ×=

θθθθρρρρββββ sin322 aa ×=

b

f

ρ



Extrinsic

( )32

321 aa

aar

××=

33

1

ar

±=

132 rrr ×= b1KT −= ρρρρ

A

=T3

T2

T1

A

a

a

a

[ ]TRK=

=

3

2

1

b

b

b

b

Estimated values

b

ρ


Calibration DemoCamera Calibration Toolbox for Matlab J. Bouguet – [1998-2000]

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples


Calibration Demo







(Q1))

19-Oct-1149

Reading:

• [FP] Chapter 3



Properties of Projection

•Points project to points

•Lines project to lines


Properties of Projection

Vanishing point•Angles are not preserved

•Parallel lines meet


Lines in a 2D plane

0cbyax =++

-c/b

-a/b

=c

b

a

l

If x = [ x1, x2]T ∈ l 0

c

b

a

1

x

xT

2

1

=

l

x

y


Lines in a 2D plane

Intersecting lines

llx ′×= l

l′

Proof

lll ⊥′×lll ′⊥′×

0l)ll( =⋅′×→0l)ll( =′⋅′×→

lx∈→

x

lx ′∈→

→ x is the intersecting point

x

y

x


Points at infinity (ideal points)

0x,

x

x

x

x 3

3

2

1

≠

=

=c

b

a

l

′=′

c

b

a

l

−′−=′×→0

a

b

)cc(llLet’s intersect two parallel lines:

Agree with the general idea of two lines intersecting at infinity

l

l′

=∞

02

1

x

x

x


Lines at infinity ∞l

Set of ideal points lies on a line called the line at infinity

How does it look like?

∞l

=∞

1

0

0

l

T

2

1

0

x

x

0

1

0

0

=

Indeed:


Projective projections of lines at infinity (2D)

lHl T−=′

=

bv

tAH

?lH T =∞−

=

=

−

b

t

t

1

0

0

bv

tAy

xT

is it a line at infinity?

…no!

?lH TA =∞−

=

−=

=

−

−−

1

0

0

1

0

0

1

0

1

0

0

10 TT

TT

At

AtA


horizon

- Recognize the horizon line

- Measure if the 2 lines meet

at the horizon

- if yes, these 2 lines are //

•Recognition helps reconstruction!

•Humans have learnt this

Are these two lines parallel or not?

∞−= lHl T

hor

Projective projections of lines at infinity (2D)


Vanishing points (= ideal points in 2D)

[ ]TRKM =

dv K=

d=direction of the line

d

C

vVanishing points

= points where

parallel lines

intersect in 3D

Image of a vanishing point =


Horizon

•Sets of parallel lines on the same plane lead to collinear vanishing

points [The line is called the horizon for that plane]

horizon


C

n

horizTK ln =

horizl

Horizon


Criminisi & Zisserman, 99

ApplicationThese transformations are used in single view metrology


Applicationthese transformations are used in single view metrology



La Trinita' (1426)

Firenze, Santa Maria

Novella; by Masaccio

(1401-1428)




Hoiem et al, 05…


http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html


A software: Make3D“Convert your image into 3d model”

Saxena, Sun, Ng, 05…

http://make3d.stanford.edu/

http://make3d.stanford.edu/images/view3D/185

http://make3d.stanford.edu/images/view3D/931?noforward=true

http://make3d.stanford.edu/images/view3D/108


19-Oct-1166


What we have learned today





(Q1))

19-Oct-1167

Reading:

• [FP] Chapter 3



Supplementary Materials

19-Oct-1168


Degeneracy and distortion in

real-world camera calibration

19-Oct-1169


Degenerate cases

•Pi’s cannot lie on the same plane!

• Points cannot lie on the intersection curve of two

quadric surfaces


Radial Distortion

No distortion

Pin cushion

Barrel

– Caused by imperfect lenses

– Deviations are most noticeable for rays that pass through the

edge of the lens


Radial Distortion

ii

ii p

v

uPM

100

00

00

1

1

=

→

λλλλ

λλλλ

d

v

vucvbuad 222 ++=

u

∑±==

3

1p

2ppdκ1λ

Polynomial function

Distortion coefficient

To model radial behavior


Radial Distortion

=

i

ii v

up

=

3

2

1

Q

q

q

q

=

i3

i2

i3

i1

P

PP

P

qqqq

ii

ii p

v

uPM

100

00

00

1

1

=

→

λλλλ

λλλλ

Q

==

PPv

PPu

2i3i

i1i3i

qq

qq

Non-linear system of equations


General Calibration Problem

)(PfX =

measurement parameter

f( ) is nonlinear

-Newton Method

-Levenberg-Marquardt Algorithm

• Iterative, starts from initial solution

• May be slow if initial solution far from real solution

• Estimated solution may be function of the initial solution

• Newton requires the computation of J, H

• Levenberg-Marquardt doesn’t require the computation of H



A possible algorithm

1. Solve linear part of the system to find approximated solution

2. Use this solution as initial condition for the full system

3. Solve full system (including distortion) using Newton or L.M.

)(PfX =


f( ) is nonlinear



)(PfX =


f( ) is nonlinear

Typical assumptions for computing initial condition :

- zero-skew, square pixel

- uo, vo = known center of the image

- no distortion

Just estimate f

and R, T


Tsai’s calibration technique

1. Estimate m1 and m2 first:

=

i

ii v

up

=

i3

i2

i3

i1

P

PP

P

1

mmmm

λλλλHow to do that?

d

v

uHint: slopev

u

i

i =



1. Estimate m1 and m2 first:

=

i

ii v

up

=

i3

i2

i3

i1

P

PP

P

1

mmmm

λλλλ

0)()( 121111 =− PuPv mm

0)()( 21 =− iiii PuPv mm

0)()( 21 =− nnnn PuPv mm

…

0Q =n

=

2

1

m

mn

i

i

i

i

i

i

i

i

P

P

P

P

P

P

v

u

2

1

3

2

3

1

)()(

)()(

mm

mmmm

==



2. Once that m1 and m2 are estimated, estimate m3:

=

i

ii v

up

=

i3

i2

i3

i1

P

PP

P

1

mmmm

λλλλ

3m is non linear function of 1m 2m λλλλ

There are some degenerate configurations for which m1 and m2 cannot be computed

Lecture 8: Camera Calibration - Artificial...

Documents

Transcript of Lecture 8: Camera Calibration - Artificial...