Lecture 7 Measurement Using a Single Camerasse.tongji.edu.cn/linzhang/DIP/slides/Lecture 07...Lin...

Lin ZHANG, SSE, 2016

Lecture 7Measurement Using a Single Camera

Lin ZHANG, PhDSchool of Software Engineering

Tongji UniversityFall 2016


If I have an image containing a coin, can you tell me the diameter of that coin?


Contents

• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation


Vector representation

{ , , }a xi y j zk x y z

Length (or norm) of a vector

2 2 2a x y z

Normalized vector (unit vector)

{ , , }a x y za a a a

We say if and only if,a 0 0, 0, 0x y z

Vector operations


if 1 1 1( , , ),a x y z

2 2 2( , , ),b x y z

1 2 1 2 1 2( , , ),a b x x y y z z

then

1 2 1 2 1 2cosa b a b x x y y z z

Dot product (inner product)

Laws of dot product:

, ( )= +a b b a a b c a b a c

Theorem

0a b a b (why?)

a

b

Vector operations


Cross product

1 1 1 1 1 1

1 1 1

2 2 2 2 2 2

2 2 2

i j ky z z x x y

a b x y z i j ky z z x x y

x y z

Vector operations


c a b

Cross product

is also a vector, whose direction is determined by the right‐hand law and

a

bc

,c a c b

sinc a b

represents the oriented area of the parallelogram taking and as two sidesc

a b(easy to prove)

1 2 2 1r r r r

(why?)

Vector operations


they are not equal to zero at the same time, and

Cross product

Theorem|| 0a b a b (why?)

Theorem

|| , ,a b

a b 0

1 2 3 1 2 1 3( )r r r r r r r

Property

(easy to understand)

Vector operations


Mixed product (scalar triple product or box product)

Geometric Interpretation: it is the (signed) volume of the parallelepiped defined by the three vectors given

1 2 3

1 2 3

1 2 3

( , , ) ( )a a ab b bc c c

a b c a b c

Vector operations



1 2 3

1 2 3

1 2 3

( , , ) ( )a a ab b bc c c

a b c a b c

( ) cos

sin cos

a b c a b c

a b c

a

cBase h

Vector operations



1 2 3

1 2 3

1 2 3

( , , ) ( )a a ab b bc c c

a b c a b c

Property:

( , , ) ( , , ) ( , , ) a b c b c a c a b( , , ) ( , , ) ( , , ) a b c b a c a c b

Vector operations

why?


Mixed product (scalar triple product or box product)Theorem

, ,a b c are coplanar ( , , ) 0 a b c

, ,a b c are coplanar , , ,v they are not equal to zero at the same time, and v a b c 0

Vector operations

why?


Foundations of Projective Geometry• What is homogeneous coordinate?

For a normal point on a plane , ,Tx y 0Its homogenous coordinate is where k can be anynon‐zero real number

, ,1 ,Tk x y

For a homogenous coordinate

Homogenous coordinate for a point is not only one

' ' ', ,T

x y z

we usually rewrite it as ' ' ' '/ , / ,1T

x z y z



Converting from homogenous coordinate to inhomogeneous coordinate

''

''

''

'

xxzyy

zz

For a normal point on a plane 0Its homogenous coordinate is where k can be anynon‐zero real number

, ,1 ,Tk x y , ,Tx y



Geometric interpretation

1e

2e

o

1o

1

00 0( , )M x y

3e

1e

2e

In plane , in the 2D frame01 1 2( : , )o e e 0 0: ( , )M x y

Coordinate of any point on line OMin the frame is the homogeneous coordinate of M

1 2 3( : , , )o e e e

These points can be represented as

0 0, ,1 Tk x y

, one point




1e

2e

o

1o

1

00 0( , )M x y

3e

1e

2e

How about a line passing through O and parallel to ?0

In plane , in the 2D frame01 1 2( : , )o e e 0 0: ( , )M x y

Coordinate of any point on line OMin the frame is the homogeneous coordinate of M

, one point

These points can be represented as

0 0, ,1 Tk x y

1 2 3( : , , )o e e e




1e

2e

o

1o

1

03e

1e

2e


Consider a line passing through Oand M(x0, y0, 0)T

0 0( , )M x y We define: it meets at an infinity point, and also the homogeneous coordinate of such a point can be represented as points on OM

0

So, infinity point has the form (kx0, ky0, 0)T


Foundations of Projective Geometry• What is homogeneous coordinate?Normal case:

line k (x0, y0, 1)

a normal point (x0, y0) on the plane 0

The homogeneous coordinate of this normal point is k(x0, y0 ,1)

abnormal case:

line (kx0, ky0, 0)

Define: it meets at an infinity point

0

The homogeneous coordinate of this infinity point is k(x0, y0 ,0)

make an analogy




1e

2e

o

1o

1

03e

1e

2e


0 0( , )M x y

One infinity point determines an orientation

We define: all infinity points on comprise an infinity line

0

In fact, plane meets at the infinity line

01 2oe e

Homogeneous equation of the infinity line is 0x + 0y + 1z = 0


Foundations of Projective Geometry

Properties of a projective plane• Two points determine a line; two lines determine a point (the

second claim is not correct in the normal Euclidean plane)• Two parallel lines intersect at an infinity point; that means one

infinity point corresponds to a specific orientation• Two parallel planes intersect at the infinity line

0 + infinity line = Projective plane



JingHu High‐speed railway: rails will “meet” at the vanishing point


Foundations of Projective Geometry• Lines in the homogeneous coordinate

On a projective plane, please determine the line passing two points

1e

2e

o1

0M

',o ox x determine two linesxx’ actually is the intersection between oxx’ and 0

'1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x

x'x



1e

2e

o1

0x

'xM Thus, locates on xx’( , , )M x y z oM resides on the plane oxx’ ', ,o oM ox x are coplanar

1 1 1

2 2 2

0x y zx y zx y z

On a projective plane, please determine the line passing two points '

1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x



1 1 1

2 2 2

0x y zx y zx y z

1 1 1 1 1 1

2 2 2 2 2 2

0y z z x x y

x y zy z z x x y

1 1 1 1 1 1

2 2 2 2 2 2

, ,T

y z z x x yy z z x x y

Homogeneous coordinate of the lineHomogeneous coordinate of the infinity line is (0,0,1)T


1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x



1 1 1

2 2 2

0x y zx y zx y z

1 1 1 1 1 1

2 2 2 2 2 2

0y z z x x y

x y zy z z x x y

TheoremOn the projective plane, the line passing two points is',x x

' l x x

1 1 1 1 1 1

2 2 2 2 2 2

, ,T

y z z x x yy z z x x y


1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x



A point is on the line0 0 0( , , )Tx y zx

0T x l

( , , )Ta b cl

0 x l(It is )


Foundations of Projective Geometry• Lines in the homogeneous coordinateTheorem: On the projective plane, the intersection of two lines is the point ' x l l',l lProof: Two lines 1 1 1 0,a x b y c z 2 2 2 0a x b y c z

'1 1 1 2 2 2( , , ) , ( , , )T Ta b c a b c l l

Inhomogeneous form ,x yX Yz z

1 1 1

2 2 2

00

a X b Y ca X b Y c

1 1 1 1

2 2 2 2

1 1 1 1

2 2 2 2

,

c b a cc b a c

X Ya b a ba b a b



1 1 1 1

2 2 2 2

1 1 1 1

2 2 2 2

, ,1

c b a cc b a c

ka b a ba b a b

x

Homogenous form of the cross point is

1 1 1 1 1 1

2 2 2 2 2 2

, ,c b a c a bc b a c a b

x

let 1 1

2 2

a bk

a b

1 1 1 1 1 1

2 2 2 2 2 2

, ,b c c a a bb c c a a b

x ' x l l

Theorem: On the projective plane, the intersection of two lines is the point ' x l l',l l



Example: find the cross point of the lines 1, 1x y

o x

y

1x

1y 1 2 3

1 2 3

1 0 ( 1) 00 1 ( 1) 0x x xx x x

Homogeneous coordinates of the two lines are (1,0, 1) , (0,1, 1)T T Cross point is

(1,0, 1) (0,1, 1) 1,1,1T T

Homogeneous form


1 2

3 3

,x xx yx x


(1,0, 1) (1,0, 2) 1 0 1 0,1,01 0 2

T T

i j k


Example: find the cross point of the lines 1, 2x x

o x

y

1x

Homogeneous coordinates of the two lines are (1,0, 1) , (1,0, 2)T T Cross point is

2x 1 2 3

1 2 3

1 0 ( 1) 01 0 ( 2) 0x x xx x x

Homogeneous form


1 2

3 3

,x xx yx x


Foundations of Projective Geometry• DualityIn projective geometry, lines and points can swap their positions

0T x lIf x is a variable, it represents the points lying on the line l;If l is a variable, it represents the lines passing a fixed point x

How to interpret?

The line passing two points is',x x ' l x xThe cross point of two lines is',l l ' x l l

Duality Principle: To any theorem of projective geometry, there corresponds a dual theorem, which may be derived by interchanging the roles of points and lines in the original theorem


Contents

• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation


Matrix differentiation

• Function is a vector and the variable is a scalar

1 2( ) ( ), ( ),..., ( ) Tnf t f t f t f t

Definition

1 2 ( )( ) ( ), ,...,T

ndf tdf t df tdfdt dt dt dt


• Function is a matrix and the variable is a scalar11 12 1

21 22 2

1 2

( ) ( ),..., ( )( ) ( ),..., ( )

( ) ( )

( ) ( ),..., ( )

m

mij n m

n n nm

f t f t f tf t f t f t

f t f t

f t f t f t

Definition111 12

221 22

1 2

( )( ) ( ) ,...,

( )( ) ( ) ( ),...,

( ) ( ) ( ),...,

m

mij

n m

n n nm

df tdf t df tdt dt dt

df tdf t df t df tdfdt dt dt

dt dt

df t df t df tdt dt dt



• Function is a scalar and the variable is a vector

1 2( ), ( , ,..., )Tnf x x xx x

Definition

1 2

, ,...,T

n

df f f fd x x x

x

In a similar way,

1 2( ), ( , ,..., )nf x x xx x

1 2

, ,...,n

df f f fd x x x

x



• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T

n mx x x y y y x y x x xDefinition

1 1 1

1 2

2 2 2

1 2

1 2

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

n

nT

m m m

n m n

y y yx x x

y y yd x x xd

y y yx x x

x x x

x x xyx

x x x



• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T

n mx x x y y y x y x x xIn a similar way,

1 2

1 1 1

1 2

2 2 2

1 2

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

m

mT

m

n n n n m

y y yx x x

y y yd x x xd

y y yx x x

x x x

x x xyx

x x x



• Function is a vector and the variable is a vectorExample:

11 2 2

2 1 1 2 2 3 22

3

( ), , ( ) , ( ) 3

( )

xy

x y x x y x xy

x

xy x x x

x

1 2

1 11

1 2

2 23

1 2

3 3

( ) ( )

2 0( ) ( ) 1 3

0 2( ) ( )

T

y yx x x

d y yd x x

xy y

x x

x x

y x xx

x x



• Function is a scalar and the variable is a matrix

11 12 1

1 2

( ) n

m m mn

f f fx x x

dfd

f f fx x x

XX

( ), m nf X X

Definition



• Useful results1, nx a

,T Td d

d d

a x x aa ax x

Then,

How to prove?

(1)



• Useful results1,m n nA x (2) Then, T

dA Ad

xx

1,m n nA x (3) Then,T T

Td A Ad

xx

1,n n nA x (4) Then, ( )T

Td A A Ad

x x xx

1 1, ,m n m n X a b (5) Then,T

Tdd

a Xb abX

1 1, ,n m m n X a b (6) Then,T T

Tdd

a X b baX

1nx (7) Then, 2Td

d

x x xx



Contents

• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird‐view Generation


Lagrange multiplier

• Single‐variable function

( )f x is differentiable in (a, b). At , f(x) achieves an extremum

0 ( , )x a b

0| 0x

dfdx

• Two‐variables function

( , )f x y is differentiable in its domain. At , f(x, y) achieves an extremum

0 0( , )x y

0 0 0 0( , ) ( , )| 0, | 0x y x yf fx y


Lagrange multiplier

• In general case1( ), nf x x If is a stationary point of 0x

0 0 01 2

| 0, | 0,..., | 0n

f f fx x x

x x x


Lagrange multiplier

• Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints

Problem: find stationary points for 1( ),x x ny f under m constraints ( ) 0, 1, 2,...,kg k m x

is a stationary point of with constraints

Solution:

11

( ; ,..., ) ( ) ( )m

m k kk

F f g

x x xIf is a stationary point of F, then,

0 10 20 0( , , ..., )m x

0x ( )f xJoseph‐Louis LagrangeJan. 25, 1736~Apr.10, 1813


Lagrange multiplier

• Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints

Solution:1

1( ; ,..., ) ( ) ( )

m

m k kk

F f g

x x x

is a stationary point of F0 10 0( , ,..., )m x

1 2 1 2

0, 0,..., 0, 0, 0,..., 0n m

F F F F F Fx x x

n + m equations!at that point

Problem: find stationary points for 1( ),x x ny f under m constraints ( ) 0, 1, 2,...,kg k m x


Lagrange multiplier

• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.

y=xp0

?

The distance is 2 2( , ) ( 1) ( 0)f x y x y

Now we want to find the stationary point of f(x, y) under the constraint

( , ) 0g x y y x According to Lagrange multiplier method, construct another function

2 2( , , ) ( ) ( ) ( 1) ( )F x y f x g x x y y x Find the stationary point for ( , , )F x y


Lagrange multiplier

• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.

y=xp0

?

0

0

0

FxFyF

2( 1) 02 0

0

xy

x y

0.50.51

xy

(0.5,0.5,1) is a stationary point of ( , , )F x y (0.5,0.5) is a stationary point of f(x,y) under constraints


Contents

• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird‐view Generation


LS for Inhomogeneous Linear SystemConsider the following linear equations system

1 2 1

1 2 2

3 1 1 32 4 2 1 4x x x

x x x

Matrix form: A x bA x b

It can be easily solved 1

2

12

xx


LS for Inhomogeneous Linear SystemHow about the following one?

1 21

1 22

1 2

3 1 1 32 4 2 1 4

1 2 62 6

x xx

x xx

x x

It does not have a solution!

What is the condition for a linear equation system can be solved?

A x b

Can we solve it in an approximate way?A: we can use least squares technique!

Carl Friedrich Gauss


LS for Inhomogeneous Linear SystemLet’s consider a system of p linear equations with q unknowns

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

...

...

......

q q

q q

p p pq q p

a x a x a x

a x a x a xA

a x a x a x

b

bx b

b

We consider the case: p>q, and rank(A)=qIn general case, there is no solution!

Instead, we want to find a vector x that minimizes the error:22

1 11

( ) ( ... )p

i iq q ii

E a x a x A

x b x b

unknowns


LS for Inhomogeneous Linear System2*2

arg min ( ) arg minE A x x

x x x b

1* T TA A A

x bPseudoinverse of A

How about the pseudoinverse of A when A is square and non-singular?


LS for Homogeneous Linear SystemLet’s consider a system of p linear equations with q unknowns

11 1 12 2 1

21 1 22 2 2

1 1 2 2

... 0

... 0

...... 0

q q

q q

p p pq q

a x a x a x

a x a x a xA

a x a x a x

x 0

We consider the case: p>q, and rank(A)=q

unknowns

Theoretically, there is only a trivial solution: x = 0

So, we add a constraint to avoid the trivial solution 2

1x


LS for Homogeneous Linear SystemWe want to minimize , subject to 2

1x 2

2( )x xE A

*2

arg min ( ), . ., 1x

x x xE s t

Use the Lagrange multiplier to solve it,

2 2*2 2

arg min 1x

x x xA

(1)

Solving the stationary point of the Lagrange function,

2 2

2 2

2 2

2 2

1

10

x x0

x

x x

A

A

(3)

(2)


LS for Homogeneous Linear System

Then, we have

2 2

2 21x x

0x

A

(3)

x xTA A

x is the eigen‐vector of ATA associated with the eigenvalue

2

2x x x x x xT T TE A A A

The unit vector x is the eigenvector associated with the minimum eigenvalue of TA A


Contents

• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation


Homography Estimation

Problem definition: Given a set of points and a corresponding set of points in a projective plane, compute the projective transformation that takes to

ix 'ix

ix'ix

' , 1, 2,...,i iH i n x xWe know there existing an H satisfying Coordinates of and are known, ix 'ix we need to find Hwhere H is a homography matrix

11 12 13

21 22 23

31 32 33

a a aH a a a

a a a

It has 8 degrees of freedom


Homography Estimation

11 12 13

21 22 23

31 32 33 1

a a acu xcv a a a yc a a a

11 12 13

21 22 23

31 32 33

a x a y a cua x a y a cva x a y a c

11 12 13

31 32 33

21 22 23

31 32 33

a x a y a ua x a y aa x a y a va x a y a

4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom


Homography Estimation4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom 11

12

13

21

22

23

31

32

33

1 0 0 00

0 0 0 1

aaaa

x y ux uy ua

x y vx vy vaaaa

Thus, four correspondence pairs generate 8 equations


Homography Estimation4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom

0 (1)A x

8 9 9 1

Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space

( ) 8Rank A


Homography Estimation• How about the case when there are more than 4 correspondence pairs?• Use the LS method (for homogeneous case) to solve the model


Contents



• Camera calibration is a necessary step in 3D computer vision in order to extract metric information from 2D images

• It estimates the parameters of a lens and image sensor of the camera; you can use these parameters to correct for lens distortion, measure the size of an object in world units, or determine the location of the camera in the scene

• These tasks are used in applications such as machine vision to detect and measure objects. They are also used in robotics, for navigation systems, and 3‐D scene reconstruction

What is camera calibration?


• Camera parameters include• Intrinsics• Extrinsics• Distortion coefficients



Contents



• For simplicity, usually we use a pinhole camera model

Single Camera Calibration


• To model the image formation process, 4 coordinate systems are required• World coordinate system (3D space)• Camera coordinate system (3D space)• Retinal coordinate system (2D space)• Pixel coordinate system (2D space)



• To model the image formation process, 4 coordinate systems are required



• From the world CS to the camera CS


, , Tw w wX Y Z is a 3D point represented in the WCS

In the camera CS, it is represented as,

R tc w

c w

c w

X XY YZ Z

a rotation matrix (orthogonal)

3 3

a translation vector3 1


• From the world CS to the camera CS


, , Tw w wX Y Z is a 3D point represented in the WCS

In the camera CS, it is represented as,

R tc w

c w

c w

X XY YZ Z

1

1 1

R t0

c w

c wT

c w

X XY YZ Z

Homogeneous form

(1)


• From the camera CS to the retinal CS


We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS





x

y

z

f

O is the optical center, f is the focal length, P = [Xc, Yc, Zc]T is a scene point while P’=(x, y) is its image on the retinal plane





cc

cc

cc

c

XfX

ZxY

y YfZ

Z

0 0 00 0 0

1 0 0 1 01

c

cc

c

Xx f

YZ y f

Z

Homogeneous form

(2)


• From the retinal CS to the pixel CS


The unit for retinal CS (x-y) is physical unit (e.g., mm, cm) while the unit for pixel CS (u-v) is pixel

One pixel represents dx physical units along the x‐axis and represents dy physical units along the y‐axis; the image of the optical center is (u0, v0)

0

0

1 0

101 1

0 0 1

udxu x

v v ydy


• From the retinal CS to the pixel CS


If the two axis of the pixel plane are not perpendicular to each other, another parameter s is introduced to represent the skewness of the two axis

0

0

1 0

101 1

0 0 1

udxu x

v v ydy

0

0

1

101 1

0 0 1

s udxu x

v v ydy

(3)


Single Camera CalibrationFrom Eqs.1~3, we can have

0 0

0 0

0 0

0

1 00 0 0

10 . 0 0 0 0 0 .1 0 0 1 0

0 0 1 0 0 1 01 1

0 00 0 .0 0 1 0

1

c c

c cc

c c

c

c

c

fs u sf uX Xdx dxu fY YfZ v v f vZ Zdy dy

Xu u

Yv

Z

0

0 00 0 . 0 .1

0 0 1 0 0 0 11 1

R tR t

0

w w

w wT

w w

X Xu

Y Yv v

Z Z

Intrinsic Extrinsic



0

0[ ] 0 [ ]1 0 0 1

1 1

K R t R t

w w

w wc

w w

X Xuu

Y YZ v v

Z Z

Intrinsic Extrinsic

, the coordinates of the principal point in the image plane 0 0,u v, the scale factors in image u and v axes and

, describing the skewness of the two image axesR and t determines the rigid transformation from the world coordinate system to the camera coordinate system Altogether, there are 11 parameters to be determined



• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion• Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion



• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion• Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion

2 4 61 2 3

2 4 61 2 3

1

1

distorted

distorted

u u k r k r k r

v v k r k r k r

where 2 2 2r u v are the radial distortion coefficients of the lens 1 2 3, ,k k k



• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion

• Tangential distortion occurs when the lens and the image plane are not parallel



• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion

• Tangential distortion occurs when the lens and the image plane are not parallel

2 21 2

2 22 1

+ 2 2

2 2

distorted

distorted

u u uv r u

v v uv r v

are the tangential distortion coefficients of the lens 1 2,



• The purpose of the camera calibration is to determine the values for the extrinsics, intrinsics, and distortion coefficients

• How to do?• Zhengyou Zhang’s method[1] is a commonly used modern approach

• A calibration board is needed; several images of the board need to be captured; based on the correspondence pairs (pixel coordinate and world coordinate of a feature point), equation systems can be obtained; by solving the equation systems, parameters can be determined

[1] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000



Calibration board



A set of Calibration board images (50~60)



• Matlab provides a “Camera Calibrator”• Straightforward to use• However, based on my experience, it is not as accurate as the routine provided in openCV3.0, especially for large FOV cameras (such as fisheye camera); thus, for some accuracy critical applications, I recommend to use the openCV function, though a little more complicated

• It exports “cameraParams” as the calibration result



• For our purpose (measuring geometric metrics of a planar object), we use the camera parameters to undistort the image

• The essence of this step is to make sure the transformation from a physical plane to the image plane can be represented by a linear projective matrix; or in other words, a straight line should be mapped to a straight line


Single Camera CalibrationOriginal image

Undistorted image


Contents



• Our task is to measure the geometric properties of objects on a plane (e.g., conveyor belt)

• Such a problem can be solved if we have its bird‐view image; bird’s‐eye‐view is easy for object detection and measurement

Bird’s‐eye‐view Generation


• Three coordinate systems are required• Bird’s‐eye‐view image coordinate system• World coordinate system• Undistorted image coordinate system

X

Y

X

Y

X

Y

Bird’s‐eye‐view image WCS Undistorted image

Similarity Projective



Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is

B WP

W IP

Then, given a position on bird’s‐eye‐view, we can get its corresponding position in the undistorted image as

, ,1 TB Bx y

1x

B

I W I B W B

xP P y

• Basic idea for bird’s‐eye‐view generation

Then, the intensity of the pixel can be determined using some interpolation technique based on the neighborhood around on the undistorted image

, ,1 TB Bx y

xI



Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is

B WP

W IP

• Basic idea for bird’s‐eye‐view generation

The key problem is how to obtain and ?B WP W IP



• Determine B WP

X

Y

X

Y

H(mm)

M (pixels)

N (pixels)



• Determine B WP

02

02

1 11 0 0 1

W B B

W B B W B

H HNM Mx x x

H Hy y P yM

For a point on bird’s‐eye‐view, the corresponding point on the world coordinate system is,

, ,1 TB Bx y

Please verify!!



• Determine W IP

The physical plane (in WCS) and the undistorted image plane can be linked via a homography matrix W IP

x xI W I WP

If we know a set of correspondence pairs , 1,x x N

Ii Wi i

W IP can be estimated using the least‐square method



• Determine W IP

A set of point correspondence pairs; for each pair, we know its coordinate on the undistorted image plane and its coordinate in the WCS



When and are known, the bird’s‐eye‐view can be generated via,

W IP B WP

1 1x

B B

I W I B W B B I B

x xP P y P y




Original image Bird’s‐eye‐view


Bird‐view GenerationAnother example

Original fish‐eye image Undistorted image


Bird‐view GenerationAnother example

Bird’s‐eye‐viewOriginal fish‐eye image


Thanks for your attention

Lecture 7 Measurement Using a Single Camerasse.tongji.edu.cn/linzhang/DIP/slides/Lecture 07...Lin...

Documents

Transcript of Lecture 7 Measurement Using a Single Camerasse.tongji.edu.cn/linzhang/DIP/slides/Lecture 07...Lin...