CMPUT 412 3D Computer Vision

Presented by Azad Shademan

Feb. 13-15, 2007

Acknowledgement

• Marc Pollefeys’ slides (University of North Carolina), comp290-89 Spring 2003

http://www.cs.unc.edu/~marc/mvg/slides.html

Motivation

• Images form one of the most important feedback signals in robotics.

• Vision-based control:– Robotic Visual Servoing– Mobile Navigation

Eye-In-Hand Visual Servoing

More motivation

• Course project:– Two cameras looking at one scene– Need to “correspond” similar points in these two

imagesLaser projection on the wall

Image planeLeft camera

Image planeRight camera

Camera Model

Mostly pinhole camera model

Pinhole Camera Model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

101

01

01

1Z

Y

X

f

f

Z

fY

fX

0|I)1,,(diagP ff

Pinhole Camera Model

Tyx

T pZfYpZfXZYX )/,/(),,(

principal pointT

yx pp ),(

101

0

0

1

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

Z

Y

X

y

x

x

x

Principle Point Offset

camX0|IKx

1y

x

pf

pf

K calibration matrix

Principle Point Offset

C~

-X~

RX~

cam

X10

RCR

1

10

C~

RRXcam

Z

Y

X

camX0|IKx XC~

|IKRx

t|RKP C~

Rt PXx

Camera Rotation and Orientation

X.xλor

1

10

t

1

1

1

11

λ3

PR

Z

Y

X

pf

pf

y

x

y

x

T

Hierarchy of transformations

vTv

tAProjective15dof

Affine12dof

Similarity7dof

Euclidean6dof

Intersection and tangency

Parallellism of planes,Volume ratios, centroids

Volume

10

tAT

10

tRT

s

10

tRT

Camera Calibration

• Procedure that finds the camera matrix, i.e.,– Intrinsic camera parameters– Extrinsic camera parameters

• Rotation• Orientation

• There is a lot to be done with calibrated cameras, BUT we are more interested in uncalibrated cameras (more challenging)

Some linear algebra to get started

• Singular value decomposition (SVD)– Decomposes a matrix as follows:

TVUΣA

Tnnnmmmnm VΣUA

IUU T

021 n

IVV T

nm

XXVT XVΣ T XVUΣ T

TTTnnn VUVUVUA 222111

000

00

00

00

Σ

2

1

n

Singular Value Decomposition

Singular Value Decomposition

• Homogeneous least-squares

• Span and null-space

• Closest rank r approximation

• Pseudo inverse

1X AXmin subject to nVX solution

nrrdiag ,,,,,, 121 0 ,, 0 ,,,,~

21 rdiag TVΣ~

UA~

TVUΣA

0000

0000

000

000

Σ 2

1

4321 UU;UU LL NS 4321 VV;VV RR NS

TUVΣA 0 ,, 0 ,,,, 112

11 rdiag

TVUΣA

Parameter estimation

• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)

• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)

• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)

Number of measurements required

• At least as many independent equations as degrees of freedom required

• Example:

Hxx'

11

λ

333231

232221

131211

y

x

hhh

hhh

hhh

y

x

2 independent equations / point8 degrees of freedom

4x2≥8

Approximate solutions

• Minimal solution4 points yield an exact solution for H

• More points– No exact solution, because

measurements are inexact (“noise”)– Search for “best” according to some cost

function– Algebraic or geometric/statistical cost

Direct Linear Transformation(DLT)

ii Hxx 0Hxx ii

i

i

i

i

xh

xh

xh

Hx3

2

1

T

T

T

iiii

iiii

iiii

ii

yx

xw

wy

xhxh

xhxh

xhxh

Hxx12

31

23

TT

TT

TT

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

Tiiii wyx ,,x

0hA i

Direct Linear Transformation(DLT)

• Equations are linear in h

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

0AAA 321 iiiiii wyx

0hA i

• Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)

0

h

h

h

x0x

xx0

3

2

1

TTT

TTT

iiii

iiii

xw

yw

(only drop third row if wi’≠0)• Holds for any homogeneous

representation, e.g. (xi’,yi’,1)

Direct Linear Transformation(DLT)

• Solving for H

0Ah 0h

A

A

A

A

4

3

2

1

size A is 8x9 or 12x9, but rank 8

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interestpick for example the one with 1h

Direct Linear Transformation(DLT)

• Over-determined solution

No exact solution because of inexact measurementi.e. “noise”

0Ah 0h

A

A

A

n

2

1

Find approximate solution- Additional constraint needed to avoid 0, e.g.

- not possible, so minimize

1h Ah0Ah

DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h

Normalized DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii inormiinormi xTx~,xTx~

norm-1

norm TH~

TH

Reprojection error

221- Hx,xxHx, dd

22 x̂,xx̂x, dd

OpenCV

• cvFindHomography()

• cvFindFundamentalMat()

• …

cvFindHomographyFinds perspective transformation between two planes void cvFindHomography( const CvMat* src_points, const CvMat* dst_points,

CvMat* homography ); src_points

Point coordinates in the original plane, 2xN, Nx2, 3xN or Nx3 array (the latter two are for representation in homogeneous coordinates), where N is the number of points.

dst_pointsPoint coordinates in the destination plane, 2xN, Nx2, 3xN or Nx3 array (the latter two are for

representation in homogeneous coordinates) homography

Output 3x3 homography matrix. The function cvFindHomography finds perspective transformation H=||hij|| between the

source and the destination planes: [x',,i,,] [x,,i,,] s,,i,,[y',,i,,]~H*[y,,i,,] [1 ] [ 1] So that the back-projection error is minimized: sum_i((x',,i,,-(h11*x,,i,, + h12*y,,i,, + h13)/(h31*x,,i,, + h32*y,,i,, + h33))^2^+ (y',,i,,-

(h21*x,,i,, + h22*y,,i,, + h23)/(h31*x,,i,, + h32*y,,i,, + h33))^2^) -> min The function is used to find initial intrinsic and extrinsic matrices. Homography matrix is determined up to a scale, thus it is normalized to make h33=1.