3-D Scene

u

u’

Study the mathematical relations between corresponding image points.

“Corresponding” means originated from the same 3D point.

Objective

Two-views geometryOutline

Background: Camera, Projection models Necessary tools: A taste of projective geometry Two view geometry:

Planar scene (homography ). Non-planar scene (epipolar geometry).

3D reconstruction (stereo).

Perspective Projection

f Xx

Zf Y

yZ

Origin (0,0,0) is the Focal center X,Y (x,y) axis are along the image axis (height / width). Z is depth = distance along the Optical axis f – Focal length

Coordinates in Projective Plane P2

k(0,0,1)

k(x,y,0)

k(1,1,1)

k(1,0,1)

k(0,1,1)

“Ideal point”

Take R3 –{0,0,0} and look at scale equivalence class (rays/lines trough the origin).

z

y

x

z

y

x

2D Projective Geometry: Basics A point:

A line:

we denote a line with a 3-vector

Line coordinates are homogenous

Points and lines are dual: p is on l if

Intersection of two lines/points

2 2( , , ) ( , )T Tx yx y z P

z z

0 ( ) ( ) 0x y

ax by cz a b cz z

0Tl p

1 2 ,l l 1 2p p

( , , )Ta b c

ll

Cross Product in matrix notation [ ]x

0

0

0

xy

xz

yz

x

tt

tt

tt

t1 2 1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

x x y z z y

y y z x x z

z z x y y x

0

0

0

x y z z y

y z x z x

z x y y x

t x t z t y t t x

t y t x t z t t y

t z t y t x t t z

Hartley & Zisserman p. 581

ptpt x

2D Projective Transformation

H is defined up to scale

9 parameters 8 degrees of freedom Determined by 4 corresponding points

how does H operate on lines?

0

1: 0 ( )( ) 0T T Tl H l l p l H Hp

Hartley & Zisserman p. 32

HH

Two-views geometryOutline

Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:

Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix

3D reconstruction from two views (Stereo algorithms)

Two View Geometry When a camera changes position and

orientation, the scene moves rigidly relative to the camera

3-D Scene

u

u’

X

Y

Z

d

p

Rotation + translation

Two View Geometry (simple cases) In two cases this results in homography:

1. Camera rotates around its focal point

2. The scene is planar

Then: Point correspondence forms 1:1mapping depth cannot be recovered

Camera Rotation

' , 0

( )

'' ' ( ' ')

' ( ' )'

P RP t

Zp P P p

f

Zp P P p

f

Zp Rp p Rp

Z

(R is 3x3 non-singular)

Planar Scenes

IntuitivelyA sequence of two perspectivities

Algebraically

Need to show:

( )

1'

1, '

' ,'

T

TT

T

n P d aX bY cZ d

n PP RP t RP t R tn P

d d

H R tn P HPd

Zp Hp

Z

Scene

Camera 1

Camera 2

Hpp '

Summary: Two Views Related by HomographyTwo images are related by homography:

One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines

2 equations 4 points are required to recover H Depth cannot be recovered

' ,'

Zp Hp

Z

The General Case: Epipolar Lines

epipolar lineepipolar line

Epipolar Plane

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

Epipole Every plane through the baseline is an epipolar

plane It determines a pair of epipolar lines (one in each image)

Two systems of epipolar lines are obtained Each system intersects in a point, the epipole The epipole is the projection of the center of the

other camera

epipolar planeepipolar linesepipolar linesepipolar linesepipolar lines

BaselineBaselineOO O’O’

Example

Epipolar Lines

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some worldcoordinates as follows:

' ' 0T

OP OO O P

Essential Matrix (algebraic constraint between corresponding image points) Set world coordinates around the first camera

What to do with O’P? Every rotation changes the observed coordinate in the second image

We need to de-rotate to make the second image plane parallel to the first

Replacing by image points

' ' 0T

OP OO O P

' 0TP t RP

, 'P OP t OO

' 0Tp t Rp Other derivations Hartley & Zisserman p. 241

Essential Matrix (cont.)

Denote this by:

Then

Define

E is called the “essential matrix”

t p t p

' ' 0T Tp t Rp p t Rp

E t R

' 0Tp Ep

' 0Tp t Rp

Properties of the Essential Matrix E is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear E, E can be recovered up to scale using 8 points. Has rank 2.

The constraint detE=0 7 points suffices In fact, there are only 5 degrees of freedom in E,

3 for rotation 2 for translation (up to scale), determined by epipole

0 ': l plpE t

' 0Tp Ep

e) trough lines ( : : 12 all PPEThus

BackgroundThe lens optical axis does not coincide with

the sensor

We model this using a 3x3 matrix the Calibration matrix

Camera Internal Parameters or Calibration matrix

Camera Calibration matrix

The difference between ideal sensor ant the real one is modeled by a 3x3 matrix K:

(cx,cy) camera center, (ax,ay) pixel dimensions, b skew

We end with

0

0 0 1

x x

y y

a b c

K a c

q Kp

Fundamental Matrix

F, is the fundamental matrix.

1 1

1

1

' 0 ( ) ( ') 0

( ) ' 0

T T

T T

T

p Ep K q E K q

q K EK q

F K EK

Properties of the Fundamental Matrix F is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear F, F can be recovered up to scale using 8 points. Has rank 2.

The constraint detF=0 7 points suffices

e) trough lines ( : 12 all PPF

0'Fpp t

Epipolar Plane

l’l’ ll

BaselineBaseline

PP

OO O’O’

Other derivations Hartley & Zisserman p. 223

x X’

ee e’e’

HomographyEpipolar

Form

ShapeOne-to-one mapConcentric epipolar lines

D.o.f.88/5 F/E

Eqs/pnt21

Minimal configuration

45+ (8, linear)

Depth NoYes, up to scale

Scene Planar

(or no translation)

3D scene

Two-views geometry Summary:

0'Fpp tHpp '

Stereo Vision

Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated

cameras Output: 3D structure of scene Steps:

Rectification Matching Depth estimation

Rectification

Image Reprojection reproject image planes onto

common plane parallel to baseline Notice, only focal point of camera

really matters(Seitz)

Rectification

Any stereo pair can be rectified by rotating and scaling the two image planes (=homography)

We will assume images have been rectified so Image planes of cameras are parallel. Focal points are at same height. Focal lengths same.

Then, epipolar lines fall along the horizontal scan lines of the images

Cyclopean Coordinates

Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras

( / 2),

( / 2),

( ) ( ),

2 2

l l

r r

l r

l r l r

l r l r l r

f X b fYx y

Z Zf X b fY

x yZ Z

fbx x

Zb x x b y y fb

X Y Zx x x x x x

Disparity

The difference is called “disparity” d is inversely related to Z: greater sensitivity to

nearby points d is directly related to b: sensitivity to small

baseline

l r

fbZ

x x

l rd x x

Main Step: Correspondence Search What to match?

Objects?

More identifiable, but difficult to compute Pixels?

Easier to handle, but maybe ambiguous Edges? Collections of pixels (regions)?

Random Dot Stereogram

Using random dot pairs Julesz showed that recognition is not needed for stereo

Random Dot in Motion

Finding Matches

SSD error

disparity

1D Search More efficient Fewer false matches

Ordering

Ordering

Comparison of Stereo Algorithms

D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,"

International Journal of Computer Vision, 47 (2002), pp. 7-42.

Ground truthScene

Results with window correlation

Window-based matching(best window size)

Ground truth

Scharstein and Szeliski

Graph Cuts (next class).

Ground truthGraph cuts