Projective Geometry for Computer Vision - romano/talks/scs_05_12_04/scs_05_12_04.pdf · Projective...

Projective Geometryfor Computer Vision

Raquel A. RomanoMIT Artificial Intelligence Laboratory

[email protected]

Scientific Computing SeminarMay 12 , 2004


Raquel A. Romano1

3D Computer VisionClassical Problem:

Given a collection of 2D images,build a model of the 3D world.

Example Applications:•virtual/immersive environments•robotics & autonomous vehicles•minimally invasive surgery



Raquel A. Romano2

Outline

1. Projective Geometry Overview2. Minimal Projective Parameters3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion



Raquel A. Romano3

Image Formation

3D scene imaging 2D images



Raquel A. Romano4

Computer Vision

3D scene model data 2D images

analysis measurement



Raquel A. Romano5

scene point

optical center

image point

optical ray

optical axis

Camera Geometry:Single View

pinhole model of perspective projection

ZYy

ZXx ==

unknown depth at each point

⎥⎦

⎤⎢⎣

⎡+⎥

⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡→⎥

⎦

⎤⎢⎣

⎡

y

x

y

x

cc

yx

fsf

yx

1unknown internal

camera parameters



Raquel A. Romano6

Camera Geometry:Multiple Views

unknown rotations and translations

jkjk TR

ijij TR

ikik TR

jx

kxix

X

TR +⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡→

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

ZYX

ZYX



Raquel A. Romano7

Measured Data:Image Points and Lines

geometric constraint: optical rays intersect in 3Dprojective geometry: express constraint in terms of

measured 2D image featuresScientific Computing SeminarMay 12 , 2004


Raquel A. Romano8

Projective Camera Model

• linear model of image formation

• depth-independent expression for optical ray intersections

• multilinear relations among point and line matches



Raquel A. Romano9

Bilinear Constraints(Longuet-Higgins ,1981, Faugeras, 1992; Hartley, 1992)

iixX λ=

[ ] XTRAx ≅ jxijT

ixijR

XiA jA

ijijj ij TxRx += ιλλ

[ ] 0=iijTj ijxRTx x

jjj

iii

xAx

xAx1

1

−

−

→

→

0=iijTj xFx

[ ] 01 =−−i

ij

iijT

jTj ij x

F

ARTAx44 344 21 x fundamental matrix



Raquel A. Romano10

Fundamental MatrixMaps a point in one image to a line in the

other image that contains its match

jx

kxix ijF kjF

Given matching points in two views, predict the matching point in a third image.



Raquel A. Romano11

Projective Models in Practice• View synthesis and interpolation: point transfer

function for dense point correspondences

• Self-calibration: automatic recovery of internal camera parameters from fundamental matrices

• Bundle adjustment initialization: initial rotation and translation for nonlinear Euclidean optimization



Raquel A. Romano12

Outline




Raquel A. Romano13

Practical Problem

• Few point matches between some views.

• Unstable for estimating geometric relationships.



Raquel A. Romano14

Geometric Consistency

Pairwise geometric relations may be inconsistent.

?



Raquel A. Romano15

Goals• Impose algebraic

geometric constraints on stationary points seen in arbitrarily many views.

• Avoid estimating too many parameters: depths, rotations, translations



Raquel A. Romano16

Geometric Dependencies

ijFijF

ijF

ijF

ijFijF

ijFijF

• Pairwise projective geometric relations are interdependent.

• Approach: define projective dependencies and restrict solutions to be globally consistent



Raquel A. Romano17

Projective Bilinear Parameters

ix jx

0=iijTj xFx

X

[ ] 1−−= iijijT

jij ARTAFx



Raquel A. Romano18

Projective Bilinear Parameters

ijhije jie

ix jx

0=iijTj xFx epipoles

jiij ee

ijhepipolar collineation

[ ] [ ] [ ]xx ijT

i

Ti

ijjjjiij ep

qhqpeF ⎥

⎦

⎤⎢⎣

⎡

−≅

(Csurka, et.al., 1997)

imaged 3Dtranslation & rotation



Raquel A. Romano19

Projective Parameters

ijjiij hee ,,ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

ijjiij hee ,,

• provide a complete projective model of camera configuration

But...• set of all pairwise

parameters are still redundant

• not all images have sufficient overlap



Raquel A. Romano20

Trifocal Dependencies• derive dependencies among

three fundamental matrices

• correctly models degreesof freedom in camera configuration

• geometrically consistent parameterized model of view triplets

kie kje

ike

ije jie jke

kih kjh

ijh



Raquel A. Romano21

Trifocal Dependencies

trifocal lines available from two fundamental matrices

• derive dependencies among three fundamental matrices

• correctly models degreesof freedom in camera configuration

• geometrically consistent parameterized model of view triplets

ktkie kje

it jtike

ije jie jke

kih kjh

ijh



Raquel A. Romano22

Outline




Raquel A. Romano23

Recovering Camera Geometryview i view jview k

fewcorrespondences



Raquel A. Romano24

Linear Initialization8-point Algorithm

(Hartley, 1995)

( )( ){ }∑

ji

iijTj

xxxFx

,

Minimize over all matching point pairs.

[ ]Tij fffffffff 333231232221131211=f

Rewrite bilinear constraints as

where

and solve linear system

0Af =ij

[ ] 0f =ijjjjjijijjiji yxyyyyxxxyxx 1



Raquel A. Romano25

Projection to Parameter Space

Map linear estimate of fundamental matrix to projective parameter space:

},,{7 ijjiijij heep =→ijF },,{4 ijji

ij hγγp =→

• parameterization requires choice of projective basis

• basis affects shape of error surface for nonlinear optimization



Raquel A. Romano26

Geometric Objective Functionpoint-to-epipolar-line distance ~ image reprojection error

weighted residual of bilinear constraint

iTjij

ijji ijwE xFxpxx

p7);,( 7 =

22

21

22

21 )()(

1)()(

1

jT

jT

iijiijij

ijij

wxFxFxFxF +

++

=



Raquel A. Romano27

Error Surface Depends on Basiscanonical basis geometrically defined basis



Raquel A. Romano28

gamma(i,j) (h1,h2)

(h1,h2) (h2,h3) (h1,h3)

(e1,e2) (e3,e4)

gamma(i,j) (h1,h2)

(e1,e2) (e3,e4)

(h1,h2) (h2,h3) (h1,h3)

Nonlinear Trifocal Estimationi jk

1. Initialize epipolar geometry• 8-point algorithm: linear solution to fundamental

matrix for all view pairs• extract epipoles and epipolar collineations

2. 7D nonlinear minimization: bifocal parameters for view pairs (i,k) (j,k)

3. Trifocally constrained estimation for view pair (i,j)• compute trifocal lines• project parameters to trifocally constrained space• 4D nonlinear minimization for bifocal parameters



Raquel A. Romano29

Convergenceeij erroreji

-4000 -2000 0

Ground Truth8-point Algorithm7-Parameter SearchTrifocal Projection4-Parameter Search



Raquel A. Romano30

Ground Truth8-Point Algorithm

7-Parameter Algorithm


Results



Raquel A. Romano31

knossos sequenceview i view k view j

fewcorrespondences



Raquel A. Romano32

Ground Truth



Raquel A. Romano33



8-Point Algorithm

Results

Summary

• Imposing projective constraints on camera geometry corrects the estimation of epipolar geometry

• Resulting camera configuration for multiple cameras is globally consistent



Raquel A. Romano34

Outline

1. Projective Geometry Overview2. Minimal Projective Parameters 3. Projective Parameter Estimation4. Motion Boundary Detection5. Conclusion



Raquel A. Romano35

Camera and Scene Motion



Raquel A. Romano36

Combining Intensity and Geometrytrifocal tensor

projective linear form relating a point-line-line(Spetsakis & Aloimonos, 1990; Shashua, 1994)

0),,( =kjiΤ llxScientific Computing SeminarMay 12 , 2004


Raquel A. Romano37

Tensor Brightness Constraint(Shashua & Hannah, 1995; Shashua & Stein, 1997)

• Horn-Schunk brightness constraint is linear in point coordinates

• Defines line in each image containing matching point

• Spatiotemporal gradient at every pixel provides test of rigid motion

u Ix + v Iy + It = 0

ax + by + c = 0

(a,b,c)T ≅Ix

⎥⎦

⎤⎢⎣

⎡Iy

It - x0 Ix – y0 Iy

u = x - x0 v = y - y0



Raquel A. Romano38

Motion Boundary Detection



Raquel A. Romano39

• Partition image into windows and solve for trifocal tensor coefficients.

• Sum residual error of tensor solution.

• Only regions with rigid 3D motion have a good fit.

• High residuals indicate regions that cross a motion boundary.

Multiple Frame Flow

• Multi-frame tracks fall into separable classes

• Track points over many frames

• Robustly fit tracks to linear approximation of instantaneous planar motion

x(t) = x0+ t [Ax0 + b]



Raquel A. Romano40

Detecting Independent Motions

Residual error of estimated motion model on all point tracks



Raquel A. Romano41

Complexity of Motion Model



Raquel A. Romano42

ConclusionsWhen possible, use domain and task

knowledge to choose model:• What type of information is needed• What aspects of the imaging conditions

are known or controlled• What types of uncertainty can be

modeled and compensated for



Raquel A. Romano43

Future NeedsRole of learning in motion analysis:

• Supervised learning of geometric motion classes

• Data-driven model selection by flow classification

• Robust estimation of appropriate motion model

• Adaptive, time-varying estimation



Raquel A. Romano44

END



Raquel A. Romano45

Projective Geometry for Computer Vision - romano/talks/scs_05_12_04/scs_05_12_04.pdf · Projective...

Documents

Transcript of Projective Geometry for Computer Vision - romano/talks/scs_05_12_04/scs_05_12_04.pdf · Projective...