N-view factorization andbundle adjustment

CMPUT 613CMPUT 613

Multiple view computation

•Tensors (2,3,4-views)Tensors (2,3,4-views)

•FactorizationFactorization– OrthographicOrthographic– PerpectivePerpective

•SequentialSequential

•Bundle adjustmentBundle adjustment

Factorization

• Factorise observations in structure of the scene and Factorise observations in structure of the scene and motion/calibration of the cameramotion/calibration of the camera

• Use Use all pointsall points in in all imagesall images at the same time at the same time

Affine factorisationAffine factorisation Projective factorisationProjective factorisation

Affine camera

The affine projection equations are The affine projection equations are

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

10001

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

P

y

x

~

~

4

4

j

j

j

yi

xi

ij

ij

yiij

xiij

Z

Y

X

P

Py

x

Py

Px

Orthographic factorization

The ortographic projection equations are The ortographic projection equations are

where where njmijiij ,...,1,,...,1,Mm P

All equations can be collected for all All equations can be collected for all ii and and jj

wherewhere

n

mmnmm

n

n

M,...,M,M,,

mmm

mmm

mmm

212

1

21

22221

11211

M

P

P

P

Pm

MPm

M ~

~m

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and M are resp. 2mx3 and 3xn matrices and

therefore the rank of m is at most 3

(Tomasi Kanade’92)

Orthographic factorization

Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

(Tomasi Kanade’92)

n

mmnmm

n

n

M,...,M,M

mmm

mmmmmm

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

0~~

1~~

1~~

1

1

1

TT

TT

TT

yi

xi

yi

yi

xi

xi

PP

PP

PP

AA

AA

AA

0~~

1~~

1~~

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP

C

C

C

A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows

Where A is computed from Where A is computed from

Orthographic factorization

Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

MAMAPP~

,~ 1

0

1

1

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP 3 linear equations per view on

symmetric matrix C (6DOF)

A can be obtained from C through Cholesky factorisation

and inversion

(Tomasi Kanade’92)

Perspective factorization

The camera equations The camera equations

for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas

where where

mjmijiijij ,...,1,,...,1,Mmλ P

MPm iii

imiii

mimiii mmm

λ,...,λ,λdiag

M,...,M,M,,...,,

21

2121

Mm

Perspective factorization

All equations can be collected for all All equations can be collected for all ii as as

wherewherePMm

mnn P

P

P

P

m

m

m

m...

,...

2

1

22

11

In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are unknown are unknown

Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a 4xx4 matrix and a 4xnn matrix, i.e. it is a rank 4 matrixmatrix, i.e. it is a rank 4 matrix

Perspective factorization algorithm

Assume that i are known, then PM is known.

Use the singular value decomposition PM=U VT

In the noise-free case

S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:

P=the first four columns of U.M=the first four rows of V.

Iterative perspective factorization

When i are unknown the following algorithm can be used:

1. Set ij=1 (affine approximation).

2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.

3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM

4. Goto 2.

In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered

up to an arbitrary projective transformation

Further Factorization work

Factorization with uncertaintyFactorization with uncertainty

Factorization for dynamic scenesFactorization for dynamic scenes

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

Sequential structure from motion

• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view

– Determine poseDetermine pose– Refine and extend structureRefine and extend structure

• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry

Initial structure and motion

eeaFeP

0IPT

x

2

1

Epipolar geometry Projective calibration

012 FmmT

compatible with F

Yields correct projective camera setup(Faugeras´92,Hartley´92)

Obtain structure through triangulation

Use reprojection error for minimizationAvoid measurements in projective space

Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine pose towards existing structure

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk

Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure

Determining close views

• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography

• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography

Distance = m´,mmedian min HD

9.8im/pt

4.8im/pt

64 images

64 images

3792

po

ints

2170

po

ints

Non-sequential image collections (2)

Refining structure and motion

• Minimize reprojection errorMinimize reprojection error

– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)

– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin