Assignment 2Compute F automatically from image

pair

(putative matches, 8-point, 7-point, iterative, RANSAC, guided matching)

(due by Wednesday 19/03/03)

Rectification and structure computation

class 15

Multiple View GeometryComp 290-089Marc Pollefeys

Multiple View Geometry course schedule(subject to change)

Jan. 7, 9 Intro & motivation Projective 2D Geometry

Jan. 14, 16

(no class) Projective 2D Geometry

Jan. 21, 23

Projective 3D Geometry (no class)

Jan. 28, 30

Parameter Estimation Parameter Estimation

Feb. 4, 6 Algorithm Evaluation Camera Models

Feb. 11, 13

Camera Calibration Single View Geometry

Feb. 18, 20

Epipolar Geometry 3D reconstruction

Feb. 25, 27

Fund. Matrix Comp. Fund. Matrix Comp.

Mar. 4, 6 Structure Comp. Planes & Homographies

Mar. 18, 20

Trifocal Tensor Three View Reconstruction

Mar. 25, 27

Multiple View Geometry

MultipleView Reconstruction

Apr. 1, 3 Bundle adjustment Papers

Apr. 8, 10

Auto-Calibration Papers

Apr. 15, 17

Dynamic SfM Papers

Apr. 22, 24

Cheirality Project Demos

Two-view geometry

Epipolar geometry

3D reconstruction

F-matrix comp.

Structure comp.

Automatic computation of F

(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)

Select strongest features (e.g. 1000/image)

Feature points

Evaluate ZNCC,SSD,SAD for all features with similar coordinates

Keep mutual best matchesKeep mutual best matches

Still many wrong matches!Still many wrong matches!

10101010 ,,´´, e.g. hhww yyxxyx

?

Feature matching

Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do

Step 3.1 select minimal sample (i.e. 7 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90%

80%

70% 60%

50%

#samples

5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

RANSAC

restrict search range to neighborhood of epipolar line (1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches: guided matching

geometric relations between two views is fully

described by recovered 3x3 matrix F

two-view geometry

Image pair rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)

try to minimize image distortion

problem when epipole in (or close to) the image

He001

Planar rectification

Bring two views Bring two views to standard stereo setupto standard stereo setup

(moves epipole to )(not possible when in/close to image)

~ image size

(calibrated)(calibrated)

Distortion minimization(uncalibrated)

(standard approach)

Polar re-parameterization around epipoles

Requires only (oriented) epipolar geometry

Preserve length of epipolar linesChoose so that no pixels are

compressed

original image rectified image

Polar rectification(Pollefeys et al. ICCV’99)

Works for all relative motionsGuarantees minimal image size

polar rectification: example

polar rectification: example

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

Example: Béguinage of Leuven

Stereo matching

• attempt to match every pixel• use additional constraints

Exploiting motion and scene constraints

• Ordering constraint• Uniqueness constraint• Disparity limit• Disparity continuity constraint

• Epipolar constraint Epipolar constraint (through rectification)

Ordering constraint

11 22 33 4,54,5 66 11 2,32,3 44 55 66

2211 33 4,54,5 6611

2,32,3

44

55

66

surface slicesurface slice surface as a pathsurface as a path

occlusion right

occlusion left

Uniqueness constraint

• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none

Disparity constraint

surface slicesurface slice surface as a pathsurface as a path

bounding box

dispa

rity b

and

use reconsructed features to determine bounding box

constantdisparitysurfaces

Disparity continuity constraint

• Assume piecewise continuous surface

piecewise continuous disparity• In general disparity changes

continuously• discontinuities at occluding

boundaries

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Hierarchical stereo matching

Dow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

Example: reconstruct image from neighboring

images

Multi-view depth fusion

• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down

sequence• Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

Point reconstruction

PXx XP'x'

linear triangulation

XP'x'PXx

0XP'x

0XpXp0XpXp0XpXp

1T2T

2T3T

1T3T

yxyx

2T3T

1T3T

2T3T

1T3T

p'p''p'p''pppp

A

yxyx

0AX

homogeneous

1X

)1,,,( ZYX

inhomogeneous

invariance?

e)(HX)(AH-1

algebraic error yes, constraint no (except for affine)

geometric error

0x̂F'x̂ subject to )'x̂,(x')x̂(x, T22 dd

X̂P''x̂ and X̂Px̂ subject toly equivalentor

possibility to compute using LM (for 2 or more points)

or directly (for 2 points)

Geometric error

Reconstruct matches in projective frame by minimizing the reprojection error

(see Hartley&Sturm,CVIU´97)Non-iterative optimal solution

Optimal 3D point in epipolar plane

Given an epipolar plane, find best 3D point for (x1,x2)

x1

x2

l1 l2

l1x1

x2l2

x1´

x2´

Select closest points (x1´,x2´) on epipolar lines

Obtain 3D point through exact triangulationGuarantees minimal reprojection error (given this epipolar plane)

Optimal epipolar plane

• Reconstruct matches in projective frame by minimizing the reprojection error

• Non-iterative methodDetermine the epipolar plane for reconstruction

Reconstruct optimal point from selected epipolar plane

222

211 XP,xXP,x dd

(Hartley and Sturm, CVIU´97)

222

211 αl,xαl,x DD

(polynomial of degree 6check all minima, incl ∞)

m1

m2

l1 l2

3DOF

1DOF

Reconstruction uncertainty

consider angle between rays

Line reconstruction

P'l'Pl

T

T

L

doesn‘t work for epipolar plane

Next class: Scene and plane homographies