Local Features

Computer Vision

Correspondence Problem & Keypoint

• Correspondence: matching points, patches, edges, or regions across images

≈

Overview of Keypoint Matching

K. Grauman, B. Leibe

Af Bf

A1

A2 A3

Tffd BA ),(

1. Find a set of

distinctive key-

points

3. Extract and

normalize the

region content

2. Define a region

around each

keypoint

4. Compute a local

descriptor from the

normalized region

5. Match local

descriptors

DoG – Efficient Computation

• Computation in Gaussian scale pyramid

K. Grauman, B. Leibe

s

Original image 4

1

2s

Sampling with

step s4 =2

s

s

s

Find local maxima in position-scale space of Difference-of-Gaussian

K. Grauman, B. Leibe

)()( ss yyxx LL

s

s2

s3

s4

s5

List of (x, y, s)

Results: Difference-of-Gaussian

K. Grauman, B. Leibe

SIFT

• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

Scale-space extrema detection

• Find the points, whose surrounding patches (with some scale) are distinctive

• An approximation to the scale-normalized Laplacian of Gaussian

Maxima and minima in a

3*3*3 neighborhood

Eliminating edge points

Orientation assignment

• Assign an orientation to each keypoint, the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation

• Compute magnitude and orientation on the Gaussian smoothed images

T. Tuytelaars, B. Leibe

Orientation normalization

• Compute orientation histogram

• Select dominant orientation

• Normalize: rotate to fixed orientation

0 2 p

Reduce effect of illumination • 128-dim vector normalized to 1

• Threshold gradient magnitudes to avoid excessive influence of high gradients

– after normalization, clamp gradients >0.2

– renormalize

A result

Structure from Motion

Computer Vision

Depth from disparity

x’ z

f

x

C C’

X

baseline

f

(X – X’) / f = baseline / z X – X’ = (baseline*f) / z z = (baseline*f) / (X – X’)

General case, with calibrated cameras

• The two cameras need not have parallel optical axes.

Vs.

• Given p in left image, where can corresponding point p’ be?

Stereo correspondence constraints

Stereo correspondence constraints

Geometry of two views constrains where the

corresponding pixel for some image point in the first view

must occur in the second view.

• It must be on the line carved out by a plane

connecting the world point and optical centers.

Epipolar constraint

• Epipolar Plane

Epipole

Epipolar Line

Baseline

Epipolar geometry

Epipole

• Baseline: line joining the camera centers

• Epipole: point of intersection of baseline with image plane

• Epipolar plane: plane containing baseline and world point

• Epipolar line: intersection of epipolar plane with the image plane

• All epipolar lines intersect at the epipole

• An epipolar plane intersects the left and right image planes in epipolar lines

Epipolar geometry: terms

Why is the epipolar constraint useful?

Epipolar constraint

This is useful because it reduces the correspondence

problem to a 1D search along an epipolar line.

Image from Andrew Zisserman

Example

What do the epipolar lines look like?

Ol Or

Ol Or

1.

2.

Example: converging cameras

Figure from Hartley & Zisserman

Figure from Hartley & Zisserman

Example: parallel cameras

Where are the

epipoles?

Example: Forward motion

What would the epipolar lines look like if the camera moves directly forward?

e

e’

Example: Forward motion

Epipole has same coordinates in both

images.

Points move along lines radiating from e:

“Focus of expansion”

Fundamental matrix

• Let p be a point in left image, p’ in right image

• Epipolar relation – p maps to epipolar line l’ – p’ maps to epipolar line l

• Epipolar mapping described by a 3x3 matrix F

• It follows that

l’ l

p p’

Fundamental matrix

• This matrix F is called – the “Essential Matrix”

• when image intrinsic parameters are known

– the “Fundamental Matrix” • more generally (uncalibrated case)

• Can solve for F from point correspondences

– Each (p, p’) pair gives one linear equation in entries of F

– F has 9 entries, but really only 7 or 8 degrees of freedom. – With 8 points it is simple to solve for F, but it is also

possible with 7. See Marc Pollefey’s notes for a nice tutorial

Active stereo with structured light

• Project “structured” light patterns onto the object – Simplifies the correspondence problem

– Allows us to use only one camera

camera

projector

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured

Light and Multi-pass Dynamic Programming. 3DPVT 2002

Structured light

https://www.youtube.com/watch?v=jmffVdU

JYv4

Kinect: Structured infrared light

http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/

Photometry stereo

https://www.youtube.com/watch?v=qlq3n5r1Xy0

Helmholtz Stereopsis

Structure from Motion

Structure from motion

• Given a set of corresponding points in two or more images, compute the camera parameters and the 3D point coordinates

Camera 1 Camera 2 Camera 3

R1,t1 R2,t2 R3,t3

? ? ? Slide credit:

Noah Snavely

?

https://www.youtube.com/watch?v=i7ierVk

XYa8

Structure from motion ambiguity

• If we scale the entire scene by some factor k and, at the same time, scale the camera matrices by the factor of 1/k, the projections of the scene points in the image remain exactly the same:

It is impossible to recover the absolute scale of the scene!

)(1

XPPXx kk

How do we know the scale of image content?

Bundle adjustment

• Non-linear method for refining structure and motion

• Minimizing reprojection error

2

1 1

,),(

m

i

n

j

jiijDE XPxXP

x1j

x2j

x3j

Xj

P1

P2

P3

P1Xj

P2Xj

P3Xj

Photo synth

Noah Snavely, Steven M. Seitz, Richard Szeliski, "Photo tourism: Exploring

photo collections in 3D," SIGGRAPH 2006

https://www.youtube.com/watch?v=IgBQC

oEfiMs