Recognition and Matching

based on local invariant features

Cordelia Schmid

INRIA, Grenoble

David Lowe

Univ. of British Columbia

Introduction

Local invariant photometric descriptors

( )local descriptor

Local : robust to occlusion/clutter + no segmentation

Photometric : distinctive

Invariant : to image transformations + illumination changes

History - Matching

Matching based on line segments

Not very discriminant

Solution : matching with interest points & correlation

[ A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry,

Z. Zhang, R. Deriche, O. Faugeras and Q. Luong,

Artificial Intelligence 1995 ]

Approach

• Extraction of interest points with the Harris detector

• Comparison of points with cross-correlation

• Verification with the fundamental matrix

Harris detector

Interest points extracted with Harris (~ 500 points)

Cross-correlation matching

Initial matches (188 pairs)

Global constraints

Robust estimation of the fundamental matrix

99 inliers 89 outliers

Summary of the approach

• Very good results in the presence of occlusion and clutter– local information– discriminant greyvalue information – robust estimation of the global relation between images– for limited view point changes

• Solution for more general view point changes– wide baseline matching (different viewpoint, scale and rotation)– local invariant descriptors based on greyvalue information

History - Recognition

Color histogram [Swain 91]

b

g

r

Each pixel is described

by a color vector

Distribution of color vectors

is described by a histogram

=> not robust to occlusion, not invariant, not distinctive

History - Recognition

Eigenimages [Turk 91]

• Each face vector is represented in the eigenimage space– eigenvectors with the highest eigenvalues = eigenimages

• The new image is projected into the eigenimage space– determine the closest face

. .1v.

3v

2v.

not robust to occlusion, requires segmentation, not invariant,

discriminant

History - Recognition

Geometric invariants [Rothwell 92]

• Function with a value independent of the transformation

• Invariant for image rotation : distance of two points

• Invariant for planar homography : cross-ratio

),(),( yxfyxf tt yxTyx ),(),( where

=> local and invariant, not discriminant, requires sub-pixel extraction of primitives

History - Recognition

Problems : occlusion, clutter, image transformations, distinctiveness

Solution : recognition with local photometric invariants

[ Local greyvalue invariants for image retrieval,

C. Schmid and R. Mohr,

PAMI 1997 ]

Approach

1) Extraction of interest points (characteristic locations)

2) Computation of local descriptors

3) Determining correspondences

4) Selection of similar images

( )local descriptor

Interest points

Geometric featuresrepeatable under transformations

2D characteristics of the signalhigh informational content

Comparison of different detectors [Schmid98] Harris detector

Harris detector

Based on the idea of auto-correlation

Important difference in all directions => interest point

Harris detector

2

),(

)),(),((),( yyxxIyxIyxf kkkWyx

k

kk

y

xyxIyxIyxIyyxxI kkykkxkkkk )),(),((),(),(with

2

),(

),(),(),(

Wyx

kkykkx

kky

xyxIyxIyxf

Auto-correlation function for a point and a shift ),( yx ),( yx

Discret shifts can be avoided with the auto-correlation matrix

Harris detector

y

x

yxIyxIyxI

yxIyxIyxI

yx

Wyxkky

Wyxkkykkx

Wyxkkykkx

Wyxkkx

kkkk

kkkk

),(

2

),(

),(),(

2

)),((),(),(

),(),()),((

Auto-correlation matrix

Harris detection

• Auto-correlation matrix– captures the structure of the local neighborhood– measure based on eigenvalues of this matrix

• 2 strong eigenvalues => interest point

• 1 strong eigenvalue => contour

• 0 eigenvalue => uniform region

• Interest point detection– threshold on the eigenvalues– local maximum for localization

Local descriptors

Descriptors characterize the local neighborhood of a point

( )local descriptor

Local descriptors

)2

),(exp(

2

1),),((

2

2

2

tt yx

yxG

ydxdyyxxIyxGGyxI ),(),()(),(

)(*),(

)(*),(

)(*),(

)(*),(

)(),(

)(),(

),(

yy

xy

xx

y

x

GyxI

GyxI

GyxI

GyxI

GyxI

GyxI

yxv

Greyvalue derivatives

Local descriptors

Invariance to image rotation : differential invariants [Koen87]

nsor epsilon te ricantisymmet theis where

2

2

)(

ij

yyyyxyxyxxxx

yyxx

yyyyyxxyxxxx

yyxx

kjiijk

lkijklij

ljiijkkkjiij

llijkklkijklij

ijij

ii

jiji

ii

LLLLLL

LL

LLLLLLLL

LLLL

L

LLLL

LLLL

LLLLLLLL

LLLLLLLL

LL

L

LLL

LL

L

Local descriptors

Robustness to illumination changes

In case of an affine transformation

baII )()( 21 xx

2

2

2

2

2/1

2/3

)(

)(

)(

)

)(

)(

)(

)(

ii

kjiijk

ii

lkijklij

ii

kjiijkkkjiij

ii

llijkklkijklij

ii

jiij

ii

ii

ii

jiji

LL

LLLL

LL

LLLL

LL

LLLLLLLL

LL

LLLLLLLL

LL

LL

LL

L

LL

LLL

Local descriptors

Robustness to illumination changes

In case of an affine transformation

baII )()( 21 xx

or normalization of the image patch with mean and variance

Determining correspondences

Vector comparison using the Mahalanobis distance

)()(),( 1 qpqpqp TMdist

( ) ( )=?

Selection of similar images

• In a large database – voting algorithm– additional constraints

• Rapid acces with an indexing mechanism

Voting algorithm

local characteristicsvector of

( )1I 1I nI2I2I

Voting algorithm

1I 1I nI2I2I} }

1 1 02 1 1

I is the corresponding model image1

2 1 1

Additional constraints

• Semi-local constraints– neighboring points should match– angles, length ratios should be similar

• Global constraints– robust estimation of the image transformation (homogaphy, epipolar geometry)

1

21

~2

~

11

2

3

2

3

Results

database with ~1000 images

Results

Results

Summary of the approach

• Very good results in the presence of occlusion and clutter– local information– discriminant greyvalue information – invariance to image rotation and illumination

• Not invariance to scale and affine changes

• Solution for more general view point changes– local invariant descriptors to scale and rotation– extraction of invariant points and regions

Approach for Matching and Recognition

• Detection of interest points/regions– Harris detector (extension to scale and affine invariance)– Blob detector based on Laplacian

• Computation of descriptors for each point

• Similarity of descriptors

• Semi-local constraints

• Global verification

Approach for Matching and Recognition

• Detection of interest points/regions

• Computation of descriptors for each point– greyvalue patch, diff. invariants, steerable filter, SIFT descriptor

• Similarity of descriptors– correlation, Mahalanobis distance, Euclidean distance

• Semi-local constraints

• Global verification

Approach for Matching and Recognition

• Detection of interest points/regions

• Computation of descriptors for each point

• Similarity of descriptors

• Semi-local constraints– geometrical or statistical relations between neighborhood points

• Global verification– robust estimation of geometry between images

Overview

8:30-8:45 Scale invariant interest points

8:45-9:00 SIFT descriptors

9:00-9:25 Affine invariance of interest points + applications

9:25-9:45 Evaluation of interest points + descriptors

9:45-10:15 Break

Overview

10:15-11:15 Object recognition system, demo, applications

11:15-11:45 Recognition of textures and object classes

11:45-12:00 Future directions + discussion