Lecture’6:’’...

Lecture 6 - !!!

Fei-Fei Li!

Lecture 6: Finding Features (part 1/2)

Professor Fei-‐Fei Li Stanford Vision Lab

2-‐Oct-‐14 1

Lecture 6 - !!!

Fei-Fei Li!

What we will learn today?

•  Local invariant features – MoHvaHon –  Requirements, invariances

•  Keypoint localizaHon – Harris corner detector

•  Scale invariant region selecHon – AutomaHc scale selecHon – Difference-‐of-‐Gaussian (DoG) detector

•  SIFT: an image region descriptor

2-‐Oct-‐14 2

Next lecture (#7)

Lecture 6 - !!!

Fei-Fei Li!

What we will learn today?





2-‐Oct-‐14 3

Some background reading: Rick Szeliski, Chapter 14.1.1; David Lowe, IJCV 2004

Lecture 6 - !!!

Fei-Fei Li!

Image matching: a challenging problem

2-‐Oct-‐14 4

Lecture 6 - !!!

Fei-Fei Li!

by Diva Sian

by swashford

Slid

e cr

edit

: St

eve

Seit

z

2-‐Oct-‐14 5

Image matching: a challenging problem

Lecture 6 - !!!

Fei-Fei Li!

Harder Case

by Diva Sian by scgbt

Slid

e cr

edit

: St

eve

Seit

z

2-‐Oct-‐14 6

Lecture 6 - !!!

Fei-Fei Li!

Harder SHll?

NASA Mars Rover images

Slid

e cr

edit

: St

eve

Seit

z

2-‐Oct-‐14 7

Lecture 6 - !!!

Fei-Fei Li!

Answer Below (Look for Hny colored squares)

NASA Mars Rover images with SIFT feature matches (Figure by Noah Snavely) Sl

ide

cred

it:

Stev

e Se

itz

2-‐Oct-‐14 8

Lecture 6 - !!!

Fei-Fei Li!

MoHvaHon for using local features •  Global representaHons have major limitaHons •  Instead, describe and match only local regions •  Increased robustness to

–  Occlusions

–  ArHculaHon

–  Intra-‐category variaHons

θqφ

dq

φ

θ

d

2-‐Oct-‐14 9

Lecture 6 - !!!

Fei-Fei Li!

General Approach N

pix

els

N pixels

Similarity measure Af

e.g. color

Bf

e.g. color

B1 B2

B3 A1

A2 A3

Tffd BA <),(

1. Find a set of distinctive key- points

3. Extract and normalize the region content

2. Define a region around each keypoint

4. Compute a local descriptor from the normalized region

5. Match local descriptors

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 10

Lecture 6 - !!!

Fei-Fei Li!

Common Requirements •  Problem 1:

–  Detect the same point independently in both images

No chance to match!

We need a repeatable detector! Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

2-‐Oct-‐14 11

Lecture 6 - !!!

Fei-Fei Li!

Common Requirements •  Problem 1:

–  Detect the same point independently in both images

•  Problem 2: –  For each point correctly recognize the corresponding one

We need a reliable and distinctive descriptor!

?

Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

2-‐Oct-‐14 12

Lecture 6 - !!!

Fei-Fei Li!

Invariance: Geometric TransformaHons

Slid

e cr

edit

: St

eve

Seit

z

2-‐Oct-‐14 13

Lecture 6 - !!!

Fei-Fei Li!

Levels of Geometric Invariance

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 14

CS131 CS231a

Lecture 6 - !!!

Fei-Fei Li!

Invariance: Photometric TransformaHons

•  Ofen modeled as a linear transformaHon: –  Scaling + Offset

Slid

e cr

edit

: Ti

nne

Tuyt

elaa

rs

2-‐Oct-‐14 15

Lecture 6 - !!!

Fei-Fei Li!

Requirements •  Region extracHon needs to be repeatable and accurate

–  Invariant to translaHon, rotaHon, scale changes –  Robust or covariant to out-‐of-‐plane (≈affine) transformaHons –  Robust to lighHng variaHons, noise, blur, quanHzaHon

•  Locality: Features are local, therefore robust to occlusion and cluier.

•  QuanHty: We need a sufficient number of regions to cover the object.

•  DisHncHveness: The regions should contain “interesHng” structure.

•  Efficiency: Close to real-‐Hme performance.

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 16

Lecture 6 - !!!

Fei-Fei Li!

Many ExisHng Detectors Available

•  Hessian & Harris [Beaudet ‘78], [Harris ‘88] •  Laplacian, DoG [Lindeberg ‘98], [Lowe ‘99] •  Harris-‐/Hessian-‐Laplace [Mikolajczyk & Schmid ‘01] •  Harris-‐/Hessian-‐Affine [Mikolajczyk & Schmid ‘04] •  EBR and IBR [Tuytelaars & Van Gool ‘04] •  MSER [Matas ‘02] •  Salient Regions [Kadir & Brady ‘01] •  Others…

•  Those detectors have become a basic building block for many recent applica8ons in Computer Vision.

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 17

Lecture 6 - !!!

Fei-Fei Li!

Keypoint LocalizaHon

•  Goals:

–  Repeatable detecHon –  Precise localizaHon –  InteresHng content

⇒ Look for two-‐dimensional signal changes

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 18

Lecture 6 - !!!

Fei-Fei Li!

Finding Corners

•  Key property: –  In the region around a corner, image gradient has two or more dominant direcHons

•  Corners are repeatable and dis8nc8ve

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference, 1988.

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

2-‐Oct-‐14 19

Lecture 6 - !!!

Fei-Fei Li!

Corners as DisHncHve Interest Points •  Design criteria

– We should easily recognize the point by looking through a small window (locality)

–  Shifing the window in any direc8on should give a large change in intensity (good localiza8on)

“edge”: no change along the edge direction

“corner”: significant change in all directions

“flat” region: no change in all directions Sl

ide

cred

it:

Alyo

sha

Efro

s

2-‐Oct-‐14 20

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector FormulaHon

•  Change of intensity for the shif [u,v]: [ ]2

,( , ) ( , ) ( , ) ( , )

x yE u v w x y I x u y v I x y= + + −∑

Intensity Shifted intensity

Window function

or Window function w(x,y) =

Gaussian 1 in window, 0 outside

Slid

e cr

edit

: Ri

ck S

zelis

ki

2-‐Oct-‐14 21

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector FormulaHon •  This measure of change can be approximated by:

where M is a 2×2 matrix computed from image derivaHves:

⎥⎦

⎤⎢⎣

⎡≈

vu

MvuvuE ][),(

M

Sum over image region – the area we are checking for corner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Slid

e cr

edit

: Ri

ck S

zelis

ki

2-‐Oct-‐14 22

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector FormulaHon

Ix Image I IxIy Iy

where M is a 2×2 matrix computed from image derivaHves:

M

Sum over image region – the area we are checking for corner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Slid

e cr

edit

: Ri

ck S

zelis

ki

2-‐Oct-‐14 23

Lecture 6 - !!!

Fei-Fei Li!

What Does This Matrix Reveal? •  First, let’s consider an axis-‐aligned corner:

•  This means: –  Dominant gradient direcHons align with x or y axis –  If either λ is close to 0, then this is not a corner, so look for locaHons where both are large.

•  What if we have a corner that is not aligned with the image axes?

⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

∑∑∑∑

2

12

2

00λ

λ

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

2-‐Oct-‐14 24

Lecture 6 - !!!

Fei-Fei Li!

What Does This Matrix Reveal? •  First, let’s consider an axis-‐aligned corner:

•  This means: –  Dominant gradient direcHons align with x or y axis –  If either λ is close to 0, then this is not a corner, so look for locaHons where both are large.

•  What if we have a corner that is not aligned with the image axes?

⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

∑∑∑∑

2

12

2

00λ

λ

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

2-‐Oct-‐14 25

Lecture 6 - !!!

Fei-Fei Li!

General Case

•  Since M is symmetric, we have

•  We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientaHon determined by R

RRM ⎥⎦

⎤⎢⎣

⎡= −

2

11

00λ

λ

Direction of the slowest change

Direction of the fastest change

(λmax)-1/2

(λmin)-1/2

adap

ted

from

Dar

ya F

rolo

va,

Den

is S

imak

ov

(Eigenvalue decomposition)

2-‐Oct-‐14 26

Lecture 6 - !!!

Fei-Fei Li!

InterpreHng the Eigenvalues •  ClassificaHon of image points using eigenvalues of M:

λ1

“Corner” λ1 and λ2 are large, λ1 ~ λ2; E increases in all direcHons

λ1 and λ2 are small; E is almost constant in all direcHons “Edge”

λ1 >> λ2

“Edge” λ2 >> λ1

“Flat” region

Slid

e cr

edit

: Kr

iste

n G

raum

an

λ2

2-‐Oct-‐14 27

Lecture 6 - !!!

Fei-Fei Li!

Corner Response FuncHon

•  Fast approximaHon –  Avoid compuHng the eigenvalues

–  α: constant (0.04 to 0.06)

λ2

“Corner” θ > 0

“Edge” θ < 0

“Edge” θ < 0

“Flat” region

θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2

λ1 Slid

e cr

edit

: Kr

iste

n G

raum

an

2-‐Oct-‐14 28

Lecture 6 - !!!

Fei-Fei Li!

Window FuncHon w(x,y)

•  OpHon 1: uniform window –  Sum over square window

–  Problem: not rotaHon invariant

•  OpHon 2: Smooth with Gaussian –  Gaussian already performs weighted sum

–  Result is rotaHon invariant

1 in window, 0 outside

2

2,( , ) x x y

x y x y y

I I IM w x y

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

Gaussian

2

2,

x x y

x y x y y

I I IM

I I I⎡ ⎤

= ⎢ ⎥⎢ ⎥⎣ ⎦

∑

2

2( ) x x y

x y y

I I IM g

I I Iσ

⎡ ⎤= ∗⎢ ⎥

⎢ ⎥⎣ ⎦

Slid

e cr

edit

: Ba

stia

n Le

ibe

2-‐Oct-‐14 29

Lecture 6 - !!!

Fei-Fei Li!

Summary: Harris Detector [Harris88] •  Compute second moment matrix

(autocorrelaHon matrix) 1. Image derivatives

Ix Iy

2. Square of derivatives

Ix2 Iy2 IxIy

3. Gaussian filter g(σI) g(Ix2) g(Iy2) g(IxIy)

R

2

2

( ) ( )( , ) ( )

( ) ( )x D x y D

I D Ix y D y D

I I IM g

I I Iσ σ

σ σ σσ σ

⎡ ⎤= ∗⎢ ⎥

⎢ ⎥⎣ ⎦

2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Iα= − − +

θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]2

4. Cornerness function – two strong eigenvalues

5. Perform non-maximum suppression

Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

2-‐Oct-‐14 30

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector: Workflow

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

2-‐Oct-‐14 31

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector: Workflow -‐ computer corner responses θ

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

2-‐Oct-‐14 32

Lecture 6 - !!!

Fei-Fei Li!

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

Harris Detector: Workflow -‐ Take only the local maxima of θ, where θ>threshold

2-‐Oct-‐14 33

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector: Workflow -‐ ResulHng Harris points

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

2-‐Oct-‐14 34

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector – Responses [Harris88]

Effect: A very precise corner detector.

Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

2-‐Oct-‐14 35

Lecture 6 - !!!

Fei-Fei Li!


Slid

e cr

edit

: Kr

ysti

an M

ikol

ajcz

yk

2-‐Oct-‐14 36

Lecture 6 - !!!

Fei-Fei Li!


Slid

e cr

edit

: Kr

iste

n G

raum

an

•  Results are well suited for finding stereo correspondences

2-‐Oct-‐14 37

Lecture 6 - !!!

Fei-Fei Li!

Harris Detector: ProperHes

•  TranslaHon invariance?

Slid

e cr

edit

: Kr

iste

n G

raum

an

2-‐Oct-‐14 38

Lecture 6 - !!!

Fei-Fei Li!


•  TranslaHon invariance •  RotaHon invariance?

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response θ is invariant to image rotation

Slid

e cr

edit

: Kr

iste

n G

raum

an

2-‐Oct-‐14 39

Lecture 6 - !!!

Fei-Fei Li!


•  TranslaHon invariance •  RotaHon invariance •  Scale invariance?

Not invariant to image scale!

All points will be classified as edges!

Corner

Slid

e cr

edit

: Kr

iste

n G

raum

an

2-‐Oct-‐14 40

Lecture 6 - !!!

Fei-Fei Li!

What we have learned today?





2-‐Oct-‐14 41

Next lecture (#7)

Some background reading: Rick Szeliski, Chapter 14.1.1; David Lowe, IJCV 2004

Lecture’6:’’...

Documents

Transcript of Lecture’6:’’...