Lecture 03 Local Feature Descriptors and … 03...Lin ZHANG, SSE, 2020 Lecture 3 Local Feature...

Lin ZHANG, SSE, 2020

Lecture 3Local Feature Descriptors and Matching

Lin ZHANG, PhDSchool of Software Engineering

Tongji UniversitySpring 2020


Content

• Scale Invariant Feature Transform• Case Study: Homography Estimation

• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation


SIFT

• Scale Invariant Feature Transform• Proposed in [1]• It uses extrema of DoG to detect key points and the associated characteristic scales

• It uses SIFT to describe a key point

[1] D.G. Lowe, Distinctive image features from scale‐invariant keypoints, IJCV 60 (2), pp. 91‐110, 2004 (cited number: 55372 by Feb. 14, 2020)

Prof. David LoweUniversity of British Columbia


SIFT


Assign Keypoints Orientations

• Assign orientation to the keypoint• Find local orientation: dominant orientation of gradient for

the image patch (its size is determined by the characteristic scale)

• Rotate the patch according to this angle; this can achieve rotation invariance description


Assign Keypoints Orientations

• Orientation normalization• Compute orientation histogram• Select dominant orientation • Normalization: rotate the patch to the selected orientation


SIFT


SIFT

• Building the descriptor• Sample the points around the keypoint• Rotate the gradients and coordinates by the previously computed orientation

• Separate the region in to sub‐regions• Create gradient‐orientation histogram for each sub‐region with 8 bins (In real implementation, each sample point is weighted by a Gaussian)

4 4


SIFT

• Building the descriptor

• Actual implementation uses 4*4 sub regions which lead to a 4*4*8 = 128 element vector


SIFT


Applications of SIFT

• Object recognition• Robot localization and mapping• Panorama stitching• 3D scene modeling, recognition and tracking• Analyzing the human brain in 3D magnetic resonance images


Applications of SIFT

• Object recognition


Content




Matrix differentiation

• Function is a vector and the variable is a scalar

1 2( ) ( ), ( ),..., ( ) Tnf t f t f t f t

Definition

1 2 ( )( ) ( ), ,...,T

ndf tdf t df tdfdt dt dt dt


• Function is a matrix and the variable is a scalar11 12 1

21 22 2

1 2

( ) ( ),..., ( )( ) ( ),..., ( )

( ) ( )

( ) ( ),..., ( )

m

mij n m

n n nm

f t f t f tf t f t f t

f t f t

f t f t f t

Definition111 12

221 22

1 2

( )( ) ( ) ,...,

( )( ) ( ) ( ),...,

( ) ( ) ( ),...,

m

mij

n m

n n nm

df tdf t df tdt dt dt

df tdf t df t df tdfdt dt dt

dt dt

df t df t df tdt dt dt



• Function is a scalar and the variable is a vector

1 2( ), ( , ,..., )Tnf x x xx x

Definition

1 2

, ,...,T

n

df f f fd x x x

x

In a similar way,

1 2( ), ( , ,..., )nf x x xx x

1 2

, ,...,n

df f f fd x x x

x



• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T

n mx x x y y y x y x x xDefinition

1 1 1

1 2

2 2 2

1 2

1 2

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

n

nT

m m m

n m n

y y yx x x

y y yd x x xd

y y yx x x

x x x

x x xyx

x x x



• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T

n mx x x y y y x y x x xIn a similar way,

1 2

1 1 1

1 2

2 2 2

1 2

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

( ) ( ) ( ), ,...,

m

mT

m

n n n n m

y y yx x x

y y yd x x xd

y y yx x x

x x x

x x xyx

x x x



• Function is a vector and the variable is a vectorExample:

11 2 2

2 1 1 2 2 3 22

3

( ), , ( ) , ( ) 3

( )

xy

x y x x y x xy

x

xy x x x

x

1 2

1 11

1 2

2 23

1 2

3 3

( ) ( )

2 0( ) ( ) 1 3

0 2( ) ( )

T

y yx x x

d y yd x x

xy y

x x

x x

y x xx

x x



• Function is a scalar and the variable is a matrix

11 12 1

1 2

( ) n

m m mn

f f fx x x

dfd

f f fx x x

XX

( ), m nf X X

Definition



• Useful results1, nx a

,T Td d

d d

a x x aa ax x

Then,

How to prove?

(1)



• Useful results1,m n nA x (2) Then, T

dA Ad

xx

1,m n nA x (3) Then,T T

Td A Ad

xx

1,n n nA x (4) Then, ( )T

Td A A Ad

x x xx

1 1, ,m n m n X a b (5) Then,T

Tdd

a Xb abX

1 1, ,n m m n X a b (6) Then,T T

Tdd

a X b baX

1nx (7) Then, 2Td

d

x x xx



Content




Lagrange multiplier

• Single‐variable function

f(x) is differentiable in (a, b). At , f(x) achieves an extremum

0 ( , )x a b

0| 0x

dfdx

• Two‐variables function

f(x, y) is differentiable in its domain. At , f(x, y)achieves an extremum

0 0( , )x y

0 0 0 0( , ) ( , )| 0, | 0x y x yf fx y


Lagrange multiplier

• In general case

( )f xIf , achieves a local extremum at x0 and it is derivable at x0, then x0 is a stationary point of f(x), i.g.,

0 0 01 2

| 0, | 0,..., | 0n

f f fx x x

x x x

Or in other words,

0( ) |f x xx 0

1nx


Lagrange multiplier

• Lagrange multiplier is a strategy for finding the stationary point of a function subject to equality constraints

Problem: find stationary points for 1( ), ny f x x under m constraints ( ) 0, 1, 2,...,kg k m x

is a stationary point of with constraints

Solution:

11

( ; ,..., ) ( ) ( )m

m k kk

F f g

x x xIf is a stationary point of F, then,

0 10 20 0( , , ..., )m x

0x ( )f xJoseph‐Louis LagrangeJan. 25, 1736~Apr.10, 1813


Lagrange multiplier

• Lagrange multiplier is a strategy for finding the stationary point of a function subject to equality constraints

Solution:1

1( ; ,..., ) ( ) ( )

m

m k kk

F f g

x x x

is a stationary point of F0 10 0( , ,..., )m x

1 2 1 2

0, 0,..., 0, 0, 0,..., 0n m

F F F F F Fx x x

n + m equations!at that point

Problem: find stationary points for 1( ), ny f x x under m constraints ( ) 0, 1, 2,...,kg k m x


Lagrange multiplier

• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.

y=xp0

?

The distance is 2 2( , ) ( 1) ( 0)f x y x y

Now we want to find the stationary point of f(x, y) under the constraint

( , ) 0g x y y x According to Lagrange multiplier method, construct another function

2 2( , , ) ( ) ( ) ( 1) ( )F x y f x g x x y y x Find the stationary point for ( , , )F x y


Lagrange multiplier

• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.

y=xp0

?

0

0

0

FxFyF

2( 1) 02 0

0

xy

x y

0.50.51

xy

(0.5,0.5,1) is a stationary point of ( , , )F x y (0.5,0.5) is a stationary point of f(x,y) under constraints


Content




LS for Inhomogeneous Linear System

Consider the following linear equations system

1 2 1

1 2 2

3 1 1 32 4 2 1 4x x x

x x x

Matrix form: A x bA x b

It can be easily solved 1

2

12

xx



How about the following one?

1 21

1 22

1 2

3 1 1 32 4 2 1 4

1 2 62 6

x xx

x xx

x x

It does not have a solution!

What is the condition for a linear equation system can be solved?

A x b

Can we solve it in an approximate way?A: we can use least squares technique!

Carl Friedrich Gauss



Let’s consider a system of p linear equations with q unknowns

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

...

...

......

q q

q q

p p pq q p

a x a x a x

a x a x a xA

a x a x a x

b

bx b

b

We consider the case: p>q, and rank(A)=qIn general case, there is no solution!

Instead, we want to find a vector x that minimizes the error:22

1 11

( ) ( ... )p

i iq q ii

E a x a x A

x b x b

unknowns



2*2

arg min ( ) arg minE A x x

x x x b

1* T TA A A

x bPseudoinverse of A

How about the pseudoinverse of A when A is square and non-singular?


LS for Homogeneous Linear System

Let’s consider a system of p linear equations with q unknowns

11 1 12 2 1

21 1 22 2 2

1 1 2 2

... 0

... 0

...... 0

q q

q q

p p pq q

a x a x a x

a x a x a xA

a x a x a x

x 0

We consider the case: p>q, and rank(A)=q

unknowns

Theoretically, there is only a trivial solution: x = 0

So, we add a constraint to avoid the trivial solution 2

1x



We want to minimize , subject to 21x 2

2( )x xE A

*2

arg min ( ), . ., 1x

x x xE s t

Use the Lagrange multiplier to solve it,

2 2*2 2

arg min 1x

x x xA

(1)

Solving the stationary point of the Lagrange function,

2 2

2 2

2 2

2 2

1

10

x x0

x

x x

A

A

(3)

(2)



Then, we have

2 2

2 21x x

0x

A

(3)

x xTA A

x is the eigen‐vector of ATA associated with the eigenvalue

2

2x x x x x xT T TE A A A

The unit vector x is the eigenvector associated with the minimum eigenvalue of TA A


Content




RANSAC‐based Homography Estimation

Problem definition: On two projective planes P1 and P2, there is a set of corresponding points , and we suppose that there is a homography matrix linking the two planes,

'1

,n

i i ix x

' , 1, 2,...,i iH i n x xCoordinates of and are known, ix 'ix we need to find H

11 12 13

21 22 23

31 32 33

a a aH a a a

a a a

Note: H is defined up to a scale factor. In other words, it has 8 DOFs



Note: Theoretically speaking, homography can only be estimatedbetween two planes, i.e., when you use such a technique to stitch twoimages, image contents should be roughly on the same plane


11 12 13

21 22 23

31 32 33 1

a a acu xcv a a a yc a a a

11 12 13

21 22 23

31 32 33

a x a y a cua x a y a cva x a y a c

11 12 13

31 32 33

21 22 23

31 32 33

a x a y a ua x a y aa x a y a va x a y a

4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom



4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom 11

12

13

21

22

23

31

32

33

1 0 0 00

0 0 0 1

aaaa

x y ux uy ua

x y vx vy vaaaa

Thus, four correspondence pairs generate 8 equations



4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom

0 (1)A x

8 9 9 1

Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space

( ) 8Rank A



• How about the case when there are more than 4 correspondence pairs?• Use the LS method to solve the model directly?• NO! Because usually, outliers exist among the correspondence pairs

RANdom SAmple Consensus (RANSAC) framework is a good candidate to solve this kind of issues



ObjectiveRobust fit a model to a data set S which contains outliersAlgorithm(1) Randomly select a sample of s data points from S and instantiate the

model from this subset(2) Determine the set of data points Si which are within a distance

threshold t of the model. The set Si is the consensus set of the sample and defines the inliers of S

(3) If the size of Si (the number of inliers) is greater than some threshold T, re‐estimate the model using all points in Si and terminate

(4) If the size of Si is less than T, select a new subset and repeat the above

(5) After N trials the largest consensus set Si is selected, and the model is re‐estimated using all points in the subset Si



Line fitting: least square



Line fitting: RANSAC



Line fitting by RANSAC




• Randomly select two points




• Randomly select two points• The hypothesized model is the

line passing through the two points




• Test another two points




• The final fitting result




Can you describe the steps of homography estimation when

using RANSAC?


Homography Estimation: Example 1


Interest points detection



Correspondence estimation

Then, the homography matrix can be estimated by using the correspondence pairs with RANSAC



Transform image one using the estimated homography matrix



Finally, stitch the transformed image one with image two



Homography Estimation: Example 2Interest points detection


Correspondence estimation

Then, the homography matrix can be estimated by using the correspondence pairs with RANSAC



Transform image one using the estimated homography matrix



Finally, stitch the transformed image one with image two



Homography Estimation: Example 3Project products of students from 2009 Media&Arts

Lecture 03 Local Feature Descriptors and … 03...Lin ZHANG, SSE, 2020 Lecture 3 Local Feature...

Documents

Transcript of Lecture 03 Local Feature Descriptors and … 03...Lin ZHANG, SSE, 2020 Lecture 3 Local Feature...