Local Features: Detection, Description & Matching
Transcript of Local Features: Detection, Description & Matching
![Page 1: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/1.jpg)
Local Features: Detection, Description & Matching
Lecture 08 Computer Vision
![Page 2: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/2.jpg)
• Dr George Stockman
Professor Emeritus, Michigan State University
• Dr David Lowe
Professor, University of British Columbia
• Dr Richard Szeliski
Head of Interactive Visual Media Group at Microsoft Research
• Dr Steve Seitz
Professor, University of Washington
• Dr Kristen Grauman
Associate Professor, University of Texas at Austin
• Dr Allan+ Jepson
Professor, University of Toronto
Material Citations
![Page 3: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/3.jpg)
Suggested Readings
• Chapter 4
Richard Szeliski , “Computer Vision: Algorithms and Applications”, Springer; 2011.
• D. Lowe, “Distinctive image features from scale invariant key points”, International Journal of Computer Vision, 2004.
![Page 4: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/4.jpg)
Since M is symmetric, we have
TXXM
2
1
0
0
The eigen values of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.
Recall: Corners as distinctive interest points
( , ) ,u
E u v u v Mv
![Page 5: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/5.jpg)
“flat” region
1 and 2 are small;
“edge”: 1 >> 2
2 >> 1
“corner”:
1 and 2 are large, 1 ~ 2;
One way to score the cornerness:
Recall: Corners as distinctive interest points
2
1 2 1 2R k
![Page 6: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/6.jpg)
• Rotation invariant?
• Scale invariant?
TXXM
2
1
0
0
Yes
Properties of the Harris corner detector
![Page 7: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/7.jpg)
• Rotation invariant?
• Scale invariant?
All points will be classified as edges
Corner !
Yes
No
Properties of the Harris corner detector
![Page 8: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/8.jpg)
by Diva Sian
by swashford
Image matching
![Page 9: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/9.jpg)
by Diva Sian by scgbt
Harder case
![Page 10: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/10.jpg)
NASA Mars Rover images
Harder still?
![Page 11: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/11.jpg)
NASA Mars Rover images
with SIFT feature matches Figure by Noah Snavely
Look for tiny colored squares..
![Page 12: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/12.jpg)
• At an interesting point, let’s define a coordinate system (x,y axis) • Use the coordinate system to pull out a patch at that point
Image Matching
![Page 13: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/13.jpg)
Image Matching
![Page 14: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/14.jpg)
Local features: main components
• Detection: Identify the interest points • Description: Extract feature vector descriptor surrounding each interest point • Matching: Determine correspondence between descriptors in two views
(1) (1)
1 1[ ,........ ]dX x x
(2) (2)
2 1[ ,........ ]dX x x
![Page 15: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/15.jpg)
• D. Lowe, “Distinctive image features from scale invariant key points”, International Journal of Computer Vision, 2004
• SIFT – Scale Invariant Feature Transform - Detector and descriptor
- SIFT is computationally expensive and copyrighted
SIFT
![Page 16: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/16.jpg)
• A SIFT keypoint is a circular image region with an orientation • It is described by a geometric frame of four parameters:
- Keypoint center coordinates x and y - Its scale (the radius of the region) - and its orientation (an angle expressed in radians)
SIFT - Feature detector
![Page 17: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/17.jpg)
Steps for Extracting Key Points
• Scale space peak selection - Potential locations for finding features
• Key point localization - Accurately locating the feature key points
• Orientation Assignment - Assigning orientation to the key points
• Key point descriptor - Describing the key point as a high dimensional vector
![Page 18: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/18.jpg)
• Algorithm for finding points and representing their patches should produce similar results even when conditions vary
• Buzzword is “invariance”
– geometric invariance: translation, rotation, scale
– photometric invariance: brightness, exposure, …
Feature Descriptors
Invariant local features
![Page 19: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/19.jpg)
• Say we have 2 images of this scene we’d like to align by matching local features • What would be the good local features (ones easy to match)?
What makes a good feature?
Uniqueness
![Page 20: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/20.jpg)
Types of invariance
• Illumination
![Page 21: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/21.jpg)
Types of invariance
• Illumination
• Scale
![Page 22: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/22.jpg)
Types of invariance
• Illumination
• Scale
• Rotation
![Page 23: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/23.jpg)
Types of invariance
• Illumination
• Scale
• Rotation
• Affine
![Page 24: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/24.jpg)
• Illumination
• Scale
• Rotation
• Affine
• Perspective
Types of invariance
![Page 25: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/25.jpg)
• Independently select interest points in each image, such that the detections are repeatable across different scales
Scale invariant interest points
![Page 26: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/26.jpg)
Intuition: Find scale that gives local maxima of some function f in both position and scale.
f
region size
Image 1
f
region size
Image 2
s1 s2
Automatic scale selection
![Page 27: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/27.jpg)
Intuition: Find scale that gives local maxima of some function f in both position and scale.
Automatic scale selection
– f is a local maximum in both position and scale
– Common definition of f: Laplacian: or difference between two Gaussian filtered images with different sigmas
![Page 28: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/28.jpg)
gdx
df
f
gdx
d
Edge
Derivative of Gaussian
Edge = maximum of derivative
Recall: Edge detection
![Page 29: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/29.jpg)
gdx
df
2
2
f
gdx
d2
2
Edge
Second derivative of Gaussian (Laplacian)
Edge = zero crossing of second derivative
Recall: Edge detection
![Page 30: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/30.jpg)
• Edge: ripple
• Blob: superposition of two ripples
Spatial selection: the magnitude of the Laplacian response will achieve a maximum at the center of the blob, provided the scale of the Laplacian is “matched” to the scale of the blob
maximum
From edges to blobs
![Page 31: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/31.jpg)
• Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D
2
2
2
22
y
g
x
gg
Blob detection in 2D
![Page 32: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/32.jpg)
• Laplacian-of-Gaussian = “blob” detector
2
2
2
22
y
g
x
gg
filt
er s
cale
s
img1 img2 img3
Blob detection in 2D: scale selection
![Page 33: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/33.jpg)
• We define the characteristic scale as the scale that produces peak of Laplacian response
characteristic scale
Blob detection in 2D
![Page 34: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/34.jpg)
Original image at ¾ the size
Original image
Scale space blob detector- Example
![Page 35: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/35.jpg)
Scale space blob detector- Example
Original image down sampled to ¾ the size but later stretched to match with original size
![Page 36: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/36.jpg)
Scale space blob detector- Example
![Page 37: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/37.jpg)
Scale space blob detector- Example
![Page 38: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/38.jpg)
Scale space blob detector- Example
![Page 39: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/39.jpg)
Scale space blob detector- Example
![Page 40: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/40.jpg)
Scale space blob detector- Example
![Page 41: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/41.jpg)
)()( yyxx LL
σ1
σ2
σ3
σ4
σ5
List of (x, y, σ)
scale
• Interest points are local maxima in both position and scale
Squared filter response maps
Scale invariant interest points
• Laplacian-of- Gaussian
![Page 42: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/42.jpg)
Scale-space blob detector: Example
![Page 43: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/43.jpg)
• Pyramids
– Smooth the image
– Subsample the smoothed image
– Repeat until image is small
• Each cycle of this process results in a smaller image with increased smoothing, but with decreased spatial sampling density (that is, decreased image resolution)
• Scale Space (DOG method)
Scale invariant interest points
![Page 44: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/44.jpg)
Pyramids
![Page 45: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/45.jpg)
• Pyramids
• Scale Space (DOG method)
– Pyramid but fill gaps with blurred images
– Take features from differences of these images
– If the feature is repeatably present in between Difference of Gaussians it is Scale Invariant and we should keep it.
Scale invariant interest points
![Page 46: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/46.jpg)
Building a Scale Space
• Scale-space representation L(x,y;t) at scale t=0,1,4,16,64,256 respectively , corresponding to the original image
![Page 47: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/47.jpg)
Building a Scale Space
• All scales must be examined to identify scale invariant features
• An efficient function is to compute the Laplacian Pyramid (Difference of Gaussian)
![Page 48: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/48.jpg)
Approximation of LoG by Difference of Gaussians
Heat Equation
Typical values : σ =1.6; k= 2
Rate of change of Gaussian
Gaussian filter with σ
![Page 49: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/49.jpg)
• We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.
2 ( , , ) ( , , )xx yyL G x y G x y
( , , ) ( , , )DoG G x y k G x y
Laplacian
Difference of Gaussians
Scale invariant interest points - Technical detail
![Page 50: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/50.jpg)
Building a Scale Space
Gaussian
Scale (next octave)
Scale (first octave) Peak detection
Local maxima/minima
Difference of Gaussian
k2σ
kσ
k3σ
k4σ
σ
![Page 51: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/51.jpg)
Building a Scale Space
http://www.vlfeat.org/api/sift.html
![Page 52: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/52.jpg)
Scale Space Peak Detection
http://www.vlfeat.org/api/sift.html
• Compare a pixel (X) with 26 pixels in current and adjacent scales (Green Circles)
• Select a pixel (X) if larger/smaller than all 26 pixels
• Large number of extrema, computationally expensive - Detect the most stable subset with a
coarse sampling of scales 9+9+8 = 26
![Page 53: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/53.jpg)
Key Point Localization
• Candidates are chosen from extrema detection
Original Image Extrema locations
![Page 54: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/54.jpg)
Initial Outlier Rejection
• Low contrast candidates
• Poorly localized candidates along an edge
• Taylor series expansion of DOG, D
• Minima or maxima is located at
• Value of D(x) at minima/maxima must be large, |D(x)|>th
![Page 55: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/55.jpg)
Initial Outlier Rejection
Reduced from 832 key points to 729 key points, th=0.03
![Page 56: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/56.jpg)
Further Outlier Rejection
• DOG has strong response along edge
• Assume DOG as a surface
- Compute principal curvatures (PC)
- Along the edge PC is very low, across the edge is high
![Page 57: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/57.jpg)
Further Outlier Rejection
• Analogous to Harris corner detector
• Compute Hessian of D
• Remove outliers by evaluating
![Page 58: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/58.jpg)
Further Outlier Rejection
• Following quantity is minimum when r=1
• It increases with r
• Eliminate key points if
![Page 59: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/59.jpg)
Further Outlier Rejection
Reduced from 729 key points to 536 key points
![Page 60: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/60.jpg)
Local features: main components
• Detection: Identify the interest points • Description: Extract feature vector descriptor surrounding each interest point • Matching: Determine correspondence between descriptors in two views
(1) (1)
1 1[ ,........ ]dX x x
(2) (2)
2 1[ ,........ ]dX x x
![Page 61: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/61.jpg)
Need both of the following:
1. Make sure your detector is invariant
– Harris is invariant to translation and rotation
– Scale is trickier
• common approach is to detect features at many scales using a Gaussian pyramid
• More sophisticated methods find “the best scale” to represent each feature (e.g., SIFT)
2. Design an invariant feature descriptor
– A descriptor captures the information in a region around the detected feature point
– The simplest descriptor: a square window of pixels
– Better approach is SIFT descriptor
How to achieve invariance
![Page 62: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/62.jpg)
SIFT Descriptor: is a 3-D spatial histogram of the image gradients in
characterizing the appearance of a keypoint.
SIFT - Feature descriptors
![Page 63: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/63.jpg)
• Use histograms to bin pixels within sub-patches according to their orientation.
0 2 p
SIFT - Feature descriptors
![Page 64: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/64.jpg)
Making descriptor rotation invariant
• To achieve rotation invariance, compute central derivatives, gradient magnitude and direction of L (smooth image) at the scale of key point (x,y)
![Page 65: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/65.jpg)
Orientation assignment
• Create a weighted direction histogram in a neighborhood of a key point (36 bins) – 10 degree for each orientation = 36x10 = 360
• Weights are - Gradient magnitudes - Spatial gaussian filter with σ =1.5 x <scale of key point>
Gradient magnitude
![Page 66: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/66.jpg)
Orientation assignment
• Select the highest peak as direction of the key point
Use of gradient orientation histograms has robust representation
![Page 67: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/67.jpg)
• Find dominant orientation of the image patch
– This is given by x+, the eigenvector of H corresponding to +
• + is the larger eigen value
– Rotate the patch according to this angle
Orientation assignment
![Page 68: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/68.jpg)
CSE 576: Computer Vision
• Rotate patch according to its dominant gradient orientation
• This puts the patches into a canonical orientation
Orientation assignment
Gradient orientation is more stable then raw intensity value
![Page 69: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/69.jpg)
• Extraordinarily robust matching technique
• Can handle changes in viewpoint
- Up to about 60 degree out of plane rotation
• Can handle significant changes in illumination
- Sometimes even day vs. night (below)
• Fast and efficient—can run in real time http://people.csail.mit.edu/albert/ladypack/wiki/index.php?title=Known_implementations_of_SIFT
SIFT - Feature descriptors
![Page 70: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/70.jpg)
• Take 16x16 square window around detected feature
• Compute edge orientation (angle of the gradient) for each pixel
• Throw out weak edges (threshold gradient magnitude)
• Create histogram of surviving edge orientations
0 2
angle histogram
SIFT - Feature descriptors
Keypoint descriptor Image gradients
![Page 71: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/71.jpg)
• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
• Compute an orientation histogram for each cell
• 16 cells * 8 orientations = 128 dimensional descriptor
Keypoint descriptor Image gradients
SIFT - Feature descriptors
![Page 72: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/72.jpg)
Local features: main components
• Detection: Identify the interest points • Description: Extract feature vector descriptor surrounding each interest point • Matching: Determine correspondence between descriptors in two views
(1) (1)
1 1[ ,........ ]dX x x
(2) (2)
2 1[ ,........ ]dX x x
![Page 73: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/73.jpg)
• Given a feature in I1, how to find the best match in I2?
– Define distance function that compares two descriptors
Matching local features
Image 1 Image 2
![Page 74: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/74.jpg)
?
• To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD)
• Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)
Image 1 Image 2
Matching local features
![Page 75: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/75.jpg)
• How to define the difference between two features f1, f2?
– Simple approach is SSD(f1, f2)
• Sum of Square Differences between entries of the two descriptors
f1 f2
Matching local features
SSD can give good scores to very ambiguous (bad) matches !!!
Image 1 Image 2
![Page 76: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/76.jpg)
• How to define the difference between two features f1, f2?
– Better approach: ratio = SSD(f1, f2) / SSD(f1, f2’)
• f2 is best SSD match to f1 in I2
• f2’ is 2nd best SSD match to f1 in I2
• Gives small values for ambiguous matches
Matching local features
f1 f2
Image 1 Image 2
f2'
![Page 77: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/77.jpg)
• To add robustness to matching, can consider ratio : Euclidean distance to best match / Euclidean distance to second best match
- If low, first match looks good
- If high, could be ambiguous match
Image 1 Image 2
? ? ? ?
Ambiguous matches
![Page 78: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/78.jpg)
• Nearest neighbor (Euclidean distance)
• Threshold ratio of nearest to 2nd nearest descriptor
Matching SIFT Descriptors
![Page 79: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/79.jpg)
• How can we measure the performance of a feature matcher?
50
75
200
feature distance
Evaluating the results
![Page 80: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/80.jpg)
• The distance threshold affects performance
– True positives = # of detected matches that are correct
• Suppose we want to maximize these—how to choose threshold?
– False positives = # of detected matches that are incorrect
• Suppose we want to minimize these—how to choose threshold?
50
75
200
feature distance
false match
true match
True/false positives
![Page 81: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/81.jpg)
0.7
• How can we measure the performance of a feature matcher?
0 1
1
false positive rate
true
positive rate
# true positives
# matching features (positives)
0.1
# false positives
# unmatched features (negatives)
Evaluating the results
![Page 82: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/82.jpg)
0.7
• How can we measure the performance of a feature matcher?
0 1
1
false positive rate
true
positive rate
# tru
e p
ositiv
es
# m
atc
hin
g f
eatu
res (p
ositiv
es)
0.1
# false positives
# unmatched features (negatives)
ROC curve (“Receiver Operator Characteristic”)
• ROC Curves
- Generated by counting # current/incorrect matches, for different threholds
- Want to maximize area under the curve (AUC)
- Useful for comparing different feature matching methods
- For more info: http://en.wikipedia.org/wiki/Receiver_operating_characteristic
Evaluating the results
![Page 83: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/83.jpg)
More on feature detection/description
![Page 84: Local Features: Detection, Description & Matching](https://reader031.fdocuments.us/reader031/viewer/2022012201/61de33d62fe44d42ea539410/html5/thumbnails/84.jpg)
Object recognition results – David Lowe