Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT !...
Transcript of Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT !...
![Page 1: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/1.jpg)
CSE 803 1
Local Patch Descriptors Slides courtesy of Steve Seitz and Larry Zitnick
![Page 2: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/2.jpg)
How do we describe an image patch?
![Page 3: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/3.jpg)
How do we describe an image patch?
Patches with similar content should have similar descriptors.
![Page 4: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/4.jpg)
What do human use?
Gabor filters…
… and many other things.
![Page 5: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/5.jpg)
Encoding the gradients
Poster
![Page 6: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/6.jpg)
11/8/17 6
Claimed Advantages of SIFT
n Locality: features are local, so robust to occlusion and clutter (no prior segmentation)
n Distinctiveness: individual features can be matched to a large database of objects
n Quantity: many features can be generated for even small objects
n Efficiency: close to real-time performance
n Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness
![Page 7: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/7.jpg)
11/8/17 7
Overall Procedure at a High Level
1. Scale-space extrema detection
2. Keypoint localization
3. Orientation assignment
4. Keypoint description
Search over multiple scales and image locations.
Fit a model to detrmine location and scale. Select keypoints based on a measure of stability.
Compute best orientation(s) for each keypoint region.
Use local image gradients at selected scale and rotation to describe each keypoint region.
![Page 8: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/8.jpg)
Basic idea: • Take 16x16 square window around detected feature • Compute gradient orientation for each pixel • Create histogram over edge orientations weighted by magnitude
Scale Invariant Feature Transform
Adapted from slide by David Lowe
0 2π angle histogram
![Page 9: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/9.jpg)
SIFT descriptor Full version
• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor
Adapted from slide by David Lowe
![Page 10: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/10.jpg)
Full version • Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor • Threshold normalize the descriptor:
SIFT descriptor
Adapted from slide by David Lowe
0.2
such that: Why?
![Page 11: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/11.jpg)
Properties of SIFT Extraordinarily robust matching technique
n Can handle changes in viewpoint n Up to about 30 degree out of plane rotation
n Can handle significant changes in illumination n Sometimes even day vs. night (below)
n Fast and efficient—can run in real time n Lots of code available
n http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT
![Page 12: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/12.jpg)
When does SIFT fail? Patches SIFT thought were the same but aren’t:
![Page 13: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/13.jpg)
Other methods: Daisy
SIFT
Daisy Picking the best DAISY, S. Winder, G. Hua, M. Brown, CVPR 09
Circular gradient binning
![Page 14: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/14.jpg)
Other methods: SURF For computational efficiency only compute gradient histogram with 4 bins:
SURF: Speeded Up Robust Features Herbert Bay, Tinne Tuytelaars, and Luc Van Gool, ECCV 2006
![Page 15: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/15.jpg)
Other methods: BRIEF
Daisy
BRIEF: binary robust independent elementary features, Calonder, V Lepetit, C Strecha, ECCV 2010
Randomly sample pair of pixels a and b. 1 if a > b, else 0. Store binary vector.
![Page 16: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/16.jpg)
Feature distance How to define the difference between two features f1, f2?
n Simple approach is SSD(f1, f2) n sum of square differences between entries of the two descriptors n can give good scores to very ambiguous (bad) matches
I1 I2
f1 f2
![Page 17: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/17.jpg)
Feature distance How to define the difference between two features f1, f2?
n Better approach: ratio distance = SSD(f1, f2) / SSD(f1, f2’) n f2 is best SSD match to f1 in I2 n f2’ is 2nd best SSD match to f1 in I2
n gives large values (~1) for ambiguous matches
I1 I2
f1 f2 f2'
![Page 18: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/18.jpg)
Eliminating bad matches Throw out features with distance > threshold
n How to choose the threshold?
50 75
200
feature distance
false match
true match
![Page 19: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/19.jpg)
True/false positives
The distance threshold affects performance n True positives = # of detected matches that are correct
n Suppose we want to maximize these—how to choose threshold?
n False positives = # of detected matches that are incorrect n Suppose we want to minimize these—how to choose threshold?
50 75
200
feature distance
false match
true match
![Page 20: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/20.jpg)
0.7
Evaluating the results How can we measure the performance of a feature matcher?
0 1
1
false positive rate
true positive
rate
# true positives matched # true positives
0.1 # false positives matched
# true negatives
features that really do have a match
the matcher correctly found a match
features that really don’t have a match
the matcher said yes when the right answer was no
![Page 21: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/21.jpg)
0.7
Evaluating the results How can we measure the performance of a feature matcher?
0 1
1
false positive rate
true positive
rate
0.1
ROC curve (“Receiver Operator
Characteristic”)
ROC Curves • Generated by counting # current/incorrect matches, for different threholds • Want to maximize area under the curve (AUC) • Useful for comparing different feature matching methods • For more info: http://en.wikipedia.org/wiki/Receiver_operating_characteristic
# true positives matched # true positives
# false positives matched # true negatives
![Page 22: Local Patch Descriptors - Semantic Scholar · 2017-11-11 · 11/8/17 6 Claimed Advantages of SIFT ! Locality: features are local, so robust to occlusion and clutter (no prior segmentation)](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f043ef6be59279e192b7a/html5/thumbnails/22.jpg)
Some actual ROC curves