Ivo Havinga United Nations Statistics Division havinga@un Rio Conventions Pavilion Event:
Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British...
-
Upload
john-norris -
Category
Documents
-
view
225 -
download
1
Transcript of Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British...
![Page 1: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/1.jpg)
Distinctive Image Features from Scale-Invariant KeypointsBy David G. Lowe, University of British Columbia
Presented by:
Tim Havinga,
Joël van Neerbos
and
Robert van der Linden
![Page 2: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/2.jpg)
Organization
• Introduction• Keypoint extraction• Applications
![Page 3: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/3.jpg)
Introduction
Matching images across affine transformation:
Change in lightingand 3D viewpoint:
![Page 4: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/4.jpg)
Introduction
• Motion tracking• Object and scene recognition• Stereo correspondence
![Page 5: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/5.jpg)
Extracting features
• Extrema detection• Keypoint localization• Orientation assignment• Local image descriptor
![Page 6: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/6.jpg)
Extrema detection
Blur copies of the image with broadening Gaussian filters.
![Page 7: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/7.jpg)
Extrema detection
Subtract these (DoG) to find local extrema.
![Page 8: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/8.jpg)
Extrema detection
Calculate the DoGs for different gaussians.
2 x
![Page 9: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/9.jpg)
Extrema detection
Calculate the DoGs for different gaussians.
2 x
![Page 10: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/10.jpg)
Extrema detection
Blur Blur
![Page 11: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/11.jpg)
Keypoint localization
Select keypoints that are higher or lower than their 26 neighbours.
![Page 12: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/12.jpg)
Keypoint localization
Reject all points where the contrast is too low.
![Page 13: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/13.jpg)
Keypoint localization
Reject all points that lie on an edge.
![Page 14: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/14.jpg)
Effects of this elimination
Extrema detection
![Page 15: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/15.jpg)
Effects of this elimination
Contrast check
![Page 16: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/16.jpg)
Effects of this elimination
Edge check
![Page 17: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/17.jpg)
Extracting features
• Extrema detection• Keypoint localization• Orientation assignment• Local image descriptor
![Page 18: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/18.jpg)
Orientation assignment
Assign an orientation to a keypoint to make its descriptor invariant to rotation
![Page 19: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/19.jpg)
Orientation assignment
The orientation of a keypoint is determined in four steps:
1. Determine sample points2. Determine the gradient magnitude and
orientation of each sample point3. Create an orientation histogram of the
sample points4. Extract the dominant directions from the
histogram
![Page 20: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/20.jpg)
Step 1: Determine sample points
• The source image is the Gaussian smoothed image with the closest scale
• Use all pixels within a certain radius
Actual scale
Used Gaussian
![Page 21: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/21.jpg)
Step 2: Determine gradient magnitude and orientation of each sample point
• Gradient magnitude:
• Gradient orientation:
![Page 22: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/22.jpg)
Step 2: Determine gradient magnitude and orientation of each sample point
• Gradient magnitude:
• Gradient orientation:
pixel
![Page 23: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/23.jpg)
Step 3: Create an orientation histogram
• The histogram has 36 bins, each covering 10 degrees
• Each sample is weighted its gradient magnitude and a Gaussian weighted circular window
![Page 24: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/24.jpg)
Step 4: Extract dominant directions
• Take the peak(s) from the orientation histogram
• Use all peaks greater than 80% of the highest peak
• Every direction gets its own keypoint
![Page 25: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/25.jpg)
The Local image descriptor
• Every keypoint now has a location, scale and orientation, from which a repeatable 2D grid can be determined
• We want distinctive descriptor vectors, partially invariant to illumination and viewpoint changes
![Page 26: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/26.jpg)
Computing the Local image descriptor
• Take the 16 x 16 sample array around the keypoint
• Compute 4 x 4 orientation histograms from this array
• Use 8 bins per histogram: 4x4x8=128 features
![Page 27: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/27.jpg)
Local image descriptor optimizations
• Normalize the obtained feature vector to enhance invariance to illumination changes
• Reduce the influence of large gradient magnitudes by capping the normalized features to 0.2
• Normalize again
![Page 28: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/28.jpg)
Possible applications for SIFT
We have a feature extraction method which yields useful keypoints, what's next?
• Some appications:• Object recognition in images• Panorama stitching• 3D scene modelling• 3D human action tracking
(for example for security surveillance)• Robot localisation and mapping
![Page 29: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/29.jpg)
Panorama stitching
![Page 30: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/30.jpg)
Brown, ICCV 2003
Panorama stitching
![Page 31: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/31.jpg)
(from Sudderth et al., 2006)
3D modelling
![Page 32: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/32.jpg)
Application: SIFT to object recognition
We can applicate SIFT to recognize objects in images.Say, we have an image which contains an object. How to recognize?
Key idea: Compare keypoints, if these are similar it is likely that it is the same object.
First problem: a lot of features arise from background clutter. How to remove these?
Possible approach: - Look for clusters of matching features- Look for distance of closest match to the second-closest match
![Page 33: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/33.jpg)
Efficiently locating the nearest neighbour
128 dimensional feature vector for each keypoint: no search optimization possible, no better way to find the nearest neighbour than exhaustive search.
But: only 3 features are enough to locate objects, for example when occluded.
Hough Transform method is used to describe clusters of keypoints as shapes and let them 'vote' for the pose of an object, described in location, orientation and scale.
![Page 34: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/34.jpg)
![Page 35: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/35.jpg)
Application: robot vision, localization and mapping
Se, S. Lowe, D. G. Little, J. Vision-based Mobile Robot Localization And Mapping using Scale-Invariant Features, 2001
Application of SIFT to mobile robotics SIFT features combined with Simultaneous
Localization And Map Building (SLAMB) Recognizing landmarks: estimation of the 10m by 10m lab, 3000 features collected Preliminary results: quite good
![Page 36: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/36.jpg)
![Page 37: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/37.jpg)
Conclusions from the paper
• The keypoints SIFT extracts are indeed invariant to image rotation, scale and robust to affine distortion, noise and change in illumination.
• SIFT can be optimized to run real-time.• The proposed approach (SIFT combined with
Hough transform for object recognition) has shown to work reliably.
![Page 38: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/38.jpg)
Discussion
• Is the SIFT method for keypoint extraction the best way to get distinctive features from images?
• Is SIFT biologically plausible? Is it important to have biologically inspired methods in object recognition / localization?
![Page 39: Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.](https://reader035.fdocuments.us/reader035/viewer/2022081603/56649dea5503460f94ae51f1/html5/thumbnails/39.jpg)
References
Main article:• Distictive Image Features from Scale-Invariant
Keypoints, D. G. Lowe. International Journal of Computer Vision 60, 91-110, 2004.
Other articles:• Depth from Familiar Objects: A Hierarchical Model for
3D Scenes, Sudderth et al, Proceedings of the 2006 IEEE Conference on computer vision and pattern recognition, volume II, 2410-2417, 2006.
• Vision-based Mobile Robot Localization And Mapping using Scale-Invariant Features, Se, S. Lowe, D. G. Little, J., 2001