Deconstruction: Discriminative learning of local image descriptorsSamantha HorvathLearning Based Methods in Vision2/14/2012

Introduction

•Computer vision makes use of many “hand-crafted” descriptors.

•These descriptors share many common components

•This paper presents a modular framework for designing and optimizing new feature descriptors

Common Descriptors

• SIFT▫ Most well-known

descriptor▫ Quantized gradient

vectors▫ Grid based spatial

histogram▫ Post-normalization

• PCA-SIFT▫ Quantized gradient

vectors▫ PCA to reduce

dimensionality• GLOH

▫ Quantized gradient vectors

▫ Polar based histogram▫ PCA to reduce

dimensionality

Common descriptors• SURF

▫Haar wavelet responses▫Grid based histograms

• HOG▫Dense SIFT

• Shape Context▫Extract points from object contour▫Polar based histogram

• Geometric blur▫Extract sparse channels▫Apply spatially varying blur▫Subsample to create descriptor

Image with Interest point Extracted Patch

Feature extraction

PoolingDimensionality reduction

Descriptor learning framework

Generalized Framework

•Represent each portion of the descriptor algorithm as an interchangeable block

•Blocks inspired by existing descriptor algorithms

•Blocks are organized into candidate descriptors, then parameters are optimized

Algorithmic building blocks

•G-block: Gaussian smoothing•T-block: non-linear transformation•S-block: Spatial summation/pooling•E-block: Embedding (dimensionality

reduction)•N-block: Normalization

General Pipeline

Transformations T1 – Gradient Orientation binning

Variants: number of orientation bins T2 – Rectified gradient binning

Variants: number of rectification bins T3 – Steerable filters

Variants: filter order and number of orientations T4 – Difference of Gaussian Responses (center-

surround) Parameter: size of center

T5 – Haar wavelet transform T6 – Fixed 4 x 4 classifier T7 – Quantized gray levels

Variant: number of gray level bins

Spatial summation blocks - parametric S1 - SIFT style

bilinear weighted grid Parameters: overall

footprint size (continuous)

S2 – GLOH style log-polar regions Variants:

number/arrangement of pooling regions

Parameters: Ring radius

Spatial summation blocks - parametric S3 – Gaussian weighted

pooling regions on a grid Variants: Grid size Parameters: Grid sample

positions, Size of the gaussians

S4 – Gaussian weighted, polar arranged pooling regions Variants: number of

pooling regions Parameters: Ring radii,

gaussian kernel size, relative angular rotation

Embedding – non-parametric

E1 – PCA (non discriminative technique)E2/E3 – Projection minimizes the ratio of

in-class variance for match pairs to the variance of all match pairs (LPP)

E4/E5 – Projection minimizes the ratio of variance between matched and non-matched pairs (LDE)

E6/E7 – Projection minimizes the ratio of in-class variance for match pairs to the total data variance (GLDE)

Smoothing and Normalization

•Gaussian smoothing▫Parameter: σ

•Normalization▫Normalize to unit vector▫Clip to threshold▫Renormalize, rinse, repeat▫Parameters: clipping threshold

New names!

•SIFT▫T1b – S1-16

•GLOH▫T1b – S2-17 – E1

•PCA-SIFT▫T1b – E1

•SURF▫T5 – S1

Data

Ground Truth dataset

Ground truth dataset

•Uses camera calibration and dense multi-view stereo data

•DoG interest points are detected•Interest points are mapped from one

image to another view of the scene•Stereo constraints are used to help match

interest points

Same to same vs. like to like

Learning/optimization•The G, S ,N blocks and one T block (T4)

contain parameters for optimization •The G, S, N, and T blocks are jointly optimized

using Powell minimization▫Powell minimization a conjugate gradient

method – does not require derivatives▫Optimization initialized with reasonable values

•E blocks are optimized separately – generalized eigenvalue problem▫Power regularization to avoid over fitting

Results…

Dimension Reduction on SIFT

•T3 blocks (rectified steerable filters) with polar summation regions (S4/S2) performed the best

•Consistently 0utperformed SIFT descriptors

Optimal Summation regions

Optimal summation regions are foveated!!!

Foveated summation regions

•The S4-25 spatial pooling variant is very similar to the DAISY descriptor (designed for dense matching)

•Foveated regions are similar to geometric blur (increased blurring away from interest point)

Results – Pipeline 2

•Performance more varied•Steerable filters still perform the best•LDE and LPP best embedding methods•Does not consistently outperform SIFT

Results – Pipeline 3

•Dimensionality reduction after learned T/S block combination

•Greatly outperforms SIFT•Straightforward PCA works the best

Thoughts

•Would like to see how the optimized spatial pooling blocks vary with the different training sets

•Ultimately, would like to see this framework tested on different dataset types

•Difficulty is getting “ground truth” matches for identification/classification tasks

Now what?

Synthesis

•Majority of my computer vision research involves medical imaging

•Medical images are very different from natural scene images▫Images represent a planar slice through

the patient▫Often poor contrast between different

structures

Synthesis

•Interest point detection has applications for medical imaging

•Non-rigid registrations▫Warp one set of interest points to overlay

the second set•Tracking

References1. M. Brown, G. Hua and S. Winder, Discriminant Learning of Local Image Descriptors. IEEE

Transactions on Pattern Analysis and Machine Intelligence. 2010. (original)2. S. Winder and M. Brown, “Learning local image descriptors,” in Proceedings of the International

Conference on Computer Vision and Pattern Recognition (CVPR07), Minneapolis, June 2007. (technical)

3. K. Mikolajczyk and C. Schmid, “Scale and affine invariant interest point detectors,” International Journal of Computer Vision, vol. 1, no. 60, pp. 63–86, 2004. (overview)

4. E. Tola, V. Lepetit, and P. Fua, “A fast local descriptor for dense matching,” in Proceedings of the International Conference on Computer Vision and Pattern Recognition, Anchorage, June 2008. (daisy)

5. Y. Ke and R. Sukthankar, “PCA-SIFT: a more distinctive representation for local image descriptors,” in Proceedings of the International Conference on Computer Vision and Pattern Recognition, vol. 2, July 2004, pp. 506–513. (PCA-sift)

6. A. Berg and J. Malik, “Geometric blur and template matching,” in International Conference on Computer Vision and Pattern Recognition, 2001, pp. I:607–614.

7. D. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004. (SIFT)

8. Bay,H,. Tuytelaars, T., &Van Gool, L.(2006). “SURF: Speeded Up Robust Features”, 9th European Conference on Computer Vision.

9. G. Sharma, F. Jurie, Learning discriminative spatial representation for image classification, British Machine Vision Conference (BMVC.11) (grid paper)

10. W. T. Freeman and E. H. Adelson, “The design and use of steerable filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, pp. 891–906, 1991.