Features, Feature descriptors, Matching

Jana KoseckaGeorge Mason University

MSRI Workshop, January 20052

Computer Vision Computer Vision

Visual Sensing Visual Sensing

Images I(x,y) – brightness patternsImages I(x,y) – brightness patterns

- image appearance depends on structure of the scene- material and reflectance properties of the objects- position and strength of light sources

MSRI Workshop, January 20053

• photometric properties of the environment• geometric properties of the environment

What gives rise to images

MSRI Workshop, January 20054

Basic ingredients

Radiance – amount of energy emitted along certain direction

Iradiance – amount of energy received along certain direction

BRDF – bidirectional reflectance distributionLambertian surfaces – the appearance depends only on radiance, not on the viewing direction

Image intensity for a Lambertian surface

MSRI Workshop, January 20055

Challenges

MSRI Workshop, January 20056

Image Primitives and Matching

Given an image point in left image, what is the (corresponding) point in the rightimage, which is the projection of the same 3-D point

MSRI Workshop, January 20057

Image Primitives and Correspondence

Difficulties – ambiguities, large changes of appearance, due to changeof viewpoint, non-uniquess

MSRI Workshop, January 20058

Correspondence

Lambertian assumption

Rigid body motion

Matching - Correspondence

radiance

MSRI Workshop, January 20059

Translational model

Affine model

Transformation of the intensity values taking into account occlusions and noise

Local Deformation Models

MSRI Workshop, January 200510

Matching and Correspondence

Motivated by problems Reconstruction of 3D scene from multiple views Object recognition using (constellation of) features modelsVarieties Small base-line matching Wide base-line matching – large view point changes For now assuming Lambertian assumption –

appearance of a local surface patch is independent of the viewpoint

MSRI Workshop, January 200511

• Translational model

• RHS approximation by the first two terms of Taylor series

• Small baseline

• Brightness constancy constraint

Feature Tracking and Optical Flow

MSRI Workshop, January 200512

• Integrate around over image patch

• Solve

Feature Tracking and Optical flow

MSRI Workshop, January 200513

rank(G) = 0 blank wall problemrank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates

Conceptually:

In reality: choice of threshold is involved

Optical Flow, Feature Tracking

MSRI Workshop, January 200514

Affine feature tracking

Intensity offsetContrast change

MSRI Workshop, January 200515

• Qualitative properties of the motion fields

• Previous method - assumption locally constant flow

• Alternative regularization techniques (locally smooth flow fields, integration along contours)

Optical Flow

MSRI Workshop, January 200516

• Compute eigenvalues of G• If smalest eigenvalue of G is bigger than - mark pixel as candidate feature point

• Alternatively feature quality function (Harris Corner Detector)

Point Feature Extraction

MSRI Workshop, January 200517

Harris Corner Detector - Example

MSRI Workshop, January 200518

Feature Selection

Compute Image Gradient Compute Feature Quality measure for each pixel

Search for local maxima

Feature Quality Function Local maxima of feature quality function

MSRI Workshop, January 200519

Feature Tracking

Translational motion model

Closed form solution

1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

MSRI Workshop, January 200520

Coarse to fine feature tracking

1. compute 2. warp the window in the second image by3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations

0

2

1

MSRI Workshop, January 200521

Tracked Features

MSRI Workshop, January 200522

Wide baseline matching

Point features detected by Harris Corner detector

MSRI Workshop, January 200523

• Sum of squared differences

• Normalize cross-correlation

• Sum of absolute differences

Region based Similarity Metric

MSRI Workshop, January 200524

NCC score for two widely separated views

NCC score

MSRI Workshop, January 200525

Advanced matching techniques

( )

1. Selected salient image locations - points, pieces of countours2. Associate Local photometric descriptors 3. Invariance to image transformations + illumination changes

NCC - is not invariant with respect to image transformation

MSRI Workshop, January 200526

Summary of the approach

Very good results in the presence of occlusion and clutter local information discriminant greyvalue information robust estimation of the global relation between

images for limited view point changes

Solution for more general view point changes wide baseline matching (different viewpoint, scale and

rotation) local invariant descriptors based on greyvalue

information

MSRI Workshop, January 200527

Local descriptors

Greyvalue derivatives

Invariance to image rotation :

differential invariants [Koenderink87]

MSRI Workshop, January 200528

Feature Detection and Matching

Detection of interest points/regions Harris detector (extension to scale and

affine invariance) Computation of descriptors for each point

(e.g. diff. invariants, steerable filters, SIFT descriptor) Similarity of descriptors (Euclidean distance, Mahalanobis

Distance)

MSRI Workshop, January 200529

Keypoint Detector and SIFT Descriptor

Each image is characterized by a set of scale-invariant keypoints and their associated descriptors [D. Lowe,2000]

Keypoints - extrema in DOG pyramid

Descriptor – 8 bin orientation histograms computed

over 4 x 4 grid overlayed over pixel neighbourhood

and stacked together to form a 128 dim feature vector

MSRI Workshop, January 200530

SIFT Keypoints

MSRI Workshop, January 200531

Overview

Scale invariance is not sufficient for large baseline changes

State of the art on affine invariant points/regions

Affine invariant interest points

Application to recognition

MSRI Workshop, January 200532

Scale invariant interest points

Invariant points + associated regions [Mikolajczyk & Schmid’01]

multi-scale Harris points

selection of points

at the characteristic scale

with Laplacian

Courtesy of Schimd’01

MSRI Workshop, January 200533

Viewpoint changes

Locally approximated by an affine transformation

A

detected scale invariant region

projected region

Courtesy of Schimd’01

MSRI Workshop, January 200534

Affine invariant Harris points

Localization & scale influence affine neighhorbood=> affine invariant Harris points (Mikolajczyk &

Schmid’02)

Iterative estimation of these parameters1. localization – local maximum of the Harris measure 2. scale – automatic scale selection with the Laplacian3. affine neighborhood – normalization with second

moment matrixRepeat estimation until convergence

Initialization with multi-scale interest points

MSRI Workshop, January 200535

Alternative features/descriptors

Affine invariant regions (Tuytelaars et al.’00) ellipses fitted to intensity maxima parallelogram formed by interest points and edges

• Maximally stable regions (Matas et al. BMVC’02) regions stable across large range of thresholds, connected components of thresholded image descriptors – rotationaly and affine invariant and color moments

MSRI Workshop, January 200536

Feature Matches

33 correct matches

Courtesy of Schimd’01

MSRI Workshop, January 200537

Pieces of Countour/Line descriprors

Select salient pieces using scale invariant detection techniques Characterize either the intensity profile along contour/or local neighbourhood with sideness information – form the descriptor Type of suitable salient regions depends of the class of objects Computational model of visual attention can guide the process of selecting salient regions

MSRI Workshop, January 200538

Additional changes of the appearance