Matching and Recognition in Matching and Recognition in 3D3D

Moving from 2D to 3D –Moving from 2D to 3D –Some Things are EasierSome Things are Easier

• No occlusion (but sometimes missing data No occlusion (but sometimes missing data instead)instead)

• Segmenting objects often simplerSegmenting objects often simpler

Moving from 2D to 3D –Moving from 2D to 3D –Many Things are HarderMany Things are Harder

• Rigid transform has 6 degrees of freedom Rigid transform has 6 degrees of freedom vs. 3vs. 3– Brute-force algorithms much less practicalBrute-force algorithms much less practical

• Rotations do not commuteRotations do not commute– Difficult to parameterize, search overDifficult to parameterize, search over

• No natural parameterization for surfaces No natural parameterization for surfaces in 3Din 3D– Hard to do FFT, convolution, PCAHard to do FFT, convolution, PCA

– Exception: range imagesException: range images

Matching / Recognition in 3DMatching / Recognition in 3D

• Project into 2D, do image matchingProject into 2D, do image matching

• Structural methods (i.e., part Structural methods (i.e., part decomposition, graph matching)decomposition, graph matching)

• Shape similarity methodsShape similarity methods

• Statistical methodsStatistical methods

• Feature-based methodsFeature-based methods

3D Medial Axis and Shock 3D Medial Axis and Shock ScaffoldsScaffolds

• Medial axis: locus of points equidistant Medial axis: locus of points equidistant fromfrom2 surfaces2 surfaces

• Shock scaffolds [Leymarie & Kimia]: do Shock scaffolds [Leymarie & Kimia]: do matching on “sheets” and “lines”matching on “sheets” and “lines”

Shape SimilarityShape Similarity

• Key difficulty – locating objects under anyKey difficulty – locating objects under anyrigid-body transformationrigid-body transformation

• Translation: relatively easy (match Translation: relatively easy (match centroids)centroids)

• Rotation:Rotation:– Align objects to each otherAlign objects to each other

– Align objects to canonical coordinate frameAlign objects to canonical coordinate frame

– Rotation-invariant methodsRotation-invariant methods

Iterative Closest Points (ICP)Iterative Closest Points (ICP)

• Besl & McKay, 1992Besl & McKay, 1992

• Start with rough guess for alignmentStart with rough guess for alignment

• Iteratively refine transformIteratively refine transform

ICPICP

• Assume Assume closestclosest points correspond to points correspond to each other, compute the best each other, compute the best transform…transform…

ICPICP

• … … and iterate to find alignmentand iterate to find alignment

• Converges to Converges to somesome local minimum local minimum

• Correct if starting position “close Correct if starting position “close enough“enough“

Aligning ScansAligning Scans

• Start with manual initial alignmentStart with manual initial alignment

[Pulli][Pulli]

Aligning ScansAligning Scans

• Improve alignment using ICP algorithmImprove alignment using ICP algorithm

[Pulli][Pulli]

Aligning Objects With MomentsAligning Objects With Moments

• For each point on object, computeFor each point on object, compute

• Canonical orientation based on Canonical orientation based on eigenvectorseigenvectors(ordered by eigenvalue)(ordered by eigenvalue)

T))(( xxxx T))(( xxxx

Problem with PCA-Based Problem with PCA-Based AlignmentAlignment

• If eigenvalues are close, axes unstableIf eigenvalues are close, axes unstable

Rotation-Invariant DescriptorsRotation-Invariant Descriptors

• Decompose model into spherical shellsDecompose model into spherical shells

• Decompose each shell into spherical Decompose each shell into spherical harmonicsharmonics

• Keep amplitude, throw away phaseKeep amplitude, throw away phase

3D ModelShape

DescriptorRotation Independent

Components

Statistical Methods for Matching Statistical Methods for Matching ShapeShape

• EGI: extended Gaussian imagesEGI: extended Gaussian images

• For each direction, what fraction of For each direction, what fraction of normals point in that directionnormals point in that direction

• Not rotation invariant, but tends to be Not rotation invariant, but tends to be peakypeaky

Shape DistributionsShape Distributions

• Osada, Funkhouser, Chazelle, and Osada, Funkhouser, Chazelle, and DobkinDobkin

• Compact representation for entire 3D Compact representation for entire 3D objectobject

• Invariant under translation, rotation, Invariant under translation, rotation, scalescale

• Application: search engine for 3D Application: search engine for 3D shapesshapes

Computing Shape DistributionsComputing Shape Distributions

• Pick Pick nn random pairs of points on the random pairs of points on the objectobject

• Compute histogram of distancesCompute histogram of distances

• Normalize for scaleNormalize for scale

3D Model3D Model ShapeShapeDistributionDistribution

Random Random samplingsampling

Comparing Shape DistributionsComparing Shape Distributions

SimilaritySimilarityMeasureMeasure

3D3DModelModel

ShapeShapeDistributionDistribution

Shape Distributions for Simple Shape Distributions for Simple ShapesShapes

Robustness ResultsRobustness Results

7 Mugs

7 Missiles

Classification ResultsClassification Results

Classification ResultsClassification Results

Features on SurfacesFeatures on Surfaces

• Can construct edge and corner detectorsCan construct edge and corner detectors

• Analogue of 1Analogue of 1stst derivative: surface normal derivative: surface normal

• Analogue of 2Analogue of 2ndnd derivative: curvature derivative: curvature– Curvature at each point in each directionCurvature at each point in each direction

– Minimum and maximum: “principal curvatures”Minimum and maximum: “principal curvatures”

– Can threshold or do nonmaximum suppressionCan threshold or do nonmaximum suppression

3D Identification Using Spin 3D Identification Using Spin ImagesImages

• Spin images: Johnson and HebertSpin images: Johnson and Hebert

• ““Signature” that captures local shapeSignature” that captures local shape

• More expressive than curvatureMore expressive than curvature

Computing Spin ImagesComputing Spin Images

• Start with a point on a 3D modelStart with a point on a 3D model

• Find (averaged) surface normal at that Find (averaged) surface normal at that pointpoint

• Define coordinate system centered at this Define coordinate system centered at this point, oriented according to surface point, oriented according to surface normal and two (arbitrary) tangentsnormal and two (arbitrary) tangents

• Express other points (within some Express other points (within some distance) in terms of the new coordinatesdistance) in terms of the new coordinates

Computing Spin ImagesComputing Spin Images

• Compute histogram of locations of other Compute histogram of locations of other points, in new coordinate system, points, in new coordinate system, ignoring rotation around normal:ignoring rotation around normal:

n̂ p n̂ p

n̂p n̂p

Computing Spin ImagesComputing Spin Images

Spin Image ParametersSpin Image Parameters

• Size of neighborhoodSize of neighborhood– Determines whether local or global shapeDetermines whether local or global shape

is capturedis captured

– Big neighborhood: more discriminatory powerBig neighborhood: more discriminatory power

– Small neighborhood: resistance to clutterSmall neighborhood: resistance to clutter

• Size of bins in histogram:Size of bins in histogram:– Big bins: less sensitive to noiseBig bins: less sensitive to noise

– Small bins: captures more detail, less storageSmall bins: captures more detail, less storage

Spin Image ResultsSpin Image Results

Range ImageRange Image Model in DatabaseModel in Database

Spin Image ResultsSpin Image Results

Detected ModelsDetected Models