Shape Classification Using the Inner-Distance

Haibin Ling

David W. Jacobs

IEEE TRANSACTION ON PATTERN ANAYSIS AND MACHINE INTELLIGENCE

FEBRUARY 2007

Outline

Introduction Related work Inner-Distance Articulation invariant signatures Inner-distance shape context Shortest path texture context Experiments

Introduction

We use the inner-distance to build shape descriptors that are robust to articulation and capture part structure.

Inner-distance is defined as the length of the shortest path between landmark points within the shape boundary.

Related work

Three categories to handle parts classification: Statistical methods to describe the articulation between

parts and often require a learning process to find the model parameters.

To measures the similarity between shapes between shapes via part-to-part matching and junction parameter distribution.

To capture the part structure by considering the interior of shape boundary. Our method belongs to. Skeleton-based approaches

The inner-distance: The definition Define a shape as a connected and closed

subset of R2. Given a shape and two points ,the inner-distance between denoted as ,is defined as the length of the shortest path connecting and within .

);,( OyxdOyx ,

OO

,, yx

O

Note:(1) In rare case where there are multiple shortest paths, we arbitrarily choose one.(2) Shapes are defined by their boundary, hence only boundary points are used as landmark points.

x y

The inner-distance: Computation Shortest path algorithms:

(1) Build a graph with the sample points. For each pair of sample points p1 and p2, if the line segment connecting p1 and p2 fall entirely within the object, Let an edge between p1 and p2 is added to the graph with its weight equal to the Euclidean distance || p1 - p2 ||.

Note:(1) Neighboring boundary points ar

e always connected.(2) (2) The inner-distance reflects t

he existence of holes without using samples points from hole boundary.

The inner-distance: Computation

(2) find the inner-distance between all pairs of points according to the graph.

The whole computation takes . It takes time to checi whether a line segment

between two points is inside the given shape. The complixity of graph construction is of .

)( 3nO

)(no

)( 3nO

The inner-distance:A model of articulate objects

}{}{1

ijji

i

n

iJOO

Articulated objects. (a) An articulated shape. (b) Overlapping junstions. (c) Ideal articulation.

The inner-distance:A model of articulate objects

is constant and very small compared to the size of the articulated parts.

An articulated to another articulated object is one-to-one continuous mapping .

)};,({max)( where,)(,

PyxdPdiamJdiamPyx

ij

fO 'O

The inner-distance: articulation insensitivity Changes of the inner-distance are due to

junction deformations. That means change is very small compared to the size of parts.

niOyxOyxdOyxd iii 1,, )';','();,(

ij

ij

Jnjiji

JyxOyxdOyxd

,,1,

,,)';','();,(

The inner-distance: articulation insensitivity Theorem:

Proof:

Is decomposed into segments.

εm,m'x',y';O'dx,y;Od

O'x',y';O'Γ

Ox,y;OΓ

Ox,yO

O

}max{|)(-)(|

then.in junctionsdifferent m' through goes )(

and in junctionsdifferent m through goes )(

path shortest thesuppose , above. defined as of

on articulatian be f andobject darticulatean be Let

);,( Oyx l

)];,(),...,;,(),;,([);,( 1221110 lll RppRppRppOyx

The inner-distance: articulation insensitivity

The inner-distance: articulation insensitivity Example:

li

iii RppdOyxd1

1 );,();,(

)]';','(),...,';','(),';','([)';','(~

1221110 lll RppRppRppOyxC

mmOyxdOyxd 1)';','(~

);,(

Note:

is not the shortest path.

)';','(~

OyxC

The inner-distance: ability to capture structures It is hard to prove because no clear part

decomposition.

Show how the inner-distance capture part structure with examples:

The inner-distance: ability to capture structures

With about the same number of sample points, the four shapes are virtually indistinguishable using distribution of Euclidean distance. However, their distributions of the inner-distance are quite different except for the first two shapes.

Note: more sample points will not affect the above statement.

Articulation Invariant Signatures The inner-distance is used to build articulation

invariant signatures for 2D shapes using multidimensional scaling (MDS).

Given sample points on the shape O.the inner-distance .MDS finds the transformed points such that the Euclidean distance minimize the stress S(Q)

niipP 1}{

njiijd 1,}{

niiqQ 1}{

njijiij qqQe 1,})({

ji ij

ji ijijij

d

QedwQS

2

2))(()(

Articulation Invariant Signatures Example:

MDS+SC+DP Use MDS to get articulation invariance signatures. Build the shape context on the signatures. Use dynamic programming for shape context matching

(a) and (c) show two shapes related by articulation. (b) and (d) show their signatures.

Related work:Shape Contexts for 2D shape The shape context was introduced by Belongi

e et al. Due to its simplicity and discriminability, the s

hape context has become quite popular recently in shape matching tasks.

It describes the relative spatial distribution.

Related work:Shape Contexts for 2D shape Given n sample on shape.

The shape context at points is defined as a histogram of the relative coordinates of the remaining n-1 points.

Where the bins uniformly divide the log-polar space. The shape context uses the Euclidean distance to m

easure the spatial relation between landmark points. This causes less discriminability for complex shapes with articulations.

ix

nxxx ,...,, 21

ixih

)}(,:{#)( kbinxxijxkh iiji

Inner-Distance Shape Context (IDSC) To extend the shape context, Euclidean

distance is directly replaced by the inner-distance.

Inner-Distance Shape Context (IDSC) The angle between the contour tangent at p

and the direction of at p is insensitive to articulation, called inner-angle, denoted .

Inner-angle is used for the orientation bins. Noise may reduces the stability of the inner-

angle, smoothing contour before computing it.

);,( Oqp

);,( Oqp

Inner-Distance Shape Context (IDSC) Example:

In the histogram, the x axis denotes the orientation bins and the y axis denotes log distance bins.

Shape matching through Dynamic programming Given two shapes A and B, points sequences

on their contour , say, for A and for B, assume .

A matching from A to B is a mapping. is matched to if , and otherwise

left unmatched. should minimize the match cost.

ni

iicC1

))(,()(

nppp 11 mqqq 11

mn

ip

)(iqip 0)( i

is the penalty for leaving unmatched, and for , is the cost of matching to .

and are the shape context histogram of and . K is the number of histogram bins.

Kk jBiA

jBiA

khkh

khkhjic

1 ,,

2,,

)()(

)]()([

2

1),(

)0,(ic ip

mj 1 ),( jicip iq

iAh , jBh ,

ip iq

Shape matching through Dynamic programming

Shape matching through Dynamic programming

DP is used to solve the matching problem since it uses the ordering information provided by shape contours.

By default, assumes the two contours are already aligned at their start and end points.

Without this assumption, one simple solution is to try different alignments at all points on the first contour and choose the best one.

)()(:complexity the 32 nOnO

Shape matching through Dynamic programming

Because shapes can be first rotated according to their moments, it is sufficient to try aligning a fixed number of points, say k points.

Usually, k is much smaller than m and n.

)()( still is complexity the 22 nOknO

Shape distance

The matching cost is used to measured the similarity between shapes.

IDSC+DP is better than SC+DP Better performance Only two parameters to tune

The penalty for a point with no matching, usually set 0.3.

The number of start points k for different alignments, usually set 4-8.

Easy to implement since it does not require the appearance and transformation model.

)(C

Shortest path texture context

The combination of texture and shape information, because Shapes from different classes sometimes are

more similar than those from the same class. Shapes are often damaged due to occlusion and

self-overlapping.

Shortest path texture context

The texture information along these paths provides a natural articulation insensitive texture description.

The angles between intensity gradient directions and shortest path directions are used, called relative orientations.

Shortest path texture context

The SPTC for each is a three dimensional histogram . The inner-distance The inner-angle The (weighted) relative orientation

The relative orientations are weighted by gradient magnitudes.

ix

ih

Shortest path texture context

Experiment

the number of inner-distance bins: or the number of inner-angle bins:the number of relative orientation bins:

The number of different starting points for alignment:

The penalty for one occlusion:

5dn12n

8rn

8dn

84 k

3.0

Experiment

(a) Articulated database. (b) MDS of the articulated database using the inner-distance.

Experiment

Retrieval result on the articulate data set

SC+DP IDSC+DP

Experiment

MPEG7 CE-Shape-1 shape database is widely tested, which consists of 1400 silhouette images from 70 classes. Each class has 20 different shapes.

Bullseye test: for every image in the database, it is matched with all other images and the top 40 most similar candidates are counted.

Experiment

The score of the test is the ratio of the number of correct hits of all images to the highest possible number of hits (which is 20x1400).

Experiment

The Kimia Database Data set 1 : 25 instance from six categories

Experiment

Data set 2 : 99 instances from nine categories

Experiment

The ETH-80 image set:

This data set contains 80 objects from eight classes, with 41 images of each object obtained from different viewpoints.

Experiment:Foliage image retrieval Swedish leaf data set

Experiment:Foliage image retrieval Smithsonian data set:

343 leaf images from 93 species.

Experiment:Foliage image retrieval

Experiment:Foliage image retrieval

Experiment:Human body matching