1Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Shape

Optimal invariant metrics for shape retrieval

Michael Bronstein

Department of Computer ScienceTechnion – Israel Institute of Technology

2Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

3Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Tagged shapes

Shapes withoutmetadata

Man, person, humanPersonText search

Content-based search

3D warehouse

4Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Outline

Feature descriptor

Geometric words

Bag of words?

5Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Invariance

Local geodesic distance histogram

Rigid Inelastic Topology

Gaussian curvature

Scale

Wang, B 2010

Heat kernel signature (HKS)

Scale-invariant HKS (SI-HKS)

6Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Heat kernels

Heat kernel is a fundamental solution of the heat equation with

point heat source at (heat value at point after time )

Initial conditions: heat distribution at time

Solution : heat distribution at time

Heat equation governs heat propagation

on a surface

7Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Heat kernel signature

Sun, Ovsjanikov & Guibas SGP 2009

Multiscale local shape descriptor

can be interpreted as probability of Brownian motion to return to

the same point after time (represents “stability” of the point)

Time (scale)

8Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Heat kernel signature

Sun, Ovsjanikov, Guibas SGP 2009Ovsjanikov, BB & Guibas NORDIA 2009

Heat kernel signatures represented in RGB space

9Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Scale invariance

B, Kokkinos CVPR 2010

Original shape Scaled by

HKS= HKS=

Not scale invariant!

10Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Scale-invariant heat kernel signature

B, Kokkinos CVPR 2010

Log scale-space

Scaling = shift and multiplicative

constant in HKS

log + d/d

Undo scaling

Fourier transformmagnitude

Undo shift

0 100 200 300-15

-10

-5

0

t0 100 200 300

-0.04

-0.03

-0.02

-0.01

0

t0 2 4 6 8 10 12 14 16 18 20

0

1

2

3

4

=2k/T

11Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Scale invariance

B, Kokkinos CVPR 2010

Heat Kernel Signature Scale-invariantHeat Kernel Signature

12Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Scale invariance

B, Kokkinos CVPR 2010

Heat Kernel Signature Scale-invariantHeat Kernel Signature

13Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Modeling vs learning

Wang, B 2010

14Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Learning invariance

Positives P Negatives N

T

15Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Similarity learning

with high probability

false positive

false negative

positive

negative

16Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Similarity-preserving hashing

-1 -1 +1 -1-1 -1 -1 +1-1 -1 +1 +1

+1 +1 -1 +1+1 +1 +1 -1+1 +1 +1 +1

with high probability

with low probability

Collision:

= # of distinct bits

Gionis, Indik, Motwani 1999Shakhnarovich 2005

17Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Boosting

Construct 1D embedding

Similarity is approximated by

Downweight pairs with

Upweight pairs with

BBK 2010; BB Ovsjanikov, Guibas 2010Shakhnarovich 2005

-1

+1

18Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Boosting

BBK 2010; BB Ovsjanikov, Guibas 2010Shakhnarovich 2005

Construct 1D embedding

Similarity is approximated by

Downweight pairs with

Upweight pairs with

-1

+1

-1 -1

+1 -1

+1 +1

-1 +1

19Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

SHREC 2010 dataset

20Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

SHREC 2010 datasetBB et al, 3DOR 2010

Total dataset size: 1K shapes (715 queries)

Positives: 10K

Negatives: 100K

21Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

ShapeGoogle with HKS descriptorBB et al, 3DOR 2010

22Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

ShapeGoogle with SI-HKS descriptorBB et al, 3DOR 2010

23Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Similarity sensitive hashing (96 bit)BB et al, 3DOR 2010

24Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

WaldHash

B2, Ovsjanikov, Guibas 2010

Construct embedding by maximizing

Remove pairs with

Downweight pairs with

Upweight pairs with

Early decision

and sample in new pairs into the training set

positive

negative

25Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

30%

B2, Ovsjanikov, Guibas 2010

26Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Incommensurablespaces!

BB, Michel, Paragios CVPR 2010

Cross-modal similarity

Modality 1 Modality 2

How to compare apples to oranges?

Triangular meshes Point clouds

27Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Modality 1 Modality 2

with high probability

Cross-modality embedding

BB, Michel, Paragios CVPR 2010

28Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Modality 1 Modality 2

Cross-modality hashing

-1 -1 +1 -1-1 -1 -1 +1-1 -1 +1 +1

+1 +1 -1 +1+1 +1 +1 -1+1 -1 -1 +1

with high probability

with low probability

Collision:

BB, Michel, Paragios CVPR 2010

29Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Query

8x8 dimensionalbag of expressions

Database

32-dimensionalbag of words

Cross-representation 3D shape retrieval

1052 shapes

BB, Michel, Paragios CVPR 2010

30Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval

Me

an

av

erag

e p

rec

isio

n

Number of bits

Retrieval performance

BB, Michel, Paragios CVPR 2010