Recovering Articulated Object Models from 3D Range Data

Dragomir Anguelov Daphne Koller Hoi-Cheung Pang Praveen Srinivasan

Sebastian Thrun

Computer Science Dept. Stanford University

Problem overview Data:

Articulated object observed in different configurations

3D range surfaces (meshes) obtained by a scanner

Task: Recover the articulated object model1. Register all surfaces2. Cluster the object surface

into rigid parts

3. Recover joints linking the parts

Applications of articulated models

Animation, object detection, scene segmentation

Surface registration Algorithm input: meshes Z0, Z1, … ZN

Pick template mesh X = Z0 Need to recover point-to-point correspondences

between X and instance meshes Zi , i={1, …, N} Any registration algorithm providing can be used

Correlated Correspondences algorithm [Anguelov et al ‘04]

Unsupervised nonrigid registration algorithm Handles large deformations, occlusion (but not clutter)

Algorithm Overview Computes embedding of mesh Z into mesh X

defined as a joint assignment to correspondences Encodes correspondence correlations (minimizing

deformation, preserving geodesic distances) as pairwise potentials in a Markov network over

Uses Loopy-BP to obtain a good joint assignment to all variables

21

3

Clustering into rigid parts:Generative model

Template X

TransformedModel

Instance Z

Transformations

z1

b1

a1

Tx1

y1

aM

xM

yM

zk

bk

…

…

…

…

Part labels

Points

Point labels

Points

j = Part (xj) k = ModelPoint (zk)

Assume values provided by anonrigid registration algorithm

1. Rigid part points share the same rigid transformaition in each instance

rotation translation

2. Each part is a connected region of the template mesh surface

a) Soft contiguity constraints:

If and are connected in X

forb) Hard contiguity constraints:

Each part consists of a single connected component of the surface of X

Clustering into rigid parts: Optimization

Model Summary: Ignoring hard contiguity constraints, generative model

defines a Markov Network over the part labels :

Objective

Iterative Optimization E-step: Given transformations , compute the part labels

maximizing Eq. (1) and satisfying the hard contiguity constraints

M-step: Given part labels , compute transformations

Point in mesh Zi , generated

by template point xj

Clustering into rigid parts: E-Step

The problem of finding a MAP assignment to given which maximizes Eq. (1) can be expressed as an integer program (IP). A relaxation of the IP produces a linear program (LP) [Kleinberg et al ’99].

For 2-part models, the optimal LP solution is also the optimal IP solution.

For n-part models, n>2: In case of a fractional LP solution, a rounding procedure produces an IP

solution within a factor of 2 of the optimal. The LP can be solved very efficiently by using multiway min-

cuts [Kolmogorov et al ’04]

The LP solution may not satisfy hard contiguity constraints. A part consisting of m connected components is split into m

separate parts which doesn’t change the objective in Eq. (1).

Clustering into rigid parts: LP formulation

Introduce variables s.t. Log-cost associated with potential becomes

where

Define separation cost of an edge as where

Putting everything together:

Relax binary constraints to obtain LP

Clustering into rigid parts: M-step and Annealing

M-step: find rigid part transformations given part labels

Optimization function decomposes into a separate equation for

each :

Eq. (2) is isomorphic to the ICP problem [Besl’ 92]. Close form estimate of is obtained by solving a simple LP.

Annealing Number of parts in solution depends on the tradeoff

where large values of result in fewer parts When estimate of is poor and is large, solving E-step LP is

hard, because multiple plausible solutions with similar scores exist

Starting with low values of and gradually increasing performs best in practice

Clustering into Rigid Parts: Initialization

Need good initial values for the transformations

Template X

Method 1: Cluster local rigid transformations [Cheung et al ’03] • Associate a vector of transformations with each point (estimate using the known correspondences for all instance meshes)• Cluster points into k parts using adaptive PCA [Archer et al ’02]• Slow: >10 mins for 2000 point models

Method 2: Break surface into k arbitrary fragments of approximately equal size• Fast: <1s for 2000 point models • Same final results!

a) Rigid parts after initializationb) Rigid parts after EM

a) b)

Optimizing the Joint Positions

A joint is a point belonging to two parts simultaneously: [Cheung et al ’03]

To solve for the joint, must optimize

Sometimes, the solution can be an entire space of points

Example: a joint with one degree of freedom Introduce an additional regularization term,

enforcing that the joint is close to the boundary between two parts in X

a) b)

a) Rigid parts after EMb) Estimated joints

Joint between parts p and q

Results: Puppet Dataset

Correctly identifies all 15 puppet parts and the skeleton

Results: Initialization

Testing EM Initialization Initialize with a different number of parts k If k > 15, the algorithm consistently finds the correct

number of parts (15)

Results: Arm Dataset

Correctly identifies 4 arm parts and the skeleton in the presence of non-rigid deformation

Results: Annealing

Results of annealing with increasing value of Increasing decreases the number of parts (A – D) Increasing too much results in a partition which tries to

reduce the number of links between parts (E)

Conclusions and Future Work

Algorithm for recovering articulate object models from 3D data

Clusters the object surface into rigid parts Recovers the articulated object skeleton

Algorithm properties Probabilistic formulation Efficient global inference

Algorithm recovers complex models consisting of a large number of parts in real world datasets

Future work Extend the algorithm to allow and explicitly model (small)

deformations of the articulated parts

Robustness to local maxima