Geometric Registration for Deformable...
Transcript of Geometric Registration for Deformable...
Geometric Registration for Deformable Shapes
3.3 Advanced Global MatchingCorrelated Correspondences [ASP*04]
A Complete Registration System [HAW*08]
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
In this session
Advanced Global Matching• Some practical applications of the optimization presented
in the last session• Correlated Correspondences [ASP*04]: Applies MRF
model• A Complete Registration System [HAW*08]: Applies
Spectral matching to filter correspondences
2
Eurographics 2010 Course – Geometric Registration for Deformable Shapes 3
Correlated correspondences• Correspondence between data and model meshes• Model mesh is a template; i.e. data is a subset of model
• Not a registration method; just computes corresponding points between data/model meshes Non-rigid ICP [Hanhel et al. 2003] (using the outputted
correspondences) used to actually generate the registration results seen in the paper
Template (Model) Data Result
=
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Basic approach
A joint probability model represents preferred correspondences• Define a “probability” of each correspondence set
between data/model meshes• Find the correspondence with the highest probability
using Loopy Belief Propagation (LBP) [Yedidia et al. 2003]
2 main components (next parts of the talk)• Probability model• Optimization
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Joint Probability Model
Compatibility constraints• Involves pair of correspondences• Represents prior knowledge of which correspondence sets
makes senseA. Minimize the amount of deformation induced by the correspondencesB. Preserve the geodesic distances in model and data
Singleton constraints• Involves a single correspondenceC. Corresponding points have same feature descriptor values
5
∏∏=k
kklk
lkkl cccZ
cP )(),(1})({,
ψψ
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Penalize unnatural deformations• Edges lengths should stay the same
Compatibility 1: Deformation potential
6
ix
jx
In model mesh
ijl
ijij ll ′≈
kz
lz
Corresponding pointsin data mesh
ijl′
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Penalize unnatural deformations• Edges should twist little as possible• Is the direction from to in ’s coord system
Compatibility 1: Deformation potential
7
ix
jxjid →
jid → ix ixjx
ijd →
ijijjiji dddd →→→→ ′≈′≈ ,
In model mesh
jid →′
ijd →′
Corresponding pointsin data mesh
kz
lz
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Encoding the preference
• Zero-mean Gaussian noise model for length and twists• Define potential for each edge in the data mesh
are “correspondence variables” indicating what is the corresponding point in the model mesh for respectively
• Caveat: additional rotation needed to measure twist For each possibility of precompute aligning rotation
matrices via rigid ICP on surrounding local patch Expand corresp. variables to be site/rotation pairs
)|()|()|(),( ijijjijiijijlkd ddGddGllGjcic →→→→ ′′′===ψ
dψ ),( lk zz),( lk cc
lk zz ,
ick =
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Penalize large changes in geodesic distance• Geodesically nearby points should stay nearby
Enforced for each edge in the data mesh
Compatibility 2: Geodesic distance potential
9
ix
jx
Corresponding pointsin model mesh
),( jiGeodesic xxdistkz
lz
Adjacent pointsin data mesh
If > 3.5p prob assigned 0otherwise prob assigned 1
pzzdist lkGeodesic ≈),(
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Penalize large changes in geodesic distance• Geodesically far points should stay far away
Enforced for each pair of points in the data mesh whose geodesic distance is > 5p
10
ix
jx
Corresponding pointsin model mesh
),( jiGeodesic xxdistkz
lz
Adjacent pointsin data mesh
If < 2p prob assigned 0otherwise prob assigned 1
pzzdist lkGeodesic 5),( >
Compatibility 2: Geodesic distance potential
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Singletons: Local surface signature potential
Spin images gives matching score for each individual correspondence
• Compute spin images & compress using PCA gives surface signature at each point
• Discrepancy between (data) and (model)• Zero-mean Gaussian noise model
11
ixixs
kzsixs
ixkz
Compare Spin Images
kzs ixs
Model meshData mesh
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Model summary
Get Pairwise Markov Random Field (MRF)• Pointwise potential for each pt in data• Pairwise potential for each edge in data
Far geodesic potentials for each pair of points > 5p apart
Model meshData mesh
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Quick intro: Markov Random Fields
Joint probability function visualized by a graph• Prob. = Product of the potentials at all edges
13
“Observed” nodes
“Hidden” nodes
),( lkkl ccψ)( kk cψ (ex) Surface signature potential
(ex) Deformation, geodesic distance potential
∏∏=k
kklk
lkkl cccZ
cP )(),(1})({,
ψψ
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Loopy Belief Propagation (LBP)
Compute marginal probability for each variable• Pick variable value that maximizes the marginal prob.
Usual way to compute marginal probabilities (tabulate and sum up) takes exponential time• BP is a dynamic programming approach to efficiently
compute marginal probabilities• Exact for tree MRFs, approximate for general MRFs
14
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Basic idea• Marginals at node proportional to product of pointwise
potential and incoming messages
Loopy Belief Propagation (LBP)
15
ix
ccbc
ac
dc
kam →
kbm →
kcm →
kdm →
∏∈
→=)(
)()()(kNl
kklkkkk cmckcb φ
kc
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Basic idea• Compute these messages (at each edge) and we are done
Loopy Belief Propagation (LBP)
16
∑lcofvaluesall
lkklll ccc ),()( ψφlc
jx
∏∑∈
→→ ←klNq
llqcofvaluesall
lkklllkkl cmccccml \)(
)(),()()( ψφ
klm →
kc
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Basic idea• Compute these messages (at each edge) and we are done
Loopy Belief Propagation (LBP)
17
cc
bc
ac
dc
lam →
lbm →
lcm → ldm →
∑lcofvaluesall
lkklll ccc ),()( ψφlc
jx
∏∑∈
→→ ←klNq
llqcofvaluesall
lkklllkkl cmccccml \)(
)(),()()( ψφ
klm →
kc
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Basic idea• Compute these messages (at each edge) and we are done
• Recursive formulation• Start at ends and work your way towards the rest
Loopy Belief Propagation (LBP)
18
cc
bc
ac
dc
lam →
lbm →
lcm → ldm →
∑lcofvaluesall
lkklll ccc ),()( ψφlc
jx
∏∑∈
→→ ←klNq
llqcofvaluesall
lkklllkkl cmccccml \)(
)(),()()( ψφ
klm →
kc
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Loops: iterate until messages converge• Start with initial values (ex: )• Apply message update rule until convergence
• Convergence not guaranteed, but works well in practice
Loopy Belief Propagation (LBP)
19
∑acofvaluesall
baabaa ccc ),()( ψφ
ccbc
ac
bam →
abm → acm →cam →
bcm →
cbm →
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Results & Applications
• Efficient, coarse-to-fine implementation• Xeon 2.4 GHz CPU, 1.5 mins for arm, 10 mins for puppet
Correspondences onhuman body models
Finding articulated parts
Interpolation between poses
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Next topic: HAW*08
An application to the spectral matching method of last session• A good illustration of how a matching method fits into a
real registration pipeline
A pairwise method• Deform the source shape to match the target shape
Gray = sourceYellow = target
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Overview
Performs both correspondence and deformation
• Correspondences based on improving closest points• After finding correspondences, deform to move shapes
closer together• Re-take correspondences from the deformed position• Deform again, and repeat until convergence
22
Correspondence Deformation
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Overview
Performs both correspondence and deformation
23
5 basic steps1.Closest points2.Improve by feature matching3.Filter by spectral matching4.Expand sparse set5.Fine-tune target locations
Correspondence Deformation
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Overview
Performs both correspondence and deformation
24
2 basic steps1.Fit per-cluster rigid transformation2.Sparse least-squares solve for deformed positions
Occasional step: Increase cluster size
Correspondence Deformation
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Detailed Overview
Sampling• Whole process works with reduced sample set
Correspondence & Deformation• Examine each step in more detail
Discussion• Discuss pros/cons
25
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Sample for robustness & efficiency
Coarse to fine approach• Use uniform subsampling of the surface and its normals• Improve efficiency, can improve robustness to local
minima
Let’s make it more concrete• Sample set denoted • In correspondence: for each , find corresponding target
points• In deformation: given , find deformed sample positions
that match while preserving local shape detail
26
is
is′
itis
itit
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #1
Find closest points• For each source sample, find
the closest target sample s = sample point on source t = sample point on target
• Usually pretty bad
Target (yellow) Source (gray)
27
2
ˆminarg ts
Tt
−∈
Closest point correspondences
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #2
Improve by feature matching• Search target’s neighbors to
see if there’s better feature match, replace target Let f(s) be feature value of s
• Iterate until we stop moving• If we move too much, discard
correspondence• Much better, but still outliers
Target (yellow) Source (gray)
28
2
)(
)()(minarg tfsfttNt
′−←∈′
Feature-matched correspondences
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #3
Filter by spectral matching• (First some preprocessing)• Construct k-nn graph on both
src & tgt sample set (k = 15)• Length of shortest path on
graph gives approx. geodesic distances on src & tgt
• Goal is to filter these ----------and keep a subset which is geodesically consistent
29
Target (yellow) Source (gray)
Feature-matched correspondences
),(),( jigjig ttdssd
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #3
Filter by spectral matching• Construct affinity matrix M
using these shortest path distances
• Consistency term & matrix
Threshold c0 = 0.7 gives how much error in consistency we are willing to accept
30
1},),(),(
,),(),(
min{ == iijig
jig
jig
jigij c
ssdttd
ttdssd
c
Target (yellow) Source (gray)
Feature-matched correspondences
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #3
Filter by spectral matching• Apply spectral matching: find
eigenvector with largest eigenvalue score for each correspondence
• Iteratively add corresp. with largest score while consistency with the rest is above c_0
• Gives kernel correspondences
• Filtered matches usually sparse
Target (yellow) Source (gray)
31
Filtered correspondences
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #4
Expand sparse set• Lots of samples have no target
position• For these, find best target
position that respects geodesic distances to kernel set
Target (yellow) Source (gray)
32
Expanded correspondences
2
( , )( , ) ( , ) ( , )
k k
K g k g kK
e d d∈
= − ∑s t
s t s s t t
( , )arg min ( , )
g j
i K iN T
e∈
=t t
t s t
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #4
Expand sparse set• Lots of samples have no target
position• Compute confidence weight
based only how well it respects geodesic distances to kernel set
Target (yellow) Source (gray)
33
Expanded correspondences
Red = not consistent ---Blue = very consistent
( , )exp( )2
K i ii
ewe
= −s t
( , )
1 ( , )k k
K k kK
e eK ∈
= ∑s t
s t
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence Step #5
Fine-tuning• So far, target points restricted
to be points in target samples• Not accurate when shapes are
close together• Relax this restriction and let
target points become any point in the original point cloud
• Replace target sample with a closer neighbor in the original point cloud
Target (yellow) Source (gray)
34
Expanded correspondences
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Deformation
Solved by energy minimization (least squares)• Last step gave target positions• Now find deformed sample positions that match
target positions
Two basic criteria: • Match correspondences: should be close to • Shape should preserve detail (as-rigid-as-possible)• Combine to give energy term:
35
corr corr rigid rigidE E Eλ λ= +
is′
is
it
it
it
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Correspondence matching term
Combination of point-to-point (α=0.6) and point-to-plane (β=0.4) metrics• Weighted by confidence weight wi of the target position
36
2' ' 2(( ) )i
Tcorr i i i i i i
S
E w α β∈
= − + − ∑s
s t s t n
Point-to-point Point-to-plane
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Shape preservation term
Deformed positions should preserve shape detail• Form an extended cluster for each sample point: the
sample itself and its neighbors• For each find the rigid transformation (R,T) from
sample positions to their deformed locations
• When solving for , constrain them to move rigidly according to each cluster that it’s associated with
37
kC~
2'
i k
k k i k is C
E∈
= + −∑ R s T s
kC~
is′
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Clusters for local rigidity
• Initially each cluster contains a single sample point• Every 10 iterations (of correspondence & deformation),
combine clusters that have similar rigid transformations (forming larger rigid parts)
38
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Advantages of features & clustering
39
Source + Target Without Features Without Clustering With Both
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Results
Efficient, robust method
42
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Conclusion
Correlated correspondence• Robust method for matching correspondences• Measure how much the correspondence “makes sense”• Probability model optimized using LBP• Requires a template
If model is incomplete, then there is no “correct” corresponding point to assign
Eurographics 2010 Course – Geometric Registration for Deformable Shapes
Conclusion
Non-rigid registration under isometric deformations• Improve closest point correspondences using features and
spectral matching• Deform shape while preserving local rigidity of clusters• Iteratively estimate correspondences and deformation
until convergence• Robust, efficient method• Relies on geodesic distances (problematic when holes are
too large)
44