Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG...
-
Upload
alvin-wright -
Category
Documents
-
view
214 -
download
0
Transcript of Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG...
1
Graph-based Deformable Matching of 3D Line Segments with
Application in Protein Fitting
HANG DOU1, MATTHEW L BAKER2, TAO JU1
Washington University in St. Louis1
Baylor College of Medicine2
2
Deformation
Applications of Feature Matching
Data alignment
Co-segmentationReconstruction
3
Our Application• Protein fitting• Protein: building blocks of life forms.
…
4
Protein
Atomic structure Low-res structure
5
Protein Fitting
Atomic structure (Cartoon Rendering)
Low-res structure (Volume)
Fitting result
6
Protein Fitting• Secondary structure (alpha helix)
7
Protein Fitting• Secondary structure (alpha helix)
8
Protein Fitting• Secondary structure (alpha helix)
9
Protein Fitting• Our method• Find feature (helix) correspondence
• Deform the protein guided by the helices correspondence
10
Protein Fitting• Our method• Find feature (helix) correspondence
• Deform the protein guided by the helices correspondence
11
Problem Statement
• Input:• Undirected line segments• |Source| <= |Target|
• Output:• One to one correspondence
• Matching Criteria:• Similar length• As rigid as possible deformation Source Target
12
Problem Statement
• Input:• Undirected line segments• |Source| <= |Target|
• Output:• One to one correspondence
• Matching Criteria:• Similar length• As rigid as possible deformation Source Target
13
Problem Statement
• Input:• Undirected line segments• |Source| <= |Target|
• Output:• One to one correspondence
• Matching Criteria:• Similar length• As rigid as possible deformation Source Target
14
Previous Work• Matching points: Lots of work• [Fitzgibbon 2003], [Angulov 2004], [Leordeanu 2005], [Chang 2009], [Chertok 2010], [Duchenne 2011], [TAM 2014], ….
15
Previous Work• Matching line segments: very few work• [Abeysinghe 2010]
TargetSource
16
Previous Work• Matching line segments: very few work• [Abeysinghe 2010]• Need line segments forming rigid body clusters
TargetSource
17
Previous Work• Matching line segments: very few work• [Abeysinghe 2010]• Need line segments forming rigid body clusters• Based on a clique-finding algorithm
TargetSource Matching Result
18
Our Work• A novel method for matching 3D line segments• Allowing fully non-rigid deformations
• Technical contributions• Formulating the problem as graph matching• Improved graph matching using continuous relaxation
19
Our Work• A novel method for matching 3D line segments• Allowing fully non-rigid deformations
• Technical contributions• Formulating the problem as graph matching• Improved graph matching using continuous relaxation
20
Background: Graph Matching• Input• The two graphs: source and target• Affinity score (between node pairs and edge pairs)• Node pair score: fnode(si, tj)• Edge pair score: fedge({si, sj}, {tk, tl})
• Goal• Find mapping π: S -> T that maximize:
• Subject to mapping constraints
Source Target
s2
s1
t2
t1
fnode
fedge
21
Our Graph Construction• Naïve method• Graph nodes:• One undirected segment one node
𝑠1
𝑠2𝑠3
Source
𝑡1
𝑡 2 𝑡 3
𝑡 4
Target
s3s2
s1
t2
t1
t4
t3
22
Our Graph Construction• Naïve method• Graph nodes:• One undirected segment one node
• Node pair affinity• Length similarity
𝑠1
𝑠2𝑠3
Source
𝑡1
𝑡 2 𝑡 3
𝑡 4
Target
s3s2
s1
t2
t1
t4
t3
23
Our Graph Construction• Naïve method• Graph nodes:• One undirected segment one node
• Node pair affinity• Length similarity
𝑠1
𝑠2𝑠3
Source
𝑡1
𝑡 2 𝑡 3
𝑡 4
Target
s3s2
s1
t2
t1
t4
t3
24
Our Graph Construction• Naïve method• Graph nodes:• One segment one node
• Node pair affinity• Length similarity
• Edge pair affinity• Rigidity in deformation
𝑠1
𝑠2𝑠3
Source
𝑡1
𝑡 2 𝑡 3
𝑡 4
Target
s3s2
s1
t2
t1
t4
t3
25
Our Graph Construction• Naïve method• Graph nodes:• One undirected segment one node
• Node pair affinity• Length similarity
• Edge pair affinity• Rigidity in deformation
𝑠1
𝑠2𝑠3
Source
𝑡1
𝑡 2 𝑡 3
𝑡 4
Target
s3s2
s1
t2
t1
t4
t3
26
Our Graph Construction• Issue of ambiguity:
Matching Source Matching Result 1
𝑠1
𝑠2𝑠3
Matching Result 2
𝑠1
𝑠2𝑠3
𝑠1
𝑠2𝑠3
𝑠4 𝑠4
27
Our Graph Construction• Issue of ambiguity:
Matching Source Matching Result 1
𝑠1
𝑠2𝑠3
Matching Result 2
𝑠1
𝑠2𝑠3
𝑠1
𝑠2𝑠3
𝑠4 𝑠4
28
Our Graph Construction• Issue of ambiguity:
Matching Source Matching Result 1
𝑠1
𝑠2𝑠3
Matching Result 2
𝑠1
𝑠2𝑠3
𝑠1
𝑠2𝑠3
𝑠4 𝑠4
29
Our Graph Construction• Issue of ambiguity:
Matching Source Matching Result 1
𝑠1
𝑠2𝑠3
Matching Result 2
𝑠1
𝑠2𝑠3
𝑠1
𝑠2𝑠3
𝑠4 𝑠4
30
Our Graph Construction• Improved graph construction• Graph nodes:• Source• One undirected segment one node
• Target• One undirected segment two nodes
𝑠1
𝑠2 𝑠3
Source Target
𝑡 3
𝑡1
𝑡 2𝑡 2𝑡 3
𝑡1𝑡 4
𝑡 4
S3S2
S1
T1
T2
T3
T4
31
Our Graph Construction• Improved graph construction• Graph nodes:• Source• One undirected segment one node
• Target• One undirected segment two nodes
• Node Affinity• Length similarity
𝑠1
𝑠2 𝑠3
Source Target
𝑡 3
𝑡1
𝑡 2𝑡 2𝑡 3
𝑡1𝑡 4
𝑡 4
S3
S1
S2
T1
T3
T4
T2
32
Our Graph Construction• Improved graph construction• Graph nodes:• Source• One undirected segment one node
• Target• One undirected segment two nodes
• Node Affinity• Length similarity
• Edge Affinity• Rigidity in deformation
𝑠1
𝑠2 𝑠3
Source Target
𝑡 3
𝑡1
𝑡 2𝑡 2𝑡 3
𝑡1𝑡 4
𝑡 4
S3S2
S1
T1
T2
T3
T4
33
Our Graph Construction• Improved graph construction• Graph nodes:• Source• One undirected segment one node
• Target• One undirected segment two nodes
• Node Affinity• Length similarity
• Edge Affinity• Rigidity in deformation
𝑠1
𝑠2 𝑠3
Source Target
𝑡 3
𝑡1
𝑡 2𝑡 2𝑡 3
𝑡1𝑡 4
𝑡 4
S3S2
S1
T1
T2
T3
T4
34
Rigidity Measure• Geometric relation descriptor• Encode each pair of line segments (s, s’) as a vector: (, , )• Rigidity between two pairs are differences in the vector
𝛼𝛽
𝜃
𝑑𝑖𝑠𝑡 (𝐬 ,𝐬 ′)𝐬 𝐬 ′
35
Our Graph Construction• No more ambiguity
𝑠1
𝑠2 𝑠3
Matching Source Matching Result
𝑡1
𝑡 2𝑡 2𝑡 3
𝑡1𝑡 4
𝑡 4
𝑡 3
36
Our Work• A novel method for matching 3D line segments• Allowing fully non-rigid deformations
• Technical contributions• Formulating the problem as graph matching• Improved graph matching using continuous relaxation
37
Our Work• A novel method for matching 3D line segments• Allowing fully non-rigid deformations
• Technical contributions• Formulating the problem as graph matching• Improved graph matching using continuous relaxation
38
Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M
s1,t1
s1,t1
Affinity Matrix (M)
s3,t3
s3,t3…. ….
….
…. s3s2
s1
t2
t1
t4
t3
Source Target
39
Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M
s1,t1
s1,t1
Affinity Matrix (M)
s3,t3
s3,t3…. ….
….
…. s3s2
s1
t2
t1
t4
t3
Source Target
40
Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M• Matching goal:
s3s2
s1
t2
t1
t4
t3
Source Target
s1,t1
s1,t1
Affinity Matrix (M)
s3,t3
s3,t3…. ….
….
….
41
Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M• Matching goal:• Find assignment vector x {0,1}|S|x|T|
• Maximize: xT M x (equation 1)• Subject to mapping constraints. Mapping constraint
Integer constraint
42
Solve Graph Matching (review)• Solve quadratic assignment• Combinatorial optimization• Usually time consuming.
• Continuous relaxation• Compute confidence vector X’ R|S|x|T| without some or all the constraints • X’(a) is the confident of the match a (Si -> Tj)
• Binarize X’ to obtain assignment vector X {0,1}|S|x|T| with all constraints• X(a) is 1 if a is an accepted match and 0 otherwise.• Widely used greedy approach [Leordeanu 2005]: sequentially pick matches with descending
confidence, while avoiding conflict based on mapping constraints.
43
Binarize Confidence Vector (Review)
TargetSource
0.4
0
Confidence Vector
44
Binarize Confidence Vector (Review)
0.4
0
Confidence VectorTargetSource
45
Binarize Confidence Vector (Review)
0.66
0
TargetSource
0.4
0
Confidence Vector
46
Binarize Confidence Vector (Review)
0.66
0
TargetSource
0.4
0
Confidence Vector
47
Binarize Continuous Solution (Review)
0.66
0
TargetSource
0.4
0
Confidence Vector
48
Binarize Confidence Vector (Review)
Correct Matches
TargetSource
0.66
0
0.4
0
Confidence Vector
49
Our Continuous Relaxation• Key idea• Only takes the assignments with high confidence• Formulate a smaller matching problem, constrained by the chosen
assignments
• Iterate until we find all the matches• Compute confidence vector (with any chosen continuous method)• Pick matches whose relative confidence (w.r.t. the highest confidence)
value is above a threshold r• Reconstruct the (smaller) affinity matrix with remaining matches• Edge affinity of already picked segments is added to the diagonal of the new matrix
50
0.66
0
Confidence vector in 1st iteration
Our Continuous Relaxation
TargetSource
r = 1
51
Our Continuous Relaxation
0.66
0
r = 1
Confidence vector in 1st iteration
TargetSource
52
Our Continuous Relaxation
0.95
0
r = 1
Confidence vector in 2nd iteration
TargetSource
53
Our Continuous Relaxation
0.95
0
r = 1
Confidence vector in 2nd iteration
TargetSource
54
Our Continuous Relaxation
0.95
0
r = 1
Confidence vector in 3rd iteration
TargetSource
55
Our Continuous Relaxation
0.95
0
r = 1
Confidence vector in 3rd iteration
TargetSource
56
Our Continuous Relaxation
0.71
0
r = 1
Confidence vector in 4th iteration
TargetSource
57
Our Continuous Relaxation
0.71
0
r = 1
Confidence vector in 4th iteration
TargetSource
58
Our Continuous Relaxation
r = 1
Confidence vector in 5th iteration
TargetSource
0.99
0
59
Our Continuous Relaxation
0.99
0
r = 1
Confidence vector in 5th iteration
TargetSource
60
0.99
0
Our Continuous Relaxation
r = 1
Confidence vector in 6th iteration
TargetSource
61
0.99
0
Our Continuous Relaxation
r = 1
Confidence vector in 6th iteration
TargetSource
62
Our Continuous Relaxation
Correct Matching Result
TargetSource
63
Experiment Result• Synthetic data and authentic data• Synthetic data: • generated by thin plate spline (TPS) deformation.
• Authentic data: • 16 pairs of proteins from Protein Data Bank.
• Accuracy and function score• Matching accuracy: • ratio of correct matches over all matches.
• Total affinity ratio: • Function score (xT M x) ratio of our method over a benchmark method [Leordeanu 2005].
64
Experiment Result• Synthetic data• Pick one protein as the matching source• Deform the source line segments at different level of normalized bending
energies as matching targets
Source Bend Energy = 0.2 Bend Energy = 0.7
65
Experiment Result• Synthetic data
Leordeanu[2005]Our, r = 0.6
Our, r = 0.3
Our, r = 1.0
Bending Energy
Tota
l Affi
nity
Rati
o
66
Experiment Result• Synthetic data
Bending Energy
Mat
chin
g Ac
cura
cy
Leordeanu[2005] Our, r = 0.3 Our, r = 0.6
Our, r = 1.0 Abeysinghe[2010]
67
Experiment Result• Synthetic data
Bending Energy
Mat
chin
g Ac
cura
cy
Leordeanu[2005] Our, r = 0.3 Our, r = 0.6
Our, r = 1.0 Abeysinghe[2010]
68
Experiment Result• Authentic data:
1sx4-A 1ss8-A
Source Target(Our) Target(Leordeanu) Target(Abeysinghe)
69
Experiment Result• Authentic data:
Protein
Mat
chin
g Ac
cura
cy
Leordeanu Our, r = 1.0, 0.6, 0.3 Abeysinghe
70
Conclusion• An algorithm for finding correspondence between two
sets of 3D line segments• Allowing fully non-rigid deformations
• Technical contributions• Formulating the problem as graph matching• Solve matching problem using a iterative continuous-discrete paradigm
• Validation• Test on both synthetic and authentic data
71
Questions?