Experiments in Graph-based Semi-Supervised Learning for...
Transcript of Experiments in Graph-based Semi-Supervised Learning for...
![Page 1: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/1.jpg)
Partha Pratim Talukdar* (Search Labs, MSR)
Fernando Pereira (Google)
Experiments in Graph-based Semi-Supervised Learning
for Class-Instance Acquisition
ACL, 2010 (Uppsala, Sweden) *Work done while at the Univ. of Pennsylvania
![Page 2: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/2.jpg)
Class-Instance Acquisition
2
![Page 3: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/3.jpg)
Class-Instance Acquisition
• Given an entity, assign human readable descriptors to it– e.g., Toyota is a car manufacturer, japanese
company, multinational company, ...
2
![Page 4: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/4.jpg)
Class-Instance Acquisition
• Given an entity, assign human readable descriptors to it– e.g., Toyota is a car manufacturer, japanese
company, multinational company, ...
• Large scale, open domain
2
![Page 5: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/5.jpg)
Class-Instance Acquisition
• Given an entity, assign human readable descriptors to it– e.g., Toyota is a car manufacturer, japanese
company, multinational company, ...
• Large scale, open domain
• Applications– Web search, Advertising, etc.
2
![Page 6: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/6.jpg)
Graph-based Class-Instance Acquisition
3
![Page 7: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/7.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
PatternTable Mining
3
![Page 8: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/8.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
PatternTable Mining
3
Musician Singer
![Page 9: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/9.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
PatternTable Mining
Cluster ID
3
Musician Singer
![Page 10: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/10.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
PatternTable Mining
Cluster ID
ExtractionConfidence
3
Musician Singer
![Page 11: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/11.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
PatternTable Mining
Cluster ID
ExtractionConfidence
3
Musician Singer
![Page 12: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/12.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
SeedClasses
PatternTable Mining
Cluster ID
ExtractionConfidence
3
Musician Singer
![Page 13: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/13.jpg)
Graph-based Class-Instance Acquisition
Bob Dylan (0.95)Johnny Cash (0.87)
Billy Joel (0.82)
Set 2Set 1Billy Joel (0.75)
Johnny Cash (0.73)
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
SeedClasses
PatternTable Mining
Cluster ID
ExtractionConfidence
3
Musician Singer
Can we infer that Bob Dylan is also a Musician, as that is missing in current
extractions?
![Page 14: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/14.jpg)
Graph-based Class-Instance Acquisition [Talukdar et al., EMNLP 2008]
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
4
![Page 15: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/15.jpg)
Graph-based Class-Instance Acquisition [Talukdar et al., EMNLP 2008]
InitializationSet 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Musician 1.0
Musician 1.0
Musician 1.0
SeedLabels
4
PredictedLabels
![Page 16: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/16.jpg)
Graph-based Class-Instance Acquisition [Talukdar et al., EMNLP 2008]
Iteration 1Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Musician 1.0
Musician 0.8Musician 1.0
Musician 1.0
SeedLabels
4
PredictedLabels
![Page 17: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/17.jpg)
Graph-based Class-Instance Acquisition [Talukdar et al., EMNLP 2008]
Iteration 2Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Musician 1.0
Musician 0.8Musician 1.0
Musician 1.0
SeedLabels
Musician 0.6
4
PredictedLabels
![Page 18: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/18.jpg)
• Graph-based Class-Instance Acquisition– Overview & previous work
• Review & comparison of three SSL algorithms– LP-ZGL (Zhu+, 2003)
– Adsorption (Baluja+, 2008)– Modified Adsorption (MAD) (Talukdar and Crammer, 2009)
• Additional Semantic Constraints
• Summary
Outline
5
![Page 19: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/19.jpg)
• Graph-based Class-Instance Acquisition– Overview & previous work
• Review & comparison of three SSL algorithms– LP-ZGL (Zhu+, 2003)
– Adsorption (Baluja+, 2008)– Modified Adsorption (MAD) (Talukdar and Crammer, 2009)
• Additional Semantic Constraints
• Summary
Outline
5
![Page 20: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/20.jpg)
Comments on the Constructed Graph
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
6
![Page 21: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/21.jpg)
Comments on the Constructed Graph
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Smoothness:Nodes connected by an edge should be assigned similar classes, as enforced by edge weight.
6
![Page 22: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/22.jpg)
Comments on the Constructed Graph
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Smoothness:Nodes connected by an edge should be assigned similar classes, as enforced by edge weight.
Initial Clusters
6
![Page 23: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/23.jpg)
Comments on the Constructed Graph
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Smoothness:Nodes connected by an edge should be assigned similar classes, as enforced by edge weight.
Initial Clusters
6
Coupling Node:Force (softly) all instance nodes connected to it to have similar class labels, exploiting the Smoothness requirement.
![Page 24: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/24.jpg)
Comments on the Constructed Graph
Set 2
Set 1
Bob Dylan
Johnny Cash
Billy Joel
0.95
0.87
0.82
0.73
0.75
Musician 1.0
Musician 1.0
Smoothness:Nodes connected by an edge should be assigned similar classes, as enforced by edge weight.
Initial Clusters
Seed classes can be different from the cluster IDs
6
Coupling Node:Force (softly) all instance nodes connected to it to have similar class labels, exploiting the Smoothness requirement.
![Page 25: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/25.jpg)
LP-ZGL[Zhu et al., ICML 2003]
![Page 26: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/26.jpg)
• m labels
• W : edge weight matrix
• Yul: weight of label l on node u
• Yul: seed weight of label l on node u
• S: diagonal matrix, non-zero for seed nodes
LP-ZGL[Zhu et al., ICML 2003]
![Page 27: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/27.jpg)
• m labels
• W : edge weight matrix
• Yul: weight of label l on node u
• Yul: seed weight of label l on node u
• S: diagonal matrix, non-zero for seed nodes
LP-ZGL[Zhu et al., ICML 2003]
arg minY
m!
l=1
!
u,v
Wuv(Yul ! Yvl)2; s.t. SYl = SYl
![Page 28: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/28.jpg)
• m labels
• W : edge weight matrix
• Yul: weight of label l on node u
• Yul: seed weight of label l on node u
• S: diagonal matrix, non-zero for seed nodes
LP-ZGL[Zhu et al., ICML 2003]
arg minY
m!
l=1
!
u,v
Wuv(Yul ! Yvl)2; s.t. SYl = SYl
Smooth
![Page 29: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/29.jpg)
• m labels
• W : edge weight matrix
• Yul: weight of label l on node u
• Yul: seed weight of label l on node u
• S: diagonal matrix, non-zero for seed nodes
LP-ZGL[Zhu et al., ICML 2003]
arg minY
m!
l=1
!
u,v
Wuv(Yul ! Yvl)2; s.t. SYl = SYl
Match Seeds (hard)Smooth
![Page 30: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/30.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
![Page 31: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/31.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
argminY
m+1�
l=1
��SY l − SY l�2 + µ1
�
u,v
Muv(Y ul − Y vl)2 + µ2�Y l −Rl�2
�
• m labels, +1 dummy label
• M = W� +W is the symmetrized weight matrix
• Y vl: weight of label l on node v
• Y vl: seed weight for label l on node v
• S: diagonal matrix, nonzero for seed nodes
• Rvl: regularization target for label l on node v
![Page 32: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/32.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
argminY
m+1�
l=1
��SY l − SY l�2 + µ1
�
u,v
Muv(Y ul − Y vl)2 + µ2�Y l −Rl�2
�
• m labels, +1 dummy label
• M = W� +W is the symmetrized weight matrix
• Y vl: weight of label l on node v
• Y vl: seed weight for label l on node v
• S: diagonal matrix, nonzero for seed nodes
• Rvl: regularization target for label l on node v
Match Seeds (soft)
![Page 33: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/33.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
argminY
m+1�
l=1
��SY l − SY l�2 + µ1
�
u,v
Muv(Y ul − Y vl)2 + µ2�Y l −Rl�2
�
• m labels, +1 dummy label
• M = W� +W is the symmetrized weight matrix
• Y vl: weight of label l on node v
• Y vl: seed weight for label l on node v
• S: diagonal matrix, nonzero for seed nodes
• Rvl: regularization target for label l on node v
Match Seeds (soft) Smooth
![Page 34: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/34.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
argminY
m+1�
l=1
��SY l − SY l�2 + µ1
�
u,v
Muv(Y ul − Y vl)2 + µ2�Y l −Rl�2
�
• m labels, +1 dummy label
• M = W� +W is the symmetrized weight matrix
• Y vl: weight of label l on node v
• Y vl: seed weight for label l on node v
• S: diagonal matrix, nonzero for seed nodes
• Rvl: regularization target for label l on node v
Match Seeds (soft) SmoothMatch Priors(Regularizer)
![Page 35: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/35.jpg)
Modified Adsorption (MAD)[Talukdar and Crammer, ECML 2009]
8
argminY
m+1�
l=1
��SY l − SY l�2 + µ1
�
u,v
Muv(Y ul − Y vl)2 + µ2�Y l −Rl�2
�
• m labels, +1 dummy label
• M = W� +W is the symmetrized weight matrix
• Y vl: weight of label l on node v
• Y vl: seed weight for label l on node v
• S: diagonal matrix, nonzero for seed nodes
• Rvl: regularization target for label l on node v
MAD has extra regularization compared
to LP-ZGL
Match Seeds (soft) SmoothMatch Priors(Regularizer)
![Page 36: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/36.jpg)
MAD (contd.)
9
Inputs Y ,R : |V |× (|L|+ 1), W : |V |× |V |, S : |V |× |V | diagonalY ← YM = W +W�
Zv ← Svv + µ1�
u �=v Mvu + µ2 ∀v ∈ Vrepeat
for all v ∈ V doY v ← 1
Zv
�(SY )v + µ1Mv·Y + µ2Rv
�
end foruntil convergence
![Page 37: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/37.jpg)
MAD (contd.)
9
Inputs Y ,R : |V |× (|L|+ 1), W : |V |× |V |, S : |V |× |V | diagonalY ← YM = W +W�
Zv ← Svv + µ1�
u �=v Mvu + µ2 ∀v ∈ Vrepeat
for all v ∈ V doY v ← 1
Zv
�(SY )v + µ1Mv·Y + µ2Rv
�
end foruntil convergence
• Easily Parallelizable: Scalable
![Page 38: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/38.jpg)
MAD (contd.)
9
Inputs Y ,R : |V |× (|L|+ 1), W : |V |× |V |, S : |V |× |V | diagonalY ← YM = W +W�
Zv ← Svv + µ1�
u �=v Mvu + µ2 ∀v ∈ Vrepeat
for all v ∈ V doY v ← 1
Zv
�(SY )v + µ1Mv·Y + µ2Rv
�
end foruntil convergence
• Easily Parallelizable: Scalable• Importance of a node can be discounted
![Page 39: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/39.jpg)
MAD (contd.)
9
Inputs Y ,R : |V |× (|L|+ 1), W : |V |× |V |, S : |V |× |V | diagonalY ← YM = W +W�
Zv ← Svv + µ1�
u �=v Mvu + µ2 ∀v ∈ Vrepeat
for all v ∈ V doY v ← 1
Zv
�(SY )v + µ1Mv·Y + µ2Rv
�
end foruntil convergence
• Easily Parallelizable: Scalable• Importance of a node can be discounted• Convergence guarantee under mild conditions
![Page 40: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/40.jpg)
Experiments with Public Datasets
10
![Page 41: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/41.jpg)
Experiments with Public Datasets
• Previous research used proprietary datasets– difficult to reproduce and extend
10
![Page 42: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/42.jpg)
Experiments with Public Datasets
• Previous research used proprietary datasets– difficult to reproduce and extend
• Public datasets are used in the paper– Freebase: relational tables from multiple sources
– TextRunner (Ritter+, 2009): Open-domain IE System
– YAGO (Suchanek+, 2007): KB curated from Wikipedia and WordNet
– Gold standard hypernyms: [Pantel+, 2009], WordNet
10
![Page 43: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/43.jpg)
Experiments with Public Datasets
• Previous research used proprietary datasets– difficult to reproduce and extend
• Public datasets are used in the paper– Freebase: relational tables from multiple sources
– TextRunner (Ritter+, 2009): Open-domain IE System
– YAGO (Suchanek+, 2007): KB curated from Wikipedia and WordNet
– Gold standard hypernyms: [Pantel+, 2009], WordNet
• Available at: http://www.talukdar.net/datasets/class_inst/10
![Page 44: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/44.jpg)
Graph Stats
11
Statistics of Graphs used in Experiments
![Page 45: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/45.jpg)
Evaluation Metric: Mean Reciprocal Rank (MRR)
12
MRR =1
|test-set|�
v∈test-set
1
rankv(class(v))
![Page 46: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/46.jpg)
Evaluation Metric: Mean Reciprocal Rank (MRR)
12
MRR =1
|test-set|�
v∈test-set
1
rankv(class(v))
Billy JoelLinguist 0.4Musician 0.3
...
Gold Label: MusicianMRR: 0.5 (1/2)
![Page 47: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/47.jpg)
Graph-based SSL Comparisons (1)
13
![Page 48: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/48.jpg)
Graph-based SSL Comparisons (1)
13
0.15
0.2
0.25
0.3
0.35
170 x 2 170 x 10
TextRunner Graph, 170 WordNet Classes
Me
an
Re
cip
roca
l R
an
k (
MR
R)
Amount of Supervision
LP-ZGL Adsorption MAD
Graph with175k nodes,529k edges.
![Page 49: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/49.jpg)
Graph-based SSL Comparisons (2)
14
0.25
0.285
0.32
0.355
0.39
192 x 2 192 x 10
Freebase-2 Graph, 192 WordNet Classes
Me
an
Re
cip
roca
l R
an
k (
MR
R)
Amount of Supervision
LP-ZGL Adsorption MAD
Graph with303k nodes,2.3m edges.
![Page 50: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/50.jpg)
When is MAD most effective?
15
![Page 51: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/51.jpg)
When is MAD most effective?
15
0
0.1
0.2
0.3
0.4
0 7.5 15 22.5 30
Re
lati
ve I
ncr
eas
e i
n M
RR
by M
AD
ove
r L
P-Z
GL
Average Degree
![Page 52: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/52.jpg)
When is MAD most effective?
15
0
0.1
0.2
0.3
0.4
0 7.5 15 22.5 30
Re
lati
ve I
ncr
eas
e i
n M
RR
by M
AD
ove
r L
P-Z
GL
Average Degree
MAD is most effective in dense graphs, where there is greater need for
regularization.
![Page 53: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/53.jpg)
• Graph-based Class-Instance Acquisition– Overview & previous work
• Review & comparison of three SSL algorithms– LP-ZGL (Zhu+, 2003)
– Adsorption (Baluja+, 2008)– Modified Adsorption (MAD) (Talukdar and Crammer, 2009)
• Additional Semantic Constraints
• Summary
Outline
16
![Page 54: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/54.jpg)
• Graph-based Class-Instance Acquisition– Overview & previous work
• Review & comparison of three SSL algorithms– LP-ZGL (Zhu+, 2003)
– Adsorption (Baluja+, 2008)– Modified Adsorption (MAD) (Talukdar and Crammer, 2009)
• Additional Semantic Constraints
• Summary
Outline
16
![Page 55: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/55.jpg)
Can Additional Semantic Constraints Help ?
17
![Page 56: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/56.jpg)
Can Additional Semantic Constraints Help ?
17
Set 2
Set 1
Isaac Newton
Johnny Cash
Bob Dylan
![Page 57: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/57.jpg)
Can Additional Semantic Constraints Help ?
17
Instances with shared
attributes are likely to be
from the same class.
Set 2
Set 1
Isaac Newton
Johnny Cash
Bob Dylan
![Page 58: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/58.jpg)
has_attribute-albums
Can Additional Semantic Constraints Help ?
17
Instances with shared
attributes are likely to be
from the same class.
Set 2
Set 1
Isaac Newton
Johnny Cash
Bob Dylan
![Page 59: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/59.jpg)
has_attribute-albums
Can Additional Semantic Constraints Help ?
17
Instances with shared
attributes are likely to be
from the same class.
Set 2
Set 1
Isaac Newton
Johnny Cash
Bob Dylan
Graph-based representation makes it easy to incorporate such constraints!
![Page 60: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/60.jpg)
Better Classes with YAGO Attributes
18
![Page 61: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/61.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
![Page 62: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/62.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
Graph constructed from TextRunner (UWash) output, 175k nodes, 529k edges
![Page 63: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/63.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
Graph constructed from output of YAGO Knowledge Base, 142k nodes, 777k edges
![Page 64: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/64.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
Combined graph, with 237k nodes, 1.3m edges
![Page 65: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/65.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
![Page 66: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/66.jpg)
Better Classes with YAGO Attributes
18 0.3
0.338
0.375
0.413
0.45
170 WordNet Classes, 10 Seeds per Class, using Adsorption
Me
an
Re
cip
roca
l R
an
k (
MR
R)
TextRunner GraphYAGO GraphTextRunner + YAGO Graph
Additional semantic constraints help improve performance significantly.
![Page 67: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/67.jpg)
Classes for Attributes
• Classes assigned to attribute nodes
19
has_isbn
wordnet_bookwordnet_maganize
![Page 68: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/68.jpg)
Classes for Attributes
• Classes assigned to attribute nodes
19
has_isbn
wordnet_bookwordnet_maganize
![Page 69: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/69.jpg)
Classes for Attributes
• Classes assigned to attribute nodes
19
has_isbn
wordnet_bookwordnet_maganize
Evidence that attributes propagate right classes.
![Page 70: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/70.jpg)
Summary
20
![Page 71: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/71.jpg)
Summary
• Compared three graph-based SSL algorithms for class-instance acquisition– MAD is consistently most effective, particularly in
dense graphs
20
![Page 72: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/72.jpg)
Summary
• Compared three graph-based SSL algorithms for class-instance acquisition– MAD is consistently most effective, particularly in
dense graphs
• Better class-instance acquisition with additional semantic constraints– easy to add such constraints in graph-based
representation
20
![Page 73: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/73.jpg)
Summary
• Compared three graph-based SSL algorithms for class-instance acquisition– MAD is consistently most effective, particularly in
dense graphs
• Better class-instance acquisition with additional semantic constraints– easy to add such constraints in graph-based
representation
• Datasets available: www.talukdar.net/datasets/class_inst/
– for reproducibility and extension
20
![Page 74: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/74.jpg)
![Page 75: Experiments in Graph-based Semi-Supervised Learning for ...talukdar.net/./papers/slides/class_inst_acl10_slides.pdf · Table Pattern Mining Cluster ID Extraction Confidence 3 Musician](https://reader030.fdocuments.us/reader030/viewer/2022040921/5e9a8163d89fb11d545a7d5e/html5/thumbnails/75.jpg)
Thank You!http://www.talukdar.net/datasets/class_inst/