Respiratory Distress in the Newborn, not RDS Dr. Alona Bin-Nun NICU Shaare Zedek.
A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian...
-
Upload
jasmin-powell -
Category
Documents
-
view
220 -
download
2
Transcript of A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian...
![Page 1: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/1.jpg)
1
A Compositional and Interpretable Semantic Space
Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell
Carnegie Mellon University
![Page 2: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/2.jpg)
2
pear
lettuce
orange
apple
carrots
VSMs and Composition
![Page 3: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/3.jpg)
How to Make a VSM
CountDim.
ReductionCorpus
Statistics
VSM
3
Many cols Few cols
![Page 4: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/4.jpg)
4
pear
lettuce
orange
apple
carrots
seedless orange
VSMs and Composition
![Page 5: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/5.jpg)
5
VSMs and Composition
f( , )
=adjective noun estimate
observed
Stats for seedless Stats for orange
Observed stats for “seedless orange”
![Page 6: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/6.jpg)
6
Previous Work
• What is “f”?(Mitchell & Lapata, 2010; Baroni and Zamparelli, 2010; Blacoe and Lapata, 2012; Socher et al., 2012; Dinu et al., 2013; Hermann & Blunsom, 2013)
• Which VSMs are best for composition?(Turney, 2012, 2013; Fyshe et al., 2013; Baroni et al., 2014)
![Page 7: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/7.jpg)
7
Our Contributions
• Can we learn a VSM that – is aware of composition function?– is interpretable?
FFIs
edib
le
![Page 8: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/8.jpg)
How to make a VSM
• Corpus– 16 billion words– 50 million documents
• Count dependencies arcs in sentences• MALT dependency parser
• Point-wise Positive Mutual Information
8
![Page 9: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/9.jpg)
Matrix Factorization in VSMs
X A
D
≈
Corpus Stats (c)
Words
9
VSM
![Page 10: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/10.jpg)
Interpretability
10
A
Latent Dims
Words
![Page 11: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/11.jpg)
Interpretability
11
• SVD (Fyshe 2013)– well, long, if, year, watch – plan, engine, e, rock, very – get, no, features, music, via
• Word2vec (pretrained on Google News)– pleasantries, draft_picks, chairman_Harley_Hotchkiss,
windstorm, Vermont_Yankee– Programme_Producers_AMPTPP, ###/mt, Al_Mehwar, NCWS,
Whereas– Ubiquitous_Sensor_Networks, KTO, discussing,
Hibernia_Terra_Nova, NASDAQ_ENWV
![Page 12: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/12.jpg)
Non-Negative Sparse Embeddings
12
X A
D
≈
(Murphy 2012)
![Page 13: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/13.jpg)
Interpretability
13
• SVD– well, long, if, year, watch – plan, engine, e, rock, very – get, no, features, music, via
• NNSE– inhibitor, inhibitors, antagonists, receptors,
inhibition – bristol, thames, southampton, brighton, poole – delhi, india, bombay, chennai, madras
![Page 14: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/14.jpg)
14
A Composition-aware VSM
![Page 15: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/15.jpg)
15
Modeling Composition
• Rows of X are words– Can also be phrases
X APhrases Phrases
Adjectives
Nouns
Adjectives
Nouns
![Page 16: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/16.jpg)
16
Modeling Composition
• Additional constraint for composition
APhrases
Adjectives w1w2
p
p = [w1 w2]
Nouns
![Page 17: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/17.jpg)
17
Weighted Addition
![Page 18: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/18.jpg)
18
Modeling Composition
![Page 19: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/19.jpg)
19
Modeling Composition
• Reformulate loss with square matrix B
AB
α β -1
adj. col. noun col. phrase col
![Page 20: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/20.jpg)
20
Modeling Composition
![Page 21: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/21.jpg)
Optimization
• Online Dictionary Learning Algorithm(Mairal 2010)
• Solve for D with gradient descent• Solve for A with ADMM– Alternating Direction Method of Multipliers
21
![Page 22: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/22.jpg)
Testing Composition
• W. add
• W. NNSE
• CNNSE
22
A
w1w2
p
SVDw1w2
p
A
w1w2
p
![Page 23: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/23.jpg)
23
Phrase Estimation
• Predict phrase vector• Sort test phrases by distance to estimate
•Rank (r/N*100)•Reciprocal rank (1/r)•Percent Perfect (δ(r==1))
r
N
![Page 24: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/24.jpg)
24
Phrase Estimation
Chance 50 ~ 0.05 1%
![Page 25: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/25.jpg)
25
Interpretable Dimensions
![Page 26: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/26.jpg)
26
Interpretability
![Page 27: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/27.jpg)
Testing Interpretability
• SVD
• NNSE
• CNNSE
27
A
w1w2
p
SVDw1w2
p
A
w1w2
p
![Page 28: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/28.jpg)
28
Interpretability
• Select the word that does not belong:• crunchy• gooey• fluffy• crispy• colt• creamy
![Page 29: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/29.jpg)
29
Interpretability
![Page 30: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/30.jpg)
Phrase Representations
30
A
phrase
top scoringwords/phrases
top scoringdimension
![Page 31: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/31.jpg)
31
Phrase Representations
Choose list of words/phrases most associated with target phrase “digital computers”• aesthetic, American music, architectural style• cellphones, laptops, monitors• both• neither
![Page 32: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/32.jpg)
32
Phrase Representation
![Page 33: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/33.jpg)
Testing Phrase Similarity• 108 adjective-noun phrase pairs
• Human judgments of similarity [1…7]
• E.g. Important part : significant role (very similar)
Northern region : early age (not similar)
33
(Mitchell & Lapata 2010)
![Page 34: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/34.jpg)
Correlation of Distances
34
Behavioral Data
Model A
Model B
![Page 35: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/35.jpg)
Testing Phrase Similarity
35
![Page 36: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/36.jpg)
36
Interpretability
![Page 37: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/37.jpg)
Better than Correlation: Interpretability
37http://www.cs.cmu.edu/~afyshe/thesis/cnnse_mitchell_lapata_all.html
(behav sim score 6.33/7)
![Page 38: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/38.jpg)
Better than Correlation: Interpretability
38http://www.cs.cmu.edu/~afyshe/thesis/cnnse_mitchell_lapata_all.html
(behav sim score 5.61/7)
![Page 39: A Compositional and Interpretable Semantic Space Alona Fyshe, Leila Wehbe, Partha Talukdar, Brian Murphy, and Tom Mitchell Carnegie Mellon University amfyshe@gmail.com.](https://reader036.fdocuments.us/reader036/viewer/2022062313/56649d705503460f94a521d3/html5/thumbnails/39.jpg)
Summary
• Composition awareness improves VSMs– Closer to behavioral measure of phrase similarity– Better phrase representations
• Interpretable dimensions– Helps to debug composition failures
39