Recursive Deep Models for Semantic Compositionality Over a ...ddekok/dl4nlp/patricia-neele.pdf ·...
Transcript of Recursive Deep Models for Semantic Compositionality Over a ...ddekok/dl4nlp/patricia-neele.pdf ·...
Philosophische FakultätSeminar für Sprachwissenschaft
Recursive Deep Models for SemanticCompositionality Over a Sentiment Treebanktext06 July 2017, Patricia Fischer & Neele Witte
OverviewSentiment Analysis
Sentiment Treebank
Neural Network Architecture
Recursive Neural NetworkMatrix Vector RNNRecursive Neural Tensor Network
ExperimentsFine-grained Sentiment for All PhrasesFull Sentence Binary SentimentContrastive ConjunctionHigh Level NegationMost Positive/Negative Phrases
2 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
3 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
• “Sentiment analysis is the measurement of positive and nega-tive language.”
• “Sentiment Analysis is the process of determining whether apiece of writing is positive, negative or neutral. It’s also knownas opinion mining, deriving the opinion or attitude of a speaker.”
• “Using NLP, statistics, or machine learning methods to extract,identify, or otherwise characterize the sentiment content of atext unit.”
4 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
• Classification of users, texts, phrases, words
• Ratings
- Binary: or or
- Scales:
- Open category:
5 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
Challenges
• Opinions expressed in complex ways
• Stylistic devices such as sarcasm, irony etc.
6 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
Examples
7 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Analysis
Examples
8 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
From
To
9 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Motivation for New Model and Database
• Not only want to represent sentiment by the sum of the senti-ments of their components, but by the composition of them
• Word order is important, especially for detecting negation
• No database with annotated single sentences (usually docu-ments)→ good results for long texts but not for short texts (e.g.Twitter Data), phrases, segments
• Accuracy for three classes on short texts: below 60%
• Aim: construct a database to train and evaluate compositionalmodels
10 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Treebank
11 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Sentiment Treebank
Normalized histogram of sentiment annotations at each n-gram length.
• 11,855 single sentences, 215,154 uniquephrases
• Movie review excerpts from rottentomato-es.com
• Stanford parser• Labeling: amazon mechanical turk→ Fine-grained sentiment classification
12 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Semantic Representation of Words
• Map words into vector space to represent their meaning (se-mantic)• Similar words are close to each other• How can we represent meaning of longer phrases?
13 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Semantic Representation of Sentences
• Can we find a semantic representation for sentences (of arbi-trary length) as well?• Map phrases into the same vector space as well• How?
14 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Semantic Representation of Sentences
• Bag of Words: represent sentence as Bag of words and createone vector per sentence
→ Problem: word order ignored
• Sentence embeddings: create embeddings for n-grams (e.g.7-gram represents a sentence embedding)
→ Problem: cannot create so many embeddings, sentences canbe very long
15 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Recursive Neural Network
Principle of CompositionalityThe meaning (vector) of a sentence is defined by1. the meaning of its words2. the rules that combine them
Recursive Neural Nets can jointly learn compositional vectorrepresentations and parse trees
16 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Recursive Structure
1. Extract a binary syntactic tree
2. Recursively merge smaller segments to get representation ofbigger segments / the whole sentence
17 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Building Blocks for Neural Network
Composition function for merging two children
f (W [c1; c2] + b) (1)
Classification function for assigning a label to each node
ya = softmax(Wsa) (2)
Loss function: the cross-entropy error between the predicteddistribution and the target distribution
18 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Recursive NN Structure
19 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
MV-RNN Structure
20 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Recursive Neural Tensor Network
p2
a
not
p1
b
very
c
good
p1 = f
([bc
]T
V [1:d ][bc
]+ W
[bc
])
p2 = f
([ap1
]T
V [1:d ][
ap1
]+ W
[ap1
])
21 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Experiments
22 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Fine-grained Sentiment for All Phrases
ModelFine-grained Positive/Negative
All Root All RootNB 67.2 41.0 82.6 81.8SVM 64.3 40.7 84.6 79.4BiNB 71.0 41.9 82.7 83.1VecAvg 73.3 32.7 85.1 80.1RNN 79.0 43.2 86.1 82.4MV-RNN 78.7 44.4 86.8 82.9RNTN 80.7 45.7 87.6 85.4
23 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Full Sentence Binary Sentiment
ModelFine-grained Positive/Negative
All Root All RootNB 67.2 41.0 82.6 81.8SVM 64.3 40.7 84.6 79.4BiNB 71.0 41.9 82.7 83.1VecAvg 73.3 32.7 85.1 80.1RNN 79.0 43.2 86.1 82.4MV-RNN 78.7 44.4 86.8 82.9RNTN 80.7 45.7 87.6 85.4
24 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Contrastive Conjunction
“There are slow and repetitive partsbut it has just enough spice to keep it interesting.”
25 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
High Level Negation
Can the Model correctly classify the reversal from positive to nega-tive sentiment?
26 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Negating Sentence with Negative Sentiment
How often did the model increase positive activation in the senti-ment?
27 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Negating Sentence with Negative Sentiment
→ Sentiment of the sentence will become less negative (not neces-sarily positive)
28 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Most Positive/Negative Phrases
29 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
ReferencesBo Pang and Lillian Lee. (2008)Opinion mining and sentiment analysis.In Foundations and Trends in Information Retrieval.
R. Socher, C. D. Manning, and A. Y. Ng. (2010)Learning continuous phrase representations and syntactic parsing with recursive neuralnetworks.In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Fea- ture LearningWorkshop.
R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. (2011a)Parsing Natural Scenes and Natural Language with Recursive Neural Networks.In ICML.
R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. (2012)Semantic compositionality through recursive matrix-vector spaces.In EMNLP.
30 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen
Thank you!Contact:
Philosophische FakultätSeminar für SprachwissenschaftWilhelmstraße 19, 72074 TübingenPhone: +49 (0)7071 29-75927Fax: +49 (0)7071 [email protected]
31 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen