Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan...
Transcript of Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan...
![Page 1: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/1.jpg)
Document-level Sentiment Inference with Social, Faction, and Discourse Context
Eunsol Choi, Hannah Rashkin, Luke Zettlemoyer, Yejin Choi
ACL 2016
1
![Page 2: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/2.jpg)
2
Targeted Sentiment InferenceExtract sentiment relation between real-world entities in news articles.
Russia Belarus
positive? negative? neutral?
Opinion Holder Opinion Target
(Stoyanov and Claire 11,Yang and Cardie 14, Deng and Wiebe 14)
Russia criticizes Belarus
![Page 3: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/3.jpg)
This work: Document-level Sentiment Inference
3
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
![Page 4: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/4.jpg)
4
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Belarus
Challenge 1: Read the entire story
Russia
positive? negative? neutral?
![Page 5: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/5.jpg)
5
• Entities form factions inside the document.
• Subset of sentiment relations can only inferred via relationships among
entities.
Challenge 2: Inference across entities
Tim Cook
C.E.O
Apple
Katy PerryRihanna
Taylor Swift
Elie Goulding
Miley Cyrus
![Page 6: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/6.jpg)
Challenge 3: Sentiment Beyond Sentence Boundaries
6
Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
![Page 7: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/7.jpg)
This work: Document-level Sentiment Inference
7
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
![Page 8: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/8.jpg)
8
Evidence: Explicit Sentiment Textual Cues
…….….Belarus for permitting Georgian President Mikheil Saakhashvili to appear on its television.…….……. Saakhashvili announced Thursday that he did not understand Russia’s claims..…….…….……s with Ge
did not understand
permit
criticize
NegativePositive
![Page 9: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/9.jpg)
9
NegativePositiveFactionEntities in the same faction shares opinions
Homophily (Lazarsfeld and Merton, 1954)
Evidence:Sentiment Inference Through Factions
president
![Page 10: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/10.jpg)
10
NegativePositiveFactionEnemy of an enemy is a friend.
Social Balance Theory (Heider, 1946)
Evidence:Sentiment Inference Through Relations
![Page 11: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/11.jpg)
This work: Document-level Sentiment Inference
11
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
![Page 12: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/12.jpg)
Related Work
Opinion argument
Opinion attribute
Opinion expression
Chavez faced America 's hostility
SubjectiveSourceTarget
IS-FROM
IS-ABOUT
Opinion Frame
Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”
12
Corpus-level Approach (KBP)2013/4 Sentiment Task
Conan O’Brien is positive towards
David Litterman, South Korea, Pokemon
Query Entity Attitude
Answer
Fine-grained Approach (MPQA)(Deng and Wiebe 15)
![Page 13: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/13.jpg)
Related Work
Opinion argument
Opinion attribute
Opinion expression
Chavez faced America 's hostility
SubjectiveSourceTarget
IS-FROM
IS-ABOUT
Opinion Frame
Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”
13
Corpus-level Approach (KBP)2013/4 Sentiment Task
Conan O’Brien is positive towards
David Litterman, South Korea, Pokemon
Query Entity Attitude
Answer
Fine-grained Approach (MPQA)(Deng and Wiebe 15)
This Work:
• Document-level approach• Considers all named entities• Focuses on entity-entity interactions
![Page 14: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/14.jpg)
Dataset
14
• Document-level sentiment data collection
• Dataset Statistics
![Page 15: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/15.jpg)
Document Annotations
15
Evaluation Dataset Collection
![Page 16: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/16.jpg)
Evaluation Dataset Collection
Mark the relationship between the pair as one of below.
16
Positive Not Negative
Unbiased Not Positive
Negative
![Page 17: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/17.jpg)
Evaluation Dataset Collection
Mark the relationship between the pair as one of below.
17
Positive Not Negative
Unbiased Not Positive
Negative
When sentiment is not explicitly stated but can be inferred, mark it as Inferred.
![Page 18: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/18.jpg)
Document Output
18
Evaluation Dataset Collection
![Page 19: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/19.jpg)
Document Output
19
Evaluation Dataset Collection
•Documents drawn from previous datasets (KBP, MPQA).
•All named entity pairs beyond sentence boundaries.
•Considers the first 15 sentences of document.
•Encouraged to capture implicit sentiment.
![Page 20: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/20.jpg)
Dataset Statistics
20
KBP
MPQA
# of Labeled Entity Pairs Per Doc
0 30 60 90 120
11.6
5
91
44.7
9.5
11.6
Positive Unbiased Negative
Same polarity
Annotator Agreement (Cohen’s Kappa) 0.71
# Docs Avg. # of entities
MPQA 54 10.6
KBP 154 7.9
Total 15,185 entity pair labels
![Page 21: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/21.jpg)
Data Characteristics
• High overlap (about 90%) with KBP and MPQA
• Denser (10x more) entity pair annotations than KBP/MPQA
• Cover all named entity pairs
• Capture inferred sentiment
21
![Page 22: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/22.jpg)
Inferred Sentiment
Positive Negative
Inferred Ratio 70% 58%
The FINRA announced the fine, saying Goldman lacked adequate procedures to ensure the required disclosure.
FINRA Goldman
22
![Page 23: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/23.jpg)
Sentiment Beyond Sentence Boundaries
~25% of entity pairs with sentiment occurs from pairs not appear in the same sentence.
23
Russia heat, smog trigger health problems..……….………………………………… “We never work with a future perspective,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
![Page 24: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/24.jpg)
Model
24
• Document-level ILP model framework
• Soft constraints capturing entity-entity interactions
• Individual entity pair model
![Page 25: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/25.jpg)
Document level ILP model
25
Objective = + + +
NegativePositive
Factual Tie
![Page 26: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/26.jpg)
Document level ILP model
NegativePositive
Factual Tie
26
Objective = + + +
Global soft constraints capturing entity-entity interactions
![Page 27: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/27.jpg)
Document level ILP model
27
Objective = + + +
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Belarus Russia
Opinion Holder Opinion Target
positive? negative? neutral?
Individual entity pair sentiment classifier model scores
![Page 28: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/28.jpg)
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
28
Saakhashvili Georgia
president of
Objective = + + +
![Page 29: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/29.jpg)
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
29
Saakhashvili Georgia
president of
Objective = + + +
Russia
![Page 30: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/30.jpg)
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
30
Saakhashvili Georgia
president of
Objective = + + +
Russia
![Page 31: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/31.jpg)
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
31
Saakhashvili Georgia
president of
Objective = + + +
Russia
![Page 32: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/32.jpg)
Inference from Faction Relation
32
Objective = + + +
![Page 33: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/33.jpg)
Faction Detector
Heuristics to decide whether an entity belongs to the other.
• modifier, compound, possessive or appositive on dependency path
33
Saakhfshvilli
Georgiapresident of
![Page 34: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/34.jpg)
Faction Detector
Taylor Swift
SelenaGomez
’s friend
34
Theresa May
Andrea Leadso
m
’s rival Gryzlov Russiaspeaker of
Good Examples Errors
On small annotated study, ~30% recall, ~60% precision
![Page 35: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/35.jpg)
Inference from Sentiment Relation
35
Russia
Georgia
Belarus
Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.
Objective = + + +
![Page 36: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/36.jpg)
Inference from Sentiment Relation
36
Russia
Georgia
Belarus
Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.
Objective = + + +
Russia
Georgia
Belarus
![Page 37: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/37.jpg)
Inference from Sentiment Relation
37
Objective = + + +
![Page 38: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/38.jpg)
38
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
Inference from Sentiment Relation
Russia Georgia Belarus
Objective = + + +
![Page 39: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/39.jpg)
39
Inference from Sentiment Relation
Russia Georgia Belarus
Objective = + + +
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
![Page 40: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/40.jpg)
40
Inference from Sentiment Relation
Objective = + + +
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
![Page 41: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/41.jpg)
Data shows that these constraints are often satisfied.
5
20
35
50
65
80
Faction Balance Reciprocity
Random Data
41
![Page 42: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/42.jpg)
• Incorporates the scores from a pairwise classifier model
42
Objective = + + +
• Factors from linear SVM classifier model
Individual Entity Pair Model
![Page 43: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/43.jpg)
• Document Features
• Dependency Path Features
• Quotation Features
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Russia Belarus
43
Individual Entity Pair Model
![Page 44: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/44.jpg)
Document Features• Capture the entity salience.
• Do they appear in headline?• Do they occur together frequently?
44
The most frequently mentioned entity is 3.4 times more likely to have sentiment.
![Page 45: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/45.jpg)
Dependency Path Features• The polarity of path by MPQA sentiment lexicon (Wilson et al 05)
• Path containing [dobj, rev_subj]
45
Skah Norway
![Page 46: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/46.jpg)
Quotation Features
46
Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
• The polarity of quotes by MPQA sentiment lexicon
![Page 47: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/47.jpg)
Evaluation
47
![Page 48: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/48.jpg)
Experimental Set-up
• Precision, Recall, and F1 score
• Positive and negative label sets
• Half as development set, half as a test set
48
![Page 49: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/49.jpg)
Comparison Systems• Random (following the data distribution of the dev set)
• Sentence-level RNN classifier:
• Sentiment model on movie domain (Socher 2013)
• Sentiment labels from sentences of the pair co-occurring
• Pairwise classifier (without global inference)
49
![Page 50: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/50.jpg)
Results (KBP)F1
Sco
re
0
10
20
30
40
Positive Negative
Random Sentence Pairwise Global
50
![Page 51: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/51.jpg)
Ablation StudyF1
Sco
re
20
25
30
35
40
Base +Reciprocity +Balance Theory +Faction
Positive Negative
51
![Page 52: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/52.jpg)
Contributions
• Document-level sentiment inference among pairs of entities
• Intuitions from social balance theory and reciprocity
• Models factual relationship affecting sentiment relation
52
![Page 53: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/53.jpg)
Future work• Incorporating additional types of factual relationships
• Refine global constraints based on entity types
53
Data is available!
http://homes.cs.washington.edu/~eunsol/project_page/acl16/
![Page 54: Document-level Sentiment Inference with Social, Faction, and … · 2017. 10. 3. · Conan O’Brien is positive towards David Litterman, South Korea, Pokemon Query Entity Attitude](https://reader036.fdocuments.us/reader036/viewer/2022071608/6146063e8f9ff812542000dc/html5/thumbnails/54.jpg)
Thanks!Questions?
54