Opinion Mining
description
Transcript of Opinion Mining
Opinion Mining
Sudeshna Sarkar
24th and 26th October 2007
Janyce Wiebe, U. PittsburghClaire Cardie, Cornell U.
Ellen Riloff, U. UtahJosef Ruppenhofer, U. Pittsburgh
Bing Liu
MIAS - June 14, 2007 3
Introduction – facts and opinions
Two main types of information on the Web. Facts and Opinions
Current search engines search for facts (assume they are true) Facts can be expressed with topic keywords.
Search engines do not search for opinions Opinions are hard to express with a few keywords
How do people think of Motorola Cell phones?
Current search ranking strategy is not appropriate for opinion retrieval/search.
MIAS - June 14, 2007 4
Introduction – user generated content
Word-of-mouth on the Web One can express personal experiences and opinions on almost
anything, at review sites, forums, discussion groups, blogs ..., (called the user generated content.)
They contain valuable information Web/global scale
No longer limited to your circle of friends Our interest: to mine opinions expressed in the user-generated
content An intellectually very challenging problem. Practically very useful.
MIAS - June 14, 2007 5
Introduction – Applications
Businesses and organizations: product and service benchmarking. Market intelligence. Business spends a huge amount of money to find consumer
sentiments and opinions. Consultants, surveys and focused groups, etc
Individuals: interested in other’s opinions when Purchasing a product or using a service, Finding opinions on political topics, Many other decision making tasks.
Ads placements: Placing ads in user-generated content Place an ad when one praises an product. Place an ad from a competitor if one criticizes an product.
Opinion retrieval/search: providing general search for opinions.
MIAS - June 14, 2007 6
Question Answering
Opinion question answering:
Q: What is the international reaction to the reelection of Robert Mugabe as President of Zimbabwe?
A: African observers generally approved of his victory while Western Governments denounced it.
MIAS - June 14, 2007 7
More motivation
Product review mining: What features of the ThinkPad T43 do customers like and which do they dislike?
Review classification: Is a review positive or negative toward the movie? Tracking sentiments toward topics over time: Is anger ratcheting up or
cooling down? Etc.
MIAS - June 14, 2007 8
Two types of evaluation
Direct Opinions: sentiment expressions on some objects, e.g., products, events, topics, persons E.g., “the picture quality of this camera is great” Subjective
Comparisons: relations expressing similarities or differences of more than one object. Usually expressing an ordering. E.g., “car x is cheaper than car y.” Objective or subjective.
MIAS - June 14, 2007 9
Opinion search (Liu, Web Data Mining book, 2007)
Can you search for opinions as conveniently as general Web search?
Whenever you need to make a decision, you may want some opinions from others, Wouldn’t it be nice? you can find them on a search
system instantly, by issuing queries such as Opinions: “Motorola cell phones” Comparisons: “Motorola vs. Nokia”
Cannot be done yet!
MIAS - June 14, 2007 10
Typical opinion search queries Find the opinion of a person or organization (opinion
holder) on a particular object or a feature of an object. E.g., what is Bill Clinton’s opinion on abortion?
Find positive and/or negative opinions on a particular object (or some features of the object), e.g., customer opinions on a digital camera, public opinions on a political topic.
Find how opinions on an object change with time. How object A compares with Object B?
Gmail vs. Yahoo mail
MIAS - June 14, 2007 11
Find the opinion of a person on X
In some cases, the general search engine can handle it, i.e., using suitable keywords. Bill Clinton’s opinion on abortion
Reason: One person or organization usually has only one
opinion on a particular topic. The opinion is likely contained in a single document. Thus, a good keyword query may be sufficient.
MIAS - June 14, 2007 12
Find opinions on an object X
We use product reviews as an example: Searching for opinions in product reviews is different
from general Web search. E.g., search for opinions on “Motorola RAZR V3”
General Web search for a fact: rank pages according to some authority and relevance scores. The user views the first page (if the search is perfect). One fact = Multiple facts
Opinion search: rank is desirable, however reading only the review ranked at the top is dangerous
because it is only the opinion of one person. One opinion Multiple opinions
MIAS - June 14, 2007 13
Search opinions (contd)
Ranking: produce two rankings
Positive opinions and negative opinions Some kind of summary of both, e.g., # of each
Or, one ranking but The top (say 30) reviews should reflect the natural distribution
of all reviews (assume that there is no spam), i.e., with the right balance of positive and negative reviews.
Questions: Should the user reads all the top reviews? OR Should the system prepare a summary of the reviews?
MIAS - June 14, 2007 14
Reviews are similar to surveys
Reviews can be regarded as traditional surveys. In traditional survey, returned survey forms are treated
as raw data. Analysis is performed to summarize the survey
results. E.g., % against or for a particular issue, etc.
In opinion search, Can a summary be produced? What should the summary be?
MIAS - June 14, 2007 15
Roadmap
Opinion mining – the abstraction Domain level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and
summarization Comparative sentence and relation extraction Summary
MIAS - June 14, 2007 16
Opinion mining – the abstraction(Hu and Liu, KDD-04)
Basic components of an opinion Opinion holder: A person or an organization that holds an specific
opinion on a particular object. Object: on which an opinion is expressed Opinion: a view, attitude, or appraisal on an object from an opinion
holder. Objectives of opinion mining: many ... We use consumer reviews of products to develop the
ideas. Other opinionated contexts are similar.
MIAS - June 14, 2007 17
Object/entity Definition (object): An object O is an entity which can be
a product, person, event, organization, or topic. O is represented as a tree or taxonomy of components (or parts), sub-components, and so on. Each node represents a component and is associated with a set of
attributes. O is the root node (which also has a set of attributes)
An opinion can be expressed on any node or attribute of the node.
To simplify our discussion, we use “features” to represent both components and attributes. The term “feature” should be understood in a broad sense
Product feature, topic or sub-topic, event or sub-event, etc Note: the object O itself is also a feature.
MIAS - June 14, 2007 18
A model of a review
An object is represented with a finite set of features, F = {f1, f2, …, fn}. Each feature fi in F can be expressed with a finite set of words or
phrases Wi, which are synonyms.
That is to say: we have a set of corresponding synonym sets W = {W1,
W2, …, Wn} for the features.
Model of a review: An opinion holder j comments on a subset of the features Sj F of an object O.
For each feature fk Sj that j comments on, he/she
chooses a word or phrase from Wk to describe the feature,
expresses a positive, negative or neutral opinion on fk.
MIAS - June 14, 2007 19
Opinion mining tasks
At the document (or review) level:
Task: sentiment classification of reviews Classes: positive, negative, and neutral Assumption: each document (or review) focuses on a single object O
(not true in many discussion posts) and contains opinion from a single opinion holder.
At the sentence level:
Task 1: identifying subjective/opinionated sentences Classes: objective and subjective (opinionated)
Task 2: sentiment classification of sentences Classes: positive, negative and neutral. Assumption: a sentence contains only one opinion
not true in many cases. Then we can also consider clauses.
MIAS - June 14, 2007 20
Opinion mining tasks (contd)
At the feature level:
Task 1: Identifying and extracting object features that have been commented on in each review.
Task 2: Determining whether the opinions on the features are positive, negative or neutral in the review.
Task 3: Grouping feature synonyms.
Produce a feature-based opinion summary of multiple reviews (more on this later).
Opinion holders: identify holders is also useful, e.g., in news articles, etc, but they are usually known in user generated content, i.e., the authors of the posts.
MIAS - June 14, 2007 21
More at the feature level
Problem 1: Both F and W are unknown. We need to perform all three tasks:
Problem 2: F is known but W is unknown. All three tasks are needed. Task 3 is easier. It
becomes the problem of matching discovered features with the set of given features F.
Problem 3: W is known (F is known too). Only task 2 is needed.
F: the set of featuresW: synonyms of each feature
MIAS - June 14, 2007 22
Roadmap
Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and summarization Comparative Sentence and relation extraction Summary
MIAS - June 14, 2007 23
Sentiment classification
Classify documents (e.g., reviews) based on the overall sentiments expressed by authors, Positive, negative, and (possibly) neutral Since in our model an object O itself is also a feature, then
sentiment classification essentially determines the opinion expressed on O in each document (e.g., review).
Similar but different from topic-based text classification. In topic-based text classification, topic words are important. In sentiment classification, sentiment words are more important,
e.g., great, excellent, horrible, bad, worst, etc.
MIAS - June 14, 2007 24
Unsupervised review classification(Turney, ACL-02)
Data: reviews from epinions.com on automobiles, banks, movies, and travel destinations.
The approach: Three steps Step 1:
Part-of-speech tagging Extracting two consecutive words (two-word phrases)
from reviews if their tags conform to some given patterns, e.g., (1) JJ, (2) NN.
MIAS - June 14, 2007 25
Step 2: Estimate the semantic orientation of the extracted phrases Use Pointwise mutual information
Semantic orientation (SO): SO(phrase) = PMI(phrase, “excellent”)
PMI(phrase, “poor”)
Using AltaVista near operator to do search to find the number of hits to compute PMI and SO.
)()(
)(log),(
21
21221 wordPwordP
wordwordPwordwordPMI
MIAS - June 14, 2007 26
Step 3: Compute the average SO of all phrases classify the review as recommended if average SO is
positive, not recommended otherwise.
Final classification accuracy: automobiles - 84% banks - 80% movies - 65.83 travel destinations - 70.53%
MIAS - June 14, 2007 27
Sentiment classification using machine learning methods (Pang et al, EMNLP-02)
The paper applied several machine learning techniques to classify movie reviews into positive and negative.
Three classification techniques were tried: Naïve Bayes Maximum entropy Support vector machine
Pre-processing settings: negation tag, unigram (single words), bigram, POS tag, position.
SVM: the best accuracy 83% (unigram)
MIAS - June 14, 2007 28
MIAS - June 14, 2007 29
Roadmap
Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and
summarization Comparative sentence and relation
extraction Summary
MIAS - June 14, 2007 30
Sentence-level sentiment analysis
Document-level sentiment classification is too coarse for most applications.
Let us move to the sentence level. Much of the work on sentence level sentiment analysis focus on
identifying subjective sentences in news articles. Classification: objective and subjective. All techniques use some forms of machine learning. E.g., using a naïve Bayesian classifier with a set of data
features/attributes extracted from training sentences (Wiebe et al. ACL-99).
MIAS - June 14, 2007 31
Using learnt patterns (Rilloff and Wiebe, EMNLP-03)
A bootstrapping approach. A high precision classifier is used to automatically identify some
subjective and objective sentences. Two high precision (low recall) classifiers were used,
a high precision subjective classifier A high precision objective classifier Based on manually collected lexical items, single words and n-
grams, which are good subjective clues. A set of patterns are then learned from these identified subjective
and objective sentences. Syntactic templates are provided to restrict the kinds of patterns to be
discovered, e.g., <subj> passive-verb. The learned patterns are then used to extract more subject and
objective sentences (the process can be repeated).
MIAS - June 14, 2007 32
Subjectivity and polarity (orientation) (Yu and Hazivassiloglou, EMNLP-03)
For subjective or opinion sentence identification, three methods were tried: Sentence similarity. Naïve Bayesian classification. Multiple naïve Bayesian (NB) classifiers.
For opinion orientation (positive, negative or neutral) (also called polarity) classification, it uses a similar method to (Turney, ACL-02), but with more seed words (rather than two) and based on
log-likelihood ratio (LLR). For classification of each word, it takes average of LLR
scores of words in the sentence and use cutoffs to decide positive, negative or neutral.
MIAS - June 14, 2007 33
Other related work
Consider gradable adjectives (Hatzivassiloglou and Wiebe, Coling-00)
Semi-supervised learning with the initial training set identified by some strong patterns and then applying NB or self-training (Wiebe and Riloff, CicLing 05)
Finding strength of opinions at the clause level (Wilson etal, AAAI-04)
Sum up orientations of opinion words in a sentence (or within some word window) Kim and Hovy, Coling-04
MIAS - June 14, 2007 34
Let us go further?
Sentiment classifications at both document and sentence (or clause) level are useful, but They do not find what the opinion holder liked and disliked.
An negative sentiment on an object does not mean that the opinion holder dislikes everything about
the object. A positive sentiment on an object
does not mean that the opinion holder likes everything about the object.
We need to go to the feature level.
MIAS - June 14, 2007 35
But before we go further
Let us discuss Opinion Words or Phrases (also called polar words, opinion bearing words, etc). E.g., Positive: beautiful, wonderful, good, amazing, Negative: bad, poor, terrible, cost someone an arm and a leg
(idiom). They are instrumental for opinion mining Three main ways to compile such a list:
Manual approach: not a bad idea, only an one- time effort Corpus-based approaches Dictionary-based approaches
Important to note: Some opinion words are context independent. (eg, good) Some are context dependent. (eg, long)
MIAS - June 14, 2007 36
Corpus-based approaches
Rely on syntactic or co-occurrence patterns in large corpuses. (Hazivassiloglou and McKeown, ACL-97; Turney, ACL-02; Yu and Hazivassiloglou, EMNLP-03; Kanayama and Nasukawa, EMNLP-06; Ding and Liu, 2007)
Can find domain (not context) dependent orientations (positive, negative, or neutral).
(Turney, ACL-02) and (Yu and Hazivassiloglou, EMNLP-03) are similar. Assign opinion orientations (polarities) to words/phrases. (Yu and Hazivassiloglou, EMNLP-03) is different from (Turney,
ACL-02) in that using more seed words (rather than two) and using log-
likelihood ratio (rather than PMI).
MIAS - June 14, 2007 37
Corpus-based approaches (contd)
Use constraints (or conventions) on connectives to identify opinion words (Hazivassiloglou and McKeown, ACL-97; Kanayama and Nasukawa, EMNLP-06; Ding and Liu, SIGIR-07). E.g., Conjunction: conjoined adjectives usually have the same orientation
(Hazivassiloglou and McKeown, ACL-97). E.g., “This car is beautiful and spacious.” (conjunction)
AND, OR, BUT, EITHER-OR, and NEITHER-NOR have similar constraints
Learning using log-linear model: determine if two conjoined adjectives are of the same or
different orientations. Clustering: produce two sets of words: positive and negative
Corpus: 21 million word 1987 Wall Street Journal corpus.
MIAS - June 14, 2007 38
Dictionary-based approaches
Typically use WordNet’s synsets and hierarchies to acquire opinion words Start with a small seed set of opinion words Use the set to search for synonyms and antonyms in
WordNet (Hu and Liu, KDD-04; Kim and Hovy, COLING-04).
Manual inspection may be used afterward. Use additional information (e.g., glosses) from WordNet
(Andreevskaia and Bergler, EACL-06) and learning (Esuti and Sebastiani, CIKM-05).
Weakness of the approach: Do not find domain and/or context dependent opinion words, e.g., small, long, fast.
MIAS - June 14, 2007 39
Roadmap
Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and
summarization Comparative sentence and relation extraction Summary
MIAS - June 14, 2007 40
Feature-based opinion mining and summarization (Hu and Liu, KDD-04)
Again focus on reviews (easier to work in a concrete domain!)
Objective: find what reviewers (opinion holders) liked and disliked Product features and opinions on the features
Since the number of reviews on an object can be large, an opinion summary should be produced. Desirable to be a structured summary. Easy to visualize and to compare. Analogous to but different from multi-document
summarization.
MIAS - June 14, 2007 41
The tasks
Recall the three tasks in our model.
Task 1: Extracting object features that have been commented on in each review.
Task 2: Determining whether the opinions on the features are positive, negative or neutral.
Task 3: Grouping feature synonyms. Summary
Task 2 may not be needed depending on the format of reviews.
MIAS - June 14, 2007 42
Different review format
Format 1 - Pros, Cons and detailed review: The reviewer is asked to describe Pros and Cons separately and also write a detailed review. Epinions.com uses this format.
Format 2 - Pros and Cons: The reviewer is asked to describe Pros and Cons separately. C|net.com used to use this format.
Format 3 - free format: The reviewer can write freely, i.e., no separation of Pros and Cons. Amazon.com uses this format.
MIAS - June 14, 2007 43
Format 1
GREAT Camera., Jun 3, 2004
Reviewer: jprice174 from Atlanta, Ga.
I did a lot of research last year before I bought this camera... It kinda hurt to leave behind my beloved nikon 35mm SLR, but I was going to Italy, and I needed something smaller, and digital.
The pictures coming out of this camera are amazing. The 'auto' feature takes great pictures most of the time. And with digital, you're not wasting film if the picture doesn't come out.
Format 2
Format 3
MIAS - June 14, 2007 44
Feature-based Summary (Hu and Liu, KDD-04)
GREAT Camera., Jun 3, 2004 Reviewer: jprice174 from Atlanta, Ga.
I did a lot of research last year before I bought this camera... It kinda hurt to leave behind my beloved nikon 35mm SLR, but I was going to Italy, and I needed something smaller, and digital. The pictures coming out of this camera are amazing. The 'auto' feature takes great pictures most of the time. And with digital, you're not wasting film if the picture doesn't come out. …
….
Feature Based Summary:
Feature1: picturePositive: 12 The pictures coming out of this camera are
amazing. Overall this is a good camera with a really good
picture clarity.…Negative: 2 The pictures come out hazy if your hands shake
even for a moment during the entire process of taking a picture.
Focusing on a display rack about 20 feet away in a brightly lit room during day time, pictures produced by this camera were blurry and in a shade of orange.
Feature2: battery life…
MIAS - June 14, 2007 45
Visual summarization & comparison
Summary of reviews of Digital camera 1
Picture Battery Size Weight Zoom
+
_
Comparison of reviews of
Digital camera 1
Digital camera 2 _
+
MIAS - June 14, 2007 46
Feature extraction from Pros and Cons of Format 1 (Liu et al WWW-03; Hu and Liu, AAAI-CAAW-05)
Observation: Each sentence segment in Pros or Cons contains only one feature. Sentence segments can be separated by commas, periods, semi-colons, hyphens, ‘&’’s, ‘and’’s, ‘but’’s, etc.
Pros in Example 1 can be separated into 3 segments:great photos <photo>easy to use <use>very small <small> <size>
Cons can be separated into 2 segments:battery usage <battery>included memory is stingy <memory>
MIAS - June 14, 2007 47
Extraction using label sequential rules
Label sequential rules (LSR) are a special kind of sequential patterns, discovered from sequences.
LSR Mining is supervised (Liu’s Web mining book 2006). The training data set is a set of sequences, e.g.,
“Included memory is stingy”
is turned into a sequence with POS tags.
{included, VB}{memory, NN}{is, VB}{stingy, JJ}
then turned into {included, VB}{$feature, NN}{is, VB}{stingy, JJ}
MIAS - June 14, 2007 48
Using LSRs for extraction
Based on a set of training sequences, we can mine label sequential rules, e.g.,
{easy, JJ }{to}{*, VB} {easy, JJ}{to}{$feature, VB} [sup = 10%, conf = 95%]
Feature Extraction Only the right hand side of each rule is needed. The word in the sentence segment of a new review that
matches $feature is extracted. We need to deal with conflict resolution also (multiple
rules are applicable.
MIAS - June 14, 2007 49
Extraction of features of formats 2 and 3
Reviews of these formats are usually complete sentencese.g., “the pictures are very clear.” Explicit feature: picture
“It is small enough to fit easily in a coat pocket or purse.” Implicit feature: size
Extraction: Frequency based approach Frequent features Infrequent features
MIAS - June 14, 2007 50
Frequency based approach(Hu and Liu, KDD-04)
Frequent features: those features that have been talked about by many reviewers.
Use sequential pattern mining Why the frequency based approach?
Different reviewers tell different stories (irrelevant) When product features are discussed, the words that they
use converge. They are main features.
Sequential pattern mining finds frequent phrases. Froogle has an implementation of the approach (no POS restriction).
MIAS - June 14, 2007 51
Using part-of relationship and the Web(Popescu and Etzioni, EMNLP-05)
Improved (Hu and Liu, KDD-04) by removing those frequent noun phrases that may not be features: better precision (a small drop in recall).
It identifies part-of relationship Each noun phrase is given a pointwise mutual information score
between the phrase and part discriminators associated with the product class, e.g., a scanner class.
The part discriminators for the scanner class are, “of scanner”, “scanner has”, “scanner comes with”, etc, which are used to find components or parts of scanners by searching on the Web: the KnowItAll approach, (Etzioni et al, WWW-04).
MIAS - June 14, 2007 52
Infrequent features extraction
How to find the infrequent features? Observation: the same opinion word can be used to describe
different features and objects. “The pictures are absolutely amazing.” “The software that comes with it is amazing.”
Frequent features
Opinion words
Infrequent features
MIAS - June 14, 2007 53
Identify feature synonyms
Liu et al (WWW-05) made an attempt using only WordNet. Carenini et al (K-CAP-05) proposed a more sophisticated method
based on several similarity metrics, but it requires a taxonomy of features to be given. The system merges each discovered feature to a feature node in
the taxonomy. The similarity metrics are defined based on string similarity,
synonyms and other distances measured using WordNet. Experimental results based on digital camera and DVD reviews
show promising results. Many ideas in information integration are applicable.
MIAS - June 14, 2007 54
Identify opinion orientation on feature
For each feature, we identify the sentiment or opinion orientation expressed by a reviewer.
We work based on sentences, but also consider, A sentence may contain multiple features. Different features may have different opinions. E.g., The battery life and picture quality are great (+), but the view
founder is small (-). Almost all approaches make use of opinion words and phrases. But
note again: Some opinion words have context independent orientations, e.g.
great. Some other opinion words have context dependent orientations,
e.g., “small” Many ways to use them.
MIAS - June 14, 2007 55
Aggregation of opinion words (Hu and Liu, KDD-04; Ding and Liu, SIGIR-07)
Input: a pair (f, s), where f is a feature and s is a sentence that contains f.
Output: whether the opinion on f in s is positive, negative, or neutral. Two steps:
Step 1: split the sentence if needed based on BUT words (but, except that, etc).
Step 2: work on the segment sf containing f. Let the set of opinion words in sf be w1, .., wn. Sum up their orientations (1, -1, 0), and assign the orientation to (f, s) accordingly.
In (Ding and Liu, SIGIR-07), step 2 is changed to
with better results. wi.o is the opinion orientation of wi. d(wi, f) is the distance from f to wi.
n
ii
i
fwd
ow1 ),(
.
MIAS - June 14, 2007 56
Context dependent opinions
Popescu and Etzioni (2005) used constraints of connectives in (Hazivassiloglou and McKeown, ACL-
97), and some additional constraints, e.g., morphological relationships, synonymy and antonymy, and
relaxation labeling to propagate opinion orientations to words and features.
Ding and Liu (2007) used constraints of connectives both at intra-sentence and inter-sentence
levels, and additional constraints of, e.g., TOO, BUT, NEGATION.to directly assign opinions to (f, s) with good results (> 0.85 of F-score).
MIAS - June 14, 2007 57
Roadmap
Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and
summarization Comparative sentence and relation extraction Summary
MIAS - June 14, 2007 58
Extraction of Comparatives
Comparative sentence mining Identify comparative sentences Extract comparative relations from them
MIAS - June 14, 2007 59
Linguistic Perspective
Comparative sentences use morphemes like more/most, -er/-est, less/least, as than and as are used to make a `standard’ against
which an entire entity is compared
Limitations Limited coverage
“In market capital, Intel is way ahead of AMD.”
Non-comparatives with comparative words “In the context of speed, faster means better.”
MIAS - June 14, 2007 60
Types of Comparatives
Gradable Non-Equal Gradable: Relations of the type greater or
less than Keywords like better, ahead, beats, etc “Optics of camera A is better than that of camera B”
Equative: Relations of type equal to Keywords and phrases like equal to, same as, both, all “Camera A and camera B both come in 7MP”
Superlative: Relations of the type greater or less than all others
Keywords and phrases like best, most, better than all “Camera A is the cheapest camera available in the market.”
MIAS - June 14, 2007 61
Types of Comparatives: non-gradable
Non-gradable: Sentences that compare features of two or more objects, but do not grade them. Sentences which imply: Object A is similar to or different from Object B with
regard to some features Object A has feature F1, Object B has feature F2 Object A has feature F, but Object B does not have
MIAS - June 14, 2007 62
Comparative Relation: gradable
Definition: A gradable comparative relation captures the essence of gradable comparative sentence and is represented with the following:
(relationWord, features, entityS1, entityS2, type) relationWord: The keyword used to express a
comparative relation in a sentence. features: a set of features being compared. entityS1 and entityS2: Sets of entities being
compared. type: non-equal gradable, equative or superlative
MIAS - June 14, 2007 63
Examples: Comparative relations
“car X has better controls than car Y”(relationWord = better, features = controls, entityS1 = carX, entityS2 = carY, type = non-equal-gradable)
“car X and car Y have equal mileage”(relationWord = equal, features = mileage, entityS1 = carX, entityS2 = carY, type = equative)
“car X is cheaper than both car Y and car Z”(relationWord = cheaper, features = null, entityS1 = carX, entityS2 = {carY, carZ}, type = non-equal-gradable)
“company X produces a variety of cars, but still best cars come from company Y”(relationWord = best, features = cars, entityS1 = companyY, entityS2 = companyX, type = superlative)
MIAS - June 14, 2007 64
Tasks
Given a collection of evaluative textsTask 1: Identify comparative sentences
Task 2: Categorize different types of comparative sentences.
Task 3: Extract comparative relations from the sentences
MIAS - June 14, 2007 65
Identify comparative sentences
Keyword strategy An observation: Its is easy to find a small set of
keywords that covers almost all comparative sentences, i.e., with a very high recall and a reasonable precision
A list of 83 keywords used in comparative sentences compiled by (Jinal and Liu, Sigir-06) including
Words with POS tags of JJR, JJS, RBR, RBS POS tags are used as keyword instead of individual
words Exceptions: more, less, most, least
Other indicative word like beat, exceed, ahead, etc. Phrases like in the lead, on par with, etc.
MIAS - June 14, 2007 66
2-step learning strategy
Step 1: Extract sentences which contain at least one keyword (recall = 98%, precision = 32% on our data set of gradables)
Step 2: Use Naïve Bayes classifier to classify sentences into two classes Comparative and non-comparative Attributes: class sequential rules (CSRs) generated
from sentences in step 1
MIAS - June 14, 2007 67
1. Sequence data preparation Use words within a radius r of a keyword to form a sequence
(words are replaced with POS tags) …
2. CSR generation Use different minimum supports for different keywords 13 manual rules, which were hard to generate automatically
3. Learning using a NB classifier Use CSRs and manual rules as attributes to build a final
classifier
MIAS - June 14, 2007 68
Classify different types of comparatives
Classify comparative sentences into three types: non-equal gradable, equative and superlative SVM learner gives the best result Asstribute set is the set of keywords If the sentence has a particular keywords in the
attribute set, the corresponding value is 1, and 0 otherwise.
MIAS - June 14, 2007 69
Extraction of comparative relations
Assumptions There is only one relation in a sentence Entities and features are nominals
Adjectival comparatives Does not deal with adverbial comparatives
3 steps Sequence data generation Label sequential rule (LSR) generation Build a sequential cover/extractor from LSRs
MIAS - June 14, 2007 70
Sequence data generation
Label Set = {$entityS1, $entityS2, $feature} Three labels are used as pivots to generate sequences.
Radius of 4 for optimal results Following words are also added
Distance words = {l1, l2, l3, l4, r1, r2, r3, r4} Special words #start and #end are used to mark the
start and the end of a sentence.
MIAS - June 14, 2007 71
Sequence data generation example
The comparative sentence
“Canon/NNP has/VBZ better/JJR optics/NNS”has $entityS1 “Canon” and $feature “optics”
Sequences are: <{#start>{l1}{$entityS1, NNP){r1}{has, VBZ}{r2}{better,
JJR}{r3}{$Feature, NNS}{r4}{#end}> <{#start>{l4}{$entityS1, NNP){l3}{has, VBZ}{l2}{better,
JJR}{l1}{$Feature, NNS}{r1}{#end}>
MIAS - June 14, 2007 72
Build a sequential cover from LSRs
LSR: {*, NN}{VBZ} { $entityS1, NN}{VBZ} Select the LSR rule with the highest confidence. Replace
the matched elements in the sentences that satisfy the rules with the labels in the rule.
Recalculate the confidence of each remaining rule based on the modified data from step 1.
Repeat step 1 and 2 until no rule left with confidence higher than minconf value (they sued 90%)
MIAS - June 14, 2007 73
Experimental Results (Jindal and Liu, AAAI 06)
Identifying Gradable Comparative Sentences Precision = 82% and recall = 81%
Classification into three gradable types SVM gave accuracy of 96%
Extraction of comparative relations LSR: F-score = 72%
MIAS - June 14, 2007 74
Summary
Two types of evaluations Direct opinions: We studied
The problem abstraction Sentiment analysis at document level, sentence level and
feature level
Comparisons: Very hard problems, but very useful
The current techniques are still in their infancy. Industrial applications are coming up
MIAS - June 14, 2007 75
END
Manual and Automatic Subjectivity and Sentiment
Analysis
MIAS - June 14, 2007 77
Outline
Corpus Annotation Pure NLP
Lexicon development Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis
Applications Product review mining
MIAS - June 14, 2007 78
Corpus AnnotationWiebe, Wilson, Cardie. Language Resources and Evaluation 39 (1-2), 2005
MIAS - June 14, 2007 79
Overview
Fine-grained: expression-level rather than sentence or document level The photo quality was the best that I have seen in a camera. The photo quality was the best that I have seen in a camera.
Annotate expressions of opinions, evaluations, emotions material attributed to a source, but presented objectively
MIAS - June 14, 2007 80
Overview
Fine-grained: expression-level rather than sentence or document level The photo quality was the best that I have seen in a camera. The photo quality was the best that I have seen in a camera.
Annotate expressions of opinions, evaluations, emotions, beliefs material attributed to a source, but presented objectively
MIAS - June 14, 2007 81
Overview
Opinions, evaluations, emotions, speculations are private states.
They are expressed in language by subjective expressions.
Private state: state that is not open to objective observation or verification.
Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.
MIAS - June 14, 2007 82
Overview
Focus on three ways private states are expressed in language
Direct subjective expressions Expressive subjective elements Objective speech events
MIAS - June 14, 2007 83
Direct Subjective Expressions
Direct mentions of private states
The United States fears a spill-over from the anti-terrorist campaign.
Private states expressed in speech events
“We foresaw electoral fraud but not daylight robbery,” Tsvangirai said.
MIAS - June 14, 2007 84
Expressive Subjective Elements [Banfield
1982]
“We foresaw electoral fraud but not daylight robbery,” Tsvangirai said
The part of the US human rights report about China is full of absurdities and fabrications
MIAS - June 14, 2007 85
Objective Speech Events
Material attributed to a source, but presented as objective fact
The government, it added, has amended the Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.
MIAS - June 14, 2007 86
MIAS - June 14, 2007 87
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007 88
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
(Writer)
MIAS - June 14, 2007 89
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
(Writer, Xirao-Nima)
MIAS - June 14, 2007 90
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
(Writer Xirao-Nima)(Writer Xirao-Nima)
MIAS - June 14, 2007 91
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
(Writer Xirao-Nima)(Writer Xirao-Nima)
(Writer)
MIAS - June 14, 2007 92
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 93
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 94
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 95
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 96
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 97
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report
Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative
MIAS - June 14, 2007 98
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
MIAS - June 14, 2007 99
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
(Writer)
MIAS - June 14, 2007 100
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
(writer, Xirao-Nima)
MIAS - June 14, 2007 101
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
(writer, Xirao-Nima, US)
MIAS - June 14, 2007 102
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
(writer, Xirao-Nima, US) (writer, Xirao-Nima)(Writer)
MIAS - June 14, 2007 103
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Objective speech event anchor: said source: <writer, Xirao-Nima>
Direct subjective anchor: fears source: <writer, Xirao-Nima, US> intensity: medium expression intensity: medium attitude type: negative target: spill-over
“The US fears a spill-over’’, said Xirao-Nima, a
professor of foreign affairs at the Central University
for Nationalities.
MIAS - June 14, 2007 104
The report has been strongly criticized and condemned bymany countries.
MIAS - June 14, 2007 105
Objective speech event anchor: the entire sentence source: <writer> implicit: true
Direct subjective anchor: strongly criticized and condemned source: <writer, many-countries> intensity: high expression intensity: high attitude type: negative target: report
The report has been strongly criticized and condemned bymany countries.
MIAS - June 14, 2007 106
As usual, the US state Department published its annual report on human rights practices in world countries last Monday.
And as usual, the portion about China contains little truth and many absurdities, exaggerations and fabrications.
MIAS - June 14, 2007 107
Objective speech event anchor : the entire 1st sentence source : <writer> implicit : true
Direct subjective anchor : the entire 2nd sentence source : <writer> implicit : true intensity : high expression intensity : medium attitude type : negative target : report
As usual, the US state Department published its annual report on human rights practices in world countries last Monday.
And as usual, the portion about China contains little truth and many absurdities, exaggerations and fabrications.
Expressive subjective element anchor : little truth source : <writer> intensity : medium attitude type : negative
Expressive subjective element anchor : many absurdities, exaggerations, and fabrications source : <writer> intensity : medium attitude type : negative
Expressive subjective element anchor : And as usual source : <writer> intensity : low attitude type : negative
MIAS - June 14, 2007 108
Corpus
www.cs.pitt.edu/mqpa/databaserelease (version 2)
English language versions of articles from the world press (187 news sources)
Also includes contextual polarity annotations (later)
Themes of the instructions: No rules about how particular words should be annotated.
Don’t take expressions out of context and think about what they could mean, but judge them as they are used in that sentence.
MIAS - June 14, 2007 109
Agreement
Inter-annotator agreement studies performed on various aspects of the scheme
Kappa is a measure of the degree of nonrandom agreement between observers and/or measurements of a specific categorical variable
Kappa values range between .70 and .80
MIAS - June 14, 2007 110
Agreement
Two council street wardens who helped lift a 14-ton bus off an injured schoolboy are to be especially commended for their heroic actions.
Nathan Thomson and Neville Sharpe will receive citations from the mayor of Croydon later this month.
Two council street wardens who helped lift a 14-ton bus off an injured schoolboy are to be especially commended for their heroic actions.
Nathan Thomson and Neville Sharpe will receive citations from the mayor of Croydon later this month.
Annotator 1 Annotator 2
MIAS - June 14, 2007 111
Agreement
Inter-annotator agreement studies performed on various aspects of the scheme
Kappa is a measure of the degree of nonrandom agreement between observers and/or measurements of a specific categorical variable
Kappa values range between .70 and .80
MIAS - June 14, 2007 112
Outline
Corpus Annotation Pure NLP
Lexicon development Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis Applications
Product review mining
MIAS - June 14, 2007 113
Who does lexicon development ?
Humans
Semi-automatic
Fully automatic
MIAS - June 14, 2007 114
What?
Find relevant words, phrases, patterns that can be used to express subjectivity
Determine the polarity of subjective expressions
MIAS - June 14, 2007 115
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive: honest important mature large patient
Ron Paul is the only honest man in Washington. Kitchell’s writing is unbelievably mature and is only likely to
get better. To humour me my patient father agrees yet again to my
choice of film
MIAS - June 14, 2007 116
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive negative: harmful hypocritical inefficient insecure
It was a macabre and hypocritical circus. Why are they being so inefficient ?
subjective: curious, peculiar, odd, likely, probably
MIAS - June 14, 2007 117
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive negative subjective: curious, peculiar, odd, likely, probable
He spoke of Sue as his probable successor. The two species are likely to flower at different times.
MIAS - June 14, 2007 118
Other parts of speech (e.g. Turney & Littman 2003, Esuli & Sebastiani 2006)
Verbs positive: praise, love negative: blame, criticize subjective: predict
Nouns positive: pleasure, enjoyment negative: pain, criticism subjective: prediction
MIAS - June 14, 2007 119
Phrases
Phrases containing adjectives and adverbs (e.g. Turney 2002,
Takamura et al. 2007 )
positive: high intelligence, low cost negative: little variation, many troubles
MIAS - June 14, 2007 120
Patterns
Lexico-syntactic patterns (Riloff & Wiebe 2003) way with <np>: … to ever let China use force to have its
way with … expense of <np>: at the expense of the world’s securty
and stability underlined <dobj>: Jiang’s subdued tone … underlined
his desire to avoid disputes …
MIAS - June 14, 2007 121
How?
How do we identify subjective items?
MIAS - June 14, 2007 122
How?
How do we identify subjective items?
Assume that contexts are coherent
MIAS - June 14, 2007 123
Conjunction
MIAS - June 14, 2007 124
Statistical association
If words of the same orientation like to co-occur together, then the presence of one makes the other more probable
Use statistical measures of association to capture this interdependence Mutual Information (Church & Hanks 1989)
MIAS - June 14, 2007 125
How?
How do we identify subjective items?
Assume that contexts are coherent Assume that alternatives are similarly subjective
MIAS - June 14, 2007 126
How?
How do we identify subjective items?
Assume that contexts are coherent Assume that alternatives are similarly subjective
MIAS - June 14, 2007 127
WordNet
MIAS - June 14, 2007 128
WordNet
MIAS - June 14, 2007 129
WordNet relations
MIAS - June 14, 2007 130
WordNet relations
MIAS - June 14, 2007 131
WordNet relations
MIAS - June 14, 2007 132
WordNet glosses
MIAS - June 14, 2007 133
WordNet examples
MIAS - June 14, 2007 134
How?
How do we identify subjective items?
Assume that contexts are coherent Assume that alternatives are similarly subjective Take advantage of word meanings
MIAS - June 14, 2007 135
*We cause great leaders
Specific papers using these ideas
MIAS - June 14, 2007 137
Hatzivassiloglou & McKeown 1997
1. Build training set: label all adjectives with frequency > 20
Test agreement with human annotators
MIAS - June 14, 2007 138
Hatzivassiloglou & McKeown 1997
1. Build training set: label all adj. with frequency > 20; test agreement with human annotators
2. Extract all conjoined adjectives
nice and comfortable
nice and scenic
MIAS - June 14, 2007 139
Hatzivassiloglou & McKeown 1997
3. A supervised learning algorithm builds a graph of adjectives linked by the same or different semantic orientation
nice
handsome
terrible
comfortable
painful
expensive
fun
scenic
MIAS - June 14, 2007 140
Hatzivassiloglou & McKeown 1997
4. A clustering algorithm partitions the adjectives into two subsets
nice
handsome
terrible
comfortable
painful
expensive
fun
scenicslow
+
MIAS - June 14, 2007 141
Wiebe 2000Learning Subjective Adjectives From Corpora
Learning evaluation and opinion clues
Distributional similarity process, based on manual annotation
Refinement with lexical features Improved results from both
MIAS - June 14, 2007 142
Lin’s (1998) Distributional Similarity
I have a brown dogsubj
mod
obj
det
Word R W I subj havehave obj dogbrown mod dog . . .
MIAS - June 14, 2007 143
Lin’s Distributional Similarity
R W R W R W R WR W R W R W R W R W R W R W R W
Word1 Word2
R={subj, obj, etc.}
MIAS - June 14, 2007 144
Bizarre
strange similar scary unusual fascinatinginteresting curious tragic different contradictory peculiar silly sad absurdpoignant crazy funny comic compellingodd
MIAS - June 14, 2007 145
Experiments
910
MIAS - June 14, 2007 146
Experiments
Seeds Separatecorpus
Distributionalsimilarity
910
Seeds +SimilarWords
MIAS - June 14, 2007 147
Experiments
Seeds Separatecorpus
Distributionalsimilarity
Seeds +SimilarWords9
10S
Words
p(subjective | s)
S > Adj > Majority
MIAS - June 14, 2007 148
Turney 2002a,b
Determine the semantic orientation of each extracted phrase based on their association with seven positive and seven negative words
)()(
)&(log),(
21
21221 wordpwordp
wordwordpwordwordPMI
)_()_(
)_()_(log)( 2 queryphitsquerynNEARwordhits
querynhitsquerypNEARwordhitswordIRPMISO
MIAS - June 14, 2007 149
Turney 2002a,b
Determine the semantic orientation of each extracted phrase based on their association with seven positive and seven negative words
)()(
)&(log),(
21
21221 wordpwordp
wordwordpwordwordPMI
)_()_(
)_()_(log)( 2 queryphitsquerynNEARwordhits
querynhitsquerypNEARwordhitswordIRPMISO
MIAS - June 14, 2007 151
Riloff & Wiebe 2003
Observation: subjectivity comes in many (low-frequency) forms better to have more data
Boot-strapping produces cheap data High-precision classifiers label sentences as subjective or objective Extraction pattern learner gathers patterns biased towards
subjective texts Learned patterns are fed back into high precision classifier
MIAS - June 14, 2007 152
MIAS - June 14, 2007 153
Riloff & Wiebe 2003
Observation: subjectivity comes in many (low-frequency) forms better to have more data
Boot-strapping produces cheap data High-precision classifiers label sentences as subjective or objective Extraction pattern learner gathers patterns biased towards
subjective texts Learned patterns are fed back into high precision classifier
MIAS - June 14, 2007 154
Yu & Hatzivassiloglou 2003Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences
Classifying documents: naïve bayes, words as features Finding opinion sentences:
2 similarity approaches Naïve bayes (n-grams, POS, counts of polar words,
counts of polar sequences, average orientation) Multiple naïve bayes
MIAS - June 14, 2007 155
Yu & Hatzivassiloglou 2003Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinino Sentences
Tagging words and sentences: modified log-likelihood ratio of collocation with pos,
neg adjectives in seed sets Adjectives, adverbs, and verbs provide best
combination for tagging polarity of sentences
MIAS - June 14, 2007 157
Kim & Hovy 2005Automatic Detection of Opinion Bearing Words and Sentences
WordNet-based method for collecting opinion-bearing adjectives and verbs manually constructed strong seed set manually labeled reference sets (opinion-bearing or
not) for synonyms/antonyms of seed set, calculate an
opinion strength relative to reference sets expand further with naïve bayes classifier
MIAS - June 14, 2007 159
Kim & Hovy 2005Automatic Detection of Opinion Bearing Words and Sentences
Corpus-based method (WSJ) Calculate bias of words for particular text genre
(Editorials and Letter to editor)
)(Pr
)(Pr)(
#rob(w)EditorialP
wobalNoneditori
wobEditorialwScore
documentseditorialinwordstotal
documentseditorialinw
MIAS - June 14, 2007 161
Esuli & Sebastiani 2005Determining the semantic orientation of termsthrough gloss analysis
use final sets as gold standard to train a classifier, which uses all or part of the glosses in some format as features
the trained classifier can then be used to label any term that has a gloss with sentiment
w(awful) … w(dire) w(direful) … w(dread) W(dreaded) … …
MIAS - June 14, 2007 162
Esuli & Sebastiani 2006Determining Term Subjectivity and Term Orientation for Opinion Mining
Uses best system of 2005 paper Additional goal of distinguishing neutral from
positive/negative Multiple variations on learning approach, learner, training
set, feature selection The new problem is harder! Their best accuracy is 66%
(83% in 2005 paper)
MIAS - June 14, 2007 163
Suzuki et al. 2006Application of Semi-supervised learning to evaluative expression classification
Automatically extract and filter “evaluative expressions": The storage capacity of this HDD is high.
Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression classifier
based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high] Unfortunately, [the laptop was too expensive].
MIAS - June 14, 2007 164
Suzuki et al. 2006
Automatically extract and filter “evaluative expressions": The storage capacity of this HDD is high.
Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression classifier
based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high] Unfortunately, [the laptop was too expensive].
Evaluation
Attribute
Subject
MIAS - June 14, 2007 165
Suzuki et al. 2006
Automatically extract and filter “evaluative expressions": The storage capacity of this HDD is high.
Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression
classifier based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high] Unfortunately, [the laptop was too expensive].
MIAS - June 14, 2007 166
Suzuki et al. 2006
Comparison of semi-supervised methods: Nigam et al.’s (2000) Naive Baiyes + EM method Naive Bayes + EM + SVM (SVM combined with Naive
Bayes + EM using Fisher kernel) And supervised methods:
Naive Bayes SVM
MIAS - June 14, 2007 167
Suzuki et al. 2006
Features: ‘Phew, [the noise] of [this HDD] is annoyingly high :-(’.
Candidate evaluative expression “Exclamation words” detected by POS tagger Emoticons and their emotional categories Words modifying words in the candidate evaluation expression Words modified by words in the candidate evaluative word
MIAS - June 14, 2007 168
Both Naive Bayes + EM, and Naive Bayes + EM + SVM work better than Naive Bayes and SVM.
Results show that Naive Bayes + EM boosted accuracy regardless of size of labeled data
Using more unlabeled data appeared to give better results. Qualitative analysis of the impact of the semi-supervised approaches by
looking at the top 100 features that had the highest probability P(feature—positive) before and after EM: more contextual features like exclamations, the happy emoticons, a
negation + ’but’, ‘therefore’ + ’interesting’, and ‘therefore‘ + ’comfortable.’
MIAS - June 14, 2007 169
Surely
… we’ve thought of everything by now?
MIAS - June 14, 2007 170
Word senses
Senses
MIAS - June 14, 2007 171
Senses
MIAS - June 14, 2007 172
Non-subjective senses of brilliant
1. Method for identifying brilliant material in paint - US Patent 7035464
2. Halley shines in a brilliant light.
3. In a classic pasodoble, an opening section in the minor mode features a brilliant trumpet melody, while the second section in the relative major begins with the violins.
MIAS - June 14, 2007 173
Andreevskaia and Bergler 2006Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from WordNet Glosses
Using wordnet relations (synonymy, antonymy and hyponymy) and glosses
Classify as positive, negative, or neutral Step algorithm with known seeds:
First expand with relations Next expand via glosses Filter out wrong POS and multiply assigned
Evaluate against General inquirer (which contains words, not word senses)
MIAS - June 14, 2007 175
Andreevskaia and Bergler 2006Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from WordNet Glosses
Disagreements between human labelers as a sign of fuzzy category structure HM and General Inquirer have 78.7% tag agreement
for shared adjectives Find way to measure the degree of centrality of words to
the category of sentiment Net overlap scores correlate with human agreement
MIAS - June 14, 2007 176
Outline
Corpus Annotation Pure NLP
Lexicon development Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis Applications
Product review mining
MIAS - June 14, 2007 177
Recognizing Contextual Polarity in Phrase-Level Sentiment AnalysisWilson, Wiebe, Hoffmann HLT-EMNLP-2005
MIAS - June 14, 2007 178
Most approaches use a lexicon of positive and negative wordsPrior polarity: out of context, positive or negative beautiful positive horrid negative
A word may appear in a phrase that expresses a different polarity in context
Contextual polarity
“Cheers to Timothy Whitfield for the wonderfully horrid visuals.”
Prior Polarity versus Contextual Polarity
MIAS - June 14, 2007 179
Example
Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.
MIAS - June 14, 2007 180
Example
Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.
MIAS - June 14, 2007 181
Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.
Example
prior polarityprior polarityContextual Contextual polaritypolarity
MIAS - June 14, 2007 182
Goal of This Work
Automatically distinguish prior and contextual polarity
MIAS - June 14, 2007 183
Approach
Use machine learning and variety of features
Achieve significant results for a large subset of sentiment expressions
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2
AllInstances
PolarInstances
MIAS - June 14, 2007 184
Outline
Introduction Manual Annotations and Corpus
Prior-Polarity Subjectivity Lexicon Experiments Conclusions
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 185
Manual Annotations
Subjective expressions of the MPQA corpus annotated with contextual polarity
MIAS - June 14, 2007 186
Annotation Scheme
Mark polarity of subjective expressions as positive, negative, both, or neutral
African observers generally approved of his victory while Western governments denounced it.
Besides, politicians refer to good and evil …
Jerome says the hospital feels no different than a hospital in the states.
positive
negative
both
neutral
MIAS - June 14, 2007 187
Annotation Scheme
Judge the contextual polarity of sentiment ultimately being conveyed
They have not succeeded, and will never succeed, in breaking the will of this valiant people.
MIAS - June 14, 2007 188
Annotation Scheme
Judge the contextual polarity of sentiment ultimately being conveyed
They have not succeeded, and will never succeed, in breaking the will of this valiant people.
MIAS - June 14, 2007 189
Annotation Scheme
Judge the contextual polarity of sentiment ultimately being conveyed
They have not succeeded, and will never succeed, in breaking the will of this valiant people.
MIAS - June 14, 2007 190
Annotation Scheme
Judge the contextual polarity of sentiment ultimately being conveyed
They have not succeeded, and will never succeed, in breaking the will of this valiant people.
MIAS - June 14, 2007 191
Corpus
425 documents from MPQA Opinion Corpus 15,991 subjective expressions in 8,984 sentences
Divided into two sets Development set for feature development
66 docs / 2,808 subjective expressions Experiment set
359 docs / 13,183 subjective expressions Divided into 10 folds for cross-validation
MIAS - June 14, 2007 192
Outline
Introduction Manual Annotations and Corpus Prior-Polarity Subjectivity Lexicon
Experiments Conclusions
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 193
Prior-Polarity Subjectivity Lexicon
Over 8,000 words from a variety of sources Both manually and automatically identified Positive/negative words from General Inquirer and
Hatzivassiloglou and McKeown (1997) All words in lexicon tagged with:
Prior polarity: positive, negative, both, neutral Reliability: strongly subjective (strongsubj),
weakly subjective (weaksubj)
MIAS - June 14, 2007 194
Outline
Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Conclusions
MIAS - June 14, 2007 195
Experiments
Give each instance its own label
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Both Steps: BoosTexter AdaBoost.HM 5000 rounds boosting 10-fold cross validation
MIAS - June 14, 2007 196
Definition of Gold Standard
Given an instance inst from the lexicon:
if inst not in a subjective expression:
goldclass(inst) = neutral
else if inst in at least one positive and one negative subjective expression:
goldclass(inst) = both
else if inst in a mixture of negative and neutral:
goldclass(inst) = negative
else if inst in a mixture of positive and neutral:
goldclass(inst) = positive
else: goldclass(inst) = contextual polarity of subjective expression
MIAS - June 14, 2007 197
Features
Many inspired by Polanyi & Zaenen (2004): Contextual Valence Shifters
Example: little threat
little truth Others capture dependency relationships between words
Example:
wonderfully horrid
pos
mod
MIAS - June 14, 2007 198
1. Word features
2. Modification features
3. Structure features
4. Sentence features
5. Document feature
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 199
1. Word features2. Modification features3. Structure features4. Sentence features5. Document feature
Word token terrifies
Word part-of-speechVB
Context that terrifies me Prior Polarity
negative Reliability
strongsubj
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 200
1. Word features
2. Modification features
3. Structure features
4. Sentence features
5. Document feature
Binary features: Preceded by
adjective adverb (other than not) intensifier
Self intensifier Modifies
strongsubj clue weaksubj clue
Modified by strongsubj clue weaksubj clue
Dependency Parse Tree
The human rights
report
poses
a substantial
challenge
…
detadj mod adj
det
subj obj
p
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 201
1. Word features
2. Modification features
3. Structure features
4. Sentence features
5. Document feature
Binary features: In subject [The human rights report]
poses In copular I am confident In passive voice
must be regarded
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
The human rights
report
poses
a substantial
challenge
…
detadj mod adj
det
subj obj
p
MIAS - June 14, 2007 202
1. Word features2. Modification features3. Structure features4. Sentence features5. Document feature
Count of strongsubj clues in previous, current, next sentence
Count of weaksubj clues in previous, current, next sentence
Counts of various parts of speech
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 203
1. Word features
2. Modification features
3. Structure features
4. Sentence features
5. Document feature
Document topic (15) economics health
Kyoto protocol presidential election in Zimbabwe
…
Example: The disease can be contracted if a person is bitten by a certain tick or if a person comes into contact with the blood of a congo fever sufferer.
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 204
75.9
63.4
82.1
40
50
60
70
80
90
Accuracy Polar F Neutral F
Word token
Word + Prior Polarity
All Features
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Results 1a
MIAS - June 14, 2007 205
30
40
50
60
70
80
Polar Recall Polar Precision
Word token
Word + Prior Polarity
All Features
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Results 1b
MIAS - June 14, 2007 206
Step 2: Polarity Classification
Classes positive, negative, both, neutral
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
19,506 5,671
MIAS - June 14, 2007 207
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 208
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
Word token terrifies
Word prior polarity negative
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 209
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
Binary features: Negated
For example: not good does not look very good not only good but amazing
Negated subject
No politically prudent Israeli could support either of them.
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 210
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
Modifies polarity
5 values: positive, negative, neutral, both, not mod
substantial: negative
Modified by polarity
5 values: positive, negative, neutral, both, not mod
challenge: positive
substantial (pos) challenge (neg)
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 211
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
Conjunction polarity
5 values: positive, negative, neutral, both, not mod
good: negative
good (pos) and evil (neg)
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 212
Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter
General polarity shifter
have few risks/rewards
Negative polarity shifter
lack of understanding
Positive polarity shifter abate the damage
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
MIAS - June 14, 2007 213
65.7 65.1
77.2
46.2
30
40
50
60
70
80
90
Accuracy Pos F Neg F Neutral F
Word token
Word + Prior Polarity
All Features
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Results 2a
MIAS - June 14, 2007 214
40
50
60
70
80
90
PosRecall
Pos Prec NegRecall
Neg Prec
Word token
Word + Prior Polarity
All Features
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Results 2b
MIAS - June 14, 2007 215
Ablation experiments removing features:
1. Negated, negated subject
2. Modifies polarity, modified by polarity
3. Conjunction polarity
4. General, negative, positive polarity shifters
Corpus
Lexicon
Neutralor
Polar?
Step 1
ContextualPolarity?
Step 2All
InstancesPolar
Instances
Combination of features is needed to achieve significant results over baseline
MIAS - June 14, 2007 217
Conclusions
Presented a two-step approach to phrase-level sentiment analysis
1. Determine if an expression is neutral or polar
2. Determines contextual polarity of the ones that are polar
Automatically identify the contextual polarity of a large subset of sentiment expression
MIAS - June 14, 2007 218
Discussion: Lexicon <> Context
Objective Subjective
#1 a bunch of #2
#1 condemn #2
headless chicken #1
#1,3,5,6,7 hot #2,4,8
MIAS - June 14, 2007 219
He is not the sharpest knife in the drawer. She is a few fries short of a Happy Meal.
Stephanie McMahon is the next Stalin.
You are no Jack Kennedy.
Whatever.
MIAS - June 14, 2007 220
That was utterly foolish. No one would say that John is smart.
My little brother could have told you that. Senator NN has voted with the president 95% of the
times.
MIAS - June 14, 2007 221
Outline
Corpus Annotation Pure NLP
Lexicon development Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis Applications
Product review mining
MIAS - June 14, 2007 222
Product review mining
MIAS - June 14, 2007 223
Product review mining
Goal: summarize a set of reviews Targeted opinion mining: topic is given Two levels:
Product Product and features
Typically done for pre-identified reviews but review identification may be necessary
MIAS - June 14, 2007 224
Laptop review 1
A Keeper Reviewed By: N.N. on 5/12/2007 Tech Level: average - Ownership: 1 week to 1 month Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight. Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem. Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007 225
Laptop review 1
A Keeper Reviewed By: N.N. on 5/12/2007 Tech Level: average - Ownership: 1 week to 1 month Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight. Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem. Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007 226
Laptop review 2
By N.N. (New York - USA) - See all my reviewsI was looking for a laptop for long time, doing search, comparing brands, technology, cost/benefits etc.... I should say that I am a normal user and this laptop satisfied all my expectations, the screen size is perfect, its very light, powerful, bright, lighter, elegant, delicate... But the only think that I regret is the Battery life, barely 2 hours... some times less... it is too short... this laptop for a flight trip is not good companion... Even the short battery life I can say that I am very happy with my Laptop VAIO and I consider that I did the best decision. I am sure that I did the best decision buying the SONY VAIO
MIAS - June 14, 2007 227
Laptop review 2
By N.N. (New York - USA) - See all my reviewsI was looking for a laptop for long time, doing search, comparing brands, technology, cost/benefits etc.... I should say that I am a normal user and this laptop satisfied all my expectations, the screen size is perfect, its very light, powerful, bright, lighter, elegant, delicate... But the only think that I regret is the Battery life, barely 2 hours... some times less... it is too short... this laptop for a flight trip is not good companion... Even the short battery life I can say that I am very happy with my Laptop VAIO and I consider that I did the best decision. I am sure that I did the best decision buying the SONY VAIO
MIAS - June 14, 2007 230
Some challenges
Available NLP tools have harder time with review data (misspellings, incomplete sentences)
Level of user experience (novice, …, prosumer) Various types and formats of reviews Additional buyer/owner narrative What rating to assume for unmentioned features? How to aggregate positive and negative evaluations? How to present results?
MIAS - June 14, 2007 231
Core problems of review mining
Finding product features Recognizing opinions
MIAS - June 14, 2007 232
Feature finding
Wide variety of linguistic expressions can evoke a product feature … you can't see the LCD very well in sunlight. … it is very difficult to see the LCD. … in the sun, the LCD screen is invisible It is very difficult to take pictures outside in the sun
with only the LCD screen.
MIAS - June 14, 2007 233
Opinions v. Polar facts
Some statements invite emotional appraisal but do not explicitly denote appraisal.
While such polar facts may in a particular context seem to have an obvious value, their evaluation may be very different in another one.
MIAS - June 14, 2007 234
A Keeper Reviewed By: N.N. on 5/12/2007 Tech Level: average - Ownership: 1 week to 1 month Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight. Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem. Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007 235
Use Coherence to resolve orientation of polar facts
Is a sentence framed by two positive sentences likely to also be positive?
Can context help settle the interpretation of inherently non-evaluative attributes (e.g. hot room v. hot water in a hotel context; Popescu & Etzioni 2005) ?
MIAS - June 14, 2007 236
Product-level review-classification Train Naïve Bayes classifier using a corpus of self-
tagged reviews available from major web sites (C|net, amazon)
Refine the classifier using the same corpus before evaluating it on sentences mined from broad web searches
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
MIAS - June 14, 2007 237
Feature selection Substitution (statistical, linguistic)
I called Kodak I called Nikon I called Fuji
Backing off to wordnet synsets Stemming N-grams arbitrary-length substrings
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
I called COMPANY
MIAS - June 14, 2007 238
Feature selection Substitution (statistical, linguistic) Backing off to wordnet synsets
brilliant -> {brainy, brilliant, smart as a whip}
Stemming N-grams arbitrary-length substrings
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
MIAS - June 14, 2007 239
Feature selection Substitution (statistical, linguistic) Backing off to wordnet synsets Stemming
bought them buying them buy them
N-grams arbitrary-length substrings
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
buy them
MIAS - June 14, 2007 240
Feature selection Substitution (statistical, linguistic)
Backing off to wordnet synsets Stemming N-grams
last long enough too hard to
arbitrary-length substrings
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
MIAS - June 14, 2007 241
Feature selection Substitution (statistical, linguistic)
Backing off to wordnet synsets Stemming N-grams arbitrary-length substrings
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
MIAS - June 14, 2007 242
Laplace (add-one) smoothing was found to be best 2 types of test (1 balanced, 1 unbalanced)
SVM did better on Test 2 (balanced data) but not Test 1
Experiments with weighting features did not give better results
Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews
MIAS - June 14, 2007 243
Hu & Liu 2004Mining Opinion Features in Customer Reviews
Here: explicit product features only, expressed as nouns or compound nouns
Use association rule mining technique rather than symbolic or statistical approach to terminology
Extract associated items (item-sets) based on support (>1%)
MIAS - June 14, 2007 244
Hu & Liu 2004Mining Opinion Features in Customer Reviews
Feature pruning compactness
“I had searched for a digital camera for 3 months.” “This is the best digital camera on the market” “The camera does not have a digital zoom”
Redundancy manual ; manual mode; manual setting
MIAS - June 14, 2007 245
Hu & Liu 2004Mining Opinion Features in Customer Reviews
For sentences with frequent feature, extract nearby adjective as opinion
Based on opinion words, gather infrequent features (N, NP nearest to an opinion adjective)
The salesman was easy going and let me try all the models on display.
MIAS - June 14, 2007 246
Yi & Niblack 2005Sentiment mining in WebFountain
MIAS - June 14, 2007 247
Yi & Niblack 2005Sentiment mining in WebFountain
Product feature terms are extracted heuristically, with high precision For all definite base noun phrases,
the NN the JJ NN the NN NN NN …
calculate a statistic based on likelihood ratio test
MIAS - June 14, 2007 249
Yi & Niblack 2005Sentiment mining in WebFountain
Manually constructed Sentiment lexicon: excellent JJ + Pattern database: impress + PP(by; with)
Sentiment miner identifies the best fitting pattern for a sentence based on the parse
MIAS - June 14, 2007 250
Yi & Niblack 2005Sentiment mining in WebFountain
Manually constructed Sentiment lexicon: excellent JJ + Pattern database: impress + PP(by; with)
Sentiment miner identifies the best fitting pattern for a sentence based on the parse
Sentiment is assigned to opinion target
MIAS - June 14, 2007 251
Yi & Niblack 2005Sentiment mining in WebFountain
Discussion of hard cases: Sentences that are ambiguous out of context Cases that did not express a sentiment at all Sentences that were not about the product:
Need to associate opinion and target
MIAS - June 14, 2007 252
Summary
Subjectivity is common in language
MIAS - June 14, 2007 253
Summary
Subjectivity is common in language Recognizing it is useful in many NLP tasks
MIAS - June 14, 2007 254
Summary
Subjectivity is common in language Recognizing it is useful in many NLP tasks It comes in many forms and often is context-dependent
MIAS - June 14, 2007 255
Summary Subjectivity is common in language Recognizing it is useful in many NLP tasks It comes in many forms and often is context-dependent Contextual coherence and distributional similarity are
important linguistic notions in lexicon building A wide variety of features seem to be necessary for
opinion and polarity recognition
MIAS - June 14, 2007 256
Summary
Goals for the future: Fine resolution on opinions (expression level) Integrating the various attributes of subjective
expressions (opinion expressions, sources, targets, etc.)
Additional material
MIAS - June 14, 2007 258
Extensions to the annotation schemeWilson & Wiebe 2005. Annotating Attributions and Private States
I think people are happy because Chavez has fallen.
direct subjective span: are happy source: <writer, I, People> attitude:
inferred attitude span: are happy because Chavez has fallen type: neg sentiment intensity: medium target:
target span: Chavez has fallen
target span: Chavez
attitude span: are happy type: pos sentiment intensity: medium target:
direct subjective span: think source: <writer, I> attitude:
attitude span: think type: positive arguing intensity: medium target:
target span: people are happy because Chavez has fallen
MIAS - June 14, 2007 259
Work on the intensity of private states
Theresa Wilson, Janyce Wiebe and Rebecca Hwa (2006). Recognizing strong and weak opinion clauses. Computational Intelligence, 22 (2), pp. 73-99.
Theresa Wilson. 2007. Ph.D. Thesis. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of private states.
MIAS - June 14, 2007 260
James R. Martin and Peter R.R. White. 2005. The Language of Evaluation: The Appraisal Framework. An approach to evaluation that comes from within the
theory of systemic-functional grammar. Website on this theory maintained by P.R. White:
http://www.grammatics.com/appraisal/index.html
More work related to lexicon building
MIAS - June 14, 2007 262
Alina Andreevskaia and Sabine Bergler. 2006. Sentiment Tag Extraction from WordNet Glosses.
Janyce Wiebe and Rada Mihalcea. 2006. Word Sense and Subjectivity
Riloff, Patwardhan, Wiebe. 2006. Feature Subsumption for Opinion Analysis.
MIAS - June 14, 2007 263
Takamura et al. 2007Extracting Semantic Orientations of Phrases from Dictionary
Use a Potts model to categorize Adj+Noun phrases Targets ambiguous adjectives like low, high, small, large Connect two nouns, if one appears in gloss of other Nodes have orientation values (pos, neg, neu) and are
connected by same or different orientation links
MIAS - June 14, 2007 264
A Sample Lexical Network
costcost
lossloss
expenditureexpenditure sacrificesacrifice
WORD GLOSS
cost loss or sacrifice, expenditure
loss something lost
loselose
MIAS - June 14, 2007 265
Takamura et al 2007
index for node
set of seed words
state of node i
class label of seed word i
constants
Probabilistic Model on the Lexical Network (Potts model)
MIAS - June 14, 2007 266
Takamura et al. 2007
The state of a seed word becomes Neighboring nodes tend to have the same
label.
“low cost” = “low expenditure”
MIAS - June 14, 2007 267
Takamura et al. 2007
Manually labeled adj+noun data provide noun seeds of known orientation
The network assigns orientation to nouns not seen in training data
Further work on review mining
MIAS - June 14, 2007 269
Morinaga et. al. 2002. Mining Product Reputations on the Web
Kobayashi et al. 2004. Collecting Evaluative Expressions for Opinion Extraction
Hu & Liu. 2006. Opinion Feature Extraction Using Class Sequential Rules
MIAS - June 14, 2007 270
Popescu & Etzioni 2005
Report on a product review mining system that extracts and labels opinion expressions their attributes
They use the relaxation-labeling technique from computer vision to perform unsupervised classification satisfying local constraints (which they call neighborhood features)
The system tries to solve several classification problems (e.g. opinion and target finding) at the same time rather than separately.
Applications of Subjectivity and
Sentiment analysis not discussed earlier
MIAS - June 14, 2007 272
Question Answering
MIAS - June 14, 2007 273
Question Answering
Much work on Subjectivity & Sentiment has been motivated by QA.
Yu, H. & Hatzivassiloglou, V. (2003) Kim, S. & Hovy, E. (AAAI-Workshop 2005)
Some QA work has also indicated that making a subjective/objective distinction would be useful for the seemingly objective task of definitional QA
Lita et al. (2005)
MIAS - June 14, 2007 274
Question Answering
Making subjective/objective distinction has been showed to be useful in answering opinion based questions in news text Stoyanov et al.(2005)
Making finer grained distinction of subjective types ( Sentiment and Arguing) further improves the QA system ( world press and online discussion forums) Somasundaran et al. (2007)
MIAS - June 14, 2007 275
Information Extraction
Information Extraction has been used to learn subjective patterns
Wiebe and Riloff (2005) Subjectivity has been shown to improve IE
Riloff et al. (2005)
MIAS - June 14, 2007 276
Summarization
Opinion Summaries from documents have been created Stoyanov & Cardie (2006)
They combine fine grained opinions from the same source to create a source specific summary of opinion
Carenini et al.(IUI-2006) They summarize a large corpora of evaluative text about a single entity (product)
Different aspects of subjectivity analysis have been used to enhance summarizing systems.
Seki et al. (2005) Summarization based on user’s needs (benefits, positive/negative factors,
commentary, etc).
MIAS - June 14, 2007 277
Blog analysis
Analysis of sentiments on Blog posts Chesley et al.(2006)
Perform subjectivity and polarity classification on blog posts
Sentiment has been used for blog analysis Balog et al. (2006)
Discover irregularities in temporal mood patterns (fear, excitement, etc) appearing in a large corpus of blogs
Kale et al. (2007) Use link polarity information to model trust and influence in the blogosphere
Blog sentiment has been used in applications Mishne and Glance (2006)
Analyze Blog sentiments about movies and correlate it with its sales
MIAS - June 14, 2007 278
Human Computer Interaction
Affect sensing Liu et al. (2003)
Human Robot Interaction Tokuhisa & Terashima (2006)
Correlate enthusiasm levels in dialogs with subjective language for human robot interaction
MIAS - June 14, 2007 279
Visualization
Visualization of sentiments Gregory et al. (2007)
Visualize affect distribution in social media (blogs) for a topic.
Gamon et al. (2005) Visualize sentiment and its orientation for a topic from large
number of customer feedback texts.
MIAS - June 14, 2007 280
Trends & Buzz
Stock market Koppel & Shtrimberg(2004)
Correlate positive/negative news stories about publicly traded companies and the stock price changes
Market Intelligence from message boards, forums, blogs. Glance et al. (2005)
MIAS - June 14, 2007 281
Source and Target Finding
MIAS - June 14, 2007 282
Bethard et al. 2004Automatic Extraction of Opinion Propositions and their Holders
Find verbs that express opinions in propositional form, and their holders Still, Vista officials realize they’re relatively fortunate.
Modify algorithms developed in earlier work on semantic parsing to perform binary classification (opinion or not)
Use presence of subjectivity clues to identify opinionated uses of verbs
MIAS - June 14, 2007 283
Choi et al.2005Identifying sources of opinions with conditional random fields and extraction patterns
Treats source finding as a combined sequential tagging and information extraction task
IE patterns are high precision, lower recall Base CRF uses information about noun phrase semantics,
morphology, syntax IE patterns connect opinion words to sources Conditional Random Fields given IE features perform better than
CRFs alone
MIAS - June 14, 2007 284
Kim & Hovy 2006Extracting opinions, opinion holders, andtopics expressed in online news media text
Perform semantic role labeling (FrameNet) for a set of adjectives and verbs (pos, neg)
Map semantic roles to holder and target E.g. for Desiring frame: Experiencer->Holder
Train on FN data, test on FN data and on news sentences collected and annotated by authors’ associates
Precision is higher for topics, recall for holders
MIAS - June 14, 2007 285
Choi, Breck, Cardie 2006 Joint extraction of entities and relations for opinion reocgnition
Find direct expressions of opinions and their sources jointly
Uses sequence-tagging CRF classifiers for opinion expressions, sources, and potential link relations
Integer linear programming combines local knowledge and incorporates constraints
Performance better even on the individual tasks
MIAS - June 14, 2007 286
Further references on Source and Target Finding
Breck & Cardie 2004. Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions.
Bloom et al. 2007. Extracting Appraisal Expressions.