Aspect Miner: Fine-grained, feature-level opinion mining from rated review corpora
-
Upload
stelios-karabasakis -
Category
Education
-
view
113 -
download
0
description
Transcript of Aspect Miner: Fine-grained, feature-level opinion mining from rated review corpora
Aspect MinerFine-grained feature level opinion mining
from rated review corpora
Stelios KarabasakisDept. of Informatics and Telecommunications National and Kapodistrian University of Athens
in association with the Knowledge Discovery in Databases Laboratory kddlab.di.uoa.gr
MSc Thesis Defense | February 2012
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 2
INTRODUCTION
What is it? The task of recognizing and classifying the opinions and sentiments expressed in unstructured text.
Opinion Mining: an overview
Use cases• product comparison• opinion summarization• opinion-aware recommendation systems• opinion-aware online advertising• reputation management• business intelligence• government intelligence
Opinion sources• news• blogs• reviews• user comments• social networks• forums• discussion groups
Our focus in this work
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 3
INTRODUCTION
Reviews
movies books
goods services
restaurantshotels
• Popular form of user generated content» consumers use them to
make informed choices» businesses use them to
gauge and monitor consumer sentiment
• Covering many distinct domains, such as…
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 4
INTRODUCTION
Ratings
• Every online review typically carries a rating» picked by the review author» summarizes the sentiment of
the text
• Corpora of rated reviews are» abundant on the web» potentially useful for
supervised opinion mining» largely ignored in the literature!
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 5
INTRODUCTION
Not as simple as counting positive vs. negative words
It is pointless to discuss why Hitchcock was a genius.
Distinct opinions about different topics in the same sentenceThe top-notch production values are not enough to distract from a clichéd story that lacks heart and soul.
Semantics of subjective expressions are domain-dependentunpredictable plot twist, gloomy atmosphere (movies) unpredictable service quality, gloomy room (hotels)
Opinion Mining is challenging
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 6
INTRODUCTION
classification dimensions• subjectivity: factual vs. subjective statements• polarity: positive vs. negative sentiment• intensity: weak vs. strong sentiment
classification granularity• binary• multiclass
Opinion Mining is a text classification problem
Motivating questionHow can we train a system to distinguish among multiple degrees of sentiment?
?
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 7
INTRODUCTION
In “Game of Thrones” (2011), the transition from book to screen is remarkably successful. The carefully chosen location and cast, the top-notch cinematography and the seamless-ness of its narrative come together brilliantly. The new HBO show offers compelling drama, even when rehashing old fantasy themes.
Classification levels
positive
document level
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 8
INTRODUCTION
In “Game of Thrones” (2011), the transition from book to screen is remarkably successful. The carefully chosen location and cast, the top-notch cinematography and the seamless-ness of its narrative come together brilliantly. The new HBO show offers compelling drama, even when rehashing old fantasy themes.
Classification levels
positive
positive
positive
sentence level
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 9
INTRODUCTION
Classification levels
adaptation: positive
production: positivecast: positivedirection: positiveplot: positive
serialization: positivesubject: negative
In “Game of Thrones” (2011), the transition from book to screen is remarkably successful. The carefully chosen location and cast, the top-notch cinematography and the seamless-ness of its narrative come together brilliantly. The new HBO show offers compelling drama, even when rehashing old fantasy themes.
feature level features = domain-specific ratable properties
Motivating questionHow can we identify feature terms and the features they refer to?
?
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 10
INTRODUCTION
Produce rich, fine-grained, feature-oriented review summaries
by analyzing reviews at the sentence level and aggregating the results
Problem description
“Avatar” (2009) aggregated summary of 90 reviews
aspect mentions sentiment mean sentiment dispersiondirectio
n 217 9/10 STRONGLY POSITIVE
17%
UNANIMOUS AGREEMENT
story 152 8/10 POSITIVE32%
GENERAL AGREEMENT
acting 177 4/10 WEAKLY NEGATIVE
56%
MIXED REACTION
Sample summary
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 11
INTRODUCTION
a sentiment lexiconmulticlass and adapted to the target domain
Solution components
term prior sentiment _masterpiece 10 (very strongly positive)good 8 (positive)mediocre 5 (very weakly negative)terrible 2 (strongly negative)
feature term featureprotagonist CASTperformance CASTdeliver CASTcamera DIRECTIONcinematography DIRECTIONdialogue WRITINGscript WRITING
a feature lexiconfor the target domain
and a set of linguistic rules for sentence classification
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 12
INTRODUCTION
The Aspect Miner system
Training subsystem
Term classifier
Feature identifier
Sentence classifier
Lexical Analyzer
Index of terms
Sentiment lexicon
Feature lexicon
Result: Feature-level sentiments
Training corpus(rated reviews)
Text to classify
(a proof-of concept implementation of our approach)
Key features: modular architecture, unsupervised,domain agnostic, configurable granularity
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 13
INTRODUCTION
Aspect Miner implementation*
• Implemented in Java with» NekoHTML for scraping» JDBC/MySQL for dataset storage» Lucene as a lexical analysis API and for indexing» Wordnet & JWNL for lemmatization» Stanford Parser for POS-tagging & typed dependency parsing» Mallet’s LDA implementation for topic modeling» GraphViz for visualizations
* source code (MIT-licensed) available from github.com/skarabasakis/ImdbAspectMiner
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 14
INTRODUCTION
107.646 movie reviews from IMDB.com, rated 1-10 stars
Training dataset*
*available as an SQL dump from http://db.tt/vAthzJRL
review length (words)
# re
view
s
mean = 291 wordsmedian = 228 words
Sentiment Lexicon ConstructionDesigning a fine-grained term classifier
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 16
SENTIMENT LEXICON
A term is a (base form, part of speech) tuple
» part of speech {VERB, NOUN, ADJECTIVE, ADVERB}» a term represents all inflected forms and spellings of a word
e.g. {choose, chooses, chose, chosen, …} [choose VERB]
{localise, localize, …} [localize VERB]
» terms can be compounde.g. [work out VERB] [common sense NOUN]
[meet up with VERB] [as a matter of fact ADVERB]
Terms
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 17
SENTIMENT LEXICON
Purpose: to extract terms from texts
» Identifies the base form of words & compounds• Uses Wordnet to look up base forms
» Eliminates non-subjective words• Stop words including very common terms (be,have,…)• Named Entities (i.e. proper nouns)• all articles, pronouns, prepositions etc.
» Eliminates words that would be misleading for sentiment classification• Comparatives & superlatives• Words within a negation scope
Lexical analyzerTraining corpus(rated reviews)
Le
xica
l An
aly
zer
Tokenization
POS tagging
Named Entity identification
Lemmatization
Comparatives annotation
Negation scope resolution
Stop word removal
Bags of terms(one per document)
Open-class word filtering
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 18
SENTIMENT LEXICON
Lexical analysis exampleThe most dramatic moment in the Sixth Sense does not occur until the
final minutes and the jaw dropping twist Shyamalan has been building up to.
Eliminate
Lemmatize
Get indexable terms
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 19
SENTIMENT LEXICON
Lexicon-based approach• Prior sentiment inferred from lexical associations
(synonyms, antonyms, hypernyms etc.) in a dictionary• High accuracy, limited coverage• Notable example: Sentiwordnet (Esuli & Sebastiani 2006)
Corpus-based approach• Prior sentiment inferred from correlation patterns
(and, or, either…or, but etc.) in a training corpus• Extended coverage, lower accuracy• Notable examples: Hatzivassiloglou & McKeown 1997, Turney & Littman
2003, Popescu & Etzioni 2005, Ding Liu & Yu 2008
Previous approaches to term classification
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 20
SENTIMENT LEXICON
• Requires a training set of rated reviews
• Prior sentiment inferred from the distribution of ratings among all the reviews where a term occurs, i.e. the rating histogram of the term
Ratings-based term classification
Our proposal: a ratings-based approachpositive term negative term
neutral term polysemous term
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 21
SENTIMENT LEXICON
IMDB dataset: Ratings distribution
rating
# reviews # terms
# reviews# terms
Caution: Ratings are not evenly distributed across the training corpus.
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 22
SENTIMENT LEXICON
Why? Weighting is necessary to» eliminate training set biases» make rating frequencies comparable to each other
How? Multiply every rating frequency in a histogram with that rating’s weight wr , calculated as follows:
» cr := cumulative term count of all reviews with rating r
» We pick wr in such a way that wr∙ cr are equal for all r
• Most predominant rating in the dataset has wr =1
• The less frequent the rating, the higher its weight
Rating frequency weighting
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 23
SENTIMENT LEXICON
Some sample histogramsextracted from the IMDB dataset
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 24
SENTIMENT LEXICON
input: weighted rating histogram for term output: one or more* sets of significant ratings
RC
* if term is polysemous
Designing a term classifier
RC
A weighted mean function can condense RC into a single rating.
71079
8107759
C
C
Rr r
Rr r
C f
rfr
This rating indicates the term’s sentiment.
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 25
SENTIMENT LEXICON
For a term to be neutral, its rating histogram must approximate a uniform distribution
Neutrality criterion
thr
rf
rf
tr
tr
)(max
)(min1
where 0 < thr ≤1
max
min
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 26
SENTIMENT LEXICON
Picks the histogram’s peak rating as the only significant rating
Term classification schemes
Pros Simplest classifier possible. Useful as a comparison baseline.Surprisingly capable at classifying polarity (almost 2/3 accurate)
Cons Can’t detect polysemyPoor at classifying intensity
Scheme 1: Peak ClassifierRC
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 27
SENTIMENT LEXICON
Term classification schemes
Pros Better at classifying intensityMakes an attempt at detecting polysemy
Cons Weak terms can be mistaken for polysemous
Scheme 2: Positive/Negative Area Classifier (PN) All ratings above a cutoff
frequency are significant Cutoff frequency should
be set a little bit above the frequency average.
Returns separate sets for positive and negative ratings
)(f*. t r11
)(ft r
RC+
RC−
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 28
SENTIMENT LEXICON
Term classification schemes
Pros Avoids detecting false polysemyAvoids biases exhibited by the other classification schemes
Scheme 3: Widest Window Classifier (WW) Looks for windows of
consecutive significant ratings Ratings are added to windows
from most to least frequent Significant rating windows must
satisfy 2 constraints minimum coverage:
W windows must contain at least 1−(2W)-1 of samples
be as wide as possible Returns as many rating classes
as the windows it detects
RC1
RC2
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 29
SENTIMENT LEXICON
Classifier evaluation: Ratings Distribution
PEAK
PN
WW
We classified 33.000 terms that appear ≥5 times in the IMDB dataset.
Conclusion: WW classifier distributes rating classes more evenly
Distribution of primary rating classes for each classifier
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 30
SENTIMENT LEXICON
Classifier evaluation: Polarity
We evaluate against a reference lexicon of 5272 terms based on the MPQA and General Inquirer subjectivity lexicons.
Accuracy Precision Recall F1-Score
POSITIVE 55.5% 44.2% 49.2% PEAK 63.6%
NEGATIVE 67.3% 65.3% 66.3%
POSITIVE 62.4% 58.4% 60.4% PN 66.2%
NEGATIVE 68.4% 72.3% 70.3%
POSITIVE 70.4% 86.2% 77.5% WW 70.1%
NEGATIVE 69.6% 60.5% 64.8%
POSITIVE 63.6% 61.3% 62.4% SentiWordnet 73.2%
NEGATIVE 83.6% 48.3% 61.3%
WW is the most accurate of the 3 proposed classifiers
But not as accurate than SentiWordnet
However, WW is more accurate for domain-specific terms
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 31
SENTIMENT LEXICON
Classifier evaluation: Intensity
We evaluate against a test set of 443 strong + 323 weak terms based on the General Inquirer subjectivity lexicon.
Using the WW classifier to classify intensity:
78% of strong terms are classified 3 and above
83% of weak terms are classified 3 and below
0.0%
10.0%
20.0%
30.0%
40.0%
1 2 3 4 5
Τιμή Έντασης WW
Πο
σο
στό
όρ
ων
WEAK STRONG
Intensity class in WW lexicon
% te
rms
in W
W le
xico
n
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 32
SENTIMENT LEXICON
A reusable sentiment lexicon for the movie review domain* downloadable from
github.com/skarabasakis/ImdbAspectMiner/blob/master/imdb_sentiment_lexicon.xls
The Aspect Miner sentiment lexicon*
Feature IdentificationUsing topic models for feature discovery
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 34
FEATURE IDENTIFICATION
The traditional approach: discovery through heuristics• frequency: commonly occurring noun phrases are often features
(Hu & Liu 2004)
• co-occurrence: terms commonly found near subjective expressions may be features (Kim & Hovy 2006, Qiu et al. 2011)
• language patterns: in phrases such as 'F of P' or 'P has F‘, P is a product and F is a feature (Popescu & Etzioni 2005)
• background knowledge: user annotations, ontologies, search engine results, Wikipedia data…
An up-and-coming approach: topic modeling
Approaches to feature identification
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 35
FEATURE IDENTIFICATION
Probabilistic Topic Models can model the abstract topics that occur in a set of documents
Topic Modeling
documents are mixtures of topics
topics are distributions over words
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 36
FEATURE IDENTIFICATION
Probabilistic topic models• require that the user specifies a number of topics
» Topics are just numbers – their semantic interpretation is not the model’s concern
• make an assumption about the probability distribution of topics• define a probabilistic procedure for generating documents from topics
» by inverting this procedure, we can infer topics from documents
A popular topic model: Latent Dirichlet Allocation (LDA)• assumes that topics follow a Dirichlet prior distribution
» i.e. each document is associated with just a small number of topics
Topic Modeling
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 37
FEATURE IDENTIFICATION
CARCHASESHOOTVEHICLECOPDRIVEKILLSTREETBULLETROBBERY
WARHEROATTACKGROUPAIRPLANEBUNCHSOLDIERKILLBOMBENEMY
POLICECASEMYSTERYVICTIMSOLVEMURDEROFFICERSUSPECTDETECTIVECRIME
Here are a few sample topics we got from running LDA on the IMDB dataset
Topics vs. Features
Motivating questionFeatures are a form of topics. Can we use topic models to discover features?
?
WARHEROATTACKGROUPAIRPLANEBUNCHSOLDIERKILLBOMBENEMY
POLICECASEMYSTERYVICTIMSOLVEMURDEROFFICERSUSPECTDETECTIVECRIME
ROLEACTORPERFORMANCEPLAYLEADCAST SUPPORTACTRESS SHINESTAR
SCRIPTIDEADIALOGUEWRITEPLOTSCREENPLAYCOME UPCRAFTEXPLAINHOLE
CARCHASESHOOTVEHICLECOPDRIVEKILLSTREETBULLETROBBERY
These topics are features.They are useful to us
These topics are themes.We are not interested in them
ROLEACTORPERFORMANCEPLAYLEADCAST SUPPORTACTRESS SHINESTAR
SCRIPTIDEADIALOGUEWRITEPLOTSCREENPLAYCOME UPCRAFTEXPLAINHOLE
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 38
FEATURE IDENTIFICATION
Problem. Topics are global, features are local
Solution. Train topic model on shorter segments (e.g. sentences) rather than full documents.
Problem. Running LDA on such short segments produces noisy topics
Solution. Implement a bootstrap aggregation scheme to filter the noise:
1. Train N topic models from different subsets of dataset
2. Merge similar topics across models to produce a single meta-model» Intuition: Valid feature-topics should occur in >1 models and share many common top
terms. Noisy topics should be isolated to specific models
Feature identification with LDA
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 39
FEATURE IDENTIFICATION
Topic Similarity for topics Tm, Tn
» More common terms with higher probabilities higher similarity
Merging topics
COMEDY 0.200JOKE 0.099LAUGH 0.096FUN 0.088FORMULA 0.025
COMEDY 0.180PARODY 0.168SATIRE 0.099JOKE 0.061RIDICULE 0.054
COMEDY 0.380PARODY 0.168JOKE 0.160LAUGH 0.096SATIRE 0.099
+ =
FUN 0.088RIDICULE 0.054FORMULA 0.025
discarded
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 40
FEATURE IDENTIFICATION
To merge 2 topic sets • Merge every topic of set A to most similar topic from set B
» but only if that similarity is above average similarity
To merge N topic sets• Merge first two, then merge the result with the third etc.• At the end
» discard topics with a low merging degree» If same term ends up in >1 topics, only keep it in the topic where it
has the highest probability
Merging topic models
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 41
FEATURE IDENTIFICATION
Movie feature lexicon
56 topics, manually labeled with 18 labels
Sentence classificationUtilizing language structure for contextual sentiment estimation and feature targeting
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 43
SENTENCE-LEVEL ANALYSIS
Sentiment
We define a mapping functionto convert ratings to sentiment classes
(preferably 1:1)
1
2
3
5
6
8
9
10
7
4
-1
+1
mbinary: R10 " S1
1
2
3
5
6
8
9
10
7
4-2
+2
m3: R10 " S3
-3
-1
+1
+3
1
2
3
5
6
8
9
10
7
4
m5: R10 " S5
-5
-4
-3
-1
+1
+3
+4
+5
+2
-2
Sentiment: a (polarity, intensity) tuple, where» polarity {+,−}» intensity {1, 2, …, n} 2n classes
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 44
SENTENCE-LEVEL ANALYSIS
Typed dependencies are binary grammatical relations between word pairs in a sentence(de Marneffe et al., 2006)
Typed Dependencies
Typed dependency trees are • semantically richer than syntax trees• easier to process, because content words are connected directly
rather than through function words
Natalie Portman comes off as very believable, gaining empathy from the audience.
amod(relations, binary)
type governor dependent
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 45
SENTENCE-LEVEL ANALYSIS
Dependency types
Stanford Parser’s representation defines a hierarchy of 48 dependency types
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 46
SENTENCE-LEVEL ANALYSIS
Contextual sentiment estimation
Our model. We empirically developed and formally defined• 6 outcome functions that model types of word interactions• 42 dependency rules that cover all possible dependency patterns
Motivating questionWhat is the contextual sentiment of a dependency, given the prior sentiment of its constituents?
?
Examples
infmod(best/+2, avoid/−4) −4
xcomp(avoid/−4, watching/+2) −2
advmod(disappointing/−2, increasingly/+3) −3
It is best to avoid watching any of the increasingly disappointing sequels.
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 47
SENTENCE-LEVEL ANALYSIS
Outcome functions
Models an interaction whereUNCHANGED base term imposes the sentiment
Ιt seems that they ran out of budget.
STRONGER stronger term imposes the sentiment
a mighty talent wasted in mass produced rom-coms
AVG both terms contribute equally to the sentiment
intelligent and ambitious
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 48
SENTENCE-LEVEL ANALYSIS
Outcome functions
Models an interaction where
INTENSIFY modifier increases the intensity of the base
increasingly disappointing sequels
REFLECT modifier overrides polarity, increases or decreases intensity of base
impossible to enjoy unless you lower your expectations
NEG modifier diminishes or negates the base
not a masterpiece, but not bad either
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 49
SENTENCE-LEVEL ANALYSIS
td(pgov, pdep) outcome_base
Dependency Rules: General form
conj(*,*) AVG_DEP
advmod({n,a,r},*) INTENSIFY_GOV
amod(*,{too}) NEGATIVE_GOV
type label term patterns
A pattern may specify:• a list of allowed parts of
speech• a white list of specific terms
outcome functionone of the following:
UNCHANGED NEGATEDSTRONGER AVGINTENSIFY REFLECTPOSITIVE NEGATIVE
base specifierGOV or DEP
Examples
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 50
SENTENCE-LEVEL ANALYSIS
Aspect Miner dependency rule setgov dep
Td pos wlist pos wlist
outcome base
1. Negation
1.1 neg * * * * NEGATE GOV
1.2.1
1.2.2
1.2.3
1.2.4
1.2.5
1.2.6
det
prt
advmod
dobj
nsubj
dep
* * * negTerms1 NEGATE GOV
1.3 pobj * negTerms1 * * NEGATE DEP
1.4 aux * * * negAux2 NEGATE GOV
1 negTerms = {n't, no, not, never, none, nothing, nobody, noone, nowhere, without, hardly, barely, rarely, seldom, against, minus, sans} 2 negAux = {should, could, would, might, ought}
2. Subjects
2.1.1
2.1.2
nsubj
nsubjpass * * * * INTENSIFY GOV
2.2.1
2.2.2
csubj
csubjpass * * * * REFLECT GOV
3. Objects
3.1.1
3.1.2
dobj
dobj
*
*
negVerbs3
*
*
*
*
*
NEGATE
REFLECT
DEP
GOV
3.2 iobj * * * * UNCHANGED GOV
3.3 pobj * * * * UNCHANGED DEP
3 negVerbs = {avoid, cease, decline , forget, fail, miss , neglect, refrain, refuse, stop}
gov dep td
pos wlist pos wlist outcome base
4. Modifiers
4.1.1
4.1.2
advmod
Amod * * * {enough} POSITIVE GOV
4.2.1
4.2.2
advmod
amod * * * {too} NEGATIVE GOV
4.3 advmod v * * * REFLECT GOV
4.4 advmod n,a,r * * * INTENSIFY GOV
4.5 amod * * * * REFLECT GOV
4.6 infmod a * * * REFLECT GOV
4.7 infmod v,n,r * * * INTENSIFY DEP
4.8 a * * * REFLECT DEP
4.9 partmod
v,n,r * * * STRONGER DEP
4.10 quantmod * * * * INTENSIFY GOV
4.11 prt * * * * STRONGER GOV
4.13 prep * * * {like} UNCHANGED GOV
4.12 prep * * * * REFLECT GOV
5. Clausal Modifiers
5.1 advcl a * * * REFLECT DEP
5.2 advcl v,n,r * * * UNCHANGED DEP
5.3 purpcl * * * * UNCHANGED DEP
6. Clausal complements
6.1.1
6.1.2
6.1.3
ccomp
xcomp
acomp
* * * * REFLECT GOV
6.2.1
6.2.2
6.2.3
conj
appos
parataxis
* * * * AVG GOV
6.3 dep * * * * STRONGER DEP
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 51
SENTENCE-LEVEL ANALYSIS
Initialization • Generate dependency tree from sentence• Annotate subjective terms with prior polarities from sentiment lexicon• Annotate feature terms with labels from feature lexicon
Sentiment estimation• Apply closest matching rule to every dependency relation in the tree
» The sentiment of the dependency replaces previous sentiment of the governor node» Dependencies are processed in reverse postfix order (bottom to top and right to left)
Feature targeting• The scope of a feature term is a subtree that contains the term and goes
» all the way down to the leaves» all the way up to the closest clausal dependency
• the sentiment at the root of the subtree gets assigned to the feature
Sentence classification algorithm
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 52
SENTENCE-LEVEL ANALYSIS
Sentence classification example
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 53
SENTENCE-LEVEL ANALYSIS
Test set: Sentence polarity dataset by Pang & Lee, 2002(5331 positive + 5331 negative sentences from movie reviews)
ResultsPolarity classification is accurate for
71.5% of positive sentences76.9% of negative sentences74.2% of all sentences
Analysis of error causes39.0% inaccurate dependency rule28.5% misclassified term (or we picked the wrong sense)21.5% erroneous sentence parsing 8.5% ambiguous sentence 2.5% dependency rules applied in the wrong order
Sentence polarity evaluation
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 54
SENTENCE-LEVEL ANALYSIS
Comparative evaluationReference Method Accuracy
Linguistic methods
Nakagawa, Irui & Kurohashi, 2005 majority voting 62,9%
Ikeda & Takamura, 2008 majority voting with negations 65.8%
Aspect Miner dependency rules 74.2%
Learning based methods
Andreevskaia & Bergler, 2008 naïve bayes 69.0%
Nakagawa, Irui & Kurohashi, 2005 SVM (bag-of-features) 76.4%
Arora, Mayfield et al., 2010 genetic programming 76.9%
Ikeda & Takamura, 2008 SVM (sentence-wise learning with polarity shifting + ngrams)
77.0%
Nakagawa, Irui & Kurohashi, 2005 dependency tree CRFs 77.3%
Conclusion: Our method fares well among linguistic techniques, but does not match the accuracy of learning based methods
ConclusionsPutting it all together
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 56
CONCLUSIONS
Training subsystem
Term classifierTraining corpus(rated reviews)
Lexi
cal A
naly
zer
Tokenization
POS tagging
Named Entity identification
Lemmatization
Comparatives annotation
Negation scope resolution
Stop word removal
Sentence classifier
Dependency tree(s)
Dependency parsing
Bags of terms(one per document)
Feature identifier
Index of terms
Corpus statistics collection
Indexing
Dependency Rule set
Open-class word filtering
Term Histogramgeneration
PEAKclassifier
PNclassifier
WWclassifier
Tra
inin
g s
et
pa
rtiti
on
ing
...
...
...
...
...
partition 1
partition 2
partition N-1
partition N
LDA
Sentiment lexicon
TΜ1 TΜ2 TΜΝ-1 TΜΝ...
Topic models
Assisted labeling
Aggregation
Feature lexicon
Text toclassify
Sentence & Feature Classification
Result:Feature-sentiment
pairs
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 57
CONCLUSIONS
• We showed the feasibility of granular prior polarity classification using review ratings» and developed a classifier that
achieved at least 70% accuracy on the training dataset
• We suggested a bagging-inspired meta-algorithm for discovering feature topics with LDA
Summary of contributions
• We developed a reusable sentiment lexicon and feature lexicon for the movie review domain
• We created a set of linguistic rules and developed a methodology that is capable fine-grained feature-level classification of sentences» and achieved 74.2% accuracy
for polarity classification on our test dataset.
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 58
CONCLUSIONS
Term classification• Assigning a special class to intensifier terms• Per-feature polysemy resolution
Feature identification• Named entities as features• Applying multi-grain topic models for
discovery of local topics, e.g. MG-LDA (Titov & MacDonald, 2008)
Sentence-level classification• Supervised learning of rules.
Replace manually-made set of rules with a set of rules inferred from frequent dependency patterns.
Suggested Improvements
intensifier term
Stelios Karabasakis Feb 2012Aspect Miner: Fine-grained feature-level opinion mining from rated review corpora 59
CONCLUSIONS
B. Liu, “Sentiment analysis and subjectivity,” Handbook of Natural Language Processing,, pp. 978–1420085921, 2010.
B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Foundations and Trends in Information Retrieval, vol. 2, no. 1-2, pp. 1–135, 2008.
A. Esuli and F. Sebastiani, “Sentiwordnet: A publicly available lexical resource for opinion mining,” in Proceedings of LREC, 2006, vol. 6, pp. 417–422.
V. Hatzivassiloglou and K. R. McKeown, “Predicting the semantic orientation of adjectives,” in Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, 1997, pp. 174–181.
P. Turney, M. L. Littman, and others, “Measuring praise and criticism: Inference of semantic orientation from association,” in ACM Transactions on Information Systems (TOIS), 2003.
A. M. Popescu and O. Etzioni, “Extracting product features and opinions from reviews,” in Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, pp. 339–346.
ReferencesFor a complete list of references, see the full report (in greek)
http://j.mp/AspectMinerM. Hu and B. Liu, “Mining and summarizing customer reviews,” in
Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 168–177.
X. Ding, B. Liu, and P. S. Yu, “A holistic lexicon-based approach to opinion mining,” in Proceedings of the international conference on Web search and web data mining, 2008, pp. 231–240.
I. Titov and R. McDonald, “Modeling online reviews with multi-grain topic models,” in Proceeding of the 17th international conference on World Wide Web, 2008, pp. 111–120.
T. Nakagawa, K. Inui, and S. Kurohashi, “Dependency tree-based sentiment classification using CRFs with hidden variables,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010, pp. 786–794.
A. Andreevskaia and S. Bergler, “When specialists and generalists work together: Overcoming domain dependence in sentiment tagging,” ACL-08: HLT, 2008.
D. Ikeda and H. Takamura, “Learning to shift the polarity of words for sentiment classification,” Comp.Intelligence, vol. 25, no. 1, pp. 296–303, 2008.