Consumer Satisfaction Rating System Using Sentiment Analysis
A Joint Model of Feature Mining and Sentiment Analysis for Product Review Rating
description
Transcript of A Joint Model of Feature Mining and Sentiment Analysis for Product Review Rating
Jorge Carrillo de Albornoz - ECIR 2011
1
A Joint Model of Feature Mining and Sentiment Analysis for Product Review RatingJorge Carrillo de AlbornozLaura PlazaPablo Gervás Alberto Díaz
Universidad Complutense de Madrid
NIL (Natural Interaction based on Language)
Jorge Carrillo de Albornoz - ECIR 2011
2
MotivationProduct review forums have become
commonplaceReviews are of great interest
◦Companies use them to exploit their marketing-mix
◦ Individuals are interested in others’ opinions when purchasing a product
Manual analysis is unfeasibleTypical NLP tasks:
◦Subjective detection◦Polarity recognition◦Rating inference, etc.
Jorge Carrillo de Albornoz - ECIR 2011
3
MotivationTraditional approaches:
◦Term frequencies, POS, etc.◦Polar expressions
They do not take into account:◦The product features on which the
opinions are expressed◦The relations between them
Jorge Carrillo de Albornoz - ECIR 2011
4
HypothesisHumans have a conceptual
model of what is relevant regarding a certain product
This model influences the polarity and strength of their opinions
It is necessary to combine feature mining and sentiment analysis strategies to◦Automatically extract the important
features◦Quantify the strength of the opinions
about such features
Jorge Carrillo de Albornoz - ECIR 2011
5
The HotelReview Corpus25 reviews from 60 different hotels
(1500 reviews)Each review:
◦The city◦The reviewer nationality◦The date◦The reviewer category◦A score in 0-10 ranking the opinion◦A free-text describing, separately,
what the reviewer liked and disliked
Jorge Carrillo de Albornoz - ECIR 2011
6
The HotelReview CorpusNo relation between the score and the
text describing the user opinion:
Two annotators◦Excellent, Good, Fair, Poor and Very poor◦Good, Fair and Poor
After removing conflicting judgments =1000 reviews
Good location. Nice roof restaurant - (I have stayed in the baglioni more than 5 times before). Maybe reshaping/redecorating the lobby.Noisy due to road traffic. The room was extremely small. Parking awkward. Shower screen was broken and there was no bulb in the bedside light.
Jorge Carrillo de Albornoz - ECIR 2011
7
The HotelReview Corpus
Download: http://nil.fdi.ucm.es/index.php?q=node/456
Jorge Carrillo de Albornoz - ECIR 2011
8
Automatic Product Review RatingStep I: Detecting Salient Product Features
◦ Identifying the features that are relevant to consumers
Step II: Extracting the User Opinion◦ Extracting from the review the opinions
expressed on such featuresStep III: Quantifying the User Opinions
◦ Predicting the polarity of the sentences associated to each feature
Step IV: Predicting the Rating of a Review◦ Translating the product review into a Vector of
Feature Intensities (VFI)
Jorge Carrillo de Albornoz - ECIR 2011
9
Step I: Detecting Salient Product FeaturesObjective: Identifying the product
features that are relevant to consumers Given a set of reviews R={r1, r2, …, rn}:
1. The set of reviews is represented as a graph Vertices = concepts Edges = is a + semantic similarity relations
2. The concepts are ranked according to its salience and a degree-based clustering algorithm is executed
3. The result is a number of clusters where each cluster represent a product feature
Jorge Carrillo de Albornoz - ECIR 2011
10
Step II: Extracting the User Opinion on Each Product FeatureObjective: Locating in the review all
textual mentions related to each product feature1. Mapping the reviews to WordNet concepts2. Associating the sentences to feature
clusters: Most Common Feature (MCF): more WordNet
concepts in common All Common Features (ACF): every feature
with some concept in common Most Salient Feature (MSF): the sentence is
associated to the highest score feature
Jorge Carrillo de Albornoz - ECIR 2011
11
Step III: Quantifying the User OpinionsObjective: Quantifying the opinion
expressed by the reviewer on the different product features
Classifying the sentences of each review into positive or negative
Any polarity classification system may be used
Our system:◦ Concepts rather than terms◦ Emotional categories◦ Negations and quantifiers
12
Step IV: Predicting the Rating of a ReviewObjective: Aggregate all previous
information to provide an overall rating for the review1. Mapping the product review to a VFI2. A VFI is a vector of N+1 values representing
the detected features and the other feature3. Two strategies for assigning values to the
VFI: Binary Polarity (BP): the position in the VFI of the feature
assigned to each sentence is increased or decreased in one according to the polarity of the sentence
Probability of Polarity (PP): the feature position is increased or decreased with the probability calculated by the classifier
Jorge Carrillo de Albornoz - ECIR 2011
[-1.0, 0.0, 0.0, 0.0, …,-1.0, 0.0, 0.0,1.0, 0.0, 1.0, 0.0, …., 1.0]
Jorge Carrillo de Albornoz - ECIR 2011
13
Experimental SetupHotelReview corpus: 1000 reviewsDifferent feature sets:
◦Feature set 1: 50 reviews 24 feature clusters and 114 concepts
◦Feature set 2: 1000 reviews 18 feature clusters and 330 concepts
◦Feature set 3: 1500 reviews 18 feature clusters and 353 concepts
Baselines:◦Carrillo de Albornoz et al. (2010)◦Pang et al. (2002)
Jorge Carrillo de Albornoz - ECIR 2011
14
Experiment 1Objectives:
1. To examine the effect of the product feature set
2. To determine the best heuristic for sentence-to-feature assignment (Most Common Feature, All Common Features and Most Salient Feature)
Task: Three classes classification (Poor, Fair and Good)
We use the Binary Polarity strategy for assigning values to the VFI vector
Jorge Carrillo de Albornoz - ECIR 2011
15
Experiment 1 - Results
Method
Feature set 1 Feature set 2 Feature set 3MCF
ACF MSF
MCF
ACF MSF
MCF
ACF MSF
Logistic 69.8
67.7
69.8
70.4
67.4
70.8
69.1
67.4 70
LibSVM 69 67.1
69.2 69 67.
8 69.2
68.8
67.7 69
FT 66.8
64.2
66.8
66.3
65.2
68.6
68.4
65.8
68.4
Average accuracies for different classifiers, using different feature sets and sentence-
to-feature assignment strategies
Jorge Carrillo de Albornoz - ECIR 2011
16
Experiment 1 - DiscussionFeature set 2 reports the best results
for all classifiersAccuracy differs little across different
feature sets and increasing the number of reviews used for extracting the features does not always improve accuracy
This is due to the fact that users are concerned about a small set of features which are also quite consistent among users
Jorge Carrillo de Albornoz - ECIR 2011
17
Experiment 1 - DiscussionThe Most Salient Feature (MSF)
heuristic for sentence-to-feature assignment produces the best outcome
The Most Common Feature (MCF) heuristic reports very close results
But the All Common Features (ACF) one behaves significantly worse
It seems that only the main feature in each sentence provides useful information for the task
Jorge Carrillo de Albornoz - ECIR 2011
18
Experiment 1IObjectives:
1. To check if the Probability of Polarity strategy produces better results than the Binary Polarity strategy
2. Test the system in a 5-classes prediction taskTasks:
◦Three classes classification (Poor, Fair and Good)
◦Five classes classification (Very Poor, Poor, Fair, Good and Excellent)
We use the Feature set 2 and the MSF strategy for these experiments
Jorge Carrillo de Albornoz - ECIR 2011
19
Experiment 1I - Results
Method 3-classes
5-classes
Logistic 71.7 46.9LibSVM 69.4 45.3FT 66.9 43.7Carrillo de Albornoz et al. [9] 66.7 43.2
Pang et al. [4] 54.2 33.5Average accuracies for different classifiers in the 3-classes and 5-classes
prediction task.
Jorge Carrillo de Albornoz - ECIR 2011
20
Experiment 1I - DiscussionThe Probability of Polarity behaves
significantly better than the Binary Polarity strategy
It allows to captures the degree of negativity/positivity of a sentence, not only its polarity
It is clearly not the same to say The bedcover was a bit dirty than The bedcover was terribly dirty
Jorge Carrillo de Albornoz - ECIR 2011
21
Experiment 1I - DiscussionThe results in the 5-classes
prediction task are considerably lower than in the 3-classes task
This was expected:1. The task is more difficult2. The borderline between Poor-Very poor
and Good-Excellent instances is fuzzyOur system significantly outperforms
both baselines in all tasks
Jorge Carrillo de Albornoz - ECIR 2011
22
Conclusions and Future WorkThe system performs significantly better
than previous approachesThe product features have different impact
on the user opinionUsers are concerned about a relatively small
set of product featuresThe salient features can be easily obtained
from a relatively small set of product reviews and without previous knowledge
Differences between the various Weka classifiers are not marked
Jorge Carrillo de Albornoz - ECIR 2011
23
Conclusions and Future WorkError propagation of the sentence
polarity classifierError assigning sentences to features
◦Not enough information: Dirty. Stinky. Unfriendly. Noisy
◦Co-reference problem: Anyway, everybody else was nice
To evaluate the system over other domains
To translate the system to other language
Jorge Carrillo de Albornoz - ECIR 2011
24
Thank you!Any question?