15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment...

15
10/29/19 1 Sentiment analysis CS440 Positive or negative movie review? unbelievably disappointing Full of zany characters and richly applied satire, and some great plot twists this is the greatest screwball comedy ever filmed It was pathetic. The worst part about it was the boxing scenes. 2

Transcript of 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment...

Page 1: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

1

Sentiment analysis

CS440

Positive or negative movie review?

•  unbelievably disappointing

•  Full of zany characters and richly applied satire, and some great plot twists

•  this is the greatest screwball comedy ever filmed

•  It was pathetic. The worst part about it was the boxing scenes.

2

Page 2: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

2

Twitter sentiment versus Gallup Poll of Consumer Confidence Brendan O'Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010. From Tweets to Polls:

Linking Text Sentiment to Public Opinion Time Series. In ICWSM-2010

Twitter sentiment

Johan Bollen, Huina Mao, Xiaojun Zeng. 2011.

Twitter mood predicts the stock market,

Journal of Computational Science 2:1, 1-8. 10.1016/j.jocs.2010.12.007.

Page 3: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

3

Dow

Jone

s •  CALM is predictive

of DJIA 3 days later

CA

LM

Bollen et al. (2011)

Why sentiment analysis?

•  Movie: is this review positive or negative?

•  Products: what do people think about the new iPhone?

•  Public sentiment: how is consumer confidence?

•  Politics: what do people think about this candidate or issue?

•  Prediction: predict election outcomes or market trends from sentiment

Page 4: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

4

Sentiment Analysis •  Sentiment analysis is the detection of attitudes

“enduring, affectively colored beliefs, dispositions towards objects or persons” Type of attitude

•  From a set of types –  Like, love, hate, value, desire, etc.

•  Or (more commonly) simple weighted polarity: –  positive, negative, neutral, together with strength

Text containing the attitude •  Sentence or entire document

Sentiment Analysis

•  Simplest task: –  Is the attitude of this text positive or negative?

•  More complex:

–  Rank the attitude of this text from 1 to 5

•  Advanced:

–  Detect complex attitude types

Page 5: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

5

Sentiment Classification in Movie Reviews

•  Polarity detection:

–  Is an IMDB movie review positive or negative?

•  Data: Polarity Data 2.0:

–  http://www.cs.cornell.edu/people/pabo/movie-review-data

Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002, 79—86.

IMDB data in the Pang and Lee database

when _star wars_ came out some twenty years ago , the image of traveling throughout the stars has become a commonplace image . […]

when han solo goes light speed , the stars change to bright lines , going towards the viewer in lines that converge at an invisible point .

cool .

“ snake eyes ” is the most aggravating kind of movie : the kind that shows so much potential then becomes unbelievably disappointing . it’s not just because this is a brian depalma film , and since he’s a great director and one who’s films are always greeted with at least some fanfare . and it’s not even because this was a film starring nicolas cage and since he gives a brauvara performance , this film is hardly worth his talents .

✓ ✗

Page 6: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

6

Baseline algorithm

•  Tokenization

•  Feature extraction

•  Classification using different classifiers –  Naïve Bayes

–  MaxEnt

–  SVM

Sentiment tokenization •  Deal with HTML and XML markup

•  Twitter mark-up (names, hash tags)

•  Capitalization (preserve for words in all caps)

•  Phone numbers, dates

•  Emoticons

•  Useful code: –  Christopher Potts sentiment tokenizer http://sentiment.christopherpotts.net/

–  Brendan O’Connor twitter tokenizer https://github.com/brendano/tweetmotif

Page 7: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

7

Extracting features for sentiment classification •  How to handle negation

–  I didn’t like this movie vs

–  I really like this movie

•  Which words to use?

–  Only adjectives

–  All words

•  All words turns out to work better, at least on this data

Negation

Add NOT_ to every word between negation and following punctuation:

didn’t like this movie, but I

didn’t NOT_like NOT_this NOT_movie but I

Das, Sanjiv and Mike Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association Annual Conference (APFA). Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002, 79—86.

Page 8: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

8

Reminder: Naïve Bayes

15

cNB = argmaxc j∈C

P(cj ) P(wi | cj )i∈positions∏

P̂(wi | c) =count(wi,c)+1count(w,c)+ |V |

w∈V∑

Binarized (Boolean feature) Multinomial Naïve Bayes

•  Intuition:

–  For sentiment (and probably for other text classification domains)

–  Word occurrence may matter more than word frequency

•  The occurrence of the word fantastic tells us a lot

•  The fact that it occurs 5 times may not tell us much more.

–  Boolean Multinomial Naïve Bayes

•  Clips all the word counts in each document at 1

16

Page 9: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

9

Boolean Multinomial Naïve Bayes: Learning

•  Calculate P(cj) terms –  For each cj in C do

docsj ← all docs with class =cj

P(cj )←| docsj |

| total # documents| P(wk | cj )←nk +αn+α |V |

•  Textj ← single doc containing all docsj •  For each word wk in Vocabulary

nk ← # of occurrences of wk in Textj

•  From training corpus, extract Vocabulary

•  Calculate P(wk | cj) terms • Remove duplicates in each doc: •  For each word type w in docj • Retain only a single instance of w

Boolean Multinomial Naïve Bayes on a test document d

18

•  First remove all duplicate words from d

•  Then compute NB using the same equation:

cNB = argmaxc j∈C

P(cj ) P(wi | cj )i∈positions∏

Page 10: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

10

Normal vs. Boolean Multinomial NB

Normal Doc Words Class Training 1 Chinese Beijing Chinese c

2 Chinese Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j

Test 5 Chinese Chinese Chinese Tokyo Japan ?

Boolean Doc Words Class Training 1 Chinese Beijing c

2 Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j

Test 5 Chinese Tokyo Japan ?

Binarized (Boolean feature) Multinomial Naïve Bayes

•  Binary seems to work better than full word counts –  This is not the same as Bernoulli Naïve Bayes

•  BNB doesn’t work well for sentiment or other text tasks

B.Pang,L.Lee,andS.Vaithyanathan.2002.Thumbsup?SenDmentClassificaDonusingMachineLearningTechniques.EMNLP-2002,79—86.V.Metsis,I.Androutsopoulos,G.Paliouras.2006.SpamFilteringwithNaiveBayes–WhichNaiveBayes?CEAS2006-ThirdConferenceonEmailandAnD-Spam.K.-M.Schneider.2004.OnwordfrequencyinformaDonandnegaDveevidenceinNaiveBayestextclassificaDon.ICANLP,474-485.JDRennie,LShih,JTeevan.2003.TacklingthepoorassumpDonsofnaivebayestextclassifiers.ICML2003

Page 11: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

11

Problems: What makes reviews hard to classify? •  Subtlety:

–  Perfume review in Perfumes: the Guide:

“If you are reading this because it is your darling fragrance, please wear it at home exclusively, and tape the windows shut.”

21

Thwarted expectations and ordering effects •  “This film should be brilliant. It sounds like a great plot, the

actors are first grade, and the supporting cast is good as well, and Stallone is attempting to deliver a good performance. However, it can’t hold up.”

•  Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented Laurence Fishbourne is not so good either, I was surprised.

Page 12: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

12

Lexicons: annotating words for their sentiment

•  There are many resources that provide annotations of words and their associated sentiment

SentiWordNet Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010 SENTIWORDNET 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. LREC-2010

Home page: https://github.com/aesuli/sentiwordnet

All elements in WordNet automatically annotated for degrees of positivity, negativity, and neutrality/objectiveness

Page 13: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

13

LIWC (Linguistic Inquiry and Word Count) Pennebaker, J.W., Booth, R.J., & Francis, M.E. (2007). Linguistic Inquiry and Word Count: LIWC 2007. Austin, TX

•  Home page: http://www.liwc.net/

•  2300 words, >70 classes

•  Affective Processes –  negative emotion (bad, weird, hate, problem, tough)

–  positive emotion (love, nice, sweet)

•  Cognitive Processes –  Tentative (maybe, perhaps, guess), Inhibition (block, constraint)

•  Pronouns, Negation (no, never), Quantifiers (few, many)

MPQA Subjectivity Cues Lexicon

•  Home page https://mpqa.cs.pitt.edu/lexicons/subj_lexicon/

•  6885 words from

–  2718 positive

–  4912 negative

•  Each word annotated for intensity (strong, weak)

•  GNU GPL

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP-2005. Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP-2003.

Page 14: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

14

How would you use lexicons to predict sentiment?

Lexicons for detecting document affect: simple unsupervised method

19.7 • OTHER TASKS: PERSONALITY 15

Class Words in 1-star reviews Class Words in 5-star reviewsNegative worst, rude, terrible, horrible,

bad, awful, disgusting, bland,tasteless, gross, mediocre, over-priced, worse, poor

Positive great, best, love(d), delicious,amazing, favorite, perfect, excel-lent, awesome, friendly, fantastic,fresh, wonderful, incredible, sweet,yum(my)

Negation no, not Emphatics/universals

very, highly, perfectly, definitely, ab-solutely, everything, every, always

1Pl pro we, us, our 2 pro you3 pro she, he, her, him Articles a, thePast verb was, were, asked, told, said, did,

charged, waited, left, tookAdvice try, recommend

Sequencers after, then Conjunct also, as, well, with, andNouns manager, waitress, waiter, cus-

tomer, customers, attitude, waste,poisoning, money, bill, minutes

Nouns atmosphere, dessert, chocolate,wine, course, menu

Irrealismodals

would, should Auxiliaries is/’s, can, ’ve, are

Comp to, that Prep, other in, of, die, city, mouthFigure 19.11 The top 50 words associated with one–star and five-star restaurant reviews in a Yelp dataset of900,000 reviews, using the Monroe et al. (2008) method (Jurafsky et al., 2014).

In such situations, lexicons can be used in a simple rule-based algorithm forclassification. The simplest version is just to use the ratio of positive to negativewords: if a document has more positive than negative words (using the lexicon todecide the polarity of each word in the document), it is classified as positive. Oftena threshold l is used, in which a document is classified as positive only if the ratiois greater than l . If the sentiment lexicon includes positive and negative weights foreach word, q+

w and q�w , these can be used as well. Here’s a simple such sentiment

algorithm:

f+ =X

w s.t. w2positivelexicon

q+w count(w)

f� =X

w s.t. w2negativelexicon

q�w count(w)

sentiment =

8>>><

>>>:

+ if f+f� > l

� if f�f+ > l

0 otherwise.

(19.12)

If supervised training data is available, these counts computed from sentiment lex-icons, sometimes weighted or normalized in various ways, can also be used as fea-tures in a classifier along with other lexical or non-lexical features. We return tosuch algorithms in Section 19.8.

19.7 Other tasks: Personality

Many other kinds of affective meaning can be extracted from text and speech. Forexample detecting a person’s personality from their language can be useful for di-personality

alog systems (users tend to prefer agents that match their personality), and can play

Sentiment = + if f+ > f-

Page 15: 15 sentiment analysis - cs.colostate.educs440/fall19/slides/15_sentiment_analysis.pdfSentiment Analysis • Sentiment analysis is the detection of attitudes “enduring, affectively

10/29/19

15

How to deal with stars?

1.  Treat as a classification problem

2.  Use regression or ordinal regression

Summary

•  Generally modeled as classification or regression task

•  Comments:

–  Negation is important

–  Using all words (in Naïve Bayes) works well for some tasks

–  Finding subsets of words may help in other tasks

–  Hand-built polarity lexicons

•  Naive Bayes is a good baseline, but other classifiers typically work better