Ling 570 Day 9: Text Classification and Sentiment Analysis 1.

1

Ling 570 Day 9:Text Classification and

Sentiment Analysis

2

Outline

Questions on HW #3 Discussion of Project #1 Text Classification Sentiment Analysis

3

Project #1

4

Your goal: political text analysis

Take a document, predict whether it is more Republican or Democratic

We have harvested blog posts from: The Democratic National Committee The Republican National Committee Fox News The Huffington Post

5

First task

Can you reconstruct the party affiliation of a given document?

We will gather some novel posts, held out from your training data

You predict the political part of each of these posts to the best of your ability

6

Second task

Is the media biased? Is a particular news source biased?

Using the classifier that you’ve learned, see whether documents from a particular news source seem to be left- or right-leaning.

What features are most indicative of the party of a given document?

Do you think your classifier is effective in detecting media bias? Why or why not?

7

Text Classification

8

Text classification

Also known as “text categorization”

Often an instance of supervised learning Start with a large body of pre-classified data Try to map new documents into one of these classes

9

train

classes – often hierarchical

test

Text classification

linguisticsphonology“acoustics”

“IPA”…

morphology“morpheme”“template”

…

…

brewingvarieties

“IPA”“hefeweizen”

..

…

“We transcribed the samples of

this unusual language in IPA…”

10

Classification methods

Manual Yahoo, back in the

day, had a manually curated hierarchy of useful web content

Can be very accurate, consistent…

…but it’s very expensive

Need to move to automatic methods

11

Text categorization

Given: A document

is the set of all possible documents But we need to represent them usefully somehow! Often times we have a high-dimensional representation

A fixed set of categories Determine:

The category of some new document

12

Machine learning:Supervised classification Given:

Instance descriptions A set of outcomes A training set

Determine: A classifier

Classification is a clear instance of this problem

13

Bayesian methods

Learning based on probability theory Bayes theorem plays a big role

Build a generative model that approximates how data is produced

Prior probability of each class Model gives a posterior probability of output given

inputs Naïve Bayes:

Bag of features (generally words) Assumes each feature is independent

14

Bag of words representation

𝑓 ¿According to a study published in the October issue of Current Biology entitled 'Spontaneous human speech mimicry by a cetacean,' whales can talk. Not to burst your bubble ring or anything, but now that we've suckered you in, let's clarify what we mean by 'talk.' A beluga whale named 'NOC' (he was named for an incredibly annoying sort of Canadian gnat), that lived at the National Marine Mammal Foundation (NMMF) in San Diego up until his death five years ago, had been heard making some weird kinds of vocalizations. At first, nobody was sure that it was him: divers hearing what sounded like 'two people were conversing in the distance just out of range for our understanding.' But then one day, a diver in NOC's tank left the water after clearly hearing someone tell him to get out. It wasn't someone, though: it was some whale, and that some whale was NOC.

15

Bag of words representation

𝑓 ( [h𝑤 𝑎𝑙𝑒 3𝑁𝑜𝑐 3𝑡𝑎𝑙𝑘 2

𝑛𝑎𝑚𝑒𝑑 2h𝑒𝑎𝑟𝑖𝑛𝑔 2h𝑤 𝑎𝑙𝑒𝑠 1

𝑤𝑎𝑡𝑒𝑟 1𝑣𝑜𝑐𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛𝑠 1

⋮ ⋮

])

16

Bayes’ Rule for text classification

For a document and a class

17



18



So…

19



So…

Divide by to get:

20

Back to text classification

Pr (𝑆𝑐𝑖𝑒𝑛𝑐𝑒|[ 𝑤h𝑎𝑙𝑒 3𝑁𝑜𝑐 3

𝑣𝑜𝑐𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛𝑠 1⋮ ⋮

])=Pr ([ 𝑤 h𝑎𝑙𝑒 3

𝑁𝑜𝑐 3𝑣𝑜𝑐𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛𝑠 1

⋮ ⋮]|𝑆𝑐𝑖𝑒𝑛𝑐𝑒)Pr (𝑆𝑐𝑖𝑒𝑛𝑐𝑒 )

Pr ( [ 𝑤h𝑎𝑙𝑒 3𝑁𝑜𝑐 3

𝑣𝑜𝑐𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛𝑠 1⋮ ⋮ ])

21


is just

22


is just the count of science docs / total docs

23


is just the count of science docs / total docsBut how do we model the whole matrix ?

24

The “Naïve” part of Naïve Bayes

Assume that everything is conditionally independent given the class:

25

Return of smoothing…

is…

26


is… The number of science documents containing whale Divided by the number of science documents

27



What is ?

28



What is ? 0! Need to smooth…

29



What is ? 0! Need to smooth… What would Add-One (Laplace) smoothing look like?

30

Exercisedocument label

TRAIN Apple poised to unveil iPad Mini TECH

Apple product leaks TECH

Researchers test apple, cherry trees SCIENCE

TEST Dangerous apple, cherry pesticides ?

31

Benchmark dataset #1:20 newsgroups 18,000 documents from 20 distinct newsgroups

A now mostly unused technology for sharing textual information, with hierarchical topical groups

comp.graphicscomp.os.ms-windows.misccomp.sys.ibm.pc.hardwarecomp.sys.mac.hardwarecomp.windows.x

rec.autosrec.motorcyclesrec.sport.baseballrec.sport.hockey

sci.cryptsci.electronicssci.medsci.space

misc.forsale talk.politics.misctalk.politics.gunstalk.politics.Mideast

talk.religion.miscalt.atheismsoc.religion.christian

32

Results:

33

Evaluation methods

“macro”-averaging: Compute Precision and Recall for each category Take average of per-category precision and recall values

gold category totals

news sports arts science

predicted category news 15 7 0 1 23

sports 6 17 0 0 23

arts 0 0 4 0 4

science 1 0 0 7 8

totals 22 24 4 8

34

Evaluation methods

There is also “macro”-averaging: Compute Precision and Recall for each category Take average of per-category precision and recall values




sports 6 17 0 0 23

arts 0 0 4 0 4

science 1 0 0 7 8

totals 22 24 4 8

35

gold category prec


predicted category news 15 7 0 1 0.65

sports 6 17 0 0 0.74

arts 0 0 4 0 1.00

science 1 0 0 7 0.88

recall 0.68 0.71 1.00 0.88

36

Evaluation methods

What is the analogue of precision and recall for multiclass classification?

We can still compute precision and recall as usual for each category Then add up these numbers to compute precision and recall This is called “micro-averaging”, and focuses on document level

accuracy

Gold standard

all other categories

Classifier output

all other categories

37

gold category prec

news sports arts science 0.82

predicted category news 15 7 0 1 0.65

sports 6 17 0 0 0.74

arts 0 0 4 0 1.00

science 1 0 0 7 0.88

recall 0.82 0.68 0.71 1.00 0.88

38

news Gold standardnews other

Classifier output

news 15 8

other 7

sports Gold standardsports other

Classifier output

sports 17 6

other 7

science Gold standardsci other

Classifier output

sci 7 1

other 1

arts Gold standardarts other

Classifier output

arts 4 0

other 0




sports 6 17 0 0 23

arts 0 0 4 0 4

science 1 0 0 7 8

totals 22 24 4 8

39

news Gold standardnews other

Classifier output

news 15 8

other 7

sports Gold standardsports other

Classifier output

sports 17 6

other 7

science Gold standardsci other

Classifier output

sci 7 1

other 1

arts Gold standardarts other

Classifier output

arts 4 0

other 0

total Gold standardcorrect other recall

Classifier output

correct 43 15 0.74

other 15

prec 0.74

40

Feature selection

41

Sentiment Analysis

42

Sentiment Analysis

Consider movie reviews: Given a review from a site like Rotten Tomatoes, try to

detect if the reviewers liked it Some observations:

Humans can quickly and easily identify sentiment Easier that performing topic classification, often Suspicion: Certain words may be indicative of

sentiment

43

Simple Experiment[Pang, Lee, Vaithyanathan, EMNLP 2002] Ask two grad students to come up with a list of words

changed with sentiment Create a very simple, deterministic classifier based on this:

Count number of positive and negative hits Break ties to increase accuracy

44

Simple Experiment[Pang, Lee, Vaithyanathan, EMNLP 2002] Ask two grad students to come up with a list of words changed with

sentiment Create a very simple, deterministic classifier based on this:

Count number of positive and negative hits Break ties to increase accuracy

Compare to automatically extracted lists

45

Toward more solid machine learning Prior decision rule was very heuristic

Just count the number of charged words Ties are a significant issue

What happens when we shift to something more complex?

46

Toward more solid machine learning Prior decision rule was very heuristic

Just count the number of charged words Ties are a significant issue

What happens when we shift to something more complex?

Naïve Bayes Maximum Entropy (aka logistic regression, aka log-

linear models) Support Vector Machines

47

Experimental results

Baseline was 69% accuracy.

Here we get just under 79% with all words, just using frequency.

What happens when we use binary features instead?

48


Unigrams are pretty good – what happens when we add bigrams?

49


Why are just bigrams worse than unigrams and bigrams together?

50


51


52

Domain Adaptation

53

What are we learning?

Primary features are unigrams.

For a movie, “unpredictable” is a good thing – likely to be an interesting thriller.

54

What are we learning?

Primary features are unigrams.

For a movie, “unpredictable” is a good thing – likely to be an interesting thriller.

For a dishwasher, “unpredictable” is not so great.

55

Domain shift[Blitzer, Dredze, Pereira, 1997] What happens when we move to another domain?

Gather Amazon reviews from four domains: Books, DVDs, Electronics, Kitchen appliances

Each review has Rating (0-5 stars) Reviewer name and location Product name Review (title, date, and body)

Ratings <3 become negative, >3 become positive; remainder considered ambiguous and discarded

1000 positive and 1000 negative in each domain

56

Domain adaptation effects

Books DVDs Electronics Kitchen50

55

60

65

70

75

80

85

90

Books DVDs Electronics Kitchen

57



55

60

65

70

75

80

85

90


58



55

60

65

70

75

80

85

90


59

Lessons learned

Be careful with your classifier: Just because you get high accuracy on one test set

doesn’t guaranteed high accuracy on another test set Domain adaptation can be a major hit

What can we do about this?

60

Lessons learned

Be careful with your classifier: Just because you get high accuracy on one test set

doesn’t guaranteed high accuracy on another test set Domain adaptation can be a major hit

What can we do about this? Supervised approaches – say we have a little bit of

training in the NEW domain, and a lot in the OLD domain, learn features from both (“Frustratingly Easy”, Daume 2007)

Unsupervised approaches (Structural Correspondence Learning)

Ling 570 Day 9: Text Classification and Sentiment Analysis 1.

Documents

Transcript of Ling 570 Day 9: Text Classification and Sentiment Analysis 1.