Text Categorization using Naïve Bayesmausam/courses/col772/spring2018/.../04-tex… · Text...
Transcript of Text Categorization using Naïve Bayesmausam/courses/col772/spring2018/.../04-tex… · Text...
1
Text Categorization
using Naïve Bayes
Mausam
(based on slides of Dan Weld, Dan
Jurafsky, Prabhakar Raghavan, Hinrich
Schutze, Guillaume Obozinski, David D.
Lewis, Fei Xia)
2
Categorization
• Given:
– A description of an instance, xX, where X is
the instance language or instance space.
– A fixed set of categories:
C={c1, c2,…cn}
• Determine:
– The category of x: c(x)C, where c(x) is a
categorization function whose domain is X and
whose range is C.
County vs. Country?
3
Who wrote which Federalist papers?
• 1787-8: anonymous essays try to convince
New York to ratify U.S Constitution: Jay,
Madison, Hamilton.
• Authorship of 12 of the letters in dispute
• 1963: solved by Mosteller and Wallace
using Bayesian methods
James Madison Alexander Hamilton
Male or female author?• The main aim of this article is to propose an exercise in stylistic analysis
which can be employed in the teaching of English language. It details the
design and results of a workshop activity on narrative carried out with
undergraduates in a university department of English. The methods proposed
are intended to enable students to obtain insights into aspects of cohesion and
narrative structure: insights, it is suggested, which are not as readily obtainable
through more traditional techniques of stylistic analysis.
• My aim in this article is to show that given a relevance theoretic approach to
utterance interpretation, it is possible to develop a better understanding of
what some of these so-called apposition markers indicate. It will be argued
that the decision to put something in other words is essentially a decision
about style, a point which is, perhaps, anticipated by Burton-Roberts when he
describes loose apposition as a rhetorical device. However, he does not justify
this suggestion by giving the criteria for classifying a mode of expression as a
rhetorical device.S. Argamon, M. Koppel, J. Fine, A. R. Shimoni, 2003. “Gender, Genre, and Writing Style in Formal Written Texts,” Text, volume 23, number 3,
pp. 321–346
Female writers use
more first person/second person pronouns
more gender laiden third person pronouns
(overall more personalization)
Positive or negative movie review?
• unbelievably disappointing
• Full of zany characters and richly applied
satire, and some great plot twists
• this is the greatest screwball comedy ever
filmed
• It was pathetic. The worst part about it was
the boxing scenes.
•6
What is the subject of this article?
• Antogonists and
Inhibitors
• Blood Supply
• Chemistry
• Drug Therapy
• Embryology
• Epidemiology
• …•7
MeSH Subject Category Hierarchy
?
MEDLINE Article
Text Classification
• Assigning documents to a fixed set of categories, e.g.
• Web pages
– Yahoo-like classification
– Assigning subject categories, topics, or genres
• Email messages
– Spam filtering
– Prioritizing
– Folderizing
• Blogs/Letters/Books
– Authorship identification
– Age/gender identification
• Reviews/Social media
– Language Identification
– Sentiment analysis
– …
Classification Methods:
Hand-coded rules
• Rules based on combinations of words or other features
– spam: black-list-address OR (“dollars” AND “have been selected”)
• Accuracy can be high
– If rules carefully refined by expert
• But building and maintaining these rules is expensive
10
Learning for Text Categorization
• Hard to construct text categorization functions.
• Learning Algorithms:
– Bayesian (naïve)
– Neural network
– Logistic Regression
– Nearest Neighbor (case based)
– Support Vector Machines (SVM)
18
Bayesian Methods
• Learning and classification methods based on probability theory.
– Bayes theorem plays a critical role in probabilistic learning and classification.
– Uses prior probability of each category given no information about an item.
• Categorization produces a posteriorprobability distribution over the possible categories given a description of an item.
Bayes’ Rule Applied to Documents and
Classes
P(c | d) =P(d | c)P(c)
P(d)
• For a document d and a class c
Naïve Bayes Classifier (I)
cMAP = argmaxcÎC
P(c | d)
= argmaxcÎC
P(d | c)P(c)
P(d)
= argmaxcÎC
P(d | c)P(c)
MAP is “maximum a
posteriori” = most likely
class
Bayes Rule
Dropping the
denominator
Naïve Bayes Classifier (II)
cMAP = argmaxcÎC
P(d | c)P(c)
Document d
represented as
features x1..xn)()|,,,(argmax 21 cPcxxxP nCc
Naïve Bayes Classifier (IV)
How often does this class
occur?
)()|,,,(argmax 21 cPcxxxPc nCc
MAP
O(|X|n•|C|) parameters
We can just count the
relative frequencies in a
corpus
Could only be estimated if a very, very large number of training examples was available.
Multinomial Naïve Bayes Independence
Assumptions
)|,,,( 21 cxxxP n
• Bag of Words assumption: Assume position doesn’t matter
The bag of words representation
I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun… It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet.
γ( )=c
I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun… It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it againwhenever I have a friend who hasn't seen it yet.
γ( )=c
The bag of words representation
The bag of words representation:
using a subset of words
x love xxxxxxxxxxxxxxxx sweetxxxxxxx satirical xxxxxxxxxxxxxxxxxxxxx great xxxxxxxxxxxxxxxxxxxxxxxxxx fun xxxxxxxxxxxxxxxxx whimsical xxxxromantic xxxx laughingxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx recommend xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx several xxxxxxxxxxxxxxxxxxxxxx happy xxxxxxxxx againxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
γ( )=c
γ( )=cgreat 2
love 2
recommend 1
laugh 1
happy 1
... ...
The bag of words representation
Planning GUIGarbageCollection
Machine
Learning NLP
parser
tag
training
translation
language...
learning
training
algorithm
shrinkage
network...
garbage
collection
memory
optimization
region...
Test
document
parser
language
label
translation
…
Bag of words for document
classification
...planning
temporal
reasoning
plan
language...
?
Multinomial Naïve Bayes Classifier
)()|,,,(argmax 21 cPcxxxPc nCc
MAP
cNB = argmaxcÎC
P(c j ) P(x | c)xÎX
Õ
Multinomial Naïve Bayes Independence
Assumptions
)|,,,( 21 cxxxP n
• Bag of Words assumption: Assume position doesn’t matter
• Conditional Independence: Assume the feature probabilities P(xi|cj) are independent given the class c.
)|(...)|()|()|()|,,( 3211 cxPcxPcxPcxPcxxP nn
Applying Multinomial Naive Bayes
Classifiers to Text Classification
cNB = argmaxcjÎC
P(c j ) P(xi | c j )iÎpositions
Õ
positions all word positions in test document
Learning the Multinomial Naïve Bayes Model
• First attempt: maximum likelihood estimates
– simply use the frequencies in the data
P̂(wi | c j ) =count(wi,c j )
count(w,c j )
wÎV
å
P̂(c j ) =doccount(C = c j )
Ndoc
• Create mega-document for topic j by concatenating all docs in this topic– Use frequency of w in mega-document
Parameter estimation
fraction of times word wi appears among all words in documents of topic cj
P̂(wi | c j ) =count(wi,c j )
count(w,c j )
wÎV
å
Problem with Maximum Likelihood
• What if we have seen no training documents with the word fantastic and classified in the topic positive (thumbs-up)?
• Zero probabilities cannot be conditioned away, no matter the other evidence!
P̂("fantastic" positive) = count("fantastic", positive)
count(w, positive
wÎV
å ) = 0
cMAP = argmaxc P̂(c) P̂(xi | c)i
Õ
Sec.13.3
Laplace (add-1) smoothing for Naïve Bayes
P̂(wi | c) =count(wi,c)+1
count(w,c)+1( )wÎV
å
=count(wi,c)+1
count(w,c
wÎV
å )æ
èçç
ö
ø÷÷ + V
P̂(wi | c) =count(wi,c)
count(w,c)( )wÎV
å
Multinomial Naïve Bayes: Learning
• Calculate P(cj) terms– For each cj in C do
docsj all docs with class =cj
P(wk | c j ) ¬nk +a
n+a |Vocabulary |
P(c j ) ¬| docs j |
| total # documents|
• Calculate P(wk | cj) terms• Textj single doc containing all docsj
• For each word wk in Vocabularynk # of occurrences of wk in Textj
• From training corpus, extract Vocabulary
47
Naïve Bayes Time Complexity
• Training Time: O(|D|Ld + |C||V|)) where Ld is the average length of a document in D.
– Assumes V and all Di , ni, and nij pre-computed in O(|D|Ld) time during one pass through all of the data.
– Generally just O(|D|Ld) since usually |C||V| < |D|Ld
• Test Time: O(|C| Lt) where Lt is the average length of a test document.
• Very efficient overall, linearly proportional to the time needed to just read in all the data.
48
Easy to Implement
• But…
• If you do… it probably won’t work…
Probabilities: Important Detail!
We are multiplying lots of small numbers
Danger of underflow!
0.557 = 7 E -18
Solution? Use logs and add!
p1 * p2 = e log(p1)+log(p2)
Always keep in log form
49
Generative Model for Multinomial Naïve Bayes
•50
c=China
X1=Shanghai X2=and X3=Shenzhen X4=issue X5=bonds
Naïve Bayes and Language Modeling
• Naïve bayes classifiers can use any sort of feature
– URL, email address, dictionaries, network features
• But if, as in the previous slides
– We use only word features
– we use all of the words in the text (not a subset)
• Then
– Naïve bayes has an important similarity to language
modeling.
•51
Each class = a unigram language model
• Assigning each word: P(word | c)
• Assigning each sentence: P(s|c)=Π P(word|c)
0.1 I
0.1 love
0.01 this
0.05 fun
0.1 film
…
I love this fun film
0.1 0.1 .05 0.01 0.1
Class pos
P(s | pos) = 0.0000005
Sec.13.2.1
Naïve Bayes as a Language Model
• Which class assigns the higher probability to s?
0.1 I
0.1 love
0.01 this
0.05 fun
0.1 film
Model pos Model neg
filmlove this funI
0.10.1 0.01 0.050.10.10.001 0.01 0.0050.2
P(s|pos) > P(s|neg)
0.2 I
0.001 love
0.01 this
0.005 fun
0.1 film
Sec.13.2.1
Example: Naïve Bayes in Spam Filtering
• SpamAssassin Features:– Mentions Generic Viagra
– Online Pharmacy
– Mentions millions of (dollar) ((dollar) NN,NNN,NNN.NN)
– Phrase: impress ... girl
– From: starts with many numbers
– Subject is all capitals
– HTML has a low ratio of text to image area
– One hundred percent guaranteed
– Claims you can be removed from the list
– 'Prestigious Non-Accredited Universities'
– http://spamassassin.apache.org/tests_3_3_x.html
Advantages
• Simple to implement
– No numerical optimization, matrix algebra, etc
• Efficient to train and use
– Easy to update with new data
– Fast to apply
• Binary/multi-class
• Good in domains with many equally important features
– Decision Trees suffer from fragmentation in such cases –
especially if little data
• Comparatively good effectiveness with small training sets
• A good dependable baseline for text classification
– But we will see other classifiers that give better accuracy
55
Disadvantages
• Independence assumption wrong
– Absurd estimates of class probabilities
• Output probabilities close to 0 or 1
– Thresholds must be tuned; not set analytically
• Generative model
– Generally lower effectiveness than
discriminative techniques
56
Experimental Evaluation
Question: How do we estimate the
performance of classifier on unseen data?
• Can’t just at accuracy on training data – this
will yield an over optimistic estimate of
performance
• Solution: Cross-validation
• Note: this is sometimes called estimating
how well the classifier will generalize
57
Evaluation: Cross Validation
• Partition examples into k disjoint sets
• Now create k training sets
– Each set is union of all equiv classes except one
– So each set has (k-1)/k of the original training data Train
Test
Test
Test
…
58
Cross-Validation (2)
• Leave-one-out
– Use if < 100 examples (rough estimate)
– Hold out one example, train on remaining examples
• 10-fold
– If have 100-1000’s of examples
• M of N fold
– Repeat M times
– Divide data into N folds, do N fold cross-validation
59
Evaluation Metrics
• Accuracy: no. of questions correctly answered
• Precision (for one label): accuracy when classification = label
• Recall (for one label): measures how many instances of a label were
missed.
• F-measure (for one label): harmonic mean of precision & recall.
• Area under Precision-recall curve (for one label): vary parameter to
show different points on p-r curve; take the area
60
Precision & Recall
P
N
“P” “N”
TP FN
FP TN
Predicted
Act
ual
Two class situation
FP
FPTP
Multi-class situation:
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)
F-measure = 2pr/(p+r)
61
A typical precision-recall curve
62
Construct Better Features
• Key to machine learning is having good
features
• In gen 2 ML, large effort devoted to
constructing appropriate features
• Ideas??
63
Issues in document representation
• Cooper’s vs. Cooper vs. Coopers.
• Full-text vs. full text vs. {full, text} vs. fulltext.
• résumé vs. resume.
Cooper’s concordance of Wordsworth was published in
1911. The applications of full-text retrieval are legion:
they include résumé scanning, litigation support and
searching published journals on-line.
slide from Raghavan, Schütze,
Larson
Punctuation
• Ne’er: use language-specific, handcrafted
“locale” to normalize.
• State-of-the-art: break up hyphenated
sequence.
• U.S.A. vs. USA
• a.out
slide from Raghavan, Schütze,
Larson
Numbers
• 3/12/91
• Mar. 12, 1991
• 55 B.C.
• B-52
• 100.2.86.144
– Generally, don’t represent as text
– Creation dates for docs
slide from Raghavan, Schütze,
Larson
Possible Feature Ideas
• Look at capitalization (may indicated a
proper noun)
• Look for commonly occurring sequences
• E.g. New York, New York City
• Limit to 2-3 consecutive words
• Keep all that meet minimum threshold (e.g.
occur at least 5 or 10 times in corpus)
67
Case folding
• Reduce all letters to lower case
• Exception: upper case in mid-sentence
– e.g., General Motors
– Fed vs. fed
– SAIL vs. sail
slide from Raghavan, Schütze,
Larson
Thesauri and Soundex
• Handle synonyms and spelling variations
– Hand-constructed equivalence classes
• e.g., car = automobile
slide from Raghavan, Schütze,
Larson
Spell Correction
• Look for all words within (say) edit distance
3 (Insert/Delete/Replace) at query time
– e.g., arfiticial inteligence
• Spell correction is expensive and slows the
processing significantly
– Invoke only when index returns zero matches?
slide from Raghavan, Schütze,
Larson
Lemmatization
• Reduce inflectional/variant forms to base form
– am, are, is be
– car, cars, car's, cars' car
the boy's cars are different colors
the boy car be different color
72
Stemming
• Are there different index terms?
– retrieve, retrieving, retrieval, retrieved, retrieves…
• Stemming algorithm:
– (retrieve, retrieving, retrieval, retrieved, retrieves)
retriev
– Strips prefixes of suffixes (-s, -ed, -ly, -ness)
– Morphological stemming
• Problems: sand / sander & wand / wander
Copyright © Weld 2002-2007 72
73
Stemming Continued
• Can reduce vocabulary by ~ 1/3
• C, Java, Perl versions, python, c#www.tartarus.org/~martin/PorterStemmer
• Criterion for removing a suffix – Does "a document is about w1" mean the same as
– a "a document about w2"
• Problems: sand / sander & wand / wander
• Commercial SEs use giant in-memory tables
Copyright © Weld 2002-2007 73
•74
Features
• Domain-specific features and weights: very important in real performance
• Upweighting: Counting a word as if it occurred twice:
– title words (Cohen & Singer 1996)
– first sentence of each paragraph (Murata, 1999)
– In sentences that contain title words (Ko et al, 2002)
Sec. 15.3.2
Properties of Text
• Word frequencies - skewed distribution
• `The’ and `of’ account for 10% of all words
• Six most common words account for 40%
From [Croft, Metzler & Strohman 2010] 75
Associate Press Corpus `AP89’
From [Croft, Metzler & Strohman 2010] 76
Middle Ground
• Very common words bad features
• Language-based stop list:
words that bear little meaning
20-500 wordshttp://www.dcs.gla.ac.uk/idom/ir_resources/linguistic_utils/stop_words
• Subject-dependent stop lists
• Very rare words also bad features
Drop words appearing less than k times / corpus
77
Word Frequency
• Which word is more indicative of document similarity?
– ‘book,’ or ‘Rumplestiltskin’?
– Need to consider “document frequency”--- how frequently the
word appears in doc collection.
• Which doc is a better match for the query “Kangaroo”?
– One with a single mention of Kangaroos… or a doc that
mentions it 10 times?
– Need to consider “term frequency”--- how many times the
word appears in the current document.
78
TF x IDF
)/log(* kikik nNtfw
C in T term of frequency document inverse idf
D document in T term of frequencytf
D document in k termT
kk
ikik
ik
nNidf
kk log
kk T contain that C indocuments of number then
C collection the indocuments of number total N
79
Inverse Document Frequency
• IDF provides high values for rare words and
low values for common words
• Add 1 to avoid 0.
41
10000log
698.220
10000log
301.05000
10000log
010000
10000log
80
TF-IDF normalization
• Normalize the term weights
– so longer docs not given more weight (fairness)
– force all values to fall within a certain range: [0, 1]
t
k kik
kikik
nNtf
nNtfw
1
22 )]/log(1[)(
))/log(1(
81
Term-Weighted Complement Naïve Bayes
82
Multi-class Problems
•91
•92
• Most (over)used data set, 21,578 docs (each 90 types, 200 tokens)
• 9603 training, 3299 test articles (ModApte/Lewis split)
• 118 categories
– An article can be in more than one category
– Learn 118 binary category distinctions
• Average document (with at least one category) has 1.24 classes
• Only about 10 out of 118 categories are large
Common categories
(#train, #test)
Evaluation: Classic Reuters-21578 Data Set
• Earn (2877, 1087) • Acquisitions (1650, 179)• Money-fx (538, 179)• Grain (433, 149)• Crude (389, 189)
• Trade (369,119)• Interest (347, 131)• Ship (197, 89)• Wheat (212, 71)• Corn (182, 56)
Sec. 15.2.4
•93
Reuters Text Categorization data set (Reuters-21578) document
<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981"
NEWID="798">
<DATE> 2-MAR-1987 16:51:43.42</DATE>
<TOPICS><D>livestock</D><D>hog</D></TOPICS>
<TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE>
<DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off tomorrow,
March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions
on a number of issues, according to the National Pork Producers Council, NPPC.
Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future
direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to
endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said.
A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry,
the NPPC added. Reuter
</BODY></TEXT></REUTERS>
Sec. 15.2.4
Precision & Recall
P
N
“P” “N”
TP FN
FP TN
Predicted
Act
ual
Two class situation
FP
FPTP
Multi-class situation:
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)
F-measure = 2pr/(p+r)
94
Micro-‐ vs. Macro-‐Averaging
95
• If we have more than one class, how do we combine
multiple performance measures into one quantity?
• Macroaveraging
– Compute performance for each class, then average.
• Microaveraging
– Collect decisions for all classes, compute contingency table,
evaluate
Precision & Recall
Multi-class Multi-label situation:
96
Precision(class 1) = 251/(Column1)
Recall(class 1) = 251/(Row1)
F-measure(class 1)) = 2piri/(pi+ri)
Precision(class i) = TPi/(TPi+FPi)
Recall(class i) = TPi/(TPi+FNi)
F-measure(class i) = 2piri/(pi+ri)
Aggregate
Average Macro Precision = Σpi/N
Average Macro Recall = Σri/N
Average Macro F-measure = 2pMrM/(pM+rM)
Average Micro Precision = ΣTPi/ ΣiColi
Average Micro Recall = ΣTPi/ ΣiRowi
Average Micro F-measure = 2pμrμ /(pμ+rμ)
Precision & Recall
Multi-class situation:
97
Precision(class 1) = 251/(Column1)
Recall(class 1) = 251/(Row1)
F-measure(class 1)) = 2piri/(pi+ri)
Precision(class i) = TPi/(TPi+FPi)
Recall(class i) = TPi/(TPi+FNi)
F-measure(class i) = 2piri/(pi+ri)
Aggregate
Average Macro Precision = Σpi/N
Average Macro Recall = Σri/N
Average Macro F-measure = 2pMrM/(pM+rM)
Average Micro Precision = ΣTPi/ ΣiColi
Average Micro Recall = ΣTPi/ ΣiRowi
Average Micro F-measure = 2pμrμ /(pμ+rμ)
Aren’t μ prec and μ recall the same?
Missed predictions
Classifier hallucinations
Multi-Class Classification
• What?
– Converting a k-class problem to a binary problem.
• Why?
– For some ML algorithms, a direct extension to the multiclass case may be problematic.
• How?
– Many methods
•99
Methods
• One-vs-all
• All-pairs
• Error-correcting Output Codes (ECOC)
• …
•100
One-vs-all
• Idea:
• Create many 1 vs other classifiers
– Classes = City, County, Country
– Classifier 1 = {City} {County, Country}
– Classifier 2 = {County} {City, Country}
– Classifier 3 = {Country} {City, County}
• Training time:
– For each class cm, train a classifier clm(x)
• replace (x,y) with
(x, 1) if y = cm
(x, -1) if y != cm•101
An example: training
• x1 c1 …
• x2 c2 …
• x3 c1 …
• x4 c3 …
for c1-vs-all:
x1 1 …
x2 -1 …
x3 1 …
x4 -1 …
for c2-vs-all:
x1 -1
x2 1 …
x3 -1 …
x4 -1 …
for c3-vs-all:
x1 -1…
x2 -1…
x3 -1 …
x4 1 …
•102
One-vs-all (cont)
• Testing time: given a new example x
– Run each of the k classifiers on x
– Choose the class cm with the highest
confidence score clm(x):
c* = arg maxm clm(x)
•103
An example: testing
• x1 c1 …
• x2 c2 …
• x3 c1 …
• x4 c3 …
three classifiers
Test data:
x ?? f1 v1 …
for c1-vs-all:
x ?? 1 0.7 -1 0.3
for c2-vs-all
x ?? 1 0.2 -1 0.8
for c3-vs-all
x ?? 1 0.6 -1 0.4
=> what’s the system prediction for x?
•104
All-pairs (All-vs-All (AVA))
• Idea:
– For each pair of classes build a classifier
– {City vs. County}, {City vs Country}, {County vs.
Country}
– Ck2 classifiers: one classifier for each class pair.
• Training:
– For each pair (cm, cn) of classes, train a classifier clmn
• replace a training instance (x,y) with
(x, 1) if y = cm
(x, -1) if y = cn
otherwise ignore the instance
•105
An example: training
• x1 c1 …
• x2 c2 …
• x3 c1 …
• x4 c3 …
for c1-vs-c2:
x1 1 …
x2 -1 …
x3 1 …
for c2-vs-c3:
x2 1 …
x4 -1 …
for c1-vs-c3:
x1 1…
x3 1 …
x4 -1 …
•106
All-pairs (cont)
• Testing time: given a new example x
– Run each of the Ck2 classifiers on x
– Max-win strategy: Choose the class cm that wins the most pairwise comparisons:
– Other coupling models have been proposed: e.g., (Hastie and Tibshirani, 1998)
•107
An example: testing
• x1 c1 …
• x2 c2 …
• x3 c1 …
• x4 c3 …
three classifiers
Test data:
x ?? f1 v1 …
for c1-vs-c2:
x ?? 1 0.7 -1 0.3
for c2-vs-c3
x ?? 1 0.2 -1 0.8
for c1-vs-c3
x ?? 1 0.6 -1 0.4
=> what’s the system prediction for x?
•108
Hierarchical Categorization
• Pick the category with max probability
• Create many OVA/AVA classifiers
• Use a hierarchical approach (wherever
hierarchy available)
Entity
Person Location
Scientist Artist City County Country
109
Error-correcting output codes
(ECOC)
• Proposed by (Dietterich and Bakiri, 1995)
• Idea:
– Each class is assigned a unique binary string of length n.
– Train n classifiers, one for each bit.
– Testing time: run n classifiers on x to get a n-bit string s, and choose the class which is closest to s.
•110
An example: Digit Recognition
•111
Meaning of each column
•112
Another example: 15-bit code for a
10-class problem
•113
Hamming distance
• Definition: the Hamming distance between two strings of equal length is the number of positions for which the corresponding symbols are different.
• Ex:
– 10111 and 10010
– 2143 and 2233
– Toned and roses
•114
How to choose a good error-
correcting code?
• Choose the one with large minimum
Hamming distance between any pair of
code words.
• If the min Hamming distance is d, then the
code can correct at least (d-1)/2 single bit
errors.
•115
Two properties of a good ECOC
• Row separations: Each codeword should be
well-separated in Hamming distance from
each of the other codewords
• Column separation: Each bit-position
function fi should be uncorrelated with each
of the other fj.
•116
Summary
• Different methods:
– Direct multiclass, if possible
– One-vs-all (a.k.a. one-per-class): k-classifiers
– All-pairs: Ck2 classifiers
– ECOC: n classifiers (n is the num of columns)
– Hierarchical classification (logC classifiers)
• Some studies report that All-pairs and
ECOC work better than one-vs-all.
•120