Text CategorizationMoshe Koppel

Lecture 3:Authorship Attribution

Mostly my own stuff together with

Jonathan Schler, Shlomo Argamon,Ido Dagan, Jamie Pennebaker, Jonathan Fine,Kfir Zigdon, Iris Zigdon, Navot Akiva, Dror Mughaz, Eran Messeri

The Authorship Attribution Problem

Given the known writings of a small number of authors A1,…,An , determine which of them wrote document X.

The vanilla version is actually not very difficult.

History of Authorship Attribution

• Researchers have been trying to solve this problem since the 19th century

• Early research focused on classical works: Bible, Shakespeare, etc.

• The most famous work is Mosteller & Wallace (1964) on the Federalist Papers

• The proliferation of online text offers many new research opportunities

Authorship – Realistic Versions

• No suspects – Can we profile the author of X?

• One suspect – Can we ascertain guilt or innocence?

• Thousands of suspects – Can we pick a needle from the haystack?

Text Categorization

Model for A vs. B

(x1,x2,...,xN)=A (x1,x2,...,xN)=A (x1,x2,...,xN)=A

: :

(x1,x2,...,xN)=B (x1,x2,...,xN)=B (x1,x2,...,xN)=B

Learning Algorithm

A

B

Text

Text

Cleaning+ Feature

Extraction

Features for Authorship Attribution

What features do we want?

• Used consistently by given author regardless of document type

Eliminates topic words

• Might be used differently by different authorsOur challenge: identify these

Olden Days

• Early researchers sought a single “magic” feature• Mainly, they looked at complexity features:

– Sentence or word length

– Type/token ratio

– Hapax legomena

• The idea was the magic feature had a distinct constant value for each author

• This approach isn’t taken seriously these days.

Features for Categorizing by Style

• Function words (and, of, the,..) [Mosteller-Wallace 64]

Features for Categorizing by Style

• Function words

• Syntax – POS n-grams

Features for Categorizing by Style

• Function words

• Syntax – POS n-grams

• SFL trees

SFL Trees

Conjunctions

ConjExtension and, or, but, yet, however,…

ConjElaboration for_example, indeed,…

ConjEnhancement

ConjSpatiotemporal then, beforehand, afterwards, while, during,…

ConjCausalConditional if, because, since, hence, thus, therefore,…

•Similar trees for prepositions, pronouns, modal verbs

•Covers all function words, some POS

•Doesn’t require POS tagging

Features for Categorizing by Style

• Function words

• Syntax

• SFL trees

• Morphology (grammatical prefixes and suffixes, e.g. –ing, re-)

Features for Categorizing by Style

• Function words

• Syntax

• SFL trees

• Morphology

• Complexity measures (e.g. avg. word length, avg. sentence length, various entropy measures)

Features for Categorizing by Style

• Function words• Syntax• SFL trees• Morphology • Complexity measures• “Unstable” words (features that might be

replaced in a rewrite, e.g. huge:large)

Measuring Instability

An interesting idea:

Translate many documents to another

language and translate back.

See what sticks.

We did just that on Reuters 21578 and BNC*.

* We used five different languages to ensure robustness.

Word Stability -- Examples

huge .0

ratio .0

device .0

onto .06

has .28

over .55

nine .98

September .99

Features for Categorizing by Style

• Function words

• Syntax

• SFL trees

• Morphology

• Complexity measures

• “Unstable” words

• Idiosyncrasies

Idiosyncrasies

Human attribution experts use idiosyncrasiese.g. spelling errors, neologisms, quirky syntax

• Use spelling/grammar checker to identify “errors”

• Categorize errors by typee.g. ie ei, nn n

Some problematic feature sets

• Many researchers use content words

• A really easy feature set is character n-grams

• These sets are problematic since they ought to fail when an author changes topics

• In practice, they work very well

Learning -- Historical

• From 1964 until 1990’s all work had flavor of Naïve Bayes

• Idea was to characterize all of each candidate author’s work and see which one X is most similar to

• Various proximity measures used (often after dimension reduction)

• Since 1990’s variety of learning methods applied

Learning Algorithms

Let’s try some old friends (and one new one):

• Naïve Bayes

• Decision Trees

• Winnow

• SVM

• Bayesian regression

Three testbed corpora

• Emails between two correspondents (date split)– Some as short as one word

• Two books by each of nine authors (book split)– Different books have different content

• Twenty bloggers (date split)– Lots of authors

Email corpus – 2 authors

features/learner nb j48 rmwinnow bmr smo FW 60.2% 58.7% 66.1% 68.2% 63.8% POS 61.0% 59.0% 66.1% 66.3% 67.1% FW+POS 65.9% 61.6% 68.0% 67.8% 71.7% SFL 57.2% 57.2% 65.6% 67.2% 62.7% CW 67.1% 66.9% 74.9% 78.4% 74.7% CNG 70.0% 62.0% 69.6% 72.1% 71.7%

Literature corpus – 9 authors

features/learner nb j48 rmwinnow bmr smo FW 51.4% 44.0% 63.0% 73.8% 77.8% POS 45.9% 50.3% 53.3% 69.6% 75.5% FW+POS 56.5% 46.2% 61.7% 75.0% 79.5% SFL 66.1% 45.7% 62.8% 76.6% 79.0% CW 68.9% 50.3% 57.0% 80.0% 84.7% CNG 68.7% 40.8% 47.9% 79.8% 82.7%

Blog corpus – 20 authors

features/learner nb j48 winnowmc bmr smo FW 38.2% 30.3% 51.8% 63.2% 63.2% POS 34.0% 30.3% 51.0% 63.2% 60.6% FW+POS 47.0% 34.3% 62.3% 70.3% 72.0% SFL 35.4% 36.3% 61.4% 69.2% 71.7% CW 56.4% 51.0% 62.9% 72.5% 70.5% CNG 63.2% 46.5% 65.2% 79.3% 79.3%

Some LessonsSome lessons:• SVM and Bayesian regression are the best learners.• FW and single POS are generally enough.• In fact, SFL trees alone are often enough.• Content words and character n-grams work surprisingly

wellSome other facts worth knowing:• Common unstable words are better than FW.• For unedited texts, idiosyncrasies are best.• For some languages, morphology is needed.