The Switchabalizer - our journey from spell checker to homophone corrrecter

IntroductionThe ProblemFirst Attempt

Second AttemptConclusion

The SwitchabalizerOur journey from spell checker to homophone correcter

Oskar Singer

July 23, 2014

Oskar Singer The Switchabalizer

How I got here

I am a rising senior in the UMass Amherst CS program specializingin machine learning and natural language processing.

Last summer, I interned at an Amherst/Boston-based textanalytics company called Lexalytics

I worked with Lexalytics’ head of software engineering on thisproject

Lexalytics often uses CommonCrawl, and it was a great option fora training data set

How I got here

Motivation

Lexalytics provides sentiment analysis software

Sentiment analysis relies heavily in sentence parsing andpart-of-speech tagging

Misspellings and misusage can do serious damage to accuracy forthose two tasks

Motivation

The ApproachThe Weaknesses

Approach

We employed an open-source spell-checker called Hunspell

Hunspell gives an unranked list of correction suggestions

So we took the argmax of a home-baked scoring function that:

penalized string edit distance

penalized keyboard distance

rewarded high word frequencies, which were harvested fromCommonCrawl data

Approach

Failure

Hunspell had an error rate of

Failure

Hunspell had an error rate of

What Happened?

How is this possible? Two reasons:

Hunspell missed all the mistakes

Hunspell made false corrections

What Happened?

Hunspell was a poor choice for a couple reasons:

Hunspell’s vocabulary is not appropriate or flexible enough forTwitter domain

Hunspell can’t detect correctly spelled words that are out ofcontext

What Happened?

Twitter’s vocabulary of abbreviations and acronyms is constantlygrowing

Hunspell’s internal dictionary is not prepared for this

What Happened?

Twitter’s vocabulary of abbreviations and acronyms is constantlygrowing

Hunspell’s internal dictionary is not prepared for this

What Happened?

Example: ur

What was Hunspell’s correction?

Ur (the ancient Sumerian city-state)

What Happened?

Example: ur

What Happened?

Example: ur

What Happened?

When the issue is misuse rather than misspelling, Hunspellcompletely ignores the problem

Specifically, commonly misused homophones were a huge problemin our data

Examples: two/too/2/to; their/there/they’re; your/you’re

What Happened?

BrainstormThe ApproachTesting and Results

Addressing Misusage

How do we capture the idea of misuse?

Context

How can we capture context?

Rule set?

Probabilistic approach!

Addressing Misusage

Context

Rule set?

Addressing Misusage

Context

Rule set?

Addressing Misusage

Context

Rule set?

Addressing Misusage

Context

Rule set?

Probability Model

Bayes network

Conditioned on the preceding and succeeding words

Assumes these two words are independent

Does not use bag-of-words approach (considers position)

Probability Model

Conditional Probability of Preceding or Succeeding Words

P(pre(wi )|wj) =#(wiwj)

#(wj),

where pre(w) is the event that w is the preceding word and #(∗)is the number of occurences of a sequence of words

P(suc(wi )|wj) =#(wjwi )

#(wj),

where suc(w) is the event that w is the succeeding word

Probability Model

Conditional Probability of Preceding or Succeeding Words

P(pre(wi )|wj) =#(wiwj)

#(wj),

where pre(w) is the event that w is the preceding word and #(∗)is the number of occurences of a sequence of words

P(suc(wi )|wj) =#(wjwi )

#(wj),

where suc(w) is the event that w is the succeeding word

Probability Model

Conditional Probability of Both Words

P(pre(wi ), suc(wk)|wj) = P(pre(wi )|wj)× P(suc(wk)|wj)

log(P(pre(wi ), suc(wk)|wj)) = log(P(pre(wi )|wj))

+ log(P(suc(wk)|wj))

The first equation holds because of our assumption ofindependence between the preceding and succeeding words

There is a missing term in the scoring function that I will addressin the Future Work section

Probability Model

Switchable Sets

Only certain groups should be compared, e.g. ”too” should not bescored against ”their”

Comparable switchables are groups in switchable sets

Each switchable is mapped to its switchable set

Switchable Sets

Picking the Word

The Final Equation

S(wi ,wj ,wk) = log(P(pre(wi ), suc(wk)|wj))

v∗ = argmaxv∈VwjS(wi , v ,wk)

where S(wi ,wj ,wk) is the score for the sequence of words wiwjwk

and Vwj is the switchable set corresponding to wj and v∗ is theideal switchable

First Pass

What about common misspellings that intersect with switchables?

Example: ”ur”

Should we put them in the switchable sets?

First Pass

Example: ”ur”

First Pass

Example: ”ur”

First Pass

My opinion: no!

Realistically, its probably okay. I opted for a more elegant solution

Replace all common mispellings with something from theappropriate switchable set

The model’s results are agnositc to the switchable that activates it

First Pass

My opinion: no!

First Pass

My opinion: no!

First Pass

My opinion: no!

Testing

Assume Wikipedia has correct usage of all switchables

Replace target words in Wikipedia articles with words from theirswitchable set

Run the Switchabilizer on corrupted articles

Testing

Results

How did we do?

20% error

Results

How did we do?

20% error

Future WorkCall to Action

Future Work

Ideal Scoring Function

S(wiwjwk) = log(P(wj , pre(wi ), suc(wk))

= log(P(wj)P(wi |wj)P(wk |wj))

Forgot the P(wj) term in the factorization of the joint distribution,which resulted in a slightly unfitting conditional distribution.Remember this for reimplementation!

Future Work

Ideal Scoring Function

S(wiwjwk) = log(P(wj , pre(wi ), suc(wk))

= log(P(wj)P(wi |wj)P(wk |wj))

Forgot the P(wj) term in the factorization of the joint distribution,which resulted in a slightly unfitting conditional distribution.Remember this for reimplementation!

Future Work

Testing conditions were not ideal because:

Test data is not target data

Mistakes are contrived

Somebody make a labeled test set, then tune the algorithm to it!

Future Work

Here are some ideas I had for future experiments:

Use a discriminative model like maximum entropy

Consider higher order neighbor words

Implement for other languages

Future Work

Start Coding!

Anyone else can do this too!

Straight-forward probability model

25-50 lines of Python

Freely accessible data from CommonCrawl!

Go learn about ML and NLP! Get your hands dirty and add yourown mods! Find new problems and try new solutions!

Start Coding!

Thank You, CommonCrawl!

Thanks so much to Lisa, Stephen, Grace and the rest of the teamfor providing such a fantastic resource and bringing me down toSan Francisco to present!

The Switchabalizer - our journey from spell checker to homophone corrrecter

Internet

Transcript of The Switchabalizer - our journey from spell checker to homophone corrrecter

￼The Switchabalizer - our journey from spell checker to homophone corrrecter

Internet

Transcript of ￼The Switchabalizer - our journey from spell checker to homophone corrrecter

The Switchabalizer - our journey from spell checker to homophone corrrecter

Transcript of The Switchabalizer - our journey from spell checker to homophone corrrecter