Ambiguity, Statistical Word Sense Discovery and Semantic...

Ambiguity,Statistical Word Sense Discovery

and Semantic Role Labeling

Andrew McCallum

Computer Science DepartmentUniversity of Massachusetts Amherst

Including slides from Chris Manning and Dan Klein.

Inherent Ambiguity in SyntaxFed raises interest rates 0.5%in effort to control inflation

NY Times headline 17 May 2000S

NP VP

NNP

FedV NP NP PP

raisesinterest rates

NN NN0.5 in NN VP

V VP

V NP

NN

CD NN PP NP%

effortto

controlinflation

Where are the ambiguities?

Fed raises interest rates 0.5 % in effort tocontrol inflation

Part-of-speech ambiguities Syntactic attachmentambiguities

Word sense ambiguities: Fed →”federal agent”interest →a feeling of wanting to know or learn more

Semantic interpretation ambiguities above the word level.

NNP NNSVBZ

NNSVBZ

NNSVBZ

VB

CD NN

Effects of V/N Ambiguity (1)

S

NP VP

NNP

Fed

V NP

raises

interest rates

NN NN


S

NP VP

N

Fed

N NP

raises interest

rates

V

N


S

NP VP

N

Fed

N NP

raises interest rates

N V

NCD

%0.5

Ambiguous Headlines

• Iraqi Head Seeks Arms• Juvenile Court to Try Shooting Defendant• Teacher Strikes Idle Kids• Stolen Painting Found by Tree• Kids Make Nutritious Snacks• Local HS Dropouts Cut in Half• British Left Waffles on Falkland Islands• Red Tape Holds Up New Bridges• Clinton Wins on Budget, but More Lies Ahead• Ban on Nude Dancing on Governor’s Desk

Natural Language Computingis hard because

• Natural language is:– highly ambiguous at all levels– complex and subtle– fuzzy, probabilistic– interpretation involves combining evidence– involves reasoning about the world– embedded a social system of people interacting

• persuading, insulting and amusing them• changing over time

Probabilistic Models of Language

The tools of probability:• Bayesian Classifiers (not rules)• Hidden Markov Models (not DFAs)• Probabilistic Context Free Grammars

• …other tools of Machine Learning, AI, Statistics

To handle this ambiguity and to integrateevidence from multiple levels we turn to:

Another Area whereProbabilistic Combination of Evidence

Won

Attachment Ambiguity

• Where to attach a phrase in the parse tree?• “I saw the man with the telescope.”

– What does “with a telescope” modify?– Is the problem AI complete? Yes, but…

– Proposed simple structural factors• Right association [Kimball 1973]

‘low’ or ‘near’ attachment = ‘early closure’ of NP• Minimal attachment [Frazier 1978]

(depends on grammar) = ‘high’ or ‘distant’ attachment= ‘late closure’ (of NP)


• Such simple structural factors dominated inearly psycholinguistics, and are still widelyinvoked.

• In the V NP PP context, right attachment getsright 55-76% of the cases…

• But this means that it gets wrong 33-45% ofthe cases!


• “The children ate the cake with a spoon.”• “The children ate the cake with frosting.”

• “Joe included the package for Susan.”• “Joe carried the package for Susan.”

• Ford, Bresnan and Kaplan (1982):“It is quite evident, then, that the closure effects inthese sentences are induced in some way by thechoice of the lexical items.”

Simple model

• (Log) likelihood ratio– A common and good way of comparing between

two exclusive alternatives– Same idea as a naïve Bayes classifier

– if >0, attach to verb, if <0 attach to noun– For example,

P(with a spoon | ate) > P(with a spoon | cake)

Attachment, Problematic Example

• “Chrysler confirmed that it would end its troubledventure with Maserati.”

• w C(w) C(w, with)end 5156 607venture 1442 155

• Get wrong answer:P(with|end) = (607/5156) = 0.118P(with|venture) = (155/1442) = 0.107

• Should also express preference for attaching ‘low’.

Other attachment issues

• There are attachment questions other thanprepositional phrases– adverbial, participial, noun compounds– Examples

door bell manufacturer[door bell] manufacturerUnix system administratorUnix [system administrator]

– Data sparseness is a bigger problem with many of these

• In general, indeterminacy is quite common– “We have not signed a settlement agreement with them.”– Either reading seems equally plausible.

Lexical acquisition, semantic similarity

• Previous models give same estimate to allunseen events.

• Unrealistic - could hope to refine that basedon semantic classes of words

• Examples– “Susan ate the cake with a durian.”– “Susan had never eaten a fresh durian before.”– Although never seen “eating pineapple” should be

more likely than “eating holograms” becausepineapple is similar to apples, and we have seen“eating apples”.

An application: selectional preferences

• Most verbs prefer arguments of a particulartype. Such regularities are called selectionalpreferences or selectional restrictions.

• “Bill drove a…” Mustang, car, truck, jeep

• Selectional preference strength: how stronglydoes a verb constrain direct objects

• “see” versus “unknotted”

Measuring selectional preference strength

• Assume we are given a clustering of (direct object) nouns.Resnick (1993) uses WordNet.

• Selectional association between a verb and a class

Proportion that its summand contributes to preference strength.

• For nouns in multiple classes, disambiguate as most likelysense:

Selection preference strength(made up data)

Noun class c P(c) P(c|eat) P(c|see) P(c|find)people 0.25 0.01 0.25 0.33furniture 0.25 0.01 0.25 0.33food 0.25 0.97 0.25 0.33action 0.25 0.01 0.25 0.01SPS S(v) 1.76 0.00 0.35

A(eat, food) = 1.08A(find, action) = -0.13

Selectional Preference Strength example(Resnick, Brown corpus)

But how might we measureword similarity for word classes?

• Vector spaces

But how might we measureword similarity for word classes?

• Vector spacesword-by-word matrix B

Similarity measures for binary vectors

Cosine measure

Example of cosine measure onword-by-word matrix on NYT

Probabilistic measures

Neighbors of word “company”[Lee]

Clustering words into topics withLatent Dirichlet Allocation

[Blei, Ng, Jordan 2003]

Sample a distributionover topics, θ

For each document:

Sample a topic, z

For each word in doc

Sample a wordfrom the topic, w

Example:

70% Iraq war30% US election

Iraq war

“bombing”

GenerativeProcess:

STORYSTORIES

TELLCHARACTER

CHARACTERSAUTHOR

READTOLD

SETTINGTALESPLOT

TELLINGSHORT

FICTIONACTION

TRUEEVENTSTELLSTALE

NOVEL

MINDWORLDDREAM

DREAMSTHOUGHT

IMAGINATIONMOMENT

THOUGHTSOWNREALLIFE

IMAGINESENSE

CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE

WATERFISHSEA

SWIMSWIMMING

POOLLIKE

SHELLSHARKTANK

SHELLSSHARKSDIVING

DOLPHINSSWAMLONGSEALDIVE

DOLPHINUNDERWATER

DISEASEBACTERIADISEASES

GERMSFEVERCAUSE

CAUSEDSPREADVIRUSES

INFECTIONVIRUS

MICROORGANISMSPERSON

INFECTIOUSCOMMONCAUSING

SMALLPOXBODY

INFECTIONSCERTAIN

Example topicsinduced from a large collection of text

FIELDMAGNETICMAGNET

WIRENEEDLE

CURRENTCOIL

POLESIRON

COMPASSLINESCORE

ELECTRICDIRECTION

FORCEMAGNETS

BEMAGNETISM

POLEINDUCED

SCIENCESTUDY

SCIENTISTSSCIENTIFIC

KNOWLEDGEWORK

RESEARCHCHEMISTRY

TECHNOLOGYMANY

MATHEMATICSBIOLOGY

FIELDPHYSICS

LABORATORYSTUDIESWORLD

SCIENTISTSTUDYINGSCIENCES

BALLGAMETEAM

FOOTBALLBASEBALLPLAYERS

PLAYFIELD

PLAYERBASKETBALL

COACHPLAYEDPLAYING

HITTENNISTEAMSGAMESSPORTS

BATTERRY

JOBWORKJOBS

CAREEREXPERIENCE

EMPLOYMENTOPPORTUNITIES

WORKINGTRAINING

SKILLSCAREERS

POSITIONSFIND

POSITIONFIELD

OCCUPATIONSREQUIRE

OPPORTUNITYEARNABLE

[Tennenbaum et al]

STORYSTORIES

TELLCHARACTER

CHARACTERSAUTHOR

READTOLD

SETTINGTALESPLOT

TELLINGSHORT

FICTIONACTION

TRUEEVENTSTELLSTALE

NOVEL

MINDWORLDDREAM

DREAMSTHOUGHT

IMAGINATIONMOMENT

THOUGHTSOWNREALLIFE

IMAGINESENSE

CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE

WATERFISHSEA

SWIMSWIMMING

POOLLIKE

SHELLSHARKTANK

SHELLSSHARKSDIVING

DOLPHINSSWAMLONGSEALDIVE

DOLPHINUNDERWATER

DISEASEBACTERIADISEASES

GERMSFEVERCAUSE

CAUSEDSPREADVIRUSES

INFECTIONVIRUS

MICROORGANISMSPERSON

INFECTIOUSCOMMONCAUSING

SMALLPOXBODY

INFECTIONSCERTAIN

FIELDMAGNETICMAGNET

WIRENEEDLE

CURRENTCOIL

POLESIRON

COMPASSLINESCORE

ELECTRICDIRECTION

FORCEMAGNETS

BEMAGNETISM

POLEINDUCED

SCIENCESTUDY

SCIENTISTSSCIENTIFIC

KNOWLEDGEWORK

RESEARCHCHEMISTRY

TECHNOLOGYMANY

MATHEMATICSBIOLOGYFIELD

PHYSICSLABORATORY

STUDIESWORLD

SCIENTISTSTUDYINGSCIENCES

BALLGAMETEAM

FOOTBALLBASEBALLPLAYERS

PLAYFIELD

PLAYERBASKETBALL

COACHPLAYEDPLAYING

HITTENNISTEAMSGAMESSPORTS

BATTERRY

JOBWORKJOBS

CAREEREXPERIENCE

EMPLOYMENTOPPORTUNITIES

WORKINGTRAINING

SKILLSCAREERS

POSITIONSFIND

POSITIONFIELD

OCCUPATIONSREQUIRE

OPPORTUNITYEARNABLE

Example topicsinduced from a large collection of text

[Tennenbaum et al]

Collocations

• An expression consisting of two or morewords that correspond to some conventionalway of saying things.

• Characterized by limited compositionality.– compositional: meaning of expression can be

predicted by meaning of its parts.– “strong tea”, “rich in calcium”– “weapons of mass destruction”– “kick the bucket”, “hear it through the grapevine”

Collocations important for…

• Terminology extraction– Finding special phrases in technical domains

• Natural language generation– To make natural output

• Computational lexicography– To automatically identify phrases to be listed in a dictionary

• Parsing– To give preference to parses with natural collocations

• Study of social phenomena– Like the reinforcement of cultural stereotypes through language (Stubbs

1996)

Contextual Theory of Meaning

• In contrast with “structural linguistics”, which emphasizes abstractions,properties of sentences

• Contextual Theory of Meaning emphasizes the importance of context– context of the social setting (not idealized speaker)– context of discourse (not sentence in isolation)– context of surrounding words

Firth: “a word is characterized by the company it keeps”• Example [Halliday]

– “strong tea”, coffee, cigarettes– “powerful drugs”, heroin, cocaine– Important for idiomatically correct English, but also social implications of

language use

Topics Modeling Phrases

• Topics based only on unigrams oftendifficult to interpret

• Topic discovery itself is confused becauseimportant meaning / distinctions carried byphrases.

• Significant opportunity to provide improvedlanguage models to ASR, MT, IR, etc.

Topical N-gram Model

z1 z2 z3 z4

w1 w2 w3 w4

y1 y2 y3 y4

θ

φ1

T

D

. . .

. . .

. . .

α

WTW

ψ γ1 γ2β φ2

[Wang, McCallum 2005]

LDA Topic

LDA

algorithmsalgorithmgenetic

problemsefficient

Topical N-grams

genetic algorithmsgenetic algorithm

evolutionary computationevolutionary algorithms

fitness function

Topic Comparison

learningoptimalreinforcementstateproblemspolicydynamicactionprogrammingactionsfunctionmarkovmethodsdecisionrlcontinuousspacessteppoliciesplanning

LDAreinforcement learningoptimal policydynamic programmingoptimal controlfunction approximatorprioritized sweepingfinite-state controllerlearning systemreinforcement learning rlfunction approximatorsmarkov decision problemsmarkov decision processeslocal searchstate-action pairmarkov decision processbelief statesstochastic policyaction selectionupright positionreinforcement learning methods

policyactionstatesactionsfunctionrewardcontrolagentq-learningoptimalgoallearningspacestepenvironmentsystemproblemstepssuttonpolicies

Topical N-grams (2) Topical N-grams (1)

Topic Comparison

motionvisualfieldpositionfiguredirectionfieldseyelocationretinareceptivevelocityvisionmovingsystemflowedgecenterlightlocal

LDAreceptive fieldspatial frequencytemporal frequencyvisual motionmotion energytuning curveshorizontal cellsmotion detectionpreferred directionvisual processingarea mtvisual cortexlight intensitydirectional selectivityhigh contrastmotion detectorsspatial phasemoving stimulidecision strategyvisual stimuli

motionresponsedirectioncellsstimulusfigurecontrastvelocitymodelresponsesstimulimovingcellintensitypopulationimagecentertuningcomplexdirections


Topic Comparison

wordsystemrecognitionhmmspeechtrainingperformancephonemewordscontextsystemsframetrainedspeakersequencespeakersmlpframessegmentationmodels

LDAspeech recognitiontraining dataneural networkerror ratesneural nethidden markov modelfeature vectorscontinuous speechtraining procedurecontinuous speech recognitiongamma filterhidden controlspeech productionneural netsinput representationoutput layerstraining algorithmtest setspeech framesspeaker dependent

speechwordtrainingsystemrecognitionhmmspeakerperformancephonemeacousticwordscontextsystemsframetrainedsequencephoneticspeakersmlphybrid


Unsupervised learning oftopic hierarchies

(Blei, Griffiths, Jordan & Tenenbaum, NIPS 2003)

Joint models of syntax and semantics (Griffiths,Steyvers, Blei & Tenenbaum, NIPS 2004)

• Embed topics model inside an nth orderHidden Markov Model:

Document-specific distribution over topics

FOODFOODSBODY

NUTRIENTSDIETFAT

SUGARENERGY

MILKEATINGFRUITS

VEGETABLESWEIGHT

FATSNEEDS

CARBOHYDRATESVITAMINSCALORIESPROTEIN

MINERALS

MAPNORTHEARTHSOUTHPOLEMAPS

EQUATORWESTLINESEAST

AUSTRALIAGLOBEPOLES

HEMISPHERELATITUDE

PLACESLAND

WORLDCOMPASS

CONTINENTS

DOCTORPATIENTHEALTH

HOSPITALMEDICAL

CAREPATIENTS

NURSEDOCTORSMEDICINENURSING

TREATMENTNURSES

PHYSICIANHOSPITALS

DRSICK

ASSISTANTEMERGENCY

PRACTICE

BOOKBOOKS

READINGINFORMATION

LIBRARYREPORT

PAGETITLE

SUBJECTPAGESGUIDE

WORDSMATERIALARTICLE

ARTICLESWORDFACTS

AUTHORREFERENCE

NOTE

GOLDIRON

SILVERCOPPERMETAL

METALSSTEELCLAYLEADADAM

OREALUMINUM

MINERALMINE

STONEMINERALS

POTMININGMINERS

TIN

BEHAVIORSELF

INDIVIDUALPERSONALITY

RESPONSESOCIAL

EMOTIONALLEARNINGFEELINGS

PSYCHOLOGISTSINDIVIDUALS

PSYCHOLOGICALEXPERIENCES

ENVIRONMENTHUMAN

RESPONSESBEHAVIORSATTITUDES

PSYCHOLOGYPERSON

CELLSCELL

ORGANISMSALGAE

BACTERIAMICROSCOPEMEMBRANEORGANISM

FOODLIVINGFUNGIMOLD

MATERIALSNUCLEUSCELLED

STRUCTURESMATERIAL

STRUCTUREGREENMOLDS

Semantic classes

PLANTSPLANT

LEAVESSEEDSSOIL

ROOTSFLOWERS

WATERFOOD

GREENSEED

STEMSFLOWER

STEMLEAF

ANIMALSROOT

POLLENGROWING

GROW

GOODSMALL

NEWIMPORTANT

GREATLITTLELARGE

*BIG

LONGHIGH

DIFFERENTSPECIAL

OLDSTRONGYOUNG

COMMONWHITESINGLE

CERTAIN

THEHIS

THEIRYOURHERITSMYOURTHIS

THESEA

ANTHATNEW

THOSEEACH

MRANYMRSALL

MORESUCHLESS

MUCHKNOWN

JUSTBETTERRATHER

GREATERHIGHERLARGERLONGERFASTER

EXACTLYSMALLER

SOMETHINGBIGGERFEWERLOWER

ALMOST

ONAT

INTOFROMWITH

THROUGHOVER

AROUNDAGAINSTACROSS

UPONTOWARDUNDERALONGNEAR

BEHINDOFF

ABOVEDOWN

BEFORE

SAIDASKED

THOUGHTTOLDSAYS

MEANSCALLEDCRIEDSHOWS

ANSWEREDTELLS

REPLIEDSHOUTED

EXPLAINEDLAUGHED

MEANTWROTE

SHOWEDBELIEVED

WHISPERED

ONESOMEMANYTWOEACHALL

MOSTANY

THREETHIS

EVERYSEVERAL

FOURFIVE

BOTHTENSIX

MUCHTWENTY

EIGHT

HEYOUTHEY

ISHEWEIT

PEOPLEEVERYONE

OTHERSSCIENTISTSSOMEONE

WHONOBODY

ONESOMETHING

ANYONEEVERYBODY

SOMETHEN

Syntactic classes

BEMAKEGET

HAVEGO

TAKEDO

FINDUSESEE

HELPKEEPGIVELOOKCOMEWORKMOVELIVEEAT

BECOME

Corpus-specific factorization(NIPS)

Sem

antic

sSy

ntax

REMAINED

5 8 14 25 26 30 33IN ARE THE SUGGEST LEVELS RESULTS BEEN

FOR WERE THIS INDICATE NUMBER ANALYSIS MAYON WAS ITS SUGGESTING LEVEL DATA CAN

BETWEEN IS THEIR SUGGESTS RATE STUDIES COULDDURING WHEN AN SHOWED TIME STUDY WELLAMONG REMAIN EACH REVEALED CONCENTRATIONS FINDINGS DIDFROM REMAINS ONE SHOW VARIETY EXPERIMENTS DOES

UNDER REMAINED ANY DEMONSTRATE RANGE OBSERVATIONS DOWITHIN PREVIOUSLY INCREASED INDICATING CONCENTRATION HYPOTHESIS MIGHT

THROUGHOUT BECOME EXOGENOUS PROVIDE DOSE ANALYSES SHOULDTHROUGH BECAME OUR SUPPORT FAMILY ASSAYS WILLTOWARD BEING RECOMBINANT INDICATES SET POSSIBILITY WOULD

INTO BUT ENDOGENOUS PROVIDES FREQUENCY MICROSCOPY MUSTAT GIVE TOTAL INDICATED SERIES PAPER CANNOT

INVOLVING MERE PURIFIED DEMONSTRATED AMOUNTS WORK

THEYAFTER APPEARED TILE SHOWS RATES EVIDENCE ALSO

ACROSS APPEAR FULL SO CLASS FINDINGAGAINST ALLOWED CHRONIC REVEAL VALUES MUTAGENESIS BECOME

WHEN NORMALLY ANOTHER DEMONSTRATES AMOUNT OBSERVATION MAGALONG EACH EXCESS SUGGESTED SITES MEASUREMENTS LIKELY

Syntactic classes in PNAS

Semantic highlighting Darker words are more likely to have been generated from the topic-based “semantics” module:

Semantic Role Labeling (SRL)

• Characterize clauses as relations with roles:

• Want to more than which NP is the subject (but not much more):• Relations like subject are syntactic, relations like agent or message are

semantic• Typical pipeline:

– Parse, then label roles– Almost all errors locked in by parser– Really, SRL is quite a lot easier than parsing

SRL Example

PropBank / FrameNet

• FrameNet: roles shared between verbs• PropBank: each verb has it’s own roles• PropBank more used, because it’s layered over the treebank (and so has

greater coverage, plus parses)• Note: some linguistic theories postulate even fewer roles than FrameNet

(e.g. 5-20 total: agent, patient, instrument, etc.)

PropBank Example

Shared Arguments

Path Features

Results

• Features:– Path from target to filler– Filler’s syntactic type, headword, case– Target’s identity– Sentence voice, etc.– Lots of other second-order features

• Gold vs parsed source trees

– SRL is fairly easy on gold trees

– Harder on automatic parses

Outline

• Role Discovery (Author-Recipient-Topic Model, ART)

• Group Discovery (Group-Topic Model, GT)

• Enhanced Topic Models– Correlations among Topics (Pachinko Allocation, PAM)

– Time Localized Topics (Topics-over-Time Model, TOT)

– Markov Dependencies in Topics (Topical N-Grams Model, TNG)

• Bibliometric Impact Measures enabled by Topics

Social Network Analysis with Topic Models

Multi-Conditional Mixtures

Groups and Topics

• Input:– Observed relations between people– Attributes on those relations (text, or categorical)

• Output:– Attributes clustered into “topics”– Groups of people---varying depending on topic

Discovering Groups fromObserved Set of Relations

Admiration relations among six high school students.

Student Roster

AdamsBennettCarterDavisEdwardsFrederking

Academic Admiration

Acad(A, B) Acad(C, B)Acad(A, D) Acad(C, D)Acad(B, E) Acad(D, E)Acad(B, F) Acad(D, F)Acad(E, A) Acad(F, A)Acad(E, C) Acad(F, C)

Adjacency Matrix Representing Relations

FEDCBA

FEDCBAFEDCBA

G3G3G2G1G2G1

G3G3G2G1G2G1

FEDCBA

FEDBCA

G3G3G2G2G1G1

G3G3G2G2G1G1

FEDBCA

Student Roster


Academic Admiration


Group Model:Partitioning Entities into Groups

2S

v

!

2G

! ! !

Stochastic Blockstructures for Relations[Nowicki, Snijders 2001]

S: number of entities

G: number of groups

Enhanced with arbitrary number of groups in [Kemp, Griffiths, Tenenbaum 2004]

BetaDirichlet

Binomial

S

gMultinomial

Two Relations with Different Attributes

FEDBCA

G3G3G2G2G1G1

G3G3G2G2G1G1FDBECA

G2G2G2G1G1G1

G2G2G2G1G1G1

FDBECA

Student Roster


Academic Admiration


Social Admiration

Soci(A, B) Soci(A, D) Soci(A, F)Soci(B, A) Soci(B, C) Soci(B, E)Soci(C, B) Soci(C, D) Soci(C, F)Soci(D, A) Soci(D, C) Soci(D, E)Soci(E, B) Soci(E, D) Soci(E, F)Soci(F, A) Soci(F, C) Soci(F, E)

FEDBCA

The Group-Topic Model:Discovering Groups and Topics Simultaneously

bN

w

t

B

T

!

!

DirichletMultinomial

Uniform

2S

v

!

2G

! ! !

BetaDirichlet

Binomial

S

gMultinomial

T

[Wang, Mohanty, McCallum 2006]

Inference and EstimationGibbs Sampling:- Many r.v.s can beintegrated out- Easy to implement- Reasonably fast

We assume the relationship is symmetric.

Dataset #1:U.S. Senate

• 16 years of voting records in the US Senate (1989 – 2005)

• a Senator may respond Yea or Nay to a resolution

• 3423 resolutions with text attributes (index terms)

• 191 Senators in total across 16 yearsS.543Title: An Act to reform Federal deposit insurance, protect the deposit insurancefunds, recapitalize the Bank Insurance Fund, improve supervision and regulationof insured depository institutions, and for other purposes.Sponsor: Sen Riegle, Donald W., Jr. [MI] (introduced 3/5/1991) Cosponsors (2)Latest Major Action: 12/19/1991 Became Public Law No: 102-242.Index terms: Banks and banking Accounting Administrative fees Cost controlCredit Deposit insurance Depressed areas and other 110 terms

Adams (D-WA), Nay Akaka (D-HI), Yea Bentsen (D-TX), Yea Biden (D-DE), YeaBond (R-MO), Yea Bradley (D-NJ), Nay Conrad (D-ND), Nay ……

Topics Discovered (U.S. Senate)

carepolicypollutionpreventionemployeelawresearchelementarybusinessaidpetrolstudents

taxcongressgasdrugaidtaxnuclearchildren

insuranceforeignwateraidlabormilitarypowerschool

federalgovernmentenergyeducation

EconomicMilitaryMisc.EnergyEducation

Mixture of Unigrams

Group-Topic Model

assistancebusinessdiseasesresearchdisabilitywagecommunicableenergymedicareminimumdrugstax

careincomecongressgovernmentmedicalcongresstariffaid

insurancetaxchemicalsfederalsecurityinsurancetradeschoolsociallaborforeigneducation

Social Security+ Medicare

EconomicForeignEducation+ Domestic

Groups Discovered (US Senate)

Groups from topic Education + Domestic

Senators Who Change Coalition the mostDependent on Topic

e.g. Senator Shelby (D-AL) votes with the Republicans on Economicwith the Democrats on Education + Domesticwith a small group of maverick Republicans on Social Security + Medicaid

Dataset #2:The UN General Assembly

• Voting records of the UN General Assembly (1990 - 2003)

• A country may choose to vote Yes, No or Abstain

• 931 resolutions with text attributes (titles)

• 192 countries in total

• Also experiments later with resolutions from 1960-2003

Vote on Permanent Sovereignty of Palestinian People, 87th plenary meeting

The draft resolution on permanent sovereignty of the Palestinian people in theoccupied Palestinian territory, including Jerusalem, and of the Arab population inthe occupied Syrian Golan over their natural resources (document A/54/591)was adopted by a recorded vote of 145 in favour to 3 against with 6 abstentions:

In favour: Afghanistan, Argentina, Belgium, Brazil, Canada, China, France,Germany, India, Japan, Mexico, Netherlands, New Zealand, Pakistan, Panama,Russian Federation, South Africa, Spain, Turkey, and other 126 countries.Against: Israel, Marshall Islands, United States.Abstain: Australia, Cameroon, Georgia, Kazakhstan, Uzbekistan, Zambia.

Topics Discovered (UN)

callsisraelcountriessecuritysituationimplementation

syriapalestineuseisraelhumanweapons

occupiedrightsnuclear

Securityin Middle East

Human RightsEverythingNuclear

Mixture ofUnigrams

Group-TopicModel

israelspacenationsoccupiedraceweaponspalestinepreventionunitedhumanarmsstatesrightsnuclearnuclear

Human RightsNuclear ArmsRace

NuclearNon-proliferation

GroupsDiscovered(UN)The countries list for eachgroup are ordered by their2005 GDP (PPP) and only 5countries are shown ingroups that have more than5 members.

Groups and Topics, Trends over Time (UN)

Outline









Social Networks in Research Literature

• Better understand structure of our ownresearch area.

• Structure helps us learn a new field.• Aid collaboration• Map how ideas travel through social networks

of researchers.

• Aids for hiring and finding reviewers!

Traditional Bibliometrics

• Analyses a small amount of data(e.g. 19 articles from a single issue of a journal)

• Uses “journal” as a proxy for “research topic”(but there is no journal for information extraction)

• Uses impact measures almost exclusivelybased on simple citation counts.

How can we use topic models to create new, interesting impact measures?

Our Data

• Over 1 million research papers,gathered as part of Rexa.info portal.

• Cross linked references / citations.

Finding Topics with TNGTraditional unigram LDA

run on 1 milliontitles / abstracts

(200 topics)

...select ~300k papers onML, NLP, robotics, vision...

Find 200 TNG topics among those papers.

Topical Bibliometric Impact Measures

• Topical Citation Counts

• Topical Impact Factors

• Topical Longevity

• Topical Diversity

• Topical Precedence

• Topical Transfer

[Mann, Mimno, McCallum, 2006]

Topical DiversityEntropy of the topic distribution among

papers that cite this paper (this topic).

LowDiversity

HighDiversity

Topical Diversity

Can also be measured on particular papers...

Topical PrecedenceWithin a topic, what are the earliest papers that received more than n citations?

“Early-ness”

Information Retrieval:

On Relevance, Probabilistic Indexing and Information Retrieval,Kuhns and Maron (1960)

Expected Search Length: A Single Measure of Retrieval Effectiveness Basedon the Weak Ordering Action of Retrieval Systems,

Cooper (1968)Relevance feedback in information retrieval,

Rocchio (1971)Relevance feedback and the optimization of retrieval effectiveness,

Salton (1971)New experiments in relevance feedback,

Ide (1971)Automatic Indexing of a Sound Database Using Self-organizing Neural Nets,

Feiten and Gunzel (1982)

Topical PrecedenceWithin a topic, what are the earliest papers that received more than n citations?

“Early-ness”

Speech Recognition:

Some experiments on the recognition of speech, with one and two ears,E. Colin Cherry (1953)

Spectrographic study of vowel reduction,B. Lindblom (1963)

Automatic Lipreading to enhance speech recognition, Eric D. Petajan (1965)

Effectiveness of linear prediction characteristics of the speech wave for...,B. Atal (1974)

Automatic Recognition of Speakers from Their Voices,B. Atal (1976)

Topical Transfer

Transfer from Digital Libraries to other topics

WebBase: a repository of Web pages11Web Pages

Trawling the Web for Emerging Cyber-Communities

12Graphs

Lessons learned from the creation anddeployment of a terabyte digital video

12Video

On being ‘Undigital’ with digital cameras:extending the dynamic...

14Computer Vision

Trawling the Web for Emerging Cyber-Communities, Kumar, Raghavan,... 1999.

31Web Pages

Paper TitleCit’sOther topic

Topical TransferCitation counts from one topic to another.

Map “producers and consumers”

Outline









Topic Model Musings

• 3 years ago Latent Dirichlet Allocationappeared as a complex innovation...but now these methods & mechanics arewell-understood.

• Innovation now is to understanddata and modeling needs,how to structure a new model to capture these.

Ambiguity, Statistical Word Sense Discovery and Semantic...

Documents

Transcript of Ambiguity, Statistical Word Sense Discovery and Semantic...