Beyond Bags of Words: A Markov Random Field Model for Information Retrieval
description
Transcript of Beyond Bags of Words: A Markov Random Field Model for Information Retrieval
![Page 1: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/1.jpg)
Beyond Bags of Words:Beyond Bags of Words:A Markov Random Field Model for A Markov Random Field Model for
Information RetrievalInformation RetrievalDon Metzler
![Page 2: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/2.jpg)
Bag of Words Representation71 the 31 garden22 and 19 of19 to 19 house18 in 18 white12 trees 12 first11 a 11 president8 for 7 as7 gardens 7 rose7 tour 6 on6 was 6 east6 tours 5 planting5 he 5 is5 grounds 5 that5 gardener 4 history4 text-decoration 4 john4 kennedy 4 april4 been 4 today4 with 4 none4 adams 4 spring4 at 4 had3 mrs 3 lawn… …
71 the 31 garden22 and 19 of19 to 19 house18 in 18 white12 trees 12 first11 a 11 president8 for 7 as7 gardens 7 rose7 tour 6 on6 was 6 east6 tours 5 planting5 he 5 is5 grounds 5 that5 gardener 4 history4 text-decoration 4 john4 kennedy 4 april4 been 4 today4 with 4 none4 adams 4 spring4 at 4 had3 mrs 3 lawn… …
![Page 3: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/3.jpg)
Bag of Words Representation
![Page 4: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/4.jpg)
Bag of Words Models
![Page 5: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/5.jpg)
Binary Independence Model
![Page 6: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/6.jpg)
Unigram Language Models
Language modeling first used in speech recognition to model speech generationIn IR, they are models of text generationTypical scenario
Estimate language model for every documentRank documents by the likelihood that the document generated the query
Documents modeled as multinomial distributions over a fixed vocabulary
![Page 7: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/7.jpg)
Unigram Language Models
Query: Document:
Ranking via Query Likelihood
P( | ) = P( | )∙P( | )∙P( | )
Estimation
P( | ) P( | θ ) P( | θ ) P( θ )
![Page 8: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/8.jpg)
Bag of Words Models
ProsSimple model
Easy to implement
Decent results
ConsToo simple
Unrealistic assumptions
Inability to explicitly model term dependencies
![Page 9: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/9.jpg)
Beyond Bags of Words
![Page 10: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/10.jpg)
Tree Dependence Model
Method of approximating complex joint distribution
Compute EMIM between every pair of terms
Build maximum spanning tree
Tree encodes first order dependencies
A
D
B
E
C
)|()|()|()|()(),,,,( CBPCEPDCPDAPDPEDCBAP
![Page 11: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/11.jpg)
n-Gram Language Models
![Page 12: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/12.jpg)
n-Gram Language Models
Query: Document:
Ranking via Query Likelihood
P( | ) = P( | )∙P( | , )∙P( | , )
Estimation
||
)1(||
)1()1(),|(1
,1
1
,11 C
cf
cf
cf
D
tf
tf
tfDwwP i
i
iii
i
iiii
![Page 13: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/13.jpg)
Dependency Models
ProsMore realistic assumptions
Improved effectiveness
ConsLess efficient
Limited notion of dependence
Not well understood
![Page 14: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/14.jpg)
State of the Art IR
![Page 15: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/15.jpg)
Desiderata
Our desired model should be able to…Support standard IR tasks (ranking, query expansion, etc.)
Easily model dependencies between terms
Handle textual and non-textual features
Consistently and significantly improve effectiveness over bag of words models and existing dependence models
Proposed solution: Markov random fields
![Page 16: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/16.jpg)
Markov Random Fields
MRFs provide a general, robust way of modeling a joint distributionThe anatomy of a MRF
Graph G • vertices representing random variables • edges encode dependence semantics
Potentials over the cliques of G• Non-negative functions over clique configurations• Measures ‘compatibility’
)(
, );(1
)(GCliquesc
G cZ
XP
![Page 17: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/17.jpg)
Markov Random Fields for IR
![Page 18: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/18.jpg)
Modeling Dependencies
![Page 19: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/19.jpg)
Parameter Tying
In theory, a potential function can be associated with every clique in a graphTypical solution is to define potentials over maximal cliques of GNeed more fine-grained control over our potentialsUse clique sets
Set of cliques that share a parameter and potential functionWe identified 7 clique sets that are relevant to IR tasks
![Page 20: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/20.jpg)
Clique SetsSingle term document/query cliques
TD = cliques w/ one query term + D
ψ(domestic, D), ψ(adoption, D), ψ(laws, D)
Ordered terms document/query cliques
OD = cliques w/ two or more contiguous query terms + D
ψ(domestic, adoption, D), ψ(adoption, laws, D),ψ(domestic, adoption, laws, D)
Unordered terms document/query cliques
UD = cliques w/ two or more query terms (in any order) + D
ψ(domestic, adoption, D), ψ(adoption, laws, D), ψ(domestic, laws, D), ψ(domestic, adoption, laws, D)
![Page 21: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/21.jpg)
Clique Sets
Single term query cliquesTQ = cliques w/ one query termψ(domestic), ψ(adoption), ψ(laws)
Ordered terms query cliquesOQ = cliques w/ two or more contiguous query terms
ψ(domestic, adoption), ψ(adoption, laws), ψ(domestic, adoption, laws)
Unordered terms query cliquesUQ = cliques w/ two or more query terms (in any order) ψ(domestic, adoption), ψ(adoption, laws), ψ(domestic, laws),
ψ(domestic, adoption, laws)
Document cliqueD = singleton clique w/ D
ψ(D)
![Page 22: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/22.jpg)
Features
![Page 23: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/23.jpg)
![Page 24: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/24.jpg)
![Page 25: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/25.jpg)
Parameter Estimation
Given a set of relevance judgments R, we want the maximum a posteriori estimate:
What is P( Λ | R )? P( R | Λ ) and P( Λ )?Depends on how model is being evaluated!Want P( Λ | R ) to be peaked around the parameter setting that maximizes the metric we are interested in
)|(maxargˆ RP
![Page 26: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/26.jpg)
Parameter Estimation
![Page 27: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/27.jpg)
Evaluation Metric Surfaces
![Page 28: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/28.jpg)
Feature Induction
![Page 29: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/29.jpg)
Query Expansion
![Page 30: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/30.jpg)
Query Expansion
![Page 31: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/31.jpg)
Query Expansion ExampleO
rigin
al q
uery
: hu
bble
tel
esco
pe a
chie
vem
ents
![Page 32: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/32.jpg)
Query Expansion ExampleO
rigin
al q
uery
: hu
bble
tel
esco
pe a
chie
vem
ents
![Page 33: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/33.jpg)
Applications
![Page 34: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/34.jpg)
Application: Ad Hoc Retrieval
![Page 35: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/35.jpg)
Ad Hoc Results
![Page 36: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/36.jpg)
Application: Web Search
![Page 37: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/37.jpg)
Application: XML Retrieval
![Page 38: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/38.jpg)
Example XML Document<PLAY><TITLE>The Tragedy of Romeo and Juliet</TITLE><ACT><TITLE>ACT I</TITLE><PROLOGUE><TITLE>PROLOGUE</TITLE><SPEECH><SPEAKER>NARRATOR</SPEAKER><LINE>Two households, both alike in dignity,</LINE><LINE>In fair Verona, where we lay our scene,</LINE><LINE>From ancient grudge break to new mutiny,</LINE><LINE>Where civil blood makes civil hands unclean.</LINE><LINE>From forth the fatal loins of these two foes</LINE><LINE>A pair of star-cross’d lovers take their life;</LINE>…</SPEECH></PROLOGUE><SCENE><TITLE>SCENE I. Verona. A public place.</TITLE><SPEECH><SPEAKER>SAMPSON</SPEAKER><LINE>Gregory, o’ my word, we’ll not carry coals.</LINE>…</PLAY>
![Page 39: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/39.jpg)
Content and Structure Queries
NEXI query languagederived from XPath
allows mixture of content and structure
//scene[about(., poison juliet)]Return scene tags that are about poison and juliet.
//*[about(., plague both houses)]Return elements (of any type) about plague both houses.
//scene[about(., juliet chamber)]//speech[about(.//line, jewel)]Return speech tags about jewel where the scene is about juliet chamber.
![Page 40: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/40.jpg)
![Page 41: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/41.jpg)
Application: Text Classification
![Page 42: Beyond Bags of Words: A Markov Random Field Model for Information Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062423/568148f7550346895db61a0c/html5/thumbnails/42.jpg)
Conclusions