Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and...
Transcript of Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and...
![Page 1: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/1.jpg)
Introduc)on to Informa)on Retrieval
Introduc)onto
Informa(onRetrieval
cs160
Introduction DavidKauchak
adapted from:
h6p://www.stanford.edu/class/cs276/handouts/lecture1‐intro.ppt
![Page 2: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/2.jpg)
Introduc)on to Informa)on Retrieval
Introduc)ons
Name/nickname
Dept.,collegeandyear Oneinteres)ngthingaboutyourself Whyareyoutakingthisclass?
Whattopics/materialwouldyouliketoseecovered? PlansaLergradua)on
![Page 3: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/3.jpg)
Introduc)on to Informa)on Retrieval
Administra)ve Webpage:www.cs.pomona.edu/classes/cs160/ Syllabus… Administra)vehandout… Classfeedback Inclasspar)cipa)on Workload
Homework1availablesoon Programmingassignment1availablesoon
Due)me?
![Page 4: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/4.jpg)
Introduc)on to Informa)on Retrieval
Informa)onretrieval(IR) WhatcomestomindwhenIsay“informa)onretrieval”?
WherehaveyouseenIR?Whataresomereal‐worldexamples/uses? Searchengines Filesearch(e.g.OSXSpotlight,WindowsInstantSearch,Google
Desktop)
Databases? Catalogsearch(e.g.library) Intranetsearch(i.e.corporatenetworks)
![Page 5: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/5.jpg)
Introduc)on to Informa)on Retrieval
Informa)onRetrieval
Informa)onRetrievalisfindingmaterialintextdocumentsofanunstructurednaturethatsa)sfyaninforma)onneedfromwithinlargecollec)onsofdigitallystoredcontent
5
![Page 6: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/6.jpg)
Introduc)on to Informa)on Retrieval
Informa)onRetrieval
Informa)onRetrievalisfindingmaterialintextdocumentsofanunstructurednaturethatsa)sfyaninforma)onneedfromwithinlargecollec)onsofdigitallystoredcontent
6
?
![Page 7: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/7.jpg)
Introduc)on to Informa)on Retrieval
Informa)onRetrieval
Informa)onRetrievalisfindingmaterialintextdocumentsofanunstructurednaturethatsa)sfyaninforma)onneedfromwithinlargecollec)onsofdigitallystoredcontent
7
• Find all documents about computer science
• Find all course web pages at Pomona
• What is the cheapest flight from LA to NY?
• Who is was the 15th president?
![Page 8: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/8.jpg)
Introduc)on to Informa)on Retrieval
Informa)onRetrieval
Informa)onRetrievalisfindingmaterialintextdocumentsofanunstructurednaturethatsa)sfyaninforma)onneedfromwithinlargecollec)onsofdigitallystoredcontent
8
What is the difference between an information need and a query?
![Page 9: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/9.jpg)
Introduc)on to Informa)on Retrieval
Informa)onRetrieval
Informa)onRetrievalisfindingmaterialintextdocumentsofanunstructurednaturethatsa)sfyaninforma)onneedfromwithinlargecollec)onsofdigitallystoredcontent
9
• Find all documents about computer science
• Find all course web pages at Pomona
• Who is was the 15th president?
Information need Query
“computer science”
Pomona AND college AND url-contains class
WHO=president NUMBER=15
![Page 10: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/10.jpg)
Introduc)on to Informa)on Retrieval
Structureddatatendstorefertoinforma)onin“tables”
Employee Manager Salary
Smith Jones 50000
Chang Smith 60000
50000 Ivy Smith
Typically allows numerical range and exact match (for text) queries, e.g., Salary < 60000 AND Manager = Smith.
IRvs.databases
![Page 11: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/11.jpg)
Introduc)on to Informa)on Retrieval
Unstructured(text)vs.structured(database)datain1996
11
![Page 12: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/12.jpg)
Introduc)on to Informa)on Retrieval
Unstructured(text)vs.structured(database)datain2006
12
![Page 13: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/13.jpg)
Introduc)on to Informa)on Retrieval
Unstructureddatain1680
WhichplaysofShakespearecontainthewordsBrutusANDCaesarbutNOTCalpurnia?
OnecouldgrepallofShakespeare’splaysforBrutusandCaesar,thenstripoutplayscontainingCalpurnia.Anyproblemswiththis? Slow(forlargecorpora) Otheropera)ons(e.g.,findthewordRomans nearcountrymen)notfeasible
Rankedretrieval(bestdocumentstoreturn) Laterlectures
13
![Page 14: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/14.jpg)
Introduc)on to Informa)on Retrieval
Unstructureddatain1680
WhichplaysofShakespearecontainthewordsBrutusANDCaesarbutNOTCalpurnia?
Howmightwespeedupthistypeofquery?
Indexing:foreachword,keeptrackofwhichdocumentsitoccursin
![Page 15: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/15.jpg)
Introduc)on to Informa)on Retrieval
Term‐documentincidencematrix
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
1 if play contains word, 0 otherwise
![Page 16: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/16.jpg)
Introduc)on to Informa)on Retrieval
Incidencevectors
Foreachterm,wehavea0/1vector Caeser=110111 Brutus=110100 Calpurnia=010000
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
![Page 17: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/17.jpg)
Introduc)on to Informa)on Retrieval
Incidencevectors
Foreachterm,wehavea0/1vector Caeser=110111 Brutus=110100 Calpurnia=010000
How can we get the answer from these vectors?
![Page 18: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/18.jpg)
Introduc)on to Informa)on Retrieval
Incidencevectors
Foreachterm,wehavea0/1vector Caeser=110111 Brutus=110100 Calpurnia=010000
BitwiseANDthevectorstogetherusingthecomplementedvectorforallNOTqueries
Caeser AND Brutus AND COMPLEMENT(Calpurnia) 110111&110100&~010000= 110111&110100&101111= 100100
![Page 19: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/19.jpg)
Introduc)on to Informa)on Retrieval
Answerstoquery
Antony and Cleopatra,Act III, Scene ii Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain.
Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me.
19
![Page 20: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/20.jpg)
Introduc)on to Informa)on Retrieval
Incidencevectors Foreachterm,wehavea0/1vector
Caeser=110111 Brutus=110100 Calpurnia=010000
BitwiseANDthevectorstogetherusingthecomplementedvectorforallNOTqueries
Any problem with this approach?
![Page 21: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/21.jpg)
Introduc)on to Informa)on Retrieval
Biggercollec)ons
ConsiderN =1milliondocuments,eachwithabout1000words
SaythereareM =500Kdis)ncttermsamongthese.Howbigistheincidencematrix?
Thematrixisa500Kby1millionmatrix=halfatrillion0’sand1’s Evenforamoderatesizeddatasetwecan’tstorethematrixinmemory
Eachvectorhas1millionentries Bitwiseopera)onsbecomemuchmoreexpensive
![Page 22: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/22.jpg)
Introduc)on to Informa)on Retrieval
Whatdoesthematrixlooklike?
ConsiderN =1milliondocuments,eachwithabout1000words
Extremelysparse!
Howmany1’sdoesthematrixcontain? nomorethanonebillion Eachofthe1milliondocumentshasatmost10001’s
Inprac)ce,we’llseethatthenumberofuniquewordsinadocumentismuchlessthanthis
What’sabe6errepresenta)on? Onlyrecordthe1posi)ons
![Page 23: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/23.jpg)
Introduc)on to Informa)on Retrieval
Invertedindex
Foreachterm,westorealistofalldocumentsthatcontainit
Whatdatastructuresmightweuseforthis?
23
Brutus
2 4 8 16 32 64 128 Brutus
2 4 8 16 32 64 128
Brutus 2 4 8 16 32 64 128
array
linked list
hashtable …
? docID
![Page 24: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/24.jpg)
Introduc)on to Informa)on Retrieval
Invertedindexrepresenta)on
24
Brutus 2 4 8 16 32 64 128 array
Pros Simpletoimplement Noextrapointersrequiredfordatastructure Con)guousmemory
Cons Howdowepickthesizeofthearray? Whatifwewanttoaddaddi)onaldocuments?
![Page 25: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/25.jpg)
Introduc)on to Informa)on Retrieval
Invertedindexrepresenta)on
25
Pros Dynamicspacealloca)on Inser)onofnewdocumentsisstraighmorward
Cons Memoryoverheadofpointers
Noncon)guousmemoryaccess
2 4 8 16 32 64 128 Brutus
linked list
![Page 26: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/26.jpg)
Introduc)on to Informa)on Retrieval
Invertedindexrepresenta)on
26
Pros Searchinconstant)me
Con)guousmemory
Cons Howdowepickthesize? Whatifwewanttoaddaddi)onaldocuments?
Mayhavetorehashifweincreaseinsize Togetconstant)meopera)ons,lotsofunusedslots/memory
Brutus 2 4 8 16 32 64 128
hashtable
![Page 27: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/27.jpg)
Introduc)on to Informa)on Retrieval
Invertedindex
27
Brutus
Calpurnia
Caesar
2 4 8 16 32 64 128
2 3 5 8 13 21 34
13 16
1
Dictionary Postings lists
Pos)ng
Themostcommonapproachistousealinkedlistrepresenta)on
![Page 28: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/28.jpg)
Introduc)on to Informa)on Retrieval
Invertedindexconstruc)onDocuments to be indexed
Friends, Romans, countrymen.
indexer
Inverted index
friend
roman
countryman
2 4
2
13 16
1
text preprocessing friend , roman , countrymen .
![Page 29: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/29.jpg)
Introduc)on to Informa)on Retrieval
Booleanretrieval Inthebooleanretrievalmodelweaskaquerythatisabooleanexpression: AbooleanqueryusesAND, ORandNOTtojoinqueryterms CaesarANDBrutusAND NOTCalpurnia PomonaANDCollege (MikeORMichael)ANDJordanAND NOT(NikeORGatorade)
Givenonlytheseopera)ons,whattypesofques)onscan’tweanswer? Phrases,e.g.“PomonaCollege” Proximity,“Michael”within2wordsof“Jordan”
29
![Page 30: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/30.jpg)
Introduc)on to Informa)on Retrieval
Booleanretrieval
Primarycommercialretrievaltoolfor3decades
Professionalsearchers(e.g.,lawyers)s)lllikebooleanqueries
Why? Youknowexactlywhatyou’regerng,aqueryeithermatchesoritdoesn’t
Throughtrialanderror,canfrequentlyfinetunethequeryappropriately
Don’thavetoworryaboutunderlyingheuris)cs(e.g.PageRank,termweigh)ngs,synonym,etc…)
30
![Page 31: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/31.jpg)
Introduc)on to Informa)on Retrieval
Example:WestLawhttp://www.westlaw.com/
Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)
Tens of terabytes of data; 700,000 users Majority of users still use boolean queries Example query:
What is the statute of limitations in cases involving the federal tort claims act?
LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM
All words starting with “LIMIT”
31
![Page 32: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/32.jpg)
Introduc)on to Informa)on Retrieval
Example:WestLawhttp://www.westlaw.com/
Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)
Tens of terabytes of data; 700,000 users Majority of users still use boolean queries Example query:
What is the statute of limitations in cases involving the federal tort claims act?
LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM
32
![Page 33: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/33.jpg)
Introduc)on to Informa)on Retrieval
Example:WestLawhttp://www.westlaw.com/
Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)
Tens of terabytes of data; 700,000 users Majority of users still use boolean queries Example query:
What is the statute of limitations in cases involving the federal tort claims act?
LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM
/3 = within 3 words, /S = in same sentence
33
![Page 34: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/34.jpg)
Introduc)on to Informa)on Retrieval
Example:WestLawhttp://www.westlaw.com/
Another example query: Requirements for disabled people to be able to access
a workplace disabl! /p acces\s! /s work-site work-place
(employment /3 place) Long,precisequeries;proximityoperators;incrementallydeveloped;notlikewebsearch
ProfessionalsearchersoLenlikeBooleansearch: Precision,transparencyandcontrol
Butthatdoesn’tmeantheyactuallyworkbe6er….
![Page 35: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/35.jpg)
Introduc)on to Informa)on Retrieval
Queryprocessing:AND Whatneedstohappentoprocess:BrutusANDCaesar
LocateBrutusandCaesarintheDic)onary; Retrievepos)ngslists
“Merge”thetwopos)ngs:
35
12834
2 4 8 16 32 641 2 3 5 8 13 21
BrutusCaesar
2 8BrutusANDCaesar
![Page 36: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/36.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
36
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar
![Page 37: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/37.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
37
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar
![Page 38: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/38.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
38
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar 2
![Page 39: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/39.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
39
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar 2
![Page 40: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/40.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
40
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar 2
![Page 41: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/41.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
41
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar 2
![Page 42: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/42.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
42
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
BrutusANDCaesar
…
2 8
![Page 43: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/43.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
43
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
What assumption are we making about the postings lists?
For efficiency, when we construct the index, we ensure that the postings lists are sorted
![Page 44: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/44.jpg)
Introduc)on to Informa)on Retrieval
Themerge
Walkthroughthetwopos)ngssimultaneously
44
128
34
2 4 8 16 32 64
1 2 3 5 8 13 21
Brutus
Caesar
What is the running time?
O(length1 + length2)
![Page 45: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/45.jpg)
Introduc)on to Informa)on Retrieval
Booleanqueries:Moregeneralmerges
Whichofthefollowingqueriescanwes)lldoin)meO(length1+length2)?
BrutusAND NOTCaesar
BrutusOR NOTCaesar
45
Sec. 1.3
![Page 46: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/46.jpg)
Introduc)on to Informa)on Retrieval
Merging
WhataboutanarbitraryBooleanformula?
(BrutusOR Caesar) AND NOT (Antony OR Cleopatra)
x=(BrutusOR Caesar) y=(Antony OR Cleopatra) x AND NOT y
Isthereanupperboundontherunning)me? O(total_terms*query_terms)
WhataboutBrutus AND Calpurnia AND Caesar?
46
![Page 47: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/47.jpg)
Introduc)on to Informa)on Retrieval
Queryop)miza)on
ConsideraquerythatisanANDoftterms.
Foreachoftheterms,getitspos)ngs,thenANDthemtogether
Whatisthebestorderforqueryprocessing?
Query: Brutus AND Calpurnia AND Caesar
47
Brutus
Calpurnia
Caesar
2 4 8 16 32 64 128
2 3 5 8 13 21 34
13 16
1
![Page 48: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/48.jpg)
Introduc)on to Informa)on Retrieval
Queryop)miza)onexample
Heuris)c:Processinorderofincreasingfreq: merge the two terms with the shortest pos)ngs list this creates a new AND query with one less term
repeat
48
Executethequeryas(CaesarANDBrutus)ANDCalpurnia.
Brutus
Calpurnia
Caesar
2 4 8 16 32 64 128
2 3 5 8 13 21 34
13 16
1
![Page 49: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/49.jpg)
Introduc)on to Informa)on Retrieval
Queryop)miza)on
ConsideraquerythatisanORoftterms.
Whatisthebestorderforqueryprocessing? Same:s)llwanttomergetheshortestpos)ngslistsfirst
Query: Brutus OR Calpurnia OR Caesar
49
Brutus
Calpurnia
Caesar
2 4 8 16 32 64 128
2 3 5 8 13 21 34
13 16
1
![Page 50: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/50.jpg)
Introduc)on to Informa)on Retrieval
50
Queryop)miza)oningeneral (maddingORcrowd)AND(ignobleORNOTstrife)
NeedtoevaluateORstatementsfirst WhichORshouldwedofirst?
Es)matethesizeofeachORbythesumofthepos)nglistlengths
NOTisjustthenumberofdocumentsminusthelength Then,itlookslikeanANDquery:
xANDy
![Page 51: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/51.jpg)
Introduc)on to Informa)on Retrieval
Exercise
Recommendaqueryprocessingorderfor
Term Freq eyes 213312 kaleidoscope 87009 marmalade 107913 skies 271658 tangerine 46653 trees 316812
51
(tangerine OR NOT trees) AND (marmalade OR skies) AND (kaleidoscope OR eyes)
![Page 52: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/52.jpg)
Introduc)on to Informa)on Retrieval
Nextsteps…
Phrases Pomona College
Proximity:FindGatesNEAR Microso;. Needindextocaptureposi)oninforma)onindocs.Morelater
Zonesindocuments:Finddocumentswith(author = Ullman) AND(textcontainsautomata)
Rankingsearchresults includeoccurrencefrequency weightdifferentzones/featuresdifferently(e.g.)tle,header,linktext,
…)
Incorporatelinkstructure
52
![Page 53: Introducon to Informaon Retrievaldkauchak/classes/f09/cs160-f09/... · 2009. 9. 23. · Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus](https://reader035.fdocuments.us/reader035/viewer/2022071512/6132d39edfd10f4dd73ab2ab/html5/thumbnails/53.jpg)
Introduc)on to Informa)on Retrieval
Resourcesfortoday’slecture
Introduc)ontoInforma)onRetrieval,ch.1
ManagingGigabytes,Chapter3.2 ModernInforma)onRetrieval,Chapter8.2 Shakespeare:h6p://www.rhymezone.com/shakespeare/ Trytheneatbrowsebykeywordsequencefeature!
53