CS276A Text Information Retrieval, Mining, and Exploitation Lecture 15 26 Nov 2002.
CS276A Text Information Retrieval, Mining, and Exploitation Lecture 4 15 Oct 2002.
-
Upload
lucinda-garrett -
Category
Documents
-
view
214 -
download
1
Transcript of CS276A Text Information Retrieval, Mining, and Exploitation Lecture 4 15 Oct 2002.
Recap of last time
Index size Index construction techniques Dynamic indices Real world considerations
Back of the envelope index size calculation
Number of docs = n = 40M Number of terms = m = 1M Use Zipf to estimate number of postings
entries: n + n/2 + n/3 + …. + n/m ~ n ln m = 560M
postings entries This is just a word-document index, not
one that includes positional information
Merge sort of 56 sorted runs
Merge tree of log256 ~ 6 layers. During each layer, read into memory runs
in blocks of 10M, merge, write back.
Disk
1
3 4
22
1
4
3
Merge sort of 56 sorted runs
How do you write back long merged runs? Wait to accumulate 10M-sized output
blocks before writing back. Thus amortize seek time over block
transfer.
Disk
1
3 4
2
2
1
4
3
Today’s topics
Ranking models The vector space model
Inverted indexes with term weighting Evaluation with ranking models
Ranking models in IR
Key idea: We wish to return in order the documents
most likely to be useful to the searcher To do this, we want to know which
documents best satisfy a query An obvious idea is that if a document talks
about a topic more then it is a better match A query should then just specify terms that
are relevant to the information need, without requiring that all of them must be present Document relevant if it has a lot of the terms
Binary term presence matrices
Record whether a document contains a word: document is binary vector in {0,1}v
What we have mainly assumed so far Idea: Query satisfaction = overlap
measure:
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
YX
Overlap matching
What are the problems with the overlap measure?
It doesn’t consider: Term frequency in document Term scarcity in collection (document
mention frequency) Length of documents
(And queries: score not normalized)
Overlap matching
One can normalize in various ways: Jaccard coefficient:
Cosine measure:
What documents would score best using Jaccard against a typical query? Does the cosine measure fix this problem?
YXYX /
YXYX /
Count term-document matrices
We haven’t considered frequency of a word
Count of a word in a document: Bag of words model Document is a vector in ℕv
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 157 73 0 0 0 0
Brutus 4 157 0 1 0 0
Caesar 232 227 0 2 1 1
Calpurnia 0 10 0 0 0 0
Cleopatra 57 0 0 0 0 0
mercy 2 0 3 5 5 1
worser 2 0 1 1 1 0
Normalization: Calpurnia
vs. Calphurnia
Weighting term frequency: tf
What is the relative importance of 0 vs. 1 occurrence of a term in a doc 1 vs. 2 occurrences 2 vs. 3 occurrences …
Unclear: but it seems that more is better, but a lot isn’t necessarily better than a few Can just use raw score Another option commonly used in practice:
0:log1?0 ,, dtdt tftf
Dot product matching
Match is dot product of query and document
[Note: 0 if orthogonal (no words in common)] Rank by match
It still doesn’t consider: Term scarcity in collection (document mention
frequency) Length of documents and queries
Not normalized
i diqi tftfdq ,,
Weighting should depend on the term overall
Which of these tells you more about a doc? 10 occurrences of hernia? 10 occurrences of the?
Suggest looking at collection frequency (cf) But document frequency (df) may be
better:Word cf dftry 10422 8760insurance 10440 3997
Document frequency weighting is only possible in known (static) collection.
tf x idf term weights
tf x idf measure combines: term frequency (tf)
measure of term density in a doc inverse document frequency (idf)
measure of informativeness of term: its rarity across the whole corpus
could just be raw count of number of documents the term occurs in (idfi = 1/dfi)
but by far the most commonly used version is:
See Kishore Papineni, NAACL 2, 2002 for theoretical justification
dfnidf
i
i log
Summary: tf x idf (or tf.idf)
Assign a tf.idf weight to each term i in each document d
Increases with the number of occurrences within a doc Increases with the rarity of the term across the whole corpus
)/log(,, ididi dfntfw
rmcontain te that documents ofnumber thedocuments ofnumber total
document in termoffrequency ,
idfn
jitf
i
di
What is the wtof a term thatoccurs in allof the docs?
Real-valued term-document matrices
Function (scaling) of count of a word in a document: Bag of words model Each is a vector in ℝv
Here log scaled tf.idf
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 13.1 11.4 0.0 0.0 0.0 0.0
Brutus 3.0 8.3 0.0 1.0 0.0 0.0
Caesar 2.3 2.3 0.0 0.5 0.3 0.3
Calpurnia 0.0 11.2 0.0 0.0 0.0 0.0
Cleopatra 17.7 0.0 0.0 0.0 0.0 0.0
mercy 0.5 0.0 0.7 0.9 0.9 0.3
worser 1.2 0.0 0.6 0.6 0.6 0.0
Documents as vectors
Each doc j can now be viewed as a vector of tfidf values, one component for each term
So we have a vector space terms are axes docs live in this space even with stemming, may have 20,000+
dimensions (The corpus of documents gives us a matrix,
which we could also view as a vector space in which words live – transposable data)
Why turn docs into vectors?
First application: Query-by-example Given a doc d, find others “like” it.
Now that d is a vector, find vectors (docs) “near” it.
Intuition
Postulate: Documents that are “close together” in vector space talk about the same things.
t1
d2
d1
d3
d4
d5
t3
t2
θ
φ
The vector space model
Query as vector: We regard query as short document We return the documents ranked by the
closeness of their vectors to the query, also represented as a vector.
Developed in the SMART system (Salton, c. 1970) and standardly used by TREC participants and web IR systems
Desiderata for proximity
If d1 is near d2, then d2 is near d1. If d1 near d2, and d2 near d3, then d1 is not
far from d3. No doc is closer to d than d itself.
First cut
Distance between vectors d1 and d2 is the length of the vector |d1 – d2|. Euclidean distance
Why is this not a great idea? We still haven’t dealt with the issue of
length normalization Long documents would be more similar to
each other by virtue of length, not topic However, we can implicitly normalize by
looking at angles instead
Cosine similarity
Distance between vectors d1 and d2 captured by the cosine of the angle x between them.
Note – this is similarity, not distance
t 1
d2
d1
t 3
t 2
θ
Cosine similarity
Cosine of angle between two vectors The denominator involves the lengths of
the vectors So the cosine measure is also known as the
normalized inner product
n
i ki
n
i ji
n
i kiji
kj
kjkj
ww
ww
dd
ddddsim
1
2,1
2,
1 ,,),(
n
i jij wd1
2,Length
Cosine similarity exercises
Exercise: Rank the following by decreasing cosine similarity: Two docs that have only frequent words
(the, a, an, of) in common. Two docs that have no words in common. Two docs that have many rare words in
common (wingspan, tailfin).
Normalized vectors
A vector can be normalized (given a length of 1) by dividing each of its components by the vector's length
This maps vectors onto the unit circle:
Then, Longer documents don’t get more weight For normalized vectors, the cosine is
simply the dot product:
11 ,
n
i jij wd
kjkj dddd
),cos(
Exercise
Euclidean distance between vectors: Euclidean distance:
Show that, for normalized vectors, Euclidean distance gives the same closeness ordering as the cosine measure
n
i kijikj wwdd1
2,,
Example
Docs: Austen's Sense and Sensibility, Pride and Prejudice; Bronte's Wuthering Heights
cos(SAS, PAP) = .996 x .993 + .087 x .120 + .017 x 0.0 = 0.999 cos(SAS, WH) = .996 x .847 + .087 x .466 + .017 x .254 =
0.929
SaS PaP WHaffection 115 58 20jealous 10 7 11gossip 2 0 6
SaS PaP WHaffection 0.996 0.993 0.847jealous 0.087 0.120 0.466gossip 0.017 0.000 0.254
Digression: spamming indices
This was all invented before the days when people were in the business of spamming web search engines: Indexing a sensible passive document
collection vs. An active document collection, where
people (and indeed, service companies) are trying to shape documents in an attempt to achieve ranking function maximization
Digression: ranking in Machine Learning
Our problem is: Given document collection D and query q,
return a ranking of D according to relevance to q.
Such ranking problems have been much less studied in machine learning than classification/regression problems
But much more interest recently, e.g., W.W. Cohen, R.E. Schapire, and Y. Singer.
Learning to order things. Journal of Artificial Intelligence Research, 10:243–270, 1999.
And subsequent research
Digression: ranking in Machine Learning
Many “WWW” applications are ranking (aka ordinal regression) problems: Text information retrieval Image similarity search (QBIC) Book/movie recommendations
Collaborative filtering Meta-search engines
Summary: What’s the real point of using vector spaces?
Key: A user’s query can be viewed as a (very) short document.
Query becomes a vector in the same space as the docs.
Can measure each doc’s proximity to it. Natural measure of scores/ranking – no
longer Boolean.
Evaluation II
Evaluation of ranked results: You can return any number of results
ordered by similarity By taking various numbers of documents
(levels of recall), you can produce a precision-recall curve
Interpolated precision
If you can increase precision by increasing recall, then you should get to count that…
Evaluation
There are various other measures Precision at fixed recall
This is perhaps the most appropriate thing for web search: all people want to know is how many good matches there are in the first one or two pages of results
11-point interpolated average precision The standard measure in the TREC competitions:
you take the precision at 11 levels of recall varying from 0 to 1 by tenths of the documents, using interpolation (the value for 0 is always interpolated!), and average them
We’ll use more notions from linear algebra next lecture
Matrix, vector Transpose and product Rank Eigenvalues and eigenvectors.