Latent Semantic Indexing 1. Term-document matrices are very large, though most cells are “zeros”...
-
Upload
joleen-fields -
Category
Documents
-
view
212 -
download
0
Transcript of Latent Semantic Indexing 1. Term-document matrices are very large, though most cells are “zeros”...
Latent Semantic Indexing
1
Latent Semantic Indexing
Term-document matrices are very large, though most cells are “zeros”
But the number of topics that people talk about is small (in some sense)
Clothes, movies, politics, … Each topic can be represented as a
cluster of (semantically) related terms, e.g.: clothes=golf, jacket, shoe..
Can we represent the term-document space by a lower dimensional “latent” space (latent space=set of topics)?
2
Searching with latent topics
3
Given a collection of documents, LSI learns clusters of frequently co-occurring terms (ex: information retrieval, ranking and web)
If you query with ranking, information retrieval LSI “automatically” extends the search to documents including also (and even ONLY) web
4
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
Selection based on ‘Golf’
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarPetrol
TopgearGTIPolo
Document base (20)
MotorBikeOil
PetrolTourer
BedlacelegalPetrolbutton
softPetrol
catline
yellow
windfullsail
harbourbeach
reportPetrol
TopgearJune
Speed
FishPondgold
PetrolKoi
PCDellRAMPetrolFloppy
CorePetrolApplePipTree
PeaPod
FreshGreenFrench
LupinPetrolSeedMay April
OfficePenDesk
Petrol VDU
FriendPal
HelpPetrolCan
PaperPetrolPastePencilRoof
CardStampGlue
HappySend
ToilPetrolWorkTimeCost
With standard VSM4 documents are selected
Golf
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
Selection based on ‘Golf’
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarPetrol
TopgearGTIPolo
20 documents
MotorBikeOil
PetrolTourer
BedlacelegalPetrolbutton
softPetrol
catline
yellow
windfullsail
harbourbeach
reportPetrol
TopgearJune
Speed
FishPondgold
PetrolKoi
PCDellRAMPetrolFloppy
CorePetrolApplePipTree
PeaPod
FreshGreenFrench
LupinPetrolSeedMay April
OfficePenDesk
Petrol VDU
FriendPal
HelpPetrolCan
PaperPetrolPastePencilRoof
CardStampGlue
HappySend
ToilPetrolWorkTimeCost
The most relevant wordsassociated with golf in these
docs are:Car, Topgear and Petrol
Rank ofselected
documents
Car 2 *(20/3) = 13
Topgear 2 *(20/3) = 13
Petrol 3 *(20/16) = 4
wf.idf
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
Rank ofselected
docsSelezione basata su ‘Golf’
Car 2 *(20/3) = 13
Topgear 2 *(20/3) = 13
Petrol 3 *(20/16) = 4
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarPetrol
TopgearGTIPolo
20 docs
MotorBikeOil
PetrolTourer
BedlacelegalPetrolbutton
softPetrol
catline
yellow
windfullsail
harbourbeach
reportPetrol
TopgearJune
Speed
FishPondgold
PetrolKoi
PCDellRAMPetrolFloppy
CorePetrolApplePipTree
PeaPod
FreshGreenFrench
LupinPetrolSeedMay April
OfficePenDesk
Petrol VDU
FriendPal
HelpPetrolCan
PaperPetrolPastePencilRoof
CardStampGlue
HappySend
ToilPetrolWorkTimeCost
Since words are weighted also wrt the idf,
car e topgear turn out to be related to Golfmore than petrol weight of
co-occurrences
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarPetrol
TopgearGTIPolo
20 docs
MotorBikeOil
PetrolTourer
BedlacelegalPetrolbutton
softPetrol
catline
yellow
windfullsail
harbourbeach
reportPetrol
TopgearJune
Speed
FishPondgold
PetrolKoi
PCDellRAMPetrolFloppy
CorePetrolApplePipTree
PeaPod
FreshGreenFrench
LupinPetrolSeedMay April
OfficePenDesk
Petrol VDU
FriendPal
HelpPetrolCan
PaperPetrolPastePencilRoof
CardStampGlue
HappySend
ToilPetrolWorkTimeCost
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
rankdf selected
docsSelection based on‘Golf’
Selection based on the semantic domain of GolfCar
2 *(20/3) = 13Topgear
2 *(20/3) = 13Petrol
3 *(20/16) = 4
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarWheel
TopgearGTIPolo
We now search with allthe words in the “semantic domain” of Golf .
The list of retrieved docs nowIncludes one new document and
does not include a previouslyretrieved document.
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
rankof
selecteddocs
Selezione basata su ‘Golf’
Selection based on semantic domainCar
2 *(20/3) = 13Topgear
2 *(20/3) = 13Petrol
3 *(20/16) = 4
Rank 2617 17 030
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarPetrol
TopgearGTIPolo
20 docs
MotorBikeOil
PetrolTourer
BedlacelegalPetrolbutton
softPetrol
catline
yellow
windfullsail
harbourbeach
reportPetrol
TopgearJune
Speed
FishPondgold
PetrolKoi
PCDellRAMPetrolFloppy
CorePetrolApplePipTree
PeaPod
FreshGreenFrench
LupinPetrolSeedMay April
OfficePenDesk
Petrol VDU
FriendPal
HelpPetrolCan
PaperPetrolPastePencilRoof
CardStampGlue
HappySend
ToilPetrolWorkTimeCost
GolfCar
TopgearPetrolGTI
GolfCar
ClarksonPetrolBadge
GolfPetrol
TopgearPoloRed
GolfTiger
WoodsBelfryTee
CarWheel
TopgearGTIPolo
The co-occurrence basedranking improves the performance.
Note that one of the most relevant doc does NOT include the word Golf, and
a doc with a “spurious”sense disappears
Linear Algebra Background
9
The LSI method: how to detect “topics”
Eigenvalues & Eigenvectors
Eigenvectors (for a square mm matrix S)
How many eigenvalues are there at most?
only has a non-zero solution if
this is a m-th order equation in λ which can have at most m distinct solutions (roots of the characteristic polynomial) – can be complex even though S is real.
eigenvalue(right) eigenvector
Example
10
Example of eigenvector/eigenvalues
Def: A vector v Rn, v ≠ 0, is an eigenvector of a matrix mxm S with corresponding eigenvalue , if:Sv = v
Example of computationremember
2 and 4 are the eigenvalues of S Characteristic
polynomial
Notice that we computed only the DIRECTION of eigenvectors
Matrix-vector multiplication
13
Therefore, any generic vector (say x= ) can be viewed as
a combination of the eigenvectors: x = 2v1 + 4v2 + 6v3
2, 4 and 6 are the coordinates of v in the orthonormal space O
Eigenvectors define an orthonormal space O; two eigenvectors of different eigenvalues of a symmetric matrix are orthogonal.
6
4
2
Notice, there is not “the” eigenvectors for a matrix, but rather an orthonormal basis (an eigenspace)
Example of 2 bidimensional orthonormal spaces
14
Difference between orthonormal and orthogonal?
Orthogonal mean that the dot product is null. Orthonormal mean that the dot product is null and the norm is equal to 1.
If two or more vectors are orthonormal they are also orthogonal but the inverse is not true.
15
Matrix vector multiplication
Thus a matrix-vector multiplication such as Sx (S, x a generic vector as in the previous slide) can be rewritten in terms of the eigenvalues/vectors of S:
Even though x is an arbitrary vector, the action of S on x (transformation) is determined by the eigenvalues/vectors.
Why?16
X=
6
4
2
Matrix vector multiplication
17
Matrix multipliication by a vector = a linear transformation of the initial vector
Geometric explanation
Multiplying a matrix and a vector has two effects over the vector: rotation (the coordinates of the vector change) and scaling (the length changes). The max compression and rotation depends on the matrix’s eigenvalues λi (s1,s2 and s3 in the picture)
x is a generic vector, v are the eigenvectors)
Geometric explanation
In the distorsion, the max effect is played by thebiggest eigenvalues (s1 and s2 in the picture )
The eigenvalues describe the distorsion operated by the matrix on the original vector
Geometric explanation
In this linear transformation, the image of “La Gioconda” is modifiedbut the central vertical axis is the same. The blue vector has changed, not the red one. Hence the red is an eigenvector of the linear transformation. Furthermore, the red vector has not been enlarged, not shortened, nor inverted, therefore its eigenvalue is 1 (the eigenvalue is a constant of translation of the image pixels along the blue direction) .
Sv=v
Let be a square matrix with m orthogonal eigenvectors
Theorem: Exists an eigen decomposition
(cf. matrix diagonalization theorem)
Columns of U are eigenvectors of S Diagonal elements of are eigenvalues of
Eigen/diagonal Decomposition
diagonal
Unique for
distinct eigen-values
21
Diagonal decomposition: why/how
nvvU ...1Let U have the eigenvectors as columns:
n
nnnn vvvvvvSSU
............
1
1111
Then, SU can be written
And S=UU–1.
Thus SU=U, or U–1SU=
22
Example
23
From which we get v21=0 and v22 any real
from which we get v11=−2v12
Diagonal decomposition – example 2
Recall .3,1 ;21
1221
S
The eigenvectors and form
1
1
1
1
11
11U
Inverting, we have
2/12/1
2/12/11U
Then, S=UU–1 =
2/12/1
2/12/1
30
01
11
11
RecallUU–1 =1.
24
So what?
What do these matrices have to do with IR? Recall M N term-document matrices … But everything so far needs square matrices – so
you need to be patient and learn one last notion
25
Singular Value Decomposition for non-square matrixes
TVUA
MM MN V is NN
For an M N matrix A of rank r there exists a factorization(Singular Value Decomposition = SVD) as follows:
The columns of U are orthogonal eigenvectors of AAT
(left singular vectors).The columns of V are orthogonal eigenvectors of ATA (right
singular eigenvector)
ii rdiag ...1 Singular values.
Eigenvalues 1 … r of AAT = eigenvalues of ATA and:
26
An example
27
Singular Value Decomposition
Illustration of SVD dimensions and sparseness
If we retain only k singular values, and set the rest to 0, then we don’t need the matrix parts in red
Then Σ is k×k, U is M×k, VT is k×N, and Ak is M×N This is referred to as the reduced SVD
It is the convenient (space-saving) and usual form for computational applications
http://www.bluebit.gr/matrix-calculator/calculate.aspx
Reduced SVD
k 29
Let’s recap
30
Since the yellow part is zero, an exact representation of A is:
But for some k<r, a good approximation is:
M<N
M>N
Example of rank approximation
31
= x x
0.981 0.000 0.000 0.000 1.9850.000 0.000 3.000 0.000 0.0000.000 0.000 0.000 0.000 0.0000.000 4.000 0.000 0.000 0.000
A*=
Approximation error
How good (bad) is this approximation? It’s the best possible, measured by the Frobenius
norm of the error:
where the i are ordered such that i i+1. Suggests why Frobenius error drops as k increases.
1)(:
min
kFkFkXrankX
AAXA
32
ii
SVD Low-rank approximation
Whereas the term-doc matrix A may have M=50000, N=10 million (and rank close to 50000)
We can construct an approximation A100 with rank 100!!! Of all rank 100 matrices, it would have the lowest
Frobenius error.
C. Eckart, G. Young, The approximation of a matrix by another of lower rank. Psychometrika, 1, 211-218, 1936.
33
Images gives a better intuition
34
K=10
35
K=20
36
K=30
37
K=100
38
K=322 (the original image)
39
We save space!! But this is only one issue
40
41
But what about texts and IR? (finally!!)
Latent Semantic Indexing via the SVD
If A is a term/document matrix, then AAT and AT A are the matrixes of term and document co-occurrences, repectively
Example 2
Term-document matrix
A
Co-occurrences of terms in documents
Lij is the number of documents in which wi and wj co-occurr
LijT is the number of co-occurring words in docs di and dj
(or viceversa if A is a document-term matrix)
Word i in doc k Word j in doc k
Term co-occurrences example
L trees,graph = (000001110) (000000111)T=2
A numerical example
2 2 0
2 2 0
0 0 1
w1
w2
w3
d1 d2 d3
1 0 1
1 0 1
0 1 0
AAT
1 1 0
0 0 1
1 1 0
=
w11 w12 w13 w21 w22 w23w31 w32 w33
But wij=wji hencew11 w12 w13 w12 w22 w23w13 w23 w33Symmetrical
LSI: the intuition
47
t1
t3t2
Projecting A in the termspace: green yellow andred vector are documents
The black vectorare the unary eigenvector of A: they represent the main “directions” of thedocument vectors
The blue segments give theintuition of eigenvalues of A:bigger values are those forwhich the projection of allvectors on the direction ofcorrespondent eigenvectorIs higher
LSI intuition
48
t1
t3t2We now project our document vectors on the reference orthonormalsystem represented by the 3 black vectors
LSI intuition
49
..and we remove the dimension(s)with lower eigenvalues
Now the two axis represent a combination of words e.g.a latent semantic space
Example
50
1.000 1.000 0.000 0.0001.000 0.000 0.000 0.0000.000 0.000 1.000 1.0000.000 0.000 1.000 1.000
t1
t2
t3
t4
d1 d2 d3 d4
0.000 -0.851 -0.526 0.000 0.000 -0.526 0.851 0.000-0.707 0.000 0.000 -0.707-0.707 0.000 0.000 0.707
2.000 0.000 0.000 0.0000.000 1.618 0.000 0.0000.000 0.000 0.618 0.0000.000 0.000 0.000 0.000
0.000 0.000 -0.707 -0.707-0.851 -0.526 0.000 0.000 0.526 -0.851 0.000 0.000 0.000 0.000 -0.707 0.707
x x
1.172 0.724 0.000 0.0000.724 0.448 0.000 0.0000.000 0.000 1.000 1.0000.000 0.000 1.000 1.000
We project termsand docs on two dimensions, v1 and v2(the principal eigenvectors)
We have two latent semanticcoordinates: s1:(t1,t2) ands2:(t3,t4)
Approximatednew term-docmatrix
Even if t2 does not occur in d2, now if we query with t2 the system will return also d2!!
Co-occurrence space
In principle, the space of document or term co-occurrences is much (much!) higher than the original space of terms!!
But with SVD we consider only the most relevant ones, trough rank reduction
51
LSI: what are the steps
From term-doc matrix A, compute the approximation Ak. with SVD
Project docs, terms and queries in a space of k<<r dimensions (the k “survived” eigenvectors) and compute cos-similarity as usual These dimensions are not the original
axes, but those defined by the orthonormal space of the reduced matrix Ak
52
Where σiqi (i=1,2..k<<r) are the new coordinates of q in the orthonormal space of Ak
Projecting terms documents and queries in the LS space
53
A
If A=UVT we also have that:
V = ATU-1
t = t∧TVT
d = d∧TU-1
q = q∧TU-1
After rank k-
approximation :A≅Ak=UkkVk
T
dk ≅ dTUkk-1
qk ≅ qTUkk-1
sim(q, d) = sim(qTUkk
-1, dTUkk-
1)
Consider a term-doc matrix MxN (M=11, N=3) and a query q
query
A
1. Compute SVD: A= UVT
2. Obtain a low rank approximation (k=2) Ak= UkkVT
k
3a. Compute doc/query similarity
For N documents, Ak has N columns, each representing the coordinates of a document d i
projected in the k LSI dimensions
A query is considered like a document, and is projected in the LSI space
3c. Compute the query vector
qk = qTUkk-1
q is projected in the 2-dimension LSI space!
Documents and queries projected in the LSI space
q/d similarity
61
An overview of a semantic network of terms based on the top 100 most significant latent semantic dimensions (Zhu&Chen)
Another LSI Example
t1t2t3t4
d1 d2 d3
AT
t1t2t3t4
Term co-occurrencesAAT
Ak=UkΣkVkT≈A
Now it is like if t3 belongs to d1!