Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR...
-
Upload
mollie-willingham -
Category
Documents
-
view
225 -
download
2
Transcript of Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR...
![Page 1: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/1.jpg)
Fast SDP Relaxations of Graph Cut Clustering,
Transduction, and Other Combinatorial Problems
(JMLR 2006)
Tijl De Bie and Nello Cristianini
Presented by Lihan HeMarch 16, 2007
![Page 2: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/2.jpg)
Outline
Statement of the problem
Spectral relaxation and eigenvector
SDP relaxation and Lagrange dual
Generalization: between spectral and SDP
Transduction and side information
Experiments
Conclusions
![Page 3: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/3.jpg)
Statement of the problem
Data set S: niixS 1}{
Affinity matrix A: ),(),( ji xxajiA
Objective: graph cut clustering -- divide the data points into two set, P and N, such that NPSNP ,
No label: clusteringWith some labels: transduction
![Page 4: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/4.jpg)
Statement of the problem
Normalized graph cut problem (NCut)
where
Cut costHow well the clusters are balanced
Cost function:
![Page 5: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/5.jpg)
Statement of the problem
Normalized graph cut problem (NCut)
ny }1 ,1{Unknown label vector
Let )( , ddiagDAd 1
Write ,2/)('),( ydSPassocs 1 2/)('),( ydSNassocs 1
1 'dsss
Rewrite the NCut problem as a combinatorial optimization problem
.
,'
,}1 ,1{ s.t.
)('4
min,,
sss
ssyd
y
yADyss
s
n
ssy
NP-complete problem, the exponent is very high.
(1)
![Page 6: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/6.jpg)
Spectral Relaxation
Let ys
dI
ss
sy
'
4~ 1
the problem becomes
sss
ys
dI
ss
sy
'
4~ 1
1~ '~ yDy
Relax the constraints by adding and dropping the combinatorial constraints on , we obtain the spectral clustering relaxationy~
(2)
1~ '~ yDy
![Page 7: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/7.jpg)
Spectral Relaxation: eigenvector
Solution: the eigenvector corresponding to the second smallest generalized eigenvalue.
Solve the constrained optimization by Lagrange dual:
)1~'~(~)('~),~( yDyyADyyL
0~~)(2~
),~(
yDyADy
yL yDyAD ~~)(
The second constraint is automatically satisfied:
1 11 ,0 v
212 ,~ vvvy 0~' i.e., ,0~' ydyD1
![Page 8: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/8.jpg)
SDP Relaxation
Let 'yy
the problem becomes
1 )( ,0 diagNote that
Relax the constraints by adding the above constraints and dropping
'yy ny }1 ,1{and
Let
ss
sq
4
2
and ,ˆ q we obtain the SDP relaxation
(3)
nnR ̂
![Page 9: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/9.jpg)
SDP Relaxation: Lagrange dual
Lagrangian:
0),,,,ˆ(
0ˆ
),,,,ˆ(
q
qL
qL
We obtain the dual problem (strong dual is hold):
(4)n+1 variables
![Page 10: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/10.jpg)
Generalization: between spectral and SDP
A cascade of relaxations tighter than spectral and looser than SDP
1qdiag )ˆ( 1')ˆ(' qWdiagW .1 , nmRW mn where
m+1 variables
n constraints m constraints, Looser than SDP
Design the structure of W design how to relax the constraints
nm 1
![Page 11: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/11.jpg)
Generalization: between spectral and SDP
rank(W)=n: original SDP relaxation.
rank(W)=1: m=1, W=d: spectral relaxation.
A relaxation is tighter than another if the column space of the matrix W used in the first one contains the full column space of W of the second.
If choose d within the column space of W, then all relaxations in the cascade are tighter than the spectral relaxation.
One approach of designing W proposed by the author:
Sort the entries of the label vector (2nd eigenvector) from spectral relaxation;Construct partition: m subsets are roughly equally large;Reorder the data points by this sorted order;W
~ n/m
mn
W=
1…1
1…1
1…1
…
1 2 m…
![Page 12: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/12.jpg)
Transduction
Given some labels, written as label vector yt -- transductive problem
Reparameterize 'ˆ :ˆ LML
Label constraints are imposed:
L=
yt 0
0 I
Labeled
Unlabeled
)1( testnn
1
1
ty
Rows (columns) corresponding to oppositely labeled training points then automatically are each other’s opposite;
Rows (columns) corresponding to same-labeled training points are equal to each other.
![Page 13: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/13.jpg)
Transduction
Transductive NCut relaxation:
ntest+2 variables)1()1( testtest nnRM
![Page 14: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/14.jpg)
General constraints
An equivalence constraint between two sets of data points specifies that they belong to the same class;
An inequivalence constraint specifies two set of data points to belong to opposite classes.
No detailed label information provided.
![Page 15: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/15.jpg)
Experiments
1. Toy problems Affinity matrix: )2/||||exp(),( 22 ji xxjiA
![Page 16: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/16.jpg)
Experiments
2. Clustering and transduction on text
Data set: 195 articles4 languages
several topics
Affinity matrix: 20-nearest neighbor: A(i,j)= 1
0.50
Distance of two articles: cosine distance on the bag of words representation
NwNwwi Rfffv ]',...,,[ 21
},...,,{ 21 NwwwDefine dictionary
),cos(1 jiij vvd
![Page 17: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/17.jpg)
Experiments
2. Clustering and transduction on text: cost
By language By topic
Spectral (randomized rounding)
SDP (randomized rounding)
Spectral (lower bound)
SDP (lower bound)
Cost: randomized rounding ≥ opt ≥ lower bound
Cos
t
Cos
t
Fraction of labeled data points Fraction of labeled data points
![Page 18: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/18.jpg)
Experiments
2. Clustering and transduction on text: accuracy
By language By topic
Spectral (randomized rounding)
SDP (randomized rounding)
Acc
urac
y
Acc
urac
y
Fraction of labeled data points Fraction of labeled data points
![Page 19: Fast SDP Relaxations of Graph Cut Clustering, Transduction, and Other Combinatorial Problems (JMLR 2006) Tijl De Bie and Nello Cristianini Presented by.](https://reader035.fdocuments.us/reader035/viewer/2022062221/56649c9b5503460f94958917/html5/thumbnails/19.jpg)
Conclusions
Proposed a new cascade of SDP relaxations of the NP-complete normalized graph cut optimization problem;
One extreme: spectral relaxation;
The other extreme: newly proposed SDP relaxation;
For unsupervised and semi-supervised learning, and more general constraints;
Balance the computational cost and the accuracy.