Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning...
Transcript of Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning...
![Page 1: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/1.jpg)
Graph partitioning algorithms
Leonid E. Zhukov
School of Data Analysis and Artificial IntelligenceDepartment of Computer Science
National Research University Higher School of Economics
Network Science
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 1 / 32
![Page 2: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/2.jpg)
Lecture outline
1 Graph paritioningMetricsAlgorithms
2 Spectral optimizationMin cutNormalized cutModularity maximization
3 Multilevel spectral
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 2 / 32
![Page 3: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/3.jpg)
Network communities
Definition
Network communities are groups of vertices such that vertices inside thegroup connected with many more edges than between groups.
Graph partitioning problem
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 3 / 32
![Page 4: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/4.jpg)
Graph partitioning
Combinatorial problem:
Number of ways to divide network of n nodes in 2 groups(bi-partition):
n!
n1!n2!, n = n1 + n2
Dividing into k non-empty groups (Stirling numbers of the secondkind)
S(n, k) =1
k!
n∑j=0
(−1)jC jk(k − j)n
Number of all possible partitions (n-th Bell number):
Bn =n∑
k=1
S(n, k)
B20 = 5, 832, 742, 205, 057
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 4 / 32
![Page 5: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/5.jpg)
Community detection
Consider only sparse graphs m� n2
Each community should be connected
Combinatorial optimization problem:- optimization criterion- optimization method
Exact solution NP-hard(bi-partition: n = n1 + n2, n!/(n1!n2!) combinations)
Solved by greedy, approximate algorithms or heuristics
Recursive top-down 2-way partition, multiway partiton
Balanced class partition vs communities
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 5 / 32
![Page 6: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/6.jpg)
Graph cut
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 6 / 32
![Page 7: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/7.jpg)
Optimization criterion: graph cut
Graph G (E ,V ) partition: V = V1 + V2
Graph cut
Q = cut(V1,V2) =∑
i∈V1,j∈V2
eij
Ratio cut:
Q =cut(V1,V2)
||V1||+
cut(V1,V2)
||V2||Normalized cut:
Q =cut(V1,V2)
Vol(V1)+
cut(V1,V2)
Vol(V2)
Quotient cut (conductance):
Q =cut(V1,V2)
min (Vol(V1),Vol(V2))
where: Vol(V1) =∑
i∈V1,j∈V eij =∑
i∈V1ki
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 7 / 32
![Page 8: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/8.jpg)
Optimization methods
Greedy optimization:– Local search [Kernighan and Lin, 1970], [Fidducia and Mettheyses,1982]
Approximate optimization:– Spectral graph partitioning [M. Fiedler, 1972], [Pothen et al 1990],[Shi and Malik, 2000]– Multicommodity flow [Leighton and Rao, 1988]
Heuristics algorithms:– Multilevel graph partitioning (METIS) [G. Karypis, Kumar 1998]
Randomized algorithms:–Randomized min cut [D. Karger, 1993]
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 8 / 32
![Page 9: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/9.jpg)
Graph cuts
Let V = V+ + V− be partitioning of the nodes
Let s = {+1,−1,+1, ...− 1,+1}T - indicator vector
x-1 x-1 x+1 x+1 x+1
s(i) =
{+1 : if v(i) ∈ V+
−1 : if v(i) ∈ V−
Number of edges, connecting V+ and V−
cut(V+,V−) =1
4
∑e(i ,j)
(s(i)− s(j))2 =1
8
∑i ,j
Aij(s(i)− s(j))2 =
=1
4
∑i ,j
(kiδijs(i)2 − Aijs(i)s(j)) =1
4
∑i ,j
(kiδij − Aij)s(i)s(j)
cut(V+,V−) =1
4
∑i ,j
(Dij − Aij)s(i)s(j)
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 9 / 32
![Page 10: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/10.jpg)
Graph cuts
Graph Laplacian: Lij = Dij − Aij , where Dii = diag(ki )
Lij =
k(i) , if i = j−1 , if ∃ e(i , j)
0 , otherwise
Laplacian matrix 5x5:
L =
1 −1−1 2 −1
−1 2 −1−1 2 −1
−1 1
x1 x2 x3 x4 x5
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 10 / 32
![Page 11: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/11.jpg)
Graph cuts
Graph Laplacian: L = D− A
Graph cut:
Q(s) =sTLs
4
Minimal cut:mins
Q(s)
Balanced cut constraint: ∑i
s(i) = 0
Integer minimization problem, exact solution is NP-hard!
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 11 / 32
![Page 12: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/12.jpg)
Spectral method - relaxation
Discrete problem → continuous problem
Discrete problem: find
mins
(1
4sTLs)
under constraints: s(i) = ±1,∑
i s(i) = 0;
Relaxation - continuous problem: find
minx
(1
4xTLx)
under constraints:∑
i x(i)2 = n ,∑
i x(i) = 0
Given x(i), round them up by s(i) = sign(x(i))
Exact constraint satisfies relaxed equation, but not other way around!
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 12 / 32
![Page 13: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/13.jpg)
Spectral method - computations
Constraint optimization problem (Lagrange multipliers):
Q(x) =1
4xTLx− λ(xTx− n), xTe = 0
Eigenvalue problem:Lx = λx, x ⊥ e
Solution:Q(xi) =
n
4λi
First (smallest) eigenvector:
Le = 0, λ = 0, x1 = e
Looking for the second smallest eigenvalue/eigenvector λ2 and x2
Minimization of Rayleigh-Ritz quotient:
minx⊥x1
(xTLx
xTx)
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 13 / 32
![Page 14: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/14.jpg)
Spectral graph theory
λ1 = 0
Number of λi = 0 equal to the number of connected components
0 ≤ λ2 ≤ 2λ2 = 0, disconnected graphλ2 = 1, totally connected
Graph diameter (longest shortest path)
D(G ) >=4
nλ2
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 14 / 32
![Page 15: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/15.jpg)
Spectral graph partitioning algorithm
Algorithm: Spectral graph partitioning - normalized cuts
Input: adjacency matrix A
Output: class indicator vector s
compute D = diag(deg(A));
compute L = D− A;
solve for second smallest eigenvector:min cut: Lx = λx;
normalized cut : Lx = λDx;
set s = sign(x2)
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 15 / 32
![Page 16: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/16.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 16 / 32
![Page 17: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/17.jpg)
Example
Eigenvalues: λ1 = 0, λ2 = 0.2, λ3 = 0.25 ...
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 17 / 32
![Page 18: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/18.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 18 / 32
![Page 19: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/19.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 19 / 32
![Page 20: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/20.jpg)
Spectral ordering
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 20 / 32
![Page 21: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/21.jpg)
Cut metrics
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 21 / 32
![Page 22: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/22.jpg)
Optimization criterion: modularity
Modularity:
Q =1
2m
∑ij
(Aij −
kikj2m
)δ(ci , cj)
where nc - number of classes and
δ(ci , cj) =
{1 : if ci = cj0 : if ci 6= cj
- kronecker delta
[Maximization!]
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 22 / 32
![Page 23: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/23.jpg)
Spectral modularity maximization
Direct modularity maximization for bi-partitioning, [Newman, 2006]
Let two classes c1 = V+, c2 = V−, indicator variable s = ±1
δ(ci , cj) =1
2(si sj + 1)
Modularity
Q =1
4m
∑ij
(Aij −
kikj2m
)(si sj + 1) =
1
4m
∑i ,j
Bijsi sj
where
Bij = Aij −kikj2m
M. Newman, 2006
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 23 / 32
![Page 24: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/24.jpg)
Spectral modularity maximization
Qudratic form:
Q(s) =1
4msTBs
Integer optimization - NP, relaxation s → x , x ∈ R
Keep norm ||x ||2 =∑
i x2i = xTx = n
Quadratic optimization
Q(x) =1
4mxTBx− λ(xTx− n)
Eigenvector problemBxi = λixi
Approximate modularity
Q(xi) =n
4mλi
Modularity maximization - largest λ = λmax
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 24 / 32
![Page 25: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/25.jpg)
Spectral modularity maximization
Can’t choose s = xk, can select optimal s
Decompose in the basis: s =∑
j ajxj, where aj = xTj s
Modularity
Q(s) =1
4msTBs =
1
4m
∑i
(xTi s)2λi
maxQ(s) reached when λ1 = λmax and max xT1 s =∑
j x1jsj
Choose s as close as possible to x, i.e. maxs(sTx) = maxsi∑
sixi :si = +1, if xi > 0si = −1, if xi < 0
Choose s ‖ x1, s = sign(x1)
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 25 / 32
![Page 26: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/26.jpg)
Modularity maximization
Algorithm: Spectral moduldarity maximization: two-way parition
Input: adjacency matrix A
Output: class indicator vector s
compute k = deg(A);
compute B = A− 12mkkT ;
solve for maximal eigenvector Bx = λx;
set s = sign(xmax)
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 26 / 32
![Page 27: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/27.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 27 / 32
![Page 28: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/28.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 28 / 32
![Page 29: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/29.jpg)
Example
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 29 / 32
![Page 30: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/30.jpg)
Multilevel spectral
recursive partitioning
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 30 / 32
![Page 31: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/31.jpg)
Multilevel spectral
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 31 / 32
![Page 32: Leonid E. ZhukovLeonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32 Title Graph partitioning algorithms Author Leonid E. Zhukov Created Date 3/11/2016 11:53:00 PM ...](https://reader033.fdocuments.us/reader033/viewer/2022042808/5f83d535d8b43d54666681bd/html5/thumbnails/32.jpg)
References
M. Fiedler. Algebraic connectivity of graphs, Czech. Math. J, 23, pp298-305, 1973A. Pothen, H. Simon and K. Liou. Partitioning sparse matrices witheigenvectors of graphs, SIAM Journal of Matrix Analysis, 11, pp430-452, 1990Bruce Hendrickson and Robert Leland. A Multilevel Algorithm forPartitioning Graphs, Sandia National Laboratories, 1995Jianbo Shi and Jitendra Malik. Normalized Cuts and ImageSegmentation, IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 22, N 8, pp 888-905, 2000M.E.J. Newman. Modularity and community structure in networks.PNAS Vol. 103, N 23, pp 8577-8582, 2006B. Good, Y.-A. de Montjoye, A. Clauset. Performance of modularitymaximization in practical contexts, Physical Review E 81, 046106,2010
Leonid E. Zhukov (HSE) Lecture 9 05.03.2016 32 / 32