Network Density of States - Cornell Universityarb/posters/Network-DOS-KDD-2019... · 2019. 10....
1
Network Density of States Kun Dong 1 Austin R. Benson 2 David Bindel 2 Applied Math 1 , CS 2 Background I Weyl’s Law connects the geometric properties of a manifold with the asymptotic spectral distribution of Laplace-Beltrami operator I Eigenvalues alone do not uniquely identify an object, but what can we learn? . How do we compute spectrum for large real-world graphs? . What geometric properties can we infer from graph spectrum? . Do non-extremal eigenpairs contain valuable information? (a) Co-spectral Domains (b) Co-spectral Graphs “Can you hear the shape of a drum?” - Mark Kac Density of States I Given a symmetric graph matrix H ∈ R N ×N , such as normalized adjacency I It has an eigendecomposition H = Q ΛQ T I Λ= diag(λ 1 , ··· , λ N ), and Q =[q 1 , ··· , q N ] orthogonal I Spectral density, also known as density of states (DOS), is μ (λ )= 1 N N ∑ i =1 δ (λ - λ i ) I Pointwise density of states (PDOS) is μ k (λ )= μ (λ ; e k )= N ∑ i =1 q i (k ) 2 δ (λ - λ i ) I Integrate these distributions over equal sized intervals for spectral histogram . On the left: Blue bins - actual count; red dot - approximated count . On the right: Vertically stacked PDOS spectral histogram. Color for heights -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 50 100 150 200 250 300 350 400 450 500 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 Node Index 10 4 0 0.2 0.4 0.6 0.8 1 Approximating Density of States with the Kernel Polynomial Method (KPM) I Chebyshev series: Fourier cosine series with a change of variable T m (cos(θ )) = cos(mθ ) I Chebyshev polynomials can also be defined with the recurrence T 0 (x )= 1, T 1 (x )= x , T m+1 (x )= 2xT m (x ) - T m-1 (x ) I Expand DOS in the dual Chebyshev basis T * m (x )= w (x )T m (x ), w (x )= 2 π (1 + δ 0m ) √ 1 - x 2 μ (λ )= ∞ ∑ m=0 d m T * m (λ ), d m = Z 1 -1 T m (λ ) μ (λ )d λ = 1 N tr(T m (H )) I Similarly for PDOS: μ k (λ )= ∞ ∑ m=0 d mk T * m (λ ), d mk = T m (H ) kk I Alternatives: Gauss Quadrature and Lanczos (GQL). Stochastic Estimators for Approximating Chebyshev Moments I For a probe vector z with i.i.d entries of mean 0 and variance 1, E[z T f (H )z ]= tr(f (H )), E[z Hz ]= diag(f (H )) I Choose N z independent probe vector Z j ’s, then tr(f (H )) ≈ 1 N z N z ∑ j =1 Z T j f (H )Z j , diag(f (H )) ≈ 1 N z N z ∑ j =1 Z j f (H )Z j I Use the three-term recurrence to efficiently compute T m (H )z Efficiently Removing DOS Spikes Coming From Motifs I Commonly observe spikes in spectrum from eigenvalues of high multiplicity I They can usually be attributed to local symmetry groups +1 -1 λ = 0 +1 -1 λ = -1/2 +1 ±1 -1 ∓1 λ = ±1/2 +1/ √ 2 -1/ √ 2 ±1 ∓1 λ = ±1/ √ 2 I Collapse the graph onto the quotient space of these localized eigenvectors I Example: High Energy Physics Theory Collaboration Network -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count No Filter -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count Filter at λ = 0 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count Filter at λ = -1/3 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count Filter at λ = -1/2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count Filter at λ = -1/4 200 400 600 800 1000 #moments 0 0.05 0.1 0.15 0.2 0.25 rel. error No Filter Zero Filter All Filter Relative Error Nested Dissection (ND) I Previous methods rely on Monte Carlo sampling, which converges at O ( 1 √ N z ) I Alternative: an exact method using nested dissection I If we can find vertex separators that bisect the graph, the recurrence becomes T m+1 (H ) 11 = 2H 11 T m (H ) 11 - T m-1 (H ) 11 + 2H 13 T m (H ) 31 . H 11 H 22 H 13 H 23 H 33 H T 13 H T 23 0 0 Vertex Separation Nested Dissection Ordering I Hierarchical partition to avoid off-diagonal computation and storage I First traverse in pre-order for the separator blocks, then fill in the leaf blocks Comparison between Our Methods Exact ND KPM Network Time[s] Time[s] Time[s] Rel-L 2 W 1 Musm 0.9 0.9 0.2 3.7e-3 2.3e-2 Yeast 1.8 1.9 0.3 4.3e-3 1.8e-2 Erd ˝ os 13.6 4.7 1.9 3.0e-4 4.8e-3 PGP 60.9 11.1 7.9 9.4e-4 2.7e-3 MvlComic 1007.3 57.2 43.0 2.9e-4 1.0e-3 Caida 5915.5 43.2 50.1 1.5e-4 1.0e-4 I All methods compute 50 Chebyshev moments I KPM uses N /8 probe vectors I Rel-L 2 is on the moments; W 1 is the Wasserstein distance on spectral histogram Gallery of DOS/PDOS -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 100 200 300 400 500 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 100 200 300 400 500 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 1200 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 100 200 300 400 500 600 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 20 40 60 80 100 Count -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1000 2000 3000 4000 5000 Node Index (a) Erd ˝ os Collaboration Network -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1000 2000 3000 4000 5000 6000 Node Index (b) Autonomous System Network (1999) -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1000 2000 3000 4000 5000 6000 Node Index (c) Marvel Characters Network -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1000 2000 3000 4000 Node Index (d) Facebook Ego Networks -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 500 1000 1500 2000 2500 Node Index (e) Minnesota Road Network -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1000 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 100 200 300 400 500 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 5 10 15 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 5 10 15 20 25 Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 0 2 4 6 8 10 Count 10 4 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 2000 4000 6000 8000 Node Index (f) HepTh Collaboration Network -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1000 2000 3000 4000 5000 6000 Node Index (g) Autonomous System Network (2000) -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 50 100 150 Node Index (h) Harry Potter Characters Network -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 50 100 150 200 Node Index (i) Twitter Ego Networks -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 5 10 15 Node Index 10 5 (j) California Road Network Computed using KPM (500 moments, 20 probe vectors). DOS Wasserstein Distance Similarity 0.00 0.25 0.10 0.12 0.28 0.47 0.25 0.44 0.43 0.40 0.25 0.00 0.15 0.14 0.29 0.46 0.24 0.43 0.19 0.16 0.10 0.15 0.00 0.03 0.32 0.52 0.29 0.49 0.34 0.31 0.12 0.14 0.03 0.00 0.35 0.55 0.31 0.51 0.33 0.29 0.28 0.29 0.32 0.35 0.00 0.23 0.05 0.19 0.47 0.44 0.47 0.46 0.52 0.55 0.23 0.00 0.25 0.09 0.51 0.49 0.25 0.24 0.29 0.31 0.05 0.25 0.00 0.21 0.42 0.39 0.44 0.43 0.49 0.51 0.19 0.09 0.21 0.00 0.52 0.50 0.43 0.19 0.34 0.33 0.47 0.51 0.42 0.52 0.00 0.04 0.40 0.16 0.31 0.29 0.44 0.49 0.39 0.50 0.04 0.00 Erdos HepTh AS1999 AS2000 MvlChars HP Facebook Twitter Road-MN Road-CA Erdos HepTh AS1999 AS2000 MvlChars HP Facebook Twitter Road-MN Road-CA 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Collaboration Network Autonomous System Fictional Characters Ego Network Road Network Computation Time KPM Single Core Average Computation Time Per Moment Network # Nodes # Edges Avg. Deg. Time (s) Enron 36,692 183,831 10.02 0.046 Gplus 107,614 13,673,453 254.12 1.133 Amazon 334,863 925,872 5.53 0.628 Neuron 1,018,524 24,735,503 48.57 9.138 RoadNetCA 1,965,206 2,766,607 2.82 2.276 Orkut 3,072,441 117,185,083 76.28 153.7 LiveJournal 3,997,962 34,681,189 17.35 14.52 Friendster 65,608,366 1,806,067,135 55.06 1,017 I Datasets from Stanford Network Analysis Project I Single Intel Xeon E5 v3 CPU at 2.30GHz with 200GB memory I Give ample opportunities for parallel computation Discussion I We develop practical spectral density computation for real-world networks I DOS and PDOS facilitate comparisons among graphs and models I Implementation available at: https://github.com/kd383/NetworkDOS https://people.cam.cornell.edu/~kd383/ [email protected]
Transcript of Network Density of States - Cornell Universityarb/posters/Network-DOS-KDD-2019... · 2019. 10....
Network Density of StatesKun Dong1 Austin R. Benson2 David Bindel2
Applied Math1, CS2
Background
I Weyl’s Law connects the geometric properties of a manifold with theasymptotic spectral distribution of Laplace-Beltrami operator
I Eigenvalues alone do not uniquely identify an object, but what can we learn?. How do we compute spectrum for large real-world graphs?. What geometric properties can we infer from graph spectrum?. Do non-extremal eigenpairs contain valuable information?
(a) Co-spectral Domains (b) Co-spectral Graphs
“Can you hear the shape of a drum?” - Mark Kac
Density of States
I Given a symmetric graph matrix H ∈RN×N, such as normalized adjacencyI It has an eigendecomposition H = QΛQT
I Λ = diag(λ1, · · · ,λN), and Q = [q1, · · · ,qN] orthogonalI Spectral density, also known as density of states (DOS), is
µ(λ ) =1N
N
∑i=1
δ (λ −λi)
I Pointwise density of states (PDOS) is
µk(λ ) = µ(λ ;ek) =N
∑i=1
qi(k )2δ (λ −λi)
I Integrate these distributions over equal sized intervals for spectral histogram. On the left: Blue bins - actual count; red dot - approximated count. On the right: Vertically stacked PDOS spectral histogram. Color for heights
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
50
100
150
200
250
300
350
400
450
500
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
2.2
No
de In
dex
104
0
0.2
0.4
0.6
0.8
1
Approximating Density of States with the Kernel Polynomial Method (KPM)
I Chebyshev series: Fourier cosine series with a change of variable
Tm(cos(θ )) = cos(mθ )
I Chebyshev polynomials can also be defined with the recurrence
T0(x) = 1, T1(x) = x, Tm+1(x) = 2xTm(x)−Tm−1(x)
I Expand DOS in the dual Chebyshev basis
T∗m(x) = w(x)Tm(x), w(x) =2
π(1 + δ0m)√
1−x2
µ(λ ) =∞
∑m=0
dmT∗m(λ ), dm =∫ 1
−1Tm(λ )µ(λ )dλ =
1N
tr(Tm(H))
I Similarly for PDOS: µk(λ ) =∞
∑m=0
dmkT∗m(λ ), dmk = Tm(H)kk
I Alternatives: Gauss Quadrature and Lanczos (GQL).
Stochastic Estimators for Approximating Chebyshev Moments
I For a probe vector z with i.i.d entries of mean 0 and variance 1,
E[zT f(H)z] = tr(f(H)), E[z�Hz] = diag(f(H))
I Choose Nz independent probe vector Zj’s, then
tr(f(H))≈ 1Nz
Nz
∑j=1
ZTj f(H)Zj , diag(f(H))≈ 1
Nz
Nz
∑j=1
Zj� f(H)Zj
I Use the three-term recurrence to efficiently compute Tm(H)z
Efficiently Removing DOS Spikes Coming From Motifs
I Commonly observe spikes in spectrum from eigenvalues of high multiplicityI They can usually be attributed to local symmetry groups
+1 −1
λ = 0
+1 −1
λ =−1/2
+1 ±1 −1 ∓1
λ =±1/2
+1/√
2 −1/√
2
±1 ∓1
λ =±1/√
2I Collapse the graph onto the quotient space of these localized eigenvectorsI Example: High Energy Physics Theory Collaboration Network
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
No Filter
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
Filter at λ = 0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
Filter at λ =−1/3
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
Filter at λ =−1/2
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
Filter at λ =−1/4
200 400 600 800 1000
#moments
0
0.05
0.1
0.15
0.2
0.25
re
l. e
rro
r
No Filter
Zero Filter
All Filter
Relative Error
Nested Dissection (ND)
I Previous methods rely on Monte Carlo sampling, which converges at O( 1√Nz
)
I Alternative: an exact method using nested dissectionI If we can find vertex separators that bisect the graph, the recurrence becomes
Tm+1(H)11 = 2H11Tm(H)11−Tm−1(H)11 + 2H13Tm(H)31 .
H11
H22
H13
H23
H33HT13 HT
23
0
0
Vertex Separation Nested Dissection Ordering
I Hierarchical partition to avoid off-diagonal computation and storageI First traverse in pre-order for the separator blocks, then fill in the leaf blocks
Comparison between Our Methods
Exact ND KPMNetwork Time[s] Time[s] Time[s] Rel-L2 W1
Musm 0.9 0.9 0.2 3.7e-3 2.3e-2Yeast 1.8 1.9 0.3 4.3e-3 1.8e-2Erdos 13.6 4.7 1.9 3.0e-4 4.8e-3PGP 60.9 11.1 7.9 9.4e-4 2.7e-3
MvlComic 1007.3 57.2 43.0 2.9e-4 1.0e-3Caida 5915.5 43.2 50.1 1.5e-4 1.0e-4
I All methods compute 50Chebyshev moments
I KPM uses N/8 probe vectorsI Rel-L2 is on the moments; W1 is
the Wasserstein distance onspectral histogram
Gallery of DOS/PDOS
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
1200
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
600
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
20
40
60
80
100
Co
un
t
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
1000
2000
3000
4000
5000
No
de
In
de
x
(a) Erdos Collaboration Network
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
1000
2000
3000
4000
5000
6000
No
de
In
de
x
(b) Autonomous SystemNetwork (1999)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
1000
2000
3000
4000
5000
6000
No
de
In
de
x
(c) Marvel Characters Network
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
1000
2000
3000
4000
No
de
In
de
x
(d) Facebook Ego Networks
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
500
1000
1500
2000
2500
No
de
In
de
x
(e) Minnesota Road Network
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
200
400
600
800
1000
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
100
200
300
400
500
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
5
10
15
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
5
10
15
20
25
Co
un
t
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
2
4
6
8
10
Co
un
t
104
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
2000
4000
6000
8000
No
de
In
de
x
(f) HepTh Collaboration Network
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
1000
2000
3000
4000
5000
6000
No
de
In
de
x
(g) Autonomous SystemNetwork (2000)
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
50
100
150
No
de
In
de
x
(h) Harry Potter CharactersNetwork
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
50
100
150
200
No
de
In
de
x
(i) Twitter EgoNetworks
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
5
10
15
No
de
In
de
x
105
(j) California Road Network
Computed using KPM (500 moments, 20 probe vectors).
DOS Wasserstein Distance Similarity
0.00
0.25
0.10
0.12
0.28
0.47
0.25
0.44
0.43
0.40
0.25
0.00
0.15
0.14
0.29
0.46
0.24
0.43
0.19
0.16
0.10
0.15
0.00
0.03
0.32
0.52
0.29
0.49
0.34
0.31
0.12
0.14
0.03
0.00
0.35
0.55
0.31
0.51
0.33
0.29
0.28
0.29
0.32
0.35
0.00
0.23
0.05
0.19
0.47
0.44
0.47
0.46
0.52
0.55
0.23
0.00
0.25
0.09
0.51
0.49
0.25
0.24
0.29
0.31
0.05
0.25
0.00
0.21
0.42
0.39
0.44
0.43
0.49
0.51
0.19
0.09
0.21
0.00
0.52
0.50
0.43
0.19
0.34
0.33
0.47
0.51
0.42
0.52
0.00
0.04
0.40
0.16
0.31
0.29
0.44
0.49
0.39
0.50
0.04
0.00
Erdos
HepTh
AS1999
AS2000
MvlC
hars
HP
Road-M
N
Road-C
A
Erdos
HepTh
AS1999
AS2000
MvlChars
HP
Road-MN
Road-CA
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Collaboration Network
Autonomous System
Fictional Characters
Ego Network
Road Network
Computation Time
KPM Single Core Average Computation Time Per MomentNetwork # Nodes # Edges Avg. Deg. Time (s)
Enron 36,692 183,831 10.02 0.046Gplus 107,614 13,673,453 254.12 1.133
Amazon 334,863 925,872 5.53 0.628Neuron 1,018,524 24,735,503 48.57 9.138
RoadNetCA 1,965,206 2,766,607 2.82 2.276Orkut 3,072,441 117,185,083 76.28 153.7
LiveJournal 3,997,962 34,681,189 17.35 14.52Friendster 65,608,366 1,806,067,135 55.06 1,017
I Datasets from StanfordNetwork Analysis Project
I Single Intel Xeon E5 v3CPU at 2.30GHz with200GB memory
I Give ample opportunitiesfor parallel computation
Discussion
I We develop practical spectral density computation for real-world networksI DOS and PDOS facilitate comparisons among graphs and modelsI Implementation available at: https://github.com/kd383/NetworkDOS
https://people.cam.cornell.edu/~kd383/ [email protected]
