Post on 19-Apr-2018
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Software and Algorithms for Graph Queries on
Massively Multithreaded Architectures
Jonathan Berry (Sandia Labs)
Bruce Hendrickson (Sandia Labs)
Simon Kahan (Google)
Petr Konecny (Cray)
March 29, 2007
Outline
• Graph-based informatics
• Massively-multithreaded architectures
• Sandia’s prototype Multithreaded Graph Library (MTGL)
• Algorithmic case studies on the Cray MTA-2
• Current and future directions, opportunities for collaboration
Graph-Based Informatics
Massive Multithreading: e.g., The Cray MTA-2
• Slow clock rate (220Mhz)
• 128 “streams” per processor
• Global address space
• Fine-grain synchronization
• Simple, serial-like programming model
• Advanced parallelizing compilers
Latency Tolerant:
important for Graph
Algorithms
No Data Cache
Hashed Memory
Cray MTA Processor
• Each thread can have up to 8 memory refs in flight
• Round trip to memory ~150 cycles
The MultiThreaded Graph Library (MTGL)
• Recent work
– Design & implement the Multithreaded Graph Library (MTGL)
• Boost GL-like, but not Boost-based
• In the process of open-sourcing
• Develop, run, debug codes on workstations and/or MTA/XMT
– Design multithreaded algorithms
• Simon Kahan’s connected components algorithm
• “Bully” connected components algorithm
• Subgraph isomorphism for semantic graphs
• Current work
– Adapt the MTGL to run on SMP’s
– Flesh out the MTGL with community finding, etc.
– Integrate the MTGL with visualization frameworks
Algorithm
Class
MTGL: C++ Design Levels
“Visitor”
class
Graph
Class
Data Str.
Class
Gives Parallelism,
Hides Most Concurrency
User provides filters,
Gets parallelism
for free
Inspired by Boost GL, but not Boost GL
Four Modes of MTGL Graph Exploration
for v in V:
visit v’s neighbors
for e in E:
visit e’s endpoints
recursive parallel searchbreadth-first search
~10 memref/edge ~10 memref/edge
~40 memref/edge ~20-40 memref/edge
visit_adj visit_edges
bfspsearch
Current MTGL Algorithms
• Connected components (psearch, visit_edges, visit_adj)
• Strongly-connected components (psearch)
• Maximal independent set (visit_edges)
• Typed subgraph isomorphism (psearch, visit_edges)
• S-t connectivity (bfs)
• Single-source shortest paths (psearch)
• Betweenness centrality (bfs-like)
• Connection subgraphs (bfs, sparse matrix, mt-quicksort)
• Find triangles (psearch)
• Find assortativity (psearch)
• Find modularity (psearch)
Under development:
• Optimize modularity via simulated annealing (visit_adj, visit_edges)
• Count edges in 1, 2 neighborhoods (various)
• more
MTGL PSearch Primitive: visitors
DefaultVisitor {
sr(v) {} # search root
d(v) {} # discover
vt(v) {} # visit test
te(v,w) {} # tree edge
oe(v,w) {} # “other” edge
…
}
The search primitives
give the programmer
these opportunities to
react to search events
Pseudocode for this talk
uses these shortened
method names
Abuse of notation: multiple
edges are allowed
MTGL Search Primitives (e.g)
PSearch<OR,vis>(v)
{
vis.d(v)
for (v,w) in E(v):
if (v,w) unvisited OR vis.vt(v,v’)
vis.te(v,w)
PSearch<OR,vis>(w)
else
vis.oe(vw)
}
PSearch<AND,vis>(v)
{
vis.d(v)
for (v,w) in E(v):
if vis.vt(v,v’)
if (v,w) unvisited
vis.te(v,w)
PSearch<AND,vis>(w)
else
vis.oe(vw)
}
More general, but details omitted here…
MTGL Synchronization Primitives
mt_write(a,v)Wait for a to be “empty,”
Write a, leave full
writeef(&a, v)
mt_readff(a)Wait for a to be “full,”
Read a, leave full
b = readff(&a)
mt_readfe(a)Wait for a to be “full,” read a,
leave empty
b = readfe(&a)
mt_incr(a,i)Atomic read, increment of ab = int_fetch_add(a,i)
MTGLPseudocodeMeaningMTA Primitive
iab , ifa←
ab fe
←
ab ff←
ab ef←
Overloaded for different platforms
Case Studies: Algorithm Kernels
• Connected Components
• Subgraph Isomorphism
• S-T Connectivity (i.e., use of BFS)
Kahan’s Algorithm for Connected Components
Phase 1:
concurrent
searches
Phase 2: CRCW
(Shiloach-Vishkin)
Phase 3:
relabel
MTGL Implementation of Kahan’s Algorithm
PSearch
(tricky)
Kahan’s
Phase I
visitor
Kahan’s
Phase III
visitor
(Trivial)
Shiloach-
Vishkin
CRCW
Kahan’s
Phase II
visitor
(Trivial)
}
}
)},{(
];[
];[{ w)oe(v,
} C[v] ][{ w) te(v,
} ][ { sr(v)
pairs int)(int, of hash table
ints |)(| ofarray user
{1
cwcvTT
wCcw
vCcv
wC
vvC
T
GVC
V
ts
ff
ff
ef
ef
∪←
←
←
←
←
←
←
=
MTGL Implementation of Kahan’s Algorithm
Phase I visitor:“component”
values were
defined by user to
start “empty”
Thread-safe
Add to hash
table
Wait until both
full
More General Filtering: The “Bully” Algorithm
}
}
C[w]
else
][][
)( OR unvisited if
];[ ];[{ w) te(v,
}
);(return
];[ ];[{ w) vt(v,
......
{
cw
vCwC
cwcvw
wCcwvCcv
cwcv
wCcwvCcv
V
ef
ef
feff
ffff
b
←
←
<
←←
<
←←
=
MTGL Implementation of the Bully Algorithm
Parallelized Shiloach-Vishkin (Bader, Feo, Cong’06)
edge **edges = g->edges.getStore();
while (graft) {
numiter++;
graft = 0;
#pragma mta assert parallel
for (int i=0; i<m; i++) {
int u = edges[i]->vert1->id;
int v = edges[i]->vert2->id;
if (D[u] < D[v] && D[v] == D[D[v]]) {
D[D[v]] = D[u];
graft = 1;
}
if (D[v] < D[u] && D[u] == D[D[u]]) {
D[D[u]] = D[v];
graft = 1;
}
}
#pragma mta assert parallel
for (int i=0; i<n; i++) {
while (D[i] != D[D[i]]) D[i] = D[D[i]];
}
}
“Graft”
“Hook”
Connected Components Performance (1)
10
100
1 10
Tim
e in
Sec
onds
Number of Processors
’Fancy Graph’ of 234M Edges
3Ghz, 64Gb Opteron Workstation: 5 minutes
C-KAHAN: edge lists are K-ary trees
Estimate possible 3x speedup of C-KAHAN
"C-KAHAN""MTGL-KAHAN""MTGL-BULLY"
"SV"
~2.9s
~40 memref/edge
Connected Components Performance (2)
1
1 10
Tim
e in
Sec
onds
Number of Processors
Power Law Out-Degree (PLOD) of 14M Edges
"MTGL-KAHAN""MTGL-BULLY"
"SV"
1
1 10
Tim
e in
Sec
onds
Number of Processors
Recursive Matrix (RMAT) of 16M Edges
"MTGL-BULLY""SV"
~80
~80
~80
~80~23
~23~25 ~27
~175
~200
~200~250
~80
~80~90
~100
~80
~80 ~80~80
memref/edge
Bug in MTGL-Kahan impl (crash)
Case Study: Subgraph Isomorphism Kernel
• Objective: find exact or inexact matches of a small pattern
graph within a large semantic graph
Instance-Specific Type Filtering for Subgraph Iso.
The Big
Graph
e
For each edge e in Big Graph:
Do the 4 types associated with
e match those of any of the
target graph edges?
Small,
Filtered
Graph
Pattern graph
Subgraph Isomorphism: Input
223132322V
T
The Target Graph:
Table of Type and Auxiliary Information:
Ideal: Euler Tour Our Experiments: Random Walk
Subgraph Isomorphism: Creating a Bipartite Graph
Small,
Filtered
Graph
(SFG)
Logical placeholders for vertices in the SFG.
k times:
visit each
edge of SFG
223132322V
T
Subgraph Isomorphism: Creating a Bipartite Graph
S-T shortest paths
(top to bottom) correspond
to candidate
matches.
Visitor object tailors
Search so that it never
goes up (similar to
“Bully” algorithm).
Branch and bound to
Find better matches.
Computational Results: Subgraph Isomorphism
100
1000
1 10
Tim
e in
Sec
onds
Number of Processors
Subgraph Isomorphism Heuristic: 234M Edges (Target of 20 Edges)
3Ghz, 64Gb Opteron Workstation: ~15 minutes
"SubgraphIsomorphism"
Computational Results: Subgraph Isomorphism
36
17
55
15
24
47
16
52 9020
28
32
55
52
13
90/10
76/24
26/74
51/49
61/3933/67
17/83
45/55
79/21
90/10
74/26
52/4822/78
85/15
85/15
86/14
54/4621/79
55/45 80 46/54
18
95
25/75
67
70
75
5136
17
55
15
2447
16
52
90
2028
32
55
52
1390/10
76/24
26/74
51/49
61/39
33/67 90/10
74/26
52/48
22/78
85/15
85/15
86/14 54/4621/7955/45
18
95
95
25/75
25/75
80
79/21
51
Target Found
Type & topological isomorphism exists between green vertices
Actual graphs from
a 234M edge
instance
Can try harder if we want a closer match
Target Found
Type & topological isomorphism exists between green vertices
Actual graphs from
a 234M edge
instance
84
17 47
16
66
82
59
16
77
90
91
68
44
38
84
4144
65
6/94
76/24
15/85
3/97
51/49
91/9
95/5
20/80
28/72
88/12
4/96
69/31
34/66
32/6835/65
15/85
79/21
98 31/69
23
14/86
47/53
17
84
17
47
16 66
82
59
16
77
90
91
6844
38
84
41
44
65
6/94
76/24
15/85
3/97
51/49
91/9
95/547/53 20/80
28/72
88/124/9669/31
34/66
32/68
35/65
15/85
79/21
14/8617
98
Traceview Output for Subgraph Isomorphism
Preprocessing: 97% utilization
Not fully utilized since filtered
graph is tiny (10k) and we don’t
branch & bound in this example.
Future
• Massively-multithreaded Graph Algorithm R&D
• Open-Sourcing of the MTGL
• Run on SMP
• Query system with shared, mmapped graphs
Acknowledgements
• Bruce Hendrickson (Graph Informatics lead)
• Keith Underwood (Eldorado performance prediction)
• Simon Kahan, Petr Konecny (Cray): help in all aspects of this project
• Wayne Wong (Cray): Eldorado simulations
• Curt Janssen (usage model)
• Bob Heaphy (matrix-vector kernel)
• David Bader, Kamesh Madduri (Ga. Tech) (s-t connectivity)
• Cindy Phillips (help with algorithms)
• John Cieslewicz (Columbia) (database operations)
• Joe Crobak (Lafayette) (single-source shortest paths)
• Andrew Lumsdaine (Indiana U.) (Parallel Boost Graph Library)
• Douglas Gregor (Indiana U.) (Parallel Boost Graph Library)
• Others!