Jianfeng Jia (UC Irvine) Pregelix: Big(ger) Graph Tyson ...pregelix.ics.uci.edu/slides/Pregelix-...
Transcript of Jianfeng Jia (UC Irvine) Pregelix: Big(ger) Graph Tyson ...pregelix.ics.uci.edu/slides/Pregelix-...
Pregelix: Big(ger) Graph Analytics on A Dataflow Engine
Yingyi Bu (UC Irvine) Joint work with:
Vinayak Borkar (UC Irvine) , Michael J. Carey (UC Irvine), Tyson Condie (UCLA), Jianfeng Jia (UC Irvine)
OutlineIntroductionPregel SemanticsThe Pregel Logical PlanThe Pregelix SystemExperimental ResultsRelated WorkConclusions
Introduction● How Big are Big Graphs?
○ Web: 8.53 Billion pages in 2012○ Facebook active users: 1.01 Billion○ de Bruijn graph: 3 Billion nodes○ ......
● Weapons for mining Big Graphs○ Pregel (Google)○ Giraph (Facebook, LinkedIn, Twitter, etc.)○ Distributed GraphLab (CMU)○ GraphX (Berkeley)
Programming Model
public abstract class Vertex<I extends WritableComparable, V extends Writable, E extends Writable, M extends Writable> implements Writable{ public abstract void compute(Iterator<M> incomingMessages);
.......}
● Vertex
● Helper methods○ sendMsg(I vertexId, M msg)○ voteToHalt()○ getSuperstep()
More APIs● Message Combiner
○ Combine messages○ Reduce network traffic
● Global Aggregator○ Aggregate statistics over all live vertices○ Done for each iteration
● Graph Mutations○ Add vertex○ Delete vertex○ A conflict resolution function
Pregel Semantics● Bulk-synchronous
○ A global barrier between iterations● Compute invocation
○ Once per active vertex in each superstep○ A halted vertex is activated when receiving messages
● Global halting○ Each vertex is halted○ No messages are in flight
● Graph mutations○ Partial ordering of operations○ User-defined resolve function
superstep:3halt: false
Process-centric runtime
Vertex { id: 1 halt: false value: 3.0 edges: (3,1.0),
(4,1.0) }Vertex { id: 3 halt: false value: 3.0 edges: (2,1.0),
(3,1.0) }
<5, 1.0>
<4, 3.0>
worker-1 worker-2
master
message <id, payload> control signal
Vertex { id: 2 halt: false value: 2.0 edges: (3,1.0),
(4,1.0)}Vertex{ id: 4 halt: false value: 1.0 edges: (1,1.0)}
<3, 1.0><2, 3.0>
Issues and Opportunities● Out-of-core support
26 similar threads on Giraph-users mailing list during the past year!
“I’m trying to run the sample connected components algorithm on a large data set on a cluster, but I get a “java.lang.OutOfMemoryError: Java heap space” error.”
Issues and Opportunities● Physical flexibility
○ PageRank, SSSP, CC, Triangle Counting○ Web graph, social network, RDF graph○ 8 machine school cluster, 200 machine Facebook
data center
One-size fits-all?
Issues and Opportunities● Software simplicity
Network management
Message delivery
Memory management
Task scheduling
PregelGraphLab GiraphHama......
Vertex/map/msg data structures
The Pregelix Approach
1.0
vid edges
vid payload
vid=vid24
haltfalsefalse
value2.01.0
(3,1.0),(4,1.0)(1,1.0)
24 3.0
Msg
Vertex
51
3.0
1.0
1 false 3.0 (3,1.0),(4,1.0)3 false 3.0 (2,1.0),(3,1.0)
3
vid edges
1
halt
falsefalse
value
3.03.0
(3,1.0),(4,1.0)(2,1.0),(3,1.0)
msg
NULL1.0
5 1.0 NULL NULL NULL2 false 2.0 (3,1.0),(4,1.0)3.04 false 1.0 (1,1.0)3.0
Relation Schema
Vertex
Msg
GS
(vid, halt, value, edges)
(vid, payload)
(halt, aggregate, superstep)
Pregel UDFs● compute
○ Executed at each active vertex in each superstep● combine
○ Aggregation function for messages● aggregate
○ Aggregate function for the global states● resolve
○ Used to resolve graph mutations
Logical Plan
…
D2
D4,D5,D6
vid combine
UDF Call (compute)
M.vid=V.vid
Vertexi(V)Msgi(M)
Vertexi+1 Msgi+1
(V.halt =false || M.payload != NULL)
D3
D7
Flow Data
D2 Vertex tuples
D3 Msg tuples
D7 Msg tuples after combination
D1
Logical Plan
D1
Agg(aggregate)Agg(bool-and)D4 D5
UDF Call (compute)
GSi+1
GSi(G)
superstep=G.superstep+1D10
Flow Data
D4 The global halting state contribution
D5 Values for aggregate
D8 The global halt state
D9 The global aggregate value
D10 The increased superstep
D9D8
D2,D3,D6 …
D2,D3,D4,D5
D1
vid(resolve)
UDF Call (compute)
Vertexi+1Flow Data
D6 Vertex tuples for deletions and insertions
D6…
The Pregelix System
Network management
Message delivery
Memory management
Task scheduling
Vertex/map/msg data structures
Connection management
Data exchanging Buffer management
Task scheduling Record/Index management
A general purpose parallel dataflow engine
Operators Access methods
Pregel Physical Plans
The Runtime
● The Hyracks data-parallel execution engine○ Out-of-core operators○ Connectors○ Access methods○ User-configurable task scheduling○ Extensibility
● Runtime Choice? Hyracks Hadoop
Parallelism
Msg-2
3
vid edgesvid msgvid=vid
24
haltfalsefalse
value2.01.0
(3,1.0) (4,1.0)(1,1.0)
24
3.03.0
vid edges
24
halt
falsefalse
value
2.01.0
(3,1.0),(4,1.0)(1,1.0)
msg
3.03.0
vid edgesvid msgvid=vid
13
haltfalsefalse
value3.03.0
(3,1.0) (4,1.0)3 1.0
1.0
vid edges
1
halt
falsefalse
value
3.03.0
(3,1.0),(4,1.0)(2,1.0),(3,1.0)
msg
NULL1.0
5 1.0 NULL NULL NULL
Worker-1 Worker-2
Msg-1 Vertex-1 Vertex-2(2,1.0),(3,1.0)
25
3.01.0
output-Msg-1
34
1.03.0
output-Msg-2
vid msg vid msg
5
Physical Choices
● Vertex storageB-TreeLSM B-Tree
● Group-by○ Pre-clustered group-by○ Sort-based group-by○ HashSort group-by
● Data redistribution○ m-to-n merging partitioning connector○ m-to-n partitioning connector
● Join○ Index Full outer join○ Index Left outer join
Data Storage
● Vertex○ Partitioned B-tree or LSM B-tree
● Msg○ Partitioned local files, sorted
● GS○ Stored on HDFS○ Cached in each worker
Physical Plan: Message Combination
vidcombine
vidcombine
(Sort-based)
(Sort-based)
(Sort-based)
(Sort-based)
(Sort-based)
(Sort-based)
Sort-Groupby-M-to-N-Partitioning HashSort-Groupby-M-to-N-Partitioning
Sort-Groupby-M-to-N-Merge-Partitioning HashSort-Groupby-M-to-N-Merge-Partitioning
M-to-N Partitioning Connector M-To-N Partitioning Merging Connector
vidcombine vidcombine
vidcombine vidcombine
vidcombine
vidcombine
(Preclustered)
(Sort-based)
(Preclustered)
(Sort-based)
(Preclustered)
(Sort-based)
vidcombine vidcombine
vidcombine vidcombine vidcombine
vidcombine
(Preclustered)
(HashSort)
(Preclustered)
(HashSort)
(Preclustered)
(HashSort)
vidcombine vidcombine
vidcombine vidcombine
vidcombine
vidcombine
(HashSort)
(HashSort)
(HashSort)
(HashSort)
(HashSort)
(HashSort)
vidcombine vidcombine
vidcombine vidcombine
D1
Physical Plan: Message Delivery
Index Left OuterJoin
UDF Call (compute)
M.vid=V.vid
Vertexi(V)Msgi(M)
(V.halt = false || M.paylod != NULL) UDF Call (compute)
Vertexi(V)Msgi(M)
…
Vidi(I)
…
Vidi+1(halt = false)
Function Call (NullMsg)
Index Full Outer Join Merge (choose())M.vid=I.vid
D11
D12
M.vid=V.vid
D1
D2 -- D6
D2 -- D6
Caching
● Iteration-aware (sticky) scheduling?○ 1 Loc: location constraints
● Caching of invariant data?○ B-tree buffer pool -- customized flushing policy: never
flush dirty pages○ File system cache -- free
Pregel, Giraph, GraphLab all have caches for this kind of iterative jobs. What do you do for caching?
Experimental Results● Setup
○ Machinesa UCI cluster ~ 32 machines4 cores, 8GB memory, 2 disk drives.
○ Datasets■ Yahoo! webmap (1,413,511,393 vertice, adjacency
list, ~70GB) and its samples.■ The Billions of Tuples Challenge dataset
(172,655,479 vertices, adjacency list, ~17GB), its samples, and its scale-ups.
○ Giraph■ Latest trunk (revision 770)■ 4 vertex computation threads, 8GB JVM heap
Related Work● Parallel Data Management
○ Gama, GRACE, Teradata○ Stratosphere (TU Berlin)○ REX (UPenn)○ AsterixDB (UCI)
● Big Graph Processing Systems○ Pregel (Google)○ Giraph (Facebook, LinkedIn, Twitter, etc.)○ Distributed GraphLab (CMU)○ GraphX (Berkeley)○ Hama (Sogou, etc.) --- Too slow!
Conclusions● Pregelix offers:
○ Transparent out-of-core support○ Physical flexibility○ Software simplicity
● We target Pregelix to be an open-source production system, rather than just a research prototype:○ http://pregelix.ics.uci.edu