Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest...
Transcript of Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest...
![Page 1: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/1.jpg)
![Page 2: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/2.jpg)
Today’s Speakers
Raman Grover
UC Irvine
Advisor: Prof. Michael Carey
Sanjay Kulhari UC Riverside
Advisor: Prof. Vassilis Tsotras
![Page 3: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/3.jpg)
AcknowledgmentsDryad: Distributed Data-Parallel Programs from
Sequential Building Blocks
Distributed Data-Parallel Computing Using a High-Level Programming Language
�
DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language
Google Tech Talks
MSDN Channel 9
![Page 4: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/4.jpg)
Parallel Distributed Computing. . . Why ?
Large-scale Internet Services Depend on clusters of hundreds or thousands of general purpose servers.
Future advances in local computing power :
Increasing the number of cores on a chip rather than improving the speed or instruction-level parallelism of a single core
![Page 5: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/5.jpg)
Hard ProblemsHigh-latency Unreliable networksControl of resources by separate federated
or competing entities, Issues of identity for authentication and
access control.The Programming Model Reliability, Efficiency and Scalability of the
applications
![Page 6: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/6.jpg)
Achieving Scalability Systems that automatically discover and exploit parallelism
in sequential programs
Those that require the developer to explicitly expose the data dependencies of a computation.
CondorShader languages developed for graphic processing units
Parallel databases
Google’s MapReduce system
![Page 7: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/7.jpg)
Reasons for SuccessDeveloper is explicitly forced to consider the data
parallelism of the computation
The developer need have no understanding of standard concurrency mechanisms such as threads and fine-grain concurrency control
Developers now work at a suitable level of abstraction for writing scalable applications since the resources available
at execution time are not generally known at the time the
code is written.
![Page 8: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/8.jpg)
LimitationsNot a free lunch !
Restrict an application’s communication flow for different reasons :
GPU shader languages Strongly tied to an efficient hardware implementation Map Reduce Designed for the widest possible class of developers, aims
for simplicity at the expense of generality performance. Parallel databases designed for relational algebra manipulations (e.g. SQL)
where the communication graph is implicit.
![Page 9: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/9.jpg)
Dryad Control over the communication graph as well as the subroutines that
live at its vertices.
Specify an arbitrary directed acyclic graph to describe the application’s communication patterns,
Express the data transport mechanisms (files, TCP pipes, and sharedmemory FIFOs) between the computation vertices.
MapReduce restricts all computations to take a single input set and generate a single output set.
SQL and shader languages allow multiple inputs but generate a single output from the user’s perspective, though SQL query plans internally use multiple-output vertices.
Dryad is notable for allowing graph vertices (and computations in general) to use an arbitrary number of inputs and outputs.
![Page 10: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/10.jpg)
In this talk !
Dryad : System OverviewDescribing a Dryad GraphCommunication ChannelDryad Job Job ExecutionFault ToleranceRuntime Graph RefinementExperimental EvaluationBuilding on Dryad
![Page 11: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/11.jpg)
Before we dive into Details…
Unix Pipes: 1-Dgrep | sed | sort | awk | perl
Dryad: 2-D grep1000 | sed500 | sort1000 | awk500 | perl50
11
![Page 12: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/12.jpg)
Dryad = Execution Layer
12
Job (Application)
Dryad
Cluster
Pipeline
Shell
Machine
≈
![Page 13: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/13.jpg)
Virtualized 2-D Pipelines
13
![Page 14: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/14.jpg)
Virtualized 2-D Pipelines
14
![Page 15: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/15.jpg)
Virtualized 2-D Pipelines
15
![Page 16: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/16.jpg)
Virtualized 2-D Pipelines
16
![Page 17: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/17.jpg)
Virtualized 2-D Pipelines
17
• 2D DAG• multi-machine• virtualized
![Page 18: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/18.jpg)
Dryad Job Structure
18
grep
sed
sortawk
perlgrep
grep
sed
sort
sort
awk
Inputfiles
Vertices (processes)
Outputfiles
ChannelsStage
grep1000 | sed500 | sort1000 | awk500 | perl50
![Page 19: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/19.jpg)
Channels
19
X
M
Items
Finite Streams of items
Distributed filesystem (persistent)
SMB/NTFS files (temporary)
TCP pipes (inter-machine)
Memory FIFOs (intra-machine)
![Page 20: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/20.jpg)
Architecture
20
Files, TCP, FIFO, Networkjob schedule
data plane
control plane
NS PD PDPD
V V V
Job manager cluster
![Page 21: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/21.jpg)
RuntimeServices
Name serverDaemon
Job ManagerCentralized coordinating processUser application to construct graphLinked with Dryad libraries for scheduling
verticesVertex executable
Dryad libraries to communicate with JMUser application sees channels in/outArbitrary application code, can use local FS
V V V
![Page 22: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/22.jpg)
Job = Directed Acyclic Graph
Processingvertices Channels
(file, pipe, shared memory)
Inputs
Outputs
![Page 23: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/23.jpg)
Job execution
Scheduler keeps track of state and history of each vertex in the graph.
When a job manager fails job is terminated but scheduler can implement checkpointing or replication to avoid this.
Execution record attached with a vertex.
Execution record paired with a available computer, remote daemon is instructed to run the vertex.
V V V
![Page 24: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/24.jpg)
Job execution (cont.)If an execution of a vertex fails it can start
again.
More than one instance of the vertex may be executing at the same time.
Each vertex names it output channels uniquely using version number.
Input ready
New execution record created and added to scheduling queue
Execution record paired with an available computer
Job manager receives periodic status updates from the vertex
Vertex
![Page 25: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/25.jpg)
Fault tolerance policyAll vertex programs are deterministic
Every terminating execution of the job will give the same results regardless of the failures over the course of execution.
Job manager will know in any case that something bad happened to a vertex.
Vertices belong to stages and stage manager can take care of slow or failed vertices of a stage.
![Page 26: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/26.jpg)
Fault tolerance policy (cont.)
If A fails, run it again
If A’s inputs are gone, run upstream vertices again.
If A is slow, run another copy elsewhere and use output from whichever finishes first.
A
![Page 27: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/27.jpg)
Run-time graph refinement
To be able to scale to large input sets while conserving scarce network bandwidth.
For associative and commutative computations aggregation tree can be helpful.
If internal vertices perform data reduction network traffic between racks will be reduced.
Keep refining when upstream vertices have completed.
Stage manager for each input layer.
![Page 28: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/28.jpg)
Run-time graph refinement (cont.)
Partial aggregation operation, to process k sets in parallel.
Data mining example follows this.
Dynamic refinement is good because the amount of data to be written is not known in advance and also the required input channels.
![Page 29: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/29.jpg)
Run-time graph refinement (cont.)
A A A A A *
B
+ + * *
+*
B
+ vertex gets 20,000 tuples, runs DISTINCT and returns 500 tuples
B vertex receives 50,000 tuples and Execute DISTINCT on them
* vertex gets 30,000 tuples, runs DISTINCT and returns 500 tuples
B vertex gets 1000 tuples, runs DISTINCT
Each A vertex sends 10,000 tuples
![Page 30: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/30.jpg)
Run-time graph refinement (cont.)
A A A A A *
B
+ + * *
+*
CC C C
B
![Page 31: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/31.jpg)
Experimental evaluationHardware:
Cluster of 10 computers (Sky server query experiment)
Cluster of 1800 computers (Data mining experiment) Each computer had 2 dual core Opteron processors
running at 2 GHz. i.e. 4 CPUs total. 8 GB of DRAM 400 GB Western Digital. 1 Gbit/sec Ethernet Windows server 2003 Enterprise X64 edition SP1.
![Page 32: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/32.jpg)
Case study I (Sky server Query)3-way join to find gravitational lens effectTable U: (objId, color) 11.8GBTable N: (objId, neighborId) 41.8GBFind neighboring stars with similar colors:
Join U+N to find
T = U.color,N.neighborId where U.objId = N.objId
Join U+T to find
U.objId where U.objId = T.neighborID and U.color ≈ T.color
![Page 33: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/33.jpg)
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
Took SQL planManually coded in DryadManually partitioned data
SkyServer DB query
u: objid, color
n: objid, neighborobjid
[partition by objid]
select
u.color,n.neighborobjid
from u join n
where
u.objid = n.objid
(u.color,n.neighborobjid)
[re-partition by n.neighborobjid]
[order by n.neighborobjid]
[distinct]
[merge outputs]
select
u.objid
from u join <temp>
where
u.objid = <temp>.neighborobjid and
|u.color - <temp>.color| < d
![Page 34: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/34.jpg)
Optimization
D
M
S
Y
X
M
S
M
S
M
S
U N
U
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
![Page 35: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/35.jpg)
Optimization
D
M
S
Y
X
M
S
M
S
M
S
U N
U
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
![Page 36: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/36.jpg)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
0 2 4 6 8 10
Number of Computers
Spe
ed-u
pDryad In-Memory
Dryad Two-pass
SQLServer 2005
![Page 37: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/37.jpg)
Case study II - Query histogram computationInput: log file (n partitions)Extract queries from log partitionsRe-partition by hash of query (k buckets)Compute histogram within each bucket
![Page 38: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/38.jpg)
Naïve histogram topology
Q Q
R
Q
R k
k
k
n
n
is:Each
R
is:
Each
MS
C
P
C
S
C
S
D
P parse lines
D hash distribute
S sort
C count occurrences
MS merge sort
![Page 39: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/39.jpg)
Efficient histogram topology
P parse lines
D hash distribute
S sort
C count occurrences
MS merge sort
M non-deterministic merge
Q' is:Each
R
is:
Each
MS
C
M
P
C
S
Q'
RR k
T
k
n
T
is:
Each
MS
D
C
![Page 40: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/40.jpg)
Final histogram refinement
Q' Q'
RR 450
TT 217
450
10,405
99,713
33.4 GB
118 GB
154 GB
10.2 TB
1,800 computers
43,171 vertices
11,072 processes
11.5 minutes
![Page 41: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/41.jpg)
Optimizing Dryad applicationsGeneral-purpose refinement rulesProcesses formed from sub graphs
Re-arrange computations, change I/O type
Application code not modifiedSystem at liberty to make optimization
choicesHigh-level front ends hide this from user
![Page 42: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/42.jpg)
All this sounds good ! But how do I interact with Dryad ?Nebula scripting language
Allows users to specify a computation as a series of stages each taking input from one or more previous stages or files system.
Dryad as generalization of UNIX piping mechanism.
Writing distributed applications using perl or grep.
Also a front end that uses perl scripts and sql select, project and join.
![Page 43: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/43.jpg)
Interacting with Dryad (Cont.)Integration with SQL Server
SQL Server Integration Services (SSIS) supports work-flow based application programming on single instance of SQL server.
SSIS input graph generated and tested on a single computer.
SSIS graph is run in distributed fashion using dryad.
Each Dryad vertex is an instance of SQL server running an SSIS sub graph of the complete Job.
Deployed in live production system.
![Page 44: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/44.jpg)
LINQMicrosoft’s Language INtegrated Query
Available in Visual Studio productsA set of operators to manipulate datasets in .NET
Support traditional relational operatorsSelect, Join, GroupBy, Aggregate, etc.
Integrated into .NET programming languagesPrograms can call operatorsOperators can invoke arbitrary .NET functions
Data modelData elements are strongly typed .NET objectsMuch more expressive than SQL tables
Highly extensibleAdd new custom operatorsAdd new execution providers
![Page 45: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/45.jpg)
PLINQ
Local machine
.Netprogra
m(C#,
VB, F#, etc)
Execution engines
Query
Objects
LINQ-to-SQL
DryadLINQ
LINQ-to-Obj
LIN
Q p
rovi
der
inte
rface
Scalability
Single-core
Multi-core
Cluster
![Page 46: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/46.jpg)
DryadLINQAutomatically distribute a LINQ programMore general than distributed SQL
Inherits flexible C# type system and libraries
Data-clustering, EM, inference, …Uniform data-parallel programming model
From SMP to clustersFew Dryad-specific extensions
Same source program runs on single-core through multi-core up to cluster
![Page 47: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/47.jpg)
47
DryadLINQClient machine
(11)
Distributedquery plan
.NET program
Query Expr
Data center
Output TablesResult
s
Input Tables
Invoke
Query
Output DryadTable
Dryad Execution
.Net Objects
JM
ToTable
foreach
Vertexcode
![Page 48: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/48.jpg)
Word Count in DryadLINQCount word frequency in a set of documents:
var docs = DryadLinq.GetTable<Doc>(“file://docs.txt”);var words = docs.SelectMany(doc => doc.words);var groups = words.GroupBy(word => word);var counts = groups.Select(g => new WordCount(g.Key, g.Count()));
counts.ToDryadTable(“counts.txt”);
![Page 49: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/49.jpg)
49
(1)
SM
GB
S
SM
Q
GB
C
D
MS
GBSum
SelectMany
sort
groupby
count
distribute
mergesort
groupby
Sum
pipelined
pipelined
![Page 50: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/50.jpg)
50
(1)
SM
GB
S
SM
Q
GB
C
D
MS
GBSum
(2)
SM
Q
GB
C
D
MS
GBSum
SM
Q
GB
C
D
MS
GBSum
SM
Q
GB
C
D
MS
GBSum
![Page 51: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/51.jpg)
Query plan
LINQ query
DryadLINQ: From LINQ to Dryad
Dryad
select
where
logs
Automatic query plan generation
Distributed query
execution by Dryad
var logentries = from line in logs where !line.StartsWith("#") select new LogEntry(line);
![Page 52: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/52.jpg)
How does it work?Sequential code “operates” on datasetsBut really just builds an expression graph
Lazy evaluationWhen a result is retrieved
Entire graph is handed to DryadLINQOptimizer builds efficient DAGProgram is executed on cluster
![Page 53: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/53.jpg)
Future DirectionsGoal: Use a cluster as if it is a single computer
Dryad/DryadLINQ represent a modest step
On-going researchWhat can we write with DryadLINQ?
Where and how to generalize the programming model?Performance, usability, etc.
How to debug/profile/analyze DryadLINQ apps?Job scheduling
How to schedule/execute N concurrent jobs?Caching and incremental computation
How to reuse previously computed results?Static program checking
A very compelling case for program analysis? Better catch bugs statically than fighting them in the
cloud?
![Page 54: Today’s Speakersalumni.cs.ucr.edu/~skulhari/Dryad.pdf · Map Reduce Designed for the widest possible class of developers, aims for simplicity at the expense of generality performance.](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f45152d856c8f5411384cd6/html5/thumbnails/54.jpg)
Conclusions
Goal: Use a compute cluster as if it is a single computerDryad/DryadLINQ represent a significant step
Requires close collaborations across many fields of computing, includingDistributed systemsDistributed and parallel databasesProgramming language design and analysis