Sparse matrix computations in MapReduce
-
Upload
david-gleich -
Category
Education
-
view
5.028 -
download
1
description
Transcript of Sparse matrix computations in MapReduce
ICME MapReduce Workshop!April 29 – May 1, 2013!
!
David F. Gleich!Computer Science!Purdue University
David Gleich · Purdue 1
!
Website www.stanford.edu/~paulcon/icme-mapreduce-2013
Paul G. Constantine!Center for Turbulence Research!
Stanford University
MRWorkshop
Goals
Learn the basics of MapReduce & Hadoop Be able to process large volumes of data from science and engineering applications … help enable you to explore on your own!
David Gleich · Purdue 2 MRWorkshop
Workshop overview
Monday!Me! Sparse matrix computations in MapReduce!Austin Benson Tall-and-skinny matrix computations in MapReduce
Tuesday!Joe Buck Extending MapReduce for scientific computing!Chunsheng Feng Large scale video analytics on pivotal Hadoop
Wednesday!Joe Nichols Post-processing CFD dynamics data in MapReduce !Lavanya Ramakrishnan Evaluating MapReduce and Hadoop for science
David Gleich · Purdue 3 MRWorkshop
Sparse matrix computations in MapReduce!
David F. Gleich!Computer Science!Purdue University
David Gleich · Purdue 4
Slides online soon! Code https://github.com/dgleich/mapreduce-matrix-tutorial
MRWorkshop
How to compute with big matrix data !A tale of two computers
224k Cores 10 PB drive
1.7 Pflops
7 MW
Custom !interconnect!
$104 M
80k cores!50 PB drive ? Pflops ? MW GB ethernet $?? M
625 GB/core!High disk to CPU
45 GB/core High CPU to disk
5
ORNL 2010 Supercomputer!Google’s 2010? !Data computer!
David Gleich · Purdue MRWorkshop
My data computers
6
Nebula Cluster @ Sandia CA!2TB/core storage, 64 nodes, 256 cores, GB ethernet Cost $150k
These systems are good for working with enormous matrix data!
ICME Hadoop @ Stanford!3TB/core storage, 11 nodes, 44 cores, GB ethernet Cost $30k
David Gleich · Purdue MRWorkshop
My data computers
7
Nebula Cluster @ Sandia CA!2TB/core storage, 64 nodes, 256 cores, GB ethernet Cost $150k
These systems are good for working with enormous matrix data!
ICME Hadoop @ Stanford!3TB/core storage, 11 nodes, 44 cores, GB ethernet Cost $30k
^
but not great,
David Gleich · Purdue
some ^
MRWorkshop
By 2013(?) all Fortune 500 companies will have a data computer
David Gleich · Purdue 8 MRWorkshop
How do you program them?
9 David Gleich · Purdue MRWorkshop
MapReduce and!Hadoop overview
10
David Gleich · Purdue MRWorkshop
MapReduce is designed to solve a different set of problems from standard parallel libraries
11
David Gleich · Purdue MRWorkshop
The MapReduce programming model
Input a list of (key, value) pairs Map apply a function f to all pairs Reduce apply a function g to !
all values with key k (for all k) Output a list of (key, value) pairs
12
David Gleich · Purdue MRWorkshop
Computing a histogram !A simple MapReduce example
13
Input!!Key ImageId Value Pixels
Map(ImageId, Pixels) for each pixel emit� Key = (r,g,b)� Value = 1
Reduce(Color, Values) emit� Key = Color Value = sum(Values)
Output!!Key Color Value ! # of pixels
David Gleich · Purdue
5 15 10 9 3 17 5 10
1 1
1
1
Map Reduce 1 1
1
1
1 1
1
1
1
1 1 1
shuffle
MRWorkshop
Many matrix computations are possible in MapReduce
Column sums are easy !Input Key (i,j) Value Aij
Other basic methods !can use common parallel/out-of-core algs!Sparse matrix-vector products y = Ax Sparse matrix-matrix products C = AB
14
Reduce(j,Values) emit Key = j, Value = sum(Values)
David Gleich · Purdue
Map((i,j), val) emit� Key = j, Value = val
A11 A12 A13 A14
A21 A22 A23 A24
A31 A32 A33 A34
A41 A42 A43 A44
(3,4) -> 5 (1,2) -> -6.0 (2,3) -> -1.2 (1,1) -> 3.14 … “Coordinate storage”
MRWorkshop
Many matrix computations are possible in MapReduce
Column sums are easy !Input Key (i,j) Value Aij
Other basic methods !can use common parallel/out-of-core algs!Sparse matrix-vector products y = Ax Sparse matrix-matrix products C = AB
15
Reduce(j,Values) emit Key = j, Value = sum(Values)
David Gleich · Purdue
Map((i,j), val) emit� Key = j, Value = val
A11 A12 A13 A14
A21 A22 A23 A24
A31 A32 A33 A34
A41 A42 A43 A44
(3,4) -> 5 (1,2) -> -6.0 (2,3) -> -1.2 (1,1) -> 3.14 … “Coordinate storage”
Beware of un-thoughtful ideas MRWorkshop
Why so many limitations?
16
David Gleich · Purdue MRWorkshop
The MapReduce programming model
Input a list of (key, value) pairs Map apply a function f to all pairs Reduce apply a function g to !
all values with key k (for all k) Output a list of (key, value) pairs Map function f must be side-effect free!
All map functions run in parallel Reduce function g must be side-effect free!
All reduce functions run in parallel
17
David Gleich · Purdue MRWorkshop
A graphical view of the MapReduce programming model
David Gleich · Purdue 18
dataMap
dataMap
dataMap
dataMap
keyvalue
keyvalue
keyvalue
keyvalue
keyvalue
keyvalue
()
Shuffle
keyvaluevalue
dataReduce
keyvaluevaluevalue
dataReduce
keyvalue dataReduce
MRWorkshop
Data scalability
The idea !Bring the computations to the data MR can schedule map functions without moving data.
1 MM R
RMMM
Maps Reduce
Shuffle
2
3
4
5
1 2 M M
3 4 M M
5 M
19
David Gleich · Purdue MRWorkshop
After waiting in the queue for a month and !after 24 hours of finding eigenvalues, one node randomly hiccups.
heartbreak on node rs252
David Gleich · Purdue 20
MRWorkshop
Fault tolerant
Redundant input helps make maps data-local Just one type of communication: shuffle
MM R
RMM
Input stored in triplicate
Map output!persisted to disk!before shuffle
Reduce input/!output on disk
David Gleich · Purdue 21
MRWorkshop
3HUIRUPDQFH�UHVXOWV��VLPXODWHG�IDXOWV�
:H�FDQ�VWLOO�UXQ�ZLWK�3�IDXOW�� �����ZLWK�RQO\�a����SHUIRUPDQFH�SHQDOW\���+RZHYHU��ZLWK�3�IDXOW��VPDOO��ZH�VWLOO�VHH�D�SHUIRUPDQFH�KLW�
Fault injection
10 100 1000
1/Prob(failure) – mean number of success per failure
Tim
e to
com
plet
ion
(sec)
200
100
No faults (200M by 200)
Faults (800M by 10)
Faults (200M by 200)
No faults !(800M by 10)
With 1/5 tasks failing, the job only takes twice as long.
David Gleich · Purdue 22
MRWorkshop
Data scalability
The idea !Bring the computations to the data MR can schedule map functions without moving data.
1 MM R
RMMM
Maps Reduce
Shuffle
2
3
4
5
1 2 M M
3 4 M M
5 M
23
David Gleich · Purdue MRWorkshop
Computing a histogram !A simple MapReduce example
24
Input!!Key ImageId Value Pixels
Map(ImageId, Pixels) for each pixel emit� Key = (r,g,b)� Value = 1
Reduce(Color, Values) emit� Key = Color Value = sum(Values)
Output!!Key Color Value ! # of pixels
David Gleich · Purdue
5 15 10 9 3 17 5 10
1 1
1
1
Map Reduce 1 1
1
1
1 1
1
1
1
1 1 1
shuffle
The entire dataset is “transposed” from images to pixels. This moves the data to the computation!
(Using a combiner helps to reduce the data moved, but it cannot always be used)
MRWorkshop
Hadoop and MapReduce are bad systems for some matrix computations.
David Gleich · Purdue 25
MRWorkshop
How should you evaluate a MapReduce algorithm?
Build a performance model! Measure the worst mapper Usually not too bad Measure the data moved Could be very bad Measure the worst reducer Could be very bad
David Gleich · Purdue 26
MRWorkshop
Tools I like
hadoop streaming dumbo mrjob hadoopy C++
David Gleich · Purdue 27
MRWorkshop
Tools I don’t use but other people seem to like …
pig java hbase mahout Eclipse Cassandra
David Gleich · Purdue 28
MRWorkshop
hadoop streaming
the map function is a program!(key,value) pairs are sent via stdin!output (key,value) pairs goes to stdout the reduce function is a program!(key,value) pairs are sent via stdin!keys are grouped!output (key,value) pairs goes to stdout
David Gleich · Purdue 29
MRWorkshop
mrjob from
a wrapper around hadoop streaming for map and reduce functions in python
class MRWordFreqCount(MRJob): def mapper(self, _, line): for word in line.split(): yield (word.lower(), 1) def reducer(self, word, counts): yield (word, sum(counts)) if __name__ == '__main__': MRWordFreqCount.run()
David Gleich · Purdue 30
MRWorkshop
How can Hadoop streaming possibly be fast?
Hadoop streaming frameworks
Iter 1QR (secs.)
Iter 1Total (secs.)
Iter 2Total (secs.)
OverallTotal (secs.)
Dumbo 67725 960 217 1177
Hadoopy 70909 612 118 730
C++ 15809 350 37 387
Java 436 66 502
Synthetic data test 100,000,000-by-500 matrix (~500GB)Codes implemented in MapReduce streamingMatrix stored as TypedBytes lists of doublesPython frameworks use Numpy+AtlasCustom C++ TypedBytes reader/writer with AtlasNew non-streaming Java implementation too
David Gleich (Sandia)
All timing results from the Hadoop job tracker
C++ in streaming beats a native Java implementation.
16/22MapReduce 2011
David Gleich · Purdue 31
Example available from github.com/dgleich/mrtsqr!
for verification
mrjob could be faster if it used typedbytes for intermediate storage see https://github.com/Yelp/mrjob/pull/447
MRWorkshop
Code samples and short tutorials at github.com/dgleich/mrmatrix github.com/dgleich/mapreduce-matrix-tutorial
David Gleich · Purdue 32
MRWorkshop
Matrix-vector product
David Gleich · Purdue 33
Ax = y
y
i
=X
k
A
ik
x
k
A x
Follow along! ���mapreduce-matrix-tutorial !/codes/smatvec.py!
MRWorkshop
Matrix-vector product
David Gleich · Purdue 34
Ax = y
y
i
=X
k
A
ik
x
k
A x
A is stored by row
$ head samples/smat_5_5.txt !0 0 0.125 3 1.024 4 0.121 !1 0 0.597 !2 2 1.247 !3 4 -1.45 !4 2 0.061 !
x is stored entry-wise !
$ head samples/vec_5.txt !0 0.241 !1 -0.98 !2 0.237 !3 -0.32 !4 0.080 !
Follow along! ���mapreduce-matrix-tutorial !/codes/smatvec.py!
MRWorkshop
Matrix-vector product!(in pictures)
David Gleich · Purdue 35
Ax = y
y
i
=X
k
A
ik
x
k
A x
A x
Input Map 1!Align on columns!
Reduce 1!Output Aik xk!keyed on row i
A
x Reduce 2!Output sum(Aik xk)!
y
MRWorkshop
Matrix-vector product!(in pictures)
David Gleich · Purdue 36
Ax = y
y
i
=X
k
A
ik
x
k
A x
A x
Input Map 1!Align on columns!
def joinmap(self, key, line): ! vals = line.split() ! if len(vals) == 2: ! # the vector ! yield (vals[0], # row ! (float(vals[1]),)) # xi ! else: ! # the matrix ! row = vals[0] ! for i in xrange(1,len(vals),2): ! yield (vals[i], # column ! (row, # i,Aij! float(vals[i+1]))) !
MRWorkshop
Matrix-vector product!(in pictures)
David Gleich · Purdue 37
Ax = y
y
i
=X
k
A
ik
x
k
A x
A x
Input Map 1!Align on columns!
Reduce 1!Output Aik xk!keyed on row i
A
x def joinred(self, key, vals): ! vecval = 0. ! matvals = [] ! for val in vals: ! if len(val) == 1: ! vecval += val[0] ! else: ! matvals.append(val) ! for val in matvals: ! yield (val[0], val[1]*vecval) !
Note that you should use a secondary sort to avoid reading both in memory
MRWorkshop
Matrix-vector product!(in pictures)
David Gleich · Purdue 38
Ax = y
y
i
=X
k
A
ik
x
k
A x
A x
Input Map 1!Align on columns!
Reduce 1!Output Aik xk!keyed on row i
A
x Reduce 2!Output sum(Aik xk)!
y def sumred(self, key, vals): ! yield (key, sum(vals)) !
MRWorkshop
Move the computations to the data? Not really!
David Gleich · Purdue 39
A x
A x
Input Map 1!Align on columns!
Reduce 1!Output Aik xk!keyed on row i
A
x Reduce 2!Output sum(Aik xk)!
y Copy data once, ���now aligned on column
Copy data again, align on row
MRWorkshop
Matrix-matrix product
David Gleich · Purdue 40
A B
AB = CCij =
X
k
Aik Bkj
Follow along! ���mapreduce-matrix-tutorial !/codes/matmat.py!
MRWorkshop
Matrix-matrix product
David Gleich · Purdue 41
A B
AB = CCij =
X
k
Aik Bkj
A is stored by row
$ head samples/smat_10_5_A.txt !0 0 0.599 4 -1.53 !1 !2 2 0.260 !3 !4 0 0.267 1 0.839
B is stored by row
$ head samples/smat_5_5.txt !0 0 0.125 3 1.024 4 0.121 !1 0 0.597 !2 2 1.247 !
Follow along! ���mapreduce-matrix-tutorial !/codes/matmat.py!
MRWorkshop
Matrix-matrix product !(in pictures)
David Gleich · Purdue 42
A B
AB = CCij =
X
k
Aik Bkj
A Map 1!Align on columns!
B Reduce 1!Output Aik Bkj!keyed on (i,j)
A
B Reduce 2!Output sum(Aik Bkj)!
C
MRWorkshop
Matrix-matrix product !(in code)
David Gleich · Purdue 43
A B
AB = CCij =
X
k
Aik Bkj
A Map 1!Align on columns!
B
def joinmap(self, key, line): ! mtype = self.parsemat() ! vals = line.split() ! row = vals[0] ! rowvals = \ ! [(vals[i],float(vals[i+1])) ! for i in xrange(1,len(vals),2)] ! if mtype==1: ! # matrix A, output by col ! for val in rowvals: ! yield (val[0], (row, val[1])) ! else: ! yield (row, (rowvals,)) !
MRWorkshop
Matrix-matrix product !(in code)
David Gleich · Purdue 44
A B
AB = CCij =
X
k
Aik Bkj
A Map 1!Align on columns!
B Reduce 1!Output Aik Bkj!keyed on (i,j)
A
B
def joinred(self, key, line): ! # load the data into memory ! brow = [] ! acol = [] ! for val in vals: ! if len(val) == 1: ! brow.extend(val[0]) ! else: ! acol.append(val) ! ! for (bcol,bval) in brow: ! for (arow,aval) in acol: ! yield ((arow,bcol),aval*bval) !
MRWorkshop
Matrix-matrix product !(in pictures)
David Gleich · Purdue 45
A B
AB = CCij =
X
k
Aik Bkj
A Map 1!Align on columns!
B Reduce 1!Output Aik Bkj!keyed on (i,j)
A
B Reduce 2!Output sum(Aik Bkj)!
C def sumred(self, key, vals): ! yield (key, sum(vals)) !
MRWorkshop
Why is MapReduce so popular?
if (root) { !PetscInt cur_nz=0; ! unsigned char* root_nz_buf; ! unsigned int *root_nz_buf_i,*root_nz_buf_j; ! double *root_nz_buf_v; ! PetscMalloc((sizeof(unsigned int)*2+sizeof(double))*root_nz_bufsize,&root_nz_buf); ! PetscMalloc(sizeof(unsigned int)*root_nz_bufsize,&root_nz_buf_i); ! PetscMalloc(sizeof(unsigned int)*root_nz_bufsize,&root_nz_buf_j); ! PetscMalloc(sizeof(double)*root_nz_bufsize,&root_nz_buf_v); ! ! unsigned long long int nzs_to_read = total_nz; ! ! while (send_rounds > 0) { ! // check if we are near the end of the file ! // and just read that amount ! size_t cur_nz_read = root_nz_bufsize; ! if (cur_nz_read > nzs_to_read) { ! cur_nz_read = nzs_to_read; ! } ! PetscInfo2(PETSC_NULL," reading %i non-zeros of %lli\n", cur_nz_read, nzs_to_read); !
600 lines of gross code in order to load a sparse matrix into memory, streaming from one processor. MapReduce offers a better alternative
David Gleich · Purdue 46
MRWorkshop
Thoughts on a better system
Default quadruple precision Matrix computations without indexing
Easy setup of MPI data jobs David Gleich · Purdue 47
Initial data load of any MPI job Compute task
MRWorkshop
Double-precision floating point was designed for the era where “big” was 1000-10000
David Gleich · Purdue 48
MRWorkshop
Error analysis of summation
s = 0; for i=1 to n: s = s + x[i] A simple summation formula has !error that is not always small if n is a billion
David Gleich · Purdue 49
fl(x + y ) = (x + y )(1 + ")
fl(X
i
x
i
) �X
i
x
i
nµX
i
|xi
| µ ⇡ 10�16
MRWorkshop
If your application matters then watch out for this issue. Use quad-precision arithmetic or compensated summation instead.
David Gleich · Purdue 50
MRWorkshop
Compensated Summation “Kahan summation algorithm” on Wikipedia s = 0.; c = 0.; for i=1 to n: y = x[i] – c t = s + y c = (t – s) – y s = t
David Gleich · Purdue 51
Mathematically, c is always zero. On a computer, c can be non-zero The parentheses matter! fl(csum(x)) �
X
i
x
i
(µ + nµ2)X
i
|xi
|
µ ⇡ 10�16
MRWorkshop
Summary
MapReduce is a powerful but limited tool that has a role in the future of computational math. … but it should be used carefully! See Austin’s talk next!
David Gleich · Purdue 52
MRWorkshop
Code samples and short tutorials at github.com/dgleich/mrmatrix github.com/dgleich/mapreduce-matrix-tutorial
David Gleich · Purdue 53
MRWorkshop