Post on 05-Jan-2016
description
The Gamma Operator for Big Data Summarizationon an Array DBMS
Carlos Ordonez
1
Acknowledgments
• Michael Stonebraker , MIT• My PhD students: Yiqun Zhang, Wellington Cabrera• SciDB team: Paul Brown, Bryan Lewis, Alex
Polyakov
2
Why SciDB?
• Large matrices beyond RAM size• Storage by row or column not good enough• Matrices natural in statistics, engineer. and science• Multidimensional arrays -> matrices, not same thing• Parallel shared-nothing best for big data analytics• Closer to DBMS technology, but some similarity with
Hadoop• Feasible to create array operators, having matrices as
input and matrix as output• Combine processing with R package and LAPACK
3
4
Old: separate sufficient statistics
5
New: Generalizing and unifying Sufficient Statistics: Z=[1,X,Y]
6
Equivalent equations with projections from Γ
7
Properties of
8
Further properties details:non-commutative and distributive
9
Storage in array chunks
10
SCAN
In SciDB we store the points in X as 2D array.
Worker11
Array storage and processing in SciDB
• Assuming d<<n it is natural to hash partition X by i=1..n• Gamma computation is fully parallel maintaining
local Gamma versions in RAM. • X can be read with a fully parallel scan• No need to write Gamma from RAM to disk during
scan, unless fault tolerant
12
Coordinator
Worker 1
OK
Coordinator Worker 1
NO!
13
Point must fit in one chunk. Otherwise, join is needed (slow)
Parallel computation
Coordinator Worker 1 Worker 2
send send
14
Dense matrix operator: O(d2 n)
15
Sparse matrix operator: O(d n) for hyper-sparse matrix
16
Pros: Algorithm evaluation with physical array operators• Since xi fits in one chunk joins are avoided (at least 2X
I/O with hash or merge join)• Since xi*xi
T can be computed in RAM we avoid an aggregation which would require sorting points by i• No need to store X twice: X, XT: half I/O, half RAM space• No need transpose X, costly reorganization even in
RAM, especially if X spans several RAM segments• Operator works in C++ compiled code: fast; vector
accessed once; direct assignment (bypass C++ functions calls)
17
System issues and limitations
• Gamma not efficiently computable in AQL or AFL: hence operator is required• Arrays of tuples in SciDB are more general, but cumbersome for
matrix manipulation: arrays of single attribute (double)• Points must be stored completely inside a chunk: wide
rectangular chunks: may not be I/O optimal• Slow: Arrays must be pre-processed to SciDB load format,
loaded to 1D array and re-dimensioned=>optimize load.• Multiple SciDB instances per node improve I/O speed:
interleaving CPU• Larger chunks are better: 8MB, especially for dense matrices;
avoid shuffling; avoid joins• Dense (alpha) and sparse (beta) versions 18
Benchmark: scale up emphasis• Small: cluster with 2 Intel Quadcore servers 4GB
RAM, 3TB disk• Large: Amazon cloud 2
19
20
Why is Gamma faster than SciDB+LAPACK?
21
Gamma operator
d Gamma op Scan mem alloc CPU merge
100 3.5 0.7 0.1 2.2 0.0200 10.9 1.0 0.1 8.6 0.0
400 38.8 2.2 0.1 33.9 0.1800 145.0 4.6 0.1 134.7 0.4
1600 599.8 11.4 0.1 575.5 1.0
SciDB and LAPACK (crossprod() call in SciDB)
TOTAL transpose subarray 1 repart 1 subarray 2 repart 2 build 0s gemm ScaLAPACK MKL
77.3 0.1 0.3 41.7 0.1 25.9 0.0 8.0 0.8 0.2163.0 0.1 0.2 84.9 0.1 55.7 0.0 17.2 1.8 0.6373.1 0.1 0.3 172.6 0.5 120.6 0.3 39.4 5.4 2.1
1497.3 0.1 0.1 553.6 0.8 537.6 0.5 169.8 21.2 8.1* * * * * * * * * 33.4
Combination: SciDB + R
22
Can Gamma operator beat LAPACK?
23
Gamma versus Open BLAS LAPACK (90% performance of MKL)
Gamma: scan, sparse/dense 2 threads; disk+RAM+CPU
LAPACK: Open BLAS~=MKL; 2 threads; RAM+CPU
d=100 LAPACK d=200 LAPACK d=400 LAPACK d=800 LAPACK
ndensitydense
sparse Op BLAS dense sparse Op BLAS dense sparse Op BLAS2 dense sparse Open BLAS
100k 0.1% 3.3 0.1 0.4 11.3 0.1 1.0 38.9 0.2 3.1 145.0 0.6 10.7
100k 1.0% 3.3 0.1 0.4 11.3 0.2 1.0 38.9 0.4 3.1 145.0 1.0 10.7
100k 10.0% 3.3 0.5 0.4 11.3 0.9 1.0 38.9 2.2 3.1 145.0 6.2 10.7
100k 100.0% 3.3 4.5 0.4 11.3 15.4 1.0 38.9 55.9 3.1 145.0 201.0 10.71M 0.1% 31.1 0.2 3.8 103.5 0.2 10.0 316.5 0.4 423.2 1475.7 0.9fail1M 1.0% 31.1 0.5 3.8 103.5 1.1 10.0 316.5 3.8 423.2 1475.7 4.0fail1M 10.0% 31.1 4.0 3.8 103.5 7.0 10.0 316.5 16.3 423.2 1475.7 46.4fail1M 100.0% 31.1 44.0 3.8 103.5 148.8 10.0 316.5 542.3 423.2 1475.7 2159.6fail
SciDB in the Cloud: massive parallelism
24
Conclusions• One pass summarization matrix operator: parallel, scalable• Optimization of outer matrix multiplication as sum (aggregation) of vector
outer products• Dense and sparse matrix versions required• Operator compatible with any parallel shared-nothing system, but better for
arrays• Gamma matrix must fit in RAM, but n unlimited• Summarization matrix can be exploited in many intermediate computations
(with appropriate projections) in linear models• Simplifies many methods to two phases:
1. Summarization2. Computing model parameters
• Requires arrays, but can work with SQL or MapReduce
25
Future work: Theory
• Use Gamma in other models like logistic regression, clustering, Factor Analysis, HMMs• Connection to frequent itemset• Sampling• Higher expected moments, co-variates• Unlikely: Numeric stability with
unnormalized sorted data
26
Future work: Systems
• DONE: Sparse matrices: layout, compression• DONE: Beat LAPACK on high d• Online model learning (cursor interface
needed, incompatible with DBMS)• Unlimited d (currently d>8000); join required
for high d? Parallel processing of high d more complicated, chunked• Interface with BLAS and MKL, not worth it?• Faster than column DBMS for sparse?
27