Post on 17-Dec-2015
Massive Scale-out of
Expensive Continuous Queries
Erik Zeitler and Tore RischUppsala Database Laboratory
Uppsala University
31 Aug 2011 Erik Zeitler and Tore Risch 2
Outline
1. Introduction2. Stream splitting strategies for scale-
out3. Evaluating stream splitting strategies4. Cost model and heuristic5. Energy efficiency6. Related work7. Conclusions and future work
31 Aug 2011 Erik Zeitler and Tore Risch 3
Input data streams
Query result data stream
DSMS
Queries
useror
programmer
meta-data
Query processing software
Stream data access software
storeddata
CQ: Continuous Queries (filters and transformations)
11001011
01001011
DSMS Data Stream Management System
SCSQ
Super Computer Stream Query processor
a Data Stream Management System
31 Aug 2011 Erik Zeitler and Tore Risch 4
CQ
CQ
CQ
split merge
Research Questions
How to ensure scalable CQ execution• with growing input stream rate?• with high CQ execution cost?
CQs are scaled out by splitting the input stream.• applications require customizable input stream splitting,
called splitstream• both tuple route and broadcast allowed
splitstream CQ
By scale-out.
31 Aug 2011 Erik Zeitler and Tore Risch 5
merge
Research Questions
How to ensure scalable CQ execution• with growing input stream rate?• with high CQ execution cost?
CQs are scaled out by splitting the input stream.• applications require customizable input stream splitting,
called splitstream• both tuple route and broadcast allowed
How to split massive streams over massively parallel CQs?• By parallelization of splitstream
By scale-out.
CQ
CQ
CQ
CQ
CQ
CQ
CQ
splitsplitstreamsplitstreamsplitstreamsplitstream
31 Aug 2011 Erik Zeitler and Tore Risch 6
Outline
1. Introduction2. Stream splitting strategies for
scale-out3. Scale-up of stream splitting
strategies4. Cost model and heuristic5. Energy efficiency6. Related work7. Conclusions and future work
31 Aug 2011 Erik Zeitler and Tore Risch 7
splitstream(stream s, integer q, function rfn, function bfn) vector of stream sv
User defines rfn and bfn rfn(object tpl, integer q) integer
rfnLRB(event e, integer q) integer as select expressway(e) where eventtype(e) = 0;
bfn(object tpl) booleanbfnLRB(event e) boolean as select eventtype(e) = 2;
rfn and bfn for streams are analogous to fragmentation and replication conditions in distributed DBMS
Unlike DDBMS, execution of rfn and bfn is parallelized
Defining stream splitting
splitstream
q
s sv
31 Aug 2011 Erik Zeitler and Tore Risch 8
Naïve (flat) splitstream implementation: fsplit
fsplit
CQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
Expensive stream splitting
computations
Bottleneck!
fsplit(stream s, integer q, function rfn, function bfn) vector of stream sv
31 Aug 2011 Erik Zeitler and Tore Risch 9
Tree shaped splitstream implementation: maxtree
fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ• Bottleneck is alleviated [Zeitler and Risch, DASFAA 2010]• but still problematic
maxtree(stream s, integer q, function rfn, function bfn) vector of stream sv
31 Aug 2011 Erik Zeitler and Tore Risch 10
Scaled-out splitstream: parasplit
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
Window router
distributes entire
windowsWindow splitter
Stream merge
parasplit(stream s, integer q, function rfn, function bfn) vector of stream sv
31 Aug 2011 Erik Zeitler and Tore Risch 11
Parasplit:route – //fsplit – //(merge – CQ)
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
Window router
distributes entire
windowsWindow splitter
Stream merge
parasplit(stream s, integer q, function rfn, function bfn) vector of stream sv
31 Aug 2011 Erik Zeitler and Tore Risch 12
Tree shaped window routing: parasplit*
PR
CQ2
CQ1
CQ3
CQ0
CQ5
CQ4
CQ7
CQ6
CQ2
CQ1
CQ3
CQ0
CQ5
CQ4
CQ7
CQ6
fsplit
fsplit
fsplit
fsplit
fsplit
fsplit
PR
PR
PR
fsplit
fsplit
fsplit
CQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
31 Aug 2011 Erik Zeitler and Tore Risch 13
Outline
1. Introduction2. Stream splitting strategies3. Scale-up of stream splitting
strategies4. Cost model and heuristic5. Energy efficiency6. Related work7. Conclusions and future work
31 Aug 2011 Erik Zeitler and Tore Risch 14
Experimental set-up
Hardware
Linux cluster Up to 70 nodes Each node has 2x
quad-core Intel® Xeon® E5430@2.66GHz, 6 MB L2$.
TCP/IP over GbE
Performance number L : Number of xways the DSMS can handle
www.cs.brandeis.edu/~linearroad
31 Aug 2011 Erik Zeitler and Tore Risch 15
LRB result
name orgyea
rL
cores
comment
Aurora Brandeis, Brown,
MIT
2004
2.5 1
Commercial sys A
2004
0.5 1
SPC IBM200
62.5 170 3GHz Xeon
Xquery ETHZ200
71.5 1
DataCell CWI200
91 4 1.4s avg RT
stream schema ETHZ201
05 4
SCSQ maxtree UU201
064 48
D disabled (later verified in mySQL)
SCSQ parasplit UU201
1512
560 D disabled
Performance number L : Number of xways the DSMS can handle
31 Aug 2011 Erik Zeitler and Tore Risch 16
Splitstream stream rate
0,00
200,00
400,00
600,00
800,00
1 000,00
0 100 200 300 400 500
q
Max
str
eam
rat
e [M
bps]
parasplit*parasplitmaxtreefsplit
CQ parallelism, q
1 Gbpswire
speed
31 Aug 2011 Erik Zeitler and Tore Risch 17
Window router stream rate
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
W – physical window size
p – number of parallel fsplit
pW
31 Aug 2011 Erik Zeitler and Tore Risch 18
Impact of window size W in window router
0,00
200,00
400,00
600,00
800,00
1000,00
0 5 10 15W [kB]
Ma
x s
tre
am
ra
te [
Mb
ps
]
p=4 p=64
network bound for large enough windows
31 Aug 2011 Erik Zeitler and Tore Risch 19
Impact of window size W in window router when scaling p
0,00
200,00
400,00
600,00
800,00
1000,00
0 5 10 15
W [kB]
Max
str
eam
rat
e [M
bp
s]
p=4 p=64
p=128 p=256
p=512
31 Aug 2011 Erik Zeitler and Tore Risch 20
0,00100,00
200,00300,00
400,00500,00
600,00700,00
800,00900,00
1 000,00
0 100 200 300 400 500p
Ma
x st
rea
m r
ate
[Mb
ps]
.
window router tree (parasplit*)
single window router (parasplit)
Parasplit*Tree shaped window router
W = 16 kB fsplit parallelism, p
31 Aug 2011 Erik Zeitler and Tore Risch 21
Outline
1. Introduction2. Stream splitting strategies3. Scale-up of stream splitting
strategies4. Cost model and heuristic5. Energy efficiency6. Related work7. Conclusions and future work
31 Aug 2011 Erik Zeitler and Tore Risch 22
Eliminate p
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
p
q
parasplit(stream s, integer q, function rfn, function bfn) vector of stream sv
Given• Input stream rate ΦD
• Parallelism of continuous query q Automatically determine
• fsplit parallelism p
31 Aug 2011 Erik Zeitler and Tore Risch 23
splitconsume ...R1
Rq
...emit(R1)
emit(Rq)
Cost model for fsplit
cr – read cost per tpl (read + de-marshal)cs – split cost per tpl (execute rfn and bfn)ce – emit cost per tpl (marshal + print)o – omit %r – routing %b – broadcast % q – number of output streams
)()( bqrcebqrocscrC fsplit
according to rfn and bfn
31 Aug 2011 Erik Zeitler and Tore Risch 24
Cost model for merge in CQ
cr – read cost per tpl (read + de-marshal)cp – poll cost per tplcm – merge cost per tplO – cost of executing the CQ and emit its
result
OcmcppcrCCQ
merge compute split
S1 consume(S1)
Sp consume(Sp)
... ... ...R1
Rw
...emit(R1)
emit(Rw)
31 Aug 2011 Erik Zeitler and Tore Risch 25
Cost model for parasplit
p can be eliminated using cost model, but
requires extensive profiling everywhere
OcmcppcrCCQ bqrcebqrocscrC Wfsplit
WWWPR cecscrC
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
p
q
31 Aug 2011 Erik Zeitler and Tore Risch 26
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
p
q
Heuristic for estimating p
Assume• 1% broadcast tuples (configurable)• 0% omitted tuples (configurable)
Measure Φfsplit(1)
on rfn and bfn, q = 1: cs +ce = 1/Φfsplit(1)
Estimate p by
qcecsC fsplit 01.099.0ˆ
bqrcebqrocscrC Wfsplit
qpfsplit
D 01.099.0ˆ)1(
fsplit
31 Aug 2011 Erik Zeitler and Tore Risch 27
0,00100,00200,00300,00400,00500,00600,00700,00800,00900,00
1 000,00
0 100 200 300 400 500q
Max
str
eam
rat
e [M
bps]
parasplitcost modelToo high p (p=q)Too low p (p=1)
p according to heuristics vs.p using exact cost model
CQ parallelism, q
31 Aug 2011 Erik Zeitler and Tore Risch 28
Outline
1. Introduction2. Stream splitting strategies3. Scale-up of stream splitting
strategies4. Cost model and heuristic5. Energy efficiency6. Related work7. Conclusions and future work
31 Aug 2011 Erik Zeitler and Tore Risch 29
Estimating energy efficiency, η
How much extra energy does parasplit consume in comparison to fsplit?
Conservatively assume energy consumption proportional to CPU usage:
Useful work • p ∙ Cfsplit
Overhead• CPR
• q ∙ CCQ(O=0)
)0(
OCQfsplitPR
fsplit
CqCpC
Cp
PR fsplit
fsplit
fsplitCQ
CQ
CQ
CQ
CQ
CQ
CQ
CQ
31 Aug 2011 Erik Zeitler and Tore Risch 30
Measuring energy efficiency
0%10%20%30%40%50%60%70%80%90%
100%
0 100 200 300 400 500q
Eff
icie
ncy
parasplit*parasplitcost modelToo high p (p=q)Too low p (p=1)
CQ parallelism, q
31 Aug 2011 Erik Zeitler and Tore Risch 31
Related work Nobody else has investigated strategies for scalable
customizable stream splitting IBM SPADE/System S [Andrade et al 2009]
• Splitstream operator with broadcast capabilities• Streaming throughput degrades when scaling q
Event based systems [Brenna et al 2009]• Custom stream splitting shown to be a bottleneck
Gigascope [Johnson et al 2008]• Assumes specialized stream splitting hardware• No customizable stream splitting
GSDM [Ivanova, Risch 2005]• Parallel execution of expensive UDFs• More limited parallelization
Streaming MapReduce [Condie et al 2010]• Does not handle scalable stream splitting
[Balkesen, Tatbul 2011]• Distributing entire windows over CQs• q ≤ 4
31 Aug 2011 Erik Zeitler and Tore Risch 32
Conclusions and future work
Naïve stream splitting is prohibitive for scale-out of CQs
Parasplit• eliminates the bottleneck of stream splitting, providing
network bound stream rates Parasplit*
• provides network bound stream rates for highly scaled-out stream splitting
Push selection predicates from CQ to rfn of splitstream
Improve energy efficiency High Availability SCSQ home page
• http://www.it.uu.se/research/group/udbl/SCSQ.html