Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Aggregation Queries
27
Ohio State University Department of Computer Science and Engineering Using Space and Attribute Partitioned Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Partial Replicas for Data Subsetting and Aggregation Queries Aggregation Queries Li Weng, Umit Catalyurek, Tahsin Umit Catalyurek, Tahsin Kurc, Kurc, Gagan Agrawal, Joel Saltz Joel Saltz
description
Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Aggregation Queries. Li Weng, Umit Catalyurek, Tahsin Kurc, Gagan Agrawal, Joel Saltz. Motivation: Data-Driven Science. Oil Reservoir Management. Magnetic Resonance Imaging. - PowerPoint PPT Presentation
Transcript of Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Aggregation Queries
Slide 1Department of Computer Science and Engineering
Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Aggregation Queries
Li Weng, Umit Catalyurek, Tahsin Kurc,
Gagan Agrawal, Joel Saltz
Motivation: Data-Driven Science
Magnetic Resonance Imaging
Oil Reservoir Management
Large Spatio-temporal datasets
Ohio State University
Replication of Scientific Datasets
Each requires different spatial-temporal region and subset of attributes
No chunking and indexing strategy can optimize for all
Replication: Create multiple copies
Large storage overhead
Ohio State University
Partial Replication
Can we get benefits of replication without the large overheads ?
Not all attributes accessed uniformly
Not all spatio-temporal regions accessed with uniform probability
Partial Replication: Each replica has
Only a subset of attributes (attribute partitioned) and/or
Only a rectilinear spatio-temporal region (space partitioned)
Challenge:
No single partial replica may be able to answer the query
Can we choose and combine partial replicas to optimize query processing ?
Ohio State University
Prior Work (CCGRID 05)
Cost models
Consider SELECT SQL queries
Implemented as an extension to Automatic Data Virtualization System (HPDC 04)
Ohio State University
Contributions
Support combined use of space and attribute partitioned partial replicas
Dynamic programming algorithm for selecting the best set of attribute partitioned replicas
New greedy strategy for recommending a combination of replicas
Extend replica selection algorithm to address queries with aggregations
-- replicas may be unevenly stored across storage units
Ohio State University
System Overview
The Replica Selection Module is coupled tightly with our prior work
of supporting SQL Select queries on scientific datasets in a cluster
environment.
STORM Runtime System
A middleware to support data selection, data partitioning, and data transfer operations on flat-file datasets hosted on a parallel system.
Services
Outline
Introduction
Motivation
Contributions
Uniformly partitioned chunks and select queries
Uneven partitioning and aggregation operation
Experimental results
Related work
Computing Goodness Value
goodness = useful dataper-chunk / costper-chunk
Chunk: an atomic unit in space partitioned replicas or a logic unit in attribute partitioned replicas
Full chunks and partial chunks of a partial replica
Cost per-chunk = tread * nread + tseek
tread : average read time for a disk page
nread : number of pages fetched
tseek : average seek time
intermediate unit between a replica and its chunks
a group of full or partial chunks having same goodness value in a replica
goodnessper-fragmen = useful dataper-fragment / costper-fragment
An Example –
3 fragments
General Structure of
Replica Selection Algorithm
Ohio State University
Dynamic Programming Algorithm
R’: the optimal combination output
l: the number of referred attributes in Q
M1..l: the referred attribute list
Calculate the Costj,j
Foreach u from 1 to l-k+1
Calculate Costu..v, Locu..v->s=-1, Locu..v->r=r1
Calculate Costu..v, Locu..v->s=-1, Locu..v->r=r2
Costu..v=∞
Costu..v=q, Locu..v->s=p, Locu..v->r=-1
Output(loc1..l)
Yes
No
Yes
No
R’
Output
Greedy Strategy
Fmax : the fragment with the maximum goodness value in F
S : the ordered list of the candidate fragments in decreasing order of their goodness value
Calculate the fragment set F
Append Fmax Into S
Re-compute the goodness value
Uneven Partitioning and Aggregation Operations
Computing Goodness Value
P : all available storage nodes
CurLoad : current workload across all storage nodes due to previously chosen candidate replicas
Cost fragment = tread*nread+tseek* nseek+tfilter*nfilter+tagg*nagg+ttrans*ntrans
tfilter : average filtering time for a tuple
nfilter : number of total tuples in all chunks
taggr : average aggregate computation time for a tuple
naggr : number of total useful tuples
ttrans : network transfer time for one unit of data
ntrans : the amount of data after aggregate operation
Ohio State University
Workload aware greedy strategy
Q : an issued query
D : the original dataset
Fmax : the fragment with the maximum goodness value in F
S : the ordered list of the candidate fragments in decreasing order of their goodness value
Append Fmax Into S
Append Fi into S
Yes
No
Yes
No
Outline
Introduction
Motivation
Contributions
Uniformly partitioned chunks and select queries
Uneven partitioning and aggregation operation
Experimental results
Related work
Experimental Setup & Design
A Linux cluster connected via a Switched Fast Ethernet. Each node has a PIII 933MHz CPU, 512 MB main Memory, and three 100GB IDE disks.
Performance evaluation of the combination of space-partitioned and attribute-partitioned replicas, and the benefit of attribute-partitioned replicas;
Scalability test when increasing the number of nodes hosting dataset;
Performance test when data query sizes are varied;
Performance evaluation for aggregate queries with unevenly partitioned replicas.
Ohio State University
Ohio State University
Department of Computer Science and Engineering
SELECT attrlist from IPARS where RID in [0,1] and TIME in [1000,1399] and X>=0 and X<=11
and Y>=0 and Y<=28 and Z>=0 and Z<=28;
Ohio State University
attr+space part :
space part :
A run-time optimization
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
#Query
SELECT * from IPARS where TIME>=1000 and TIME<=1599 and X>=0 and X<=11
and Y>=0 and Y<=31 and Z>=0 and Z<=31;
Upto 4 nodes, query execution time scales linearly.
Due to the dominating seek cost in the total I/O overhead, execution time is not reduced by half while using 8 nodes.
Chart6
1
1
2
2
4
4
8
8
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
# Query
SELECT * from IPARS where TIME>=1000 and TIME<=TIMEVAL and X>=0 and X<=11
and Y>=0 and Y<=28 and Z>=0 and Z<=28;
Our algorithm has chosen {1,3,4,6} out of all replicas in Table #1.
The query filters 83% of the retrieved data when using the original dataset only; however,
it need to filter about 50% of the retrieved data in the presence of replicas.
Chart2
[1000,1199]
[1000,1199]
[1000,1399]
[1000,1399]
[1000,1599]
[1000,1599]
[1000,1799]
[1000,1799]
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
Ohio State University
Ohio State University
Alg+Ref – solution after the refinement step
Solution-1 & 2 – two manually created solutions
Chart1
Estimated
Estimated
Estimated
Estimated
Real
Real
Real
Real
Solution-2: {0, 1, 4}
16.8462
110.1677469238
118
12.3925
81.0422412029
67.3
Solution-1
17.9651
117.4849277736
120.3
Solution-2
Alg: {0, 2, 3, 4, 5, 6}
16.93
51.240550427
50
Alg+Ref: {0, 2, 3, 5, 6}
16.2382
49.1467398667
47
Solution-1
29.8557
90.3616362305
93.5
Solution-2
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
16.8462
110.1677469238
118
12.3925
81.0422412029
67.3
Solution-1
17.9651
117.4849277736
120.3
Solution-2
Alg: {0, 2, 3, 4, 5, 6}
16.93
51.240550427
50
Alg+Ref: {0, 2, 3, 5, 6}
16.2382
49.1467398667
47
Solution-1
29.8557
90.3616362305
93.5
Solution-2
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
Related Work
Replication research
Data availability and reliability
Data caching techniques
Management and replacement of replicas
Our previous work on performance optimization using space partitioned replicas
Ohio State University
Conclusions
The proposed cost models are capable of estimating execution time trends.
The designed greedy strategy together with dynamic programming algorithm can choose a good set of candidate replicas that decrease the query execution time.
Our implementations show good scalability.
When data transfer bandwidth is the limiting factor, using a combination of space and attribute partitioned replicas should be preferred.
Processing Remotely
(AVHRR)
•
are gathered to form an
instantaneous field of view
A single file of
IFOV’s
}Solution-2: {0, 1, 4
Using Space and Attribute Partitioned Partial Replicas for Data Subsetting and Aggregation Queries
Li Weng, Umit Catalyurek, Tahsin Kurc,
Gagan Agrawal, Joel Saltz
Motivation: Data-Driven Science
Magnetic Resonance Imaging
Oil Reservoir Management
Large Spatio-temporal datasets
Ohio State University
Replication of Scientific Datasets
Each requires different spatial-temporal region and subset of attributes
No chunking and indexing strategy can optimize for all
Replication: Create multiple copies
Large storage overhead
Ohio State University
Partial Replication
Can we get benefits of replication without the large overheads ?
Not all attributes accessed uniformly
Not all spatio-temporal regions accessed with uniform probability
Partial Replication: Each replica has
Only a subset of attributes (attribute partitioned) and/or
Only a rectilinear spatio-temporal region (space partitioned)
Challenge:
No single partial replica may be able to answer the query
Can we choose and combine partial replicas to optimize query processing ?
Ohio State University
Prior Work (CCGRID 05)
Cost models
Consider SELECT SQL queries
Implemented as an extension to Automatic Data Virtualization System (HPDC 04)
Ohio State University
Contributions
Support combined use of space and attribute partitioned partial replicas
Dynamic programming algorithm for selecting the best set of attribute partitioned replicas
New greedy strategy for recommending a combination of replicas
Extend replica selection algorithm to address queries with aggregations
-- replicas may be unevenly stored across storage units
Ohio State University
System Overview
The Replica Selection Module is coupled tightly with our prior work
of supporting SQL Select queries on scientific datasets in a cluster
environment.
STORM Runtime System
A middleware to support data selection, data partitioning, and data transfer operations on flat-file datasets hosted on a parallel system.
Services
Outline
Introduction
Motivation
Contributions
Uniformly partitioned chunks and select queries
Uneven partitioning and aggregation operation
Experimental results
Related work
Computing Goodness Value
goodness = useful dataper-chunk / costper-chunk
Chunk: an atomic unit in space partitioned replicas or a logic unit in attribute partitioned replicas
Full chunks and partial chunks of a partial replica
Cost per-chunk = tread * nread + tseek
tread : average read time for a disk page
nread : number of pages fetched
tseek : average seek time
intermediate unit between a replica and its chunks
a group of full or partial chunks having same goodness value in a replica
goodnessper-fragmen = useful dataper-fragment / costper-fragment
An Example –
3 fragments
General Structure of
Replica Selection Algorithm
Ohio State University
Dynamic Programming Algorithm
R’: the optimal combination output
l: the number of referred attributes in Q
M1..l: the referred attribute list
Calculate the Costj,j
Foreach u from 1 to l-k+1
Calculate Costu..v, Locu..v->s=-1, Locu..v->r=r1
Calculate Costu..v, Locu..v->s=-1, Locu..v->r=r2
Costu..v=∞
Costu..v=q, Locu..v->s=p, Locu..v->r=-1
Output(loc1..l)
Yes
No
Yes
No
R’
Output
Greedy Strategy
Fmax : the fragment with the maximum goodness value in F
S : the ordered list of the candidate fragments in decreasing order of their goodness value
Calculate the fragment set F
Append Fmax Into S
Re-compute the goodness value
Uneven Partitioning and Aggregation Operations
Computing Goodness Value
P : all available storage nodes
CurLoad : current workload across all storage nodes due to previously chosen candidate replicas
Cost fragment = tread*nread+tseek* nseek+tfilter*nfilter+tagg*nagg+ttrans*ntrans
tfilter : average filtering time for a tuple
nfilter : number of total tuples in all chunks
taggr : average aggregate computation time for a tuple
naggr : number of total useful tuples
ttrans : network transfer time for one unit of data
ntrans : the amount of data after aggregate operation
Ohio State University
Workload aware greedy strategy
Q : an issued query
D : the original dataset
Fmax : the fragment with the maximum goodness value in F
S : the ordered list of the candidate fragments in decreasing order of their goodness value
Append Fmax Into S
Append Fi into S
Yes
No
Yes
No
Outline
Introduction
Motivation
Contributions
Uniformly partitioned chunks and select queries
Uneven partitioning and aggregation operation
Experimental results
Related work
Experimental Setup & Design
A Linux cluster connected via a Switched Fast Ethernet. Each node has a PIII 933MHz CPU, 512 MB main Memory, and three 100GB IDE disks.
Performance evaluation of the combination of space-partitioned and attribute-partitioned replicas, and the benefit of attribute-partitioned replicas;
Scalability test when increasing the number of nodes hosting dataset;
Performance test when data query sizes are varied;
Performance evaluation for aggregate queries with unevenly partitioned replicas.
Ohio State University
Ohio State University
Department of Computer Science and Engineering
SELECT attrlist from IPARS where RID in [0,1] and TIME in [1000,1399] and X>=0 and X<=11
and Y>=0 and Y<=28 and Z>=0 and Z<=28;
Ohio State University
attr+space part :
space part :
A run-time optimization
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
#Query
SELECT * from IPARS where TIME>=1000 and TIME<=1599 and X>=0 and X<=11
and Y>=0 and Y<=31 and Z>=0 and Z<=31;
Upto 4 nodes, query execution time scales linearly.
Due to the dominating seek cost in the total I/O overhead, execution time is not reduced by half while using 8 nodes.
Chart6
1
1
2
2
4
4
8
8
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
# Query
SELECT * from IPARS where TIME>=1000 and TIME<=TIMEVAL and X>=0 and X<=11
and Y>=0 and Y<=28 and Z>=0 and Z<=28;
Our algorithm has chosen {1,3,4,6} out of all replicas in Table #1.
The query filters 83% of the retrieved data when using the original dataset only; however,
it need to filter about 50% of the retrieved data in the presence of replicas.
Chart2
[1000,1199]
[1000,1199]
[1000,1399]
[1000,1399]
[1000,1599]
[1000,1599]
[1000,1799]
[1000,1799]
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
=
5
horizontal
partial
replicas
and
10
vertical
partial
replicas
with
different
partial
ranges
and
chunk
size/shape
Hot
Range
1
Hot
Range
2
Hot
Range
3
Hot
Range
4
Hot
Range
5
Hot
Range
6
#1
Query1
(8
nodes)
select
=
each
tuple=
84
Bytes
Hot
Range
1
one
horizontal
replica
1
Hot
Range
2
one
horizontal
replica
2
Hot
Range
3
one
horizontal
replica
3
Hot
Range
4
one
horizontal
replica
4(a)
1720KB
11
vertical
replicas
4(b)
{x,y,z,rid,time},
327KB
4(c)
{x,y,z,rid,time,
coil,soil,
poil,
oilvx,
oilvy,
oilvz}
819KB
4(d)
{oilvx/y/z,
soil,
poil,
coil}
491KB
4(e)
{gasvx/y/z,
sgas,
cgas,
pgas}
491KB
4(f)
{watvx/y/z,
pwat,
cwat}
409KB
4(g)
{x,
oilvx,
gasvx,
watvx}
327KB
4(h)
{y,
oilvy,
gasvy,
watvy}
327KB
4(i)
{z,
oilvz,
gasvz,
watvz}
327KB
4(j)
{sgas,
coil,
pwat}
245KB
4(k)
{pgas,
cwat}
164KB
4(l)
{soil,
cgas}
164KB
Hot
Range
5
one
horizontal
replica
5
Hot
Range
6
one
horizontal
replica
6
#query1
select
Ohio State University
Ohio State University
Alg+Ref – solution after the refinement step
Solution-1 & 2 – two manually created solutions
Chart1
Estimated
Estimated
Estimated
Estimated
Real
Real
Real
Real
Solution-2: {0, 1, 4}
16.8462
110.1677469238
118
12.3925
81.0422412029
67.3
Solution-1
17.9651
117.4849277736
120.3
Solution-2
Alg: {0, 2, 3, 4, 5, 6}
16.93
51.240550427
50
Alg+Ref: {0, 2, 3, 5, 6}
16.2382
49.1467398667
47
Solution-1
29.8557
90.3616362305
93.5
Solution-2
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
16.8462
110.1677469238
118
12.3925
81.0422412029
67.3
Solution-1
17.9651
117.4849277736
120.3
Solution-2
Alg: {0, 2, 3, 4, 5, 6}
16.93
51.240550427
50
Alg+Ref: {0, 2, 3, 5, 6}
16.2382
49.1467398667
47
Solution-1
29.8557
90.3616362305
93.5
Solution-2
Solution-2: {0, 1, 4}
Solution-1: {0, 1, 3, 4}
Solution-2: {0, 1, 4}
Related Work
Replication research
Data availability and reliability
Data caching techniques
Management and replacement of replicas
Our previous work on performance optimization using space partitioned replicas
Ohio State University
Conclusions
The proposed cost models are capable of estimating execution time trends.
The designed greedy strategy together with dynamic programming algorithm can choose a good set of candidate replicas that decrease the query execution time.
Our implementations show good scalability.
When data transfer bandwidth is the limiting factor, using a combination of space and attribute partitioned replicas should be preferred.
Processing Remotely
(AVHRR)
•
are gathered to form an
instantaneous field of view
A single file of
IFOV’s
}Solution-2: {0, 1, 4
