dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data...
Transcript of dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data...
![Page 1: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/1.jpg)
![Page 2: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/2.jpg)
![Page 3: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/3.jpg)
https://openclipart.org/detail/20542/dell-1u-server https://openclipart.org/detail/216936/dell-2u
![Page 4: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/4.jpg)
https://openclipart.org/detail/172422/forklift and https://openclipart.org/detail/252316/database-cylinder-red
![Page 5: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/5.jpg)
Efficiently
Everywhere
Effortlessly
![Page 6: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/6.jpg)
CPUcaches
Magnetic Hard Disk Drives
DRAM
NVRAMNVRAM
RDMARDMARDMA
In-memory is really “in”Efficiently
Everywhere
Effortlessly
![Page 7: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/7.jpg)
EfficientlyOn modern hardware
many cores, large
memory configuratio
ns
Effortlessly
Azure: Pricing vs Configuration (Series: D, F and G)
16
20
16
32
0
0.2
0.4
0.6
0.8
0 2 4 6 8 10 12 14 16
$/c
ore
/ho
ur
GB/core
D5v2
D15v2
F16
G5Bubble volume = #cores
Everywhere
![Page 8: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/8.jpg)
EfficientlyOn modern hardware
many cores, large
memory configuratio
ns
Everywhere
Containers of any size
infinite and instant
elasticity
https://openclipart.org/detail/29048/recepticle
Effortlessly
![Page 9: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/9.jpg)
EfficientlyOn modern
hardware
many cores,
large memory
configurations
EverywhereContainers of
any size
infinite and
instant
elasticity
Effortlessly Auto-tuninga true utility
model
![Page 10: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/10.jpg)
Storage Manager Design
Block #4(Row Store)
`
Meta data
Block #1(Data Morphed)
Meta data
Block #2(Column Store)
Meta data
Block #3Row Store + 2 Indices
`
Meta data
B-tre
e
Hash
In
dex
Block #5(Row Store)
`
Meta data
A database = A collection of tables
A table = set of blocks
A block = bag of tuples
A block = a mini self-contained database. Only tuples from one table per
block.A sub-block = a storage organization within a block
e.g. row sub-block, column sub-block, or index sub-
block
Chasseur and Patel, VLDB’13
![Page 11: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/11.jpg)
11
⨝
σ σ
R S
SELECT * FROM R, SWHERE R.b > 10
AND S.c >100
AND R.a= S.a
The “traditional” way
![Page 12: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/12.jpg)
Pool of Worker
Threads
12
Clean Separation of Data Flow and Control Flow
⨝
σ σ
R S
SELECT * FROM R, SWHERE R.b > 10
AND S.c >100
AND R.a= S.a
r1
The “traditional” way The Quickstep way
r2 s1 s2
s3 s4
Scheduler Queue:
σ (r1)σ (r2)σ (s1). . .
TMB
Buffer Pool
r1
r2
s1 s2
s3 s4
r
’s’
s’’
+ Cleaner Abstraction+ Dynamic Optimization
+ Better p9X
+ Manageability and Debug-ability
![Page 13: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/13.jpg)
0
10
20
30
Q1.1
Q1.2
Q1.3
Q2.1
Q2.2
Q2.3
Q3.1
Q3.2
Q3.3
Q3.4
Q4.1
Q4.2
Q4.3
Execu
tio
n T
ime
(seco
nd
s)
Query
Quickstep MonetDB PostgreSQL 9.6 (β1) Spark
SSB 10GB on 4 core container
![Page 14: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/14.jpg)
Priority scheduling: Example of elastic behavior
![Page 15: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/15.jpg)
EfficientlyBare metal
performance on core
data kernels, and on
any hardware
Everywhereelasticity for inter and
intra-operator
parallelism, even for
complex operators
Effortlessly
Unified memory
management, hybrid
storage formats,
learning-based
scheduler/optimizer
An experimental data platform for the cloud-
native data world
www.quickstep.io
![Page 16: dell-2u · 2018-01-04 · Storage Manager Design Block #4 (Row Store) ` Meta data Block #1 (Data Morphed) Meta data Block #2 (Column Store) Block #3 Row Store + 2 Indices ` Meta data](https://reader034.fdocuments.us/reader034/viewer/2022043003/5f80ea56f7893067a4169bc8/html5/thumbnails/16.jpg)
Rogers
Jeffrey
Saket
SaurabhHarshad
Deshmukh
Hakan
Memisoglu
Navneet
Potti
Subasree
Venkatsub
h-
ramaniyen
Marc
Spehlmann
Jianqiao
Zhu
Zuyu
Zhang