Post on 07-Jun-2020
On The [Ir]relevance of Network Performance for Data Processing
Animesh Trivedi, Patrick Stuedi, Jonas Pfefferle,Radu Stoica, Bernard Metzler, Ioannis Koltsidas,
Nikolas Ioannou
IBM Research, Zurich
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 2
How [Ir]relevant is the Network?
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 3
How [Ir]relevant is the Network?
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 4
How [Ir]relevant is the Network?
TeraSort PageRank SQL WordCount GroupBy0
50
100
150
200
250
3001 Gbps 10 Gbps 40 Gbps
Ru
nti
me
in s
ecs
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 5
How [Ir]relevant is the Network?
TeraSort PageRank SQL WordCount GroupBy0
50
100
150
200
250
3001 Gbps 10 Gbps 40 Gbps
Ru
nti
me
in s
ecs
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 6
How [Ir]relevant is the Network?
TeraSort PageRank SQL WordCount GroupBy0
50
100
150
200
250
3001 Gbps 10 Gbps 40 Gbps
Network IO is very relevant - up to 64%
Ru
nti
me
in s
ecs
60%
64%
47% 33%
28%
1
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 7
How [Ir]relevant is the Network?
TeraSort PageRank SQL WordCount GroupBy0
50
100
150
200
250
3001 Gbps 10 Gbps 40 Gbps
Network IO is very relevant - up to 64% ??
Ru
nti
me
in s
ecs
1
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 8
Is It Spark Specific?
Flink-TS Flink-PR GraphLab Timely0
50
100
150
200
250
3001 Gbps 10 Gbps 40 Gbps
Ru
nti
me
in s
ecs
725s
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 9
Spark TeraSort: The Shuffle Story
outputinput
distributed sorting
- simple - shuffle data is input data- highest chance of improvements
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 10
Spark TeraSort: The Shuffle Story
Shuffledata
Reduce tasks
output
Map tasks
Cores
input
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 11
Spark TeraSort: The Shuffle Story
Shuffledata
Reduce tasks
output
net
net
net
reading in shuffle data
Cores
input
Map tasks
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 12
Spark TeraSort: The Shuffle Story
Shuffledata
Reduce tasks
output
net CPU
net CPU
net CPU
reading in shuffle data
sortingshuffle data
Cores
input
Map tasks
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 13
Spark TeraSort: The Shuffle Story
Shuffledata
Reduce tasks
output
net CPU
net CPU
net CPU
reading in shuffle data
sortingshuffle data
performance
Cores
input
Map tasks
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 14
How Important is the Network?
1 Gbps 10 Gbps 40 Gbps 100 Gbps*0%
20%
40%
60%
80%
100%
CPUNetwork
Gains from the networks are shadowed by the high CPU footprint
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 15
How Important is the Network?
1 Gbps 10 Gbps 40 Gbps 100 Gbps*0%
20%
40%
60%
80%
100%
CPUNetwork
Gains from the networks are shadowed by the high CPU footprint
52%
48%
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 16
How Important is the Network?
1 Gbps 10 Gbps 40 Gbps 100 Gbps*0%
20%
40%
60%
80%
100%
CPUNetwork
Gains from the networks are shadowed by the high CPU footprint
52%
48%
8%
92%
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 17
How Important is the Network?
1 Gbps 10 Gbps 40 Gbps 100 Gbps*0%
20%
40%
60%
80%
100%
CPUNetwork
Gains from the networks are shadowed by the high CPU footprint
52%
48%
8%
92% 97%
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 18
How Important is the Network?
1 Gbps 10 Gbps 40 Gbps 100 Gbps*0%
20%
40%
60%
80%
100%
CPUNetwork
Gains from the networks are shadowed by the high CPU footprint
52%
48%
8%
92% 97% 99%
Network gains are shadowed by the CPU
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 19
What Exactly is the CPU Doing?
Map Reduce0%
20%
40%
60%
80%
100%
Misc.IteratorSerializationSortingIOJVMLinux
Spark
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 20
What Exactly is the CPU Doing?
Map Reduce0%
20%
40%
60%
80%
100%
Misc.IteratorSerializationSortingIOJVMLinux
Spark
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 21
What Exactly is the CPU Doing?
Map Reduce0%
20%
40%
60%
80%
100%
Misc.IteratorSerializationSortingIOJVMLinux
Overheads are spread across the entire stack - serialization, abstration, execution model etc.2
Spark
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 22
The Balancing Act: CPU vs Network
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 23
The Balancing Act: CPU vs Network
I.Balance out the CPU
with the network time
Sorting : O(nlog(n))Network: O(n)
use smaller 'n'
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 24
The Balancing Act: CPU vs Network
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 25
The Balancing Act: CPU vs Network
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 26
The Balancing Act: CPU vs Network
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 27
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
if a single corecannot do 40 Gbps
then use more
Needs a more careful analysis of at the entire stack
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 28
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
1 2 4 8 160
20
40
60
Number of cores
idealmeasured
Ban
dw
idth
(Gb
ps)
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 29
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
1 2 4 8 160
20
40
60
Number of cores
idealmeasured
Ban
dw
idth
(Gb
ps)
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 30
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
1 2 4 8 160
20
40
60
Number of cores
idealmeasured
Ban
dw
idth
(Gb
ps)
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 31
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
Number of coresR
un
tim
e (s
ecs)
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
1 2 4 8 160
100
200
300reduce map
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 32
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
Number of coresR
un
tim
e (s
ecs)
I.Balance out the CPU
with the network time
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
1 2 4 8 160
100
200
300reduce map
260
_____coresruntime = 9 +
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 33
The Balancing Act: CPU vs Network
II.Use more cores to
scale up
Classical techniques are ineffective
I.Balance out the CPU
with the network time
3
Smaller Partitions
Ru
nti
me
(se
cs)
020406080
100
Number of cores1 2 4 8 16
0
100
200
300reduce map
Ru
nti
me
(se
cs)
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 34
ConclusionFaster networks (IO) are very relevant – as long as you have CPU cycles – differentiate between user vs framework CPU usage
Framework's CPU usage is bad – CPU-network imbalance : sorting, serialization, volcano
execution model, etc. – scalability (serial vs parallel components)– ineffective classical balancing techniques
Knowing today's usec-era IO and CPU hardware, how would you re-design modern data processing framework?
1
2
3
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 35
ConclusionFaster networks (IO) are very relevant – as long as you have CPU cycles – differentiate between user vs framework CPU usage
Framework's CPU usage is bad – CPU-network imbalance : sorting, serialization, volcano
execution model, etc. – scalability (serial vs parallel components)– ineffective classical balancing techniques
Knowing today's usec-era IO and CPU hardware, how would you re-design modern data processing framework?
1
2
3
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 36
ConclusionFaster networks (IO) are very relevant – as long as you have CPU cycles – differentiate between user vs framework CPU usage
Framework's CPU usage is bad – CPU-network imbalance : sorting, serialization, volcano
execution model, etc. – scalability (serial vs parallel components)– ineffective classical balancing techniques
Knowing today's usec-era IO and CPU hardware, how would you re-design modern data processing framework?
1
2
3
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 37
Backup
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 38
Spark
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 39
Spark
The 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16) 40
Runtime
1 2 4 8 160
50
100
150
200
250
300reduce map
What Exactly is the CPU Doing?Sp
ark
Map Reduce Reduce/Count0%
20%
40%
60%
80%
100%
Misc.IteratorSerializationSortingIOJVMLinux