Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5...
Transcript of Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5...
![Page 1: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/1.jpg)
Elasecutor: Elastic Executor Scheduling in Data Analytics Systems
Libin Liu, Hong Xu
City University of Hong Kong
ACM Symposium on Cloud Computing 2018
1
![Page 2: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/2.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
2
![Page 3: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/3.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
2
![Page 4: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/4.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
2
![Page 5: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/5.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
2
![Page 6: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/6.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
3
![Page 7: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/7.jpg)
Data Analytics Systems
• Various workloads running in data analytics systems concurrently
• The workflow of an analytics application can be expressed as a DAG
3
Directed Acyclic Graph (DAG)
Stage 1parallelize
filter
map
Stage 2reduceByKey
map
Stage 3parallelize
filter
map
Stage 4join
●
●
![Page 8: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/8.jpg)
Resource Scheduling
• Resource schedulers for various objectives, e.g., fairness, cluster utilization, application completion time, etc.
4
![Page 9: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/9.jpg)
Resource Scheduling
• Resource schedulers for various objectives, e.g., fairness, cluster utilization, application completion time, etc.
4
Efficient resource scheduling is an important and practical issue in data analytics systems
![Page 10: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/10.jpg)
Current Solutions
• Static allocation according to peak demands
• “Task-based” resource schedulers adopted in “executor-based” systems
• Assign executors to machines randomly
5
![Page 11: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/11.jpg)
Need for an Elastic Scheduler
6
![Page 12: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/12.jpg)
Need for an Elastic Scheduler
7
![Page 13: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/13.jpg)
Need for an Elastic Scheduler
7
Executor resource usage exhibits significant temporal variations
![Page 14: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/14.jpg)
Need for an Elastic Scheduler
8
Resource CPU Memory Network Disk
Terasort
Peak/Avg. 1.8 1.7 6.2 1.5
Peak/Trough 60 3.3 237 6.1
K-means
Peak/Avg. 1.7 1.2 11.5 5.6
Peak/Trough 75 6 53 100
PagerankPeak/Avg. 3.9 1.3 20.2 9.1
Peak/Trough 50 11.5 119 50
Logistic Regression
Peak/Avg. 2.1 1.4 5.5 6.1
Peak/Trough 50 12 409.6 42.5
![Page 15: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/15.jpg)
Need for an Elastic Scheduler
8
Resource CPU Memory Network Disk
Terasort
Peak/Avg. 1.8 1.7 6.2 1.5
Peak/Trough 60 3.3 237 6.1
K-means
Peak/Avg. 1.7 1.2 11.5 5.6
Peak/Trough 75 6 53 100
PagerankPeak/Avg. 3.9 1.3 20.2 9.1
Peak/Trough 50 11.5 119 50
Logistic Regression
Peak/Avg. 2.1 1.4 5.5 6.1
Peak/Trough 50 12 409.6 42.5
![Page 16: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/16.jpg)
Need for an Elastic Scheduler
8
Resource CPU Memory Network Disk
Terasort
Peak/Avg. 1.8 1.7 6.2 1.5
Peak/Trough 60 3.3 237 6.1
K-means
Peak/Avg. 1.7 1.2 11.5 5.6
Peak/Trough 75 6 53 100
PagerankPeak/Avg. 3.9 1.3 20.2 9.1
Peak/Trough 50 11.5 119 50
Logistic Regression
Peak/Avg. 2.1 1.4 5.5 6.1
Peak/Trough 50 12 409.6 42.5
![Page 17: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/17.jpg)
Need for an Elastic Scheduler
8
Resource CPU Memory Network Disk
Terasort
Peak/Avg. 1.8 1.7 6.2 1.5
Peak/Trough 60 3.3 237 6.1
K-means
Peak/Avg. 1.7 1.2 11.5 5.6
Peak/Trough 75 6 53 100
PagerankPeak/Avg. 3.9 1.3 20.2 9.1
Peak/Trough 50 11.5 119 50
Logistic Regression
Peak/Avg. 2.1 1.4 5.5 6.1
Peak/Trough 50 12 409.6 42.5
Static allocation using peak demands would cause severe resource wastage and performance issues
![Page 18: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/18.jpg)
Our Idea
9
Dynamically allocate and explicitly size resources to executors over time, and strategically assign
executors to machines
![Page 19: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/19.jpg)
Our Idea
9
Dynamically allocate and explicitly size resources to executors over time, and strategically assign
executors to machines
![Page 20: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/20.jpg)
Our Idea
9
Dynamically allocate and explicitly size resources to executors over time, and strategically assign
executors to machines
Elasecutor, a novel executor scheduler for data analytics systems
![Page 21: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/21.jpg)
Outline
• Motivation
• Elasecutor Design ‒Elastic Executor Scheduling ‒Demand Prediction ‒Dynamic Reprovisioning
• Implementation
• Evaluation
• Conclusion
10
![Page 22: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/22.jpg)
Elastic Executor Scheduling
• Challenge − Scheduling executors with their multi-resource demand
time-series −Multi-dimensional packing −APX-hard −Analyzed in detail in section 3.2.1
• Objective −Minimizing makespan − i.e., avoid resource underutilization and minimize
machine-level resource fragmentation
11
![Page 23: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/23.jpg)
Elastic Executor Scheduling - DRR
• Dominant Remaining Resource: “dominant” = “maximum”
• An example: We select as the time point to calculate DRR for machine 1. and , and its DRR is
12
![Page 24: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/24.jpg)
Elastic Executor Scheduling - DRR
• Dominant Remaining Resource: “dominant” = “maximum”
• An example: We select as the time point to calculate DRR for machine 1. and , and its DRR is
12
DRR is defined as the maximum remaining resource
along the time dimension up to time 𝑡
![Page 25: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/25.jpg)
Why DRR
• Convert multi-dimensional metrics into scalars
• Better reflect resource utilization − “Maximum”, not “Minimum”
• Better than alternative metric TRC − TRC sums up the relative remaining capacity of each
resource
13
Improvement of DRR over TRC as an alternative metric for executor placement
![Page 26: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/26.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
![Page 27: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/27.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
![Page 28: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/28.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
Search executors in the queue
![Page 29: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/29.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
Search executors in the queue
Calculate DRR for any executor placed on the machine
![Page 30: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/30.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
Search executors in the queue
Calculate DRR for any executor placed on the machine
Choose the one producing minimum DRR to schedule
![Page 31: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/31.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
Search executors in the queue
Calculate DRR for any executor placed on the machine
Choose the one producing minimum DRR to schedule
Update placement results
![Page 32: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/32.jpg)
Elastic Executor Scheduling - MinFrag
• Base on BFD (Best Fit Decreasing)
• Iteratively assigning the “largest” executor to a machine that yields the minimum DRR
14
Heartbeat received
Search executors in the queue
Calculate DRR for any executor placed on the machine
Choose the one producing minimum DRR to schedule
Update placement results
Termination
Repeat the process
![Page 33: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/33.jpg)
Elastic Executor Scheduling - MinFrag
15
(a) Available resources of machine
(b) Resource demands of executor 1
(c) Resource demands of executor 2
![Page 34: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/34.jpg)
Elastic Executor Scheduling - MinFrag
15
(a) Available resources of machine
(b) Resource demands of executor 1
(c) Resource demands of executor 2
𝐷𝑅𝑅(1,𝑗) = max{ 53112
,165448
} =53112
![Page 35: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/35.jpg)
Elastic Executor Scheduling - MinFrag
15
(a) Available resources of machine
(b) Resource demands of executor 1
(c) Resource demands of executor 2
𝐷𝑅𝑅(1,𝑗) = max{ 53112
,165448
} =53112
𝐷𝑅𝑅(2,𝑗) = max{ 1332
,43128
} =1332
![Page 36: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/36.jpg)
Prediction Module
• Recurring workloads −Average resource time series of the latest 3 runs as the
prediction result
• New workloads − Support Vector Regression
16
![Page 37: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/37.jpg)
Dynamic Reprovisioning
• To prevent possible prediction errors and unpredicted issues
• Mechanism −Monitoring stage execution time −Once observing longer than 1.1x expected one −Allocating all remaining resource to the executor for
one monitoring period
17
![Page 38: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/38.jpg)
Implementation
• Spark 2.1.0
• Allocation Module (Cgroups, modified OpenJDK)
• Scheduling Module
• Resource Usage Depository
• Reprovisioning Module
• Prediction Module
• Monitor Surrogate
18
![Page 39: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/39.jpg)
Elasecutor System
19
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Scheduling Module
Reprovisioning Module
Resource Manager
Master
Workers
![Page 40: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/40.jpg)
Elasecutor System
20
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling Module
Reprovisioning Module
Resource Manager
Master
Workers
![Page 41: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/41.jpg)
Elasecutor System
21
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
Demands
Reprovisioning Module
Resource Manager
Master
Workers
![Page 42: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/42.jpg)
Elasecutor System
22
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
DemandsAvailable
Resources
Reprovisioning Module
Resource Manager
Master
Workers
![Page 43: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/43.jpg)
Elasecutor System
23
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
DemandsAvailable
Resources
Reprovisioning Module
Resource ManagerScheduling
Decision
Master
Workers
![Page 44: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/44.jpg)
Elasecutor System
24
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
DemandsAvailable
Resources
Reprovisioning Module
Resource ManagerScheduling
Decision
Launch Adjust
Master
Workers
![Page 45: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/45.jpg)
Elasecutor System
25
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
DemandsAvailable
Resources
Reprovisioning Module
Resource ManagerScheduling
Decision
TriggerLaunch Adjust
Master
Workers
![Page 46: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/46.jpg)
Elasecutor System
26
●●●
Monitor Surrogate
Executor TasksCPU Mem Net Disk
Allocation Module
Prediction Module
Resource Usage
Depository
Repo
rt
Profiles
Scheduling ModulePredicted
DemandsAvailable
Resources
Reprovisioning Module
Resource ManagerScheduling
Decision
Trigger
Reprovision
Launch Adjust
Master
Workers
![Page 47: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/47.jpg)
Testbed Experiments
• Testbed Setup ‒35 dell servers ‒Each server with two CPUs, 64GB RAM, and a quad-port
10GbE NIC ‒A 10GbE Switch
• Methodology ‒120 recurring applications with different workloads,
input data sizes, and resource settings ‒12 new applications ‒Arriving according to a Poisson process
27
![Page 48: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/48.jpg)
Schemes Compared
• Static − Statically allocating CPU and memory for each executor
based on peak demands − Launching a fixed number of executors
• Dynamic − Scaling the number of executors dynamically, − each executor allocated a multiple of <1 core, 2GB
RAM>
• Tetris (SIGCOMM’14)
−Allocating peak demanded resources to executors −BFD-like algorithm for executor placement
28
![Page 49: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/49.jpg)
Evaluation - Makespan
29Makespan measures the total time used to complete all applications
(a) Makespan Reduction (b) Stability of makespan
![Page 50: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/50.jpg)
Evaluation - Makespan
29Makespan measures the total time used to complete all applications
(a) Makespan Reduction (b) Stability of makespan
High makespan reduction and more stable performance guarantee
![Page 51: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/51.jpg)
Evaluation - ACT
30
The CDFs of reduction in application completion time
![Page 52: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/52.jpg)
Evaluation - ACT
30
The CDFs of reduction in application completion time
Significant ACT improvement and more consistent application level performance
![Page 53: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/53.jpg)
Evaluation - Resource Utilization
31
Elasecutor’s average utilization improvement over other policies
![Page 54: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/54.jpg)
Evaluation - Resource Utilization
31
Elasecutor’s average utilization improvement over other policies
![Page 55: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/55.jpg)
Evaluation - Resource Utilization
31
Elasecutor’s average utilization improvement over other policies
![Page 56: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/56.jpg)
Evaluation - Microbenchmark
32
CDFs of reductions in ACT and AET by comparing Elasecutor with and without reprovisioning module
![Page 57: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/57.jpg)
Evaluation - Microbenchmark
32
CDFs of reductions in ACT and AET by comparing Elasecutor with and without reprovisioning module
Reprovisioning is important for prediction based resource schedulers to improve application QoS
![Page 58: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/58.jpg)
Conclusion
• Elasecutor − Elastically allocating resources to avoid overallocation −Placing executors strategically to minimize multi-
resource fragmentation
• Experiment results −Reducing makespan by more than 42% on average −Reducing the median application completion time by
up to 40% − Improving cluster resource utilization by up to 55%
33
![Page 59: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/59.jpg)
Thanks!Q & A
34
![Page 60: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/60.jpg)
Overhead
35
Monitor surrogate ‘s resource consumption
Resource scheduler’s processing delay
![Page 61: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/61.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
36
Statistical analysis of CoVsThe CDFs of coefficient of variations
![Page 62: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/62.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
37
Statistical analysis of CoVsThe CDFs of coefficient of variations
![Page 63: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/63.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
37
Statistical analysis of CoVsThe CDFs of coefficient of variations
![Page 64: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/64.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
38
Statistical analysis of CoVsThe CDFs of coefficient of variations
![Page 65: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/65.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
38
Statistical analysis of CoVsThe CDFs of coefficient of variations
![Page 66: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/66.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
38
Statistical analysis of CoVsThe CDFs of coefficient of variations
5.5
![Page 67: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/67.jpg)
Predictability
• Workloads: Sort, WordCount, Terasort, Bayes, K-means, LR, PageRank, NWeight
38
Statistical analysis of CoVsThe CDFs of coefficient of variations
5.5For most recurring workloads, it is accurate enough to use the profiling results from previous runs with
the same setting to represent the resource demands
![Page 68: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/68.jpg)
Resource Utilization
39
Tetris Elasecutor
![Page 69: Elasecutor: Elastic Executor Scheduling in Data Analytics ... · Terasort Peak/Avg. 1.8 1.7 6.2 1.5 Peak/Trough 60 3.3 237 6.1 K-means Peak/Avg. 1.7 1.2 11.5 5.6 Peak/Trough 75 6](https://reader036.fdocuments.us/reader036/viewer/2022070805/5f0383487e708231d4096dd9/html5/thumbnails/69.jpg)
Resource Utilization
39
Tetris Elasecutor
Utilize resource efficiently and save cost for operators