April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc...
Transcript of April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc...
![Page 1: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/1.jpg)
3
April 2020
1 Harvard University and intern at Google; 2 University of St Andrews and visiting researcher at Google; 3 CMU and visiting researcher at Google
1
![Page 2: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/2.jpg)
Proprietary + Confidential
BorgGoogle's internal cluster manager.
Cell: a set of machines managed by Borg as one unit.
Cell
2
![Page 3: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/3.jpg)
Proprietary + Confidential
BorgUsers submit work in the form of jobs
Cell
Job
each of which contains one or more tasks.
Task Task
3
![Page 4: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/4.jpg)
Proprietary + Confidential
BorgA job may run in an alloc set
Cell
Job
making each of its tasks run in an alloc instance
Alloc set
Allocinstance
4
Task Task
![Page 5: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/5.jpg)
Proprietary + Confidential
Borg
Cell
Job
Jobs have tiers: production, mid, best-effort batch, free.
Alloc set
Allocinstance
5
Task Task
![Page 6: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/6.jpg)
Proprietary + Confidential
Borg
Cell
JobAlloc set
Allocinstance
More info: "Large scale cluster management at Google with Borg" (EuroSys '15)
6
Task Task
![Page 7: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/7.jpg)
Proprietary + Confidential
traces
A single Borg trace describes the workload in a Borg cell:
● {Jobs, tasks}, {alloc sets, alloc instances}○ arrivals and departures: submit, update, finish○ scheduling decisions: place, evict
● Resource allocations and usage
2011 trace: 1 cell from May, 2011
7
![Page 8: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/8.jpg)
Proprietary + Confidential
new
2019 trace: 8 cells for May 2019
● ~96k machines in 3 continents
github.com/google/cluster-data 8
Job
Job
Job
● CPU usage histograms● Job-parent information● Autopilot (see companion paper in session 5)
![Page 9: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/9.jpg)
Proprietary + Confidential
Two metrics:
9
Job
● used
● allocated
![Page 10: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/10.jpg)
Proprietary + Confidential
used2011 2019
New “mid” tier
Time (days)
Fraction of cell capacity
10
![Page 11: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/11.jpg)
Proprietary + Confidential
used2011 2019
Much more “best effort
batch”
Time (days)
Fraction of cell capacity
11
![Page 12: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/12.jpg)
Proprietary + Confidential
used
201112
2019
CPU memory
![Page 13: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/13.jpg)
Proprietary + Confidential
allocated2011 2019
Time (days)
Fraction of cell capacity
13
![Page 14: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/14.jpg)
Proprietary + Confidential
Memory2011 2019
Time (days)
Fraction of cell capacity
14
![Page 15: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/15.jpg)
Proprietary + Confidential
allocated
201115
2019
CPU memory
![Page 16: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/16.jpg)
Proprietary + Confidential
used allocation
16
allocatedused
100%
![Page 17: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/17.jpg)
Proprietary + Confidential
x - utilization
P(utilization > x)
2011
17
![Page 18: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/18.jpg)
Proprietary + Confidential
x - utilization
P(utilization > x)
Median machine in 2011: ~ 30% utilized
Median machine in 2019:
50 - 77% utilized
18
Median utilization is higher in 2019
![Page 19: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/19.jpg)
Proprietary + Confidential
x - tasks submitted per hour
P(tasks submitted > x)
Scheduler load today: ~ 4 times higher
19
4×
![Page 20: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/20.jpg)
Proprietary + Confidential
VERY
C2 = variance / mean2 for CPU-hours and memory-hours
● CPU-hours of UNIX jobs (1996): C2≈50● CPU-hours of supercomputing jobs (2005): C2≈250● CPU-hours of Google Borg jobs (2011): C2≈8400
2019 Google Borg trace: 23k
20
![Page 21: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/21.jpg)
Proprietary + Confidential
Largest 1% of jobs: hogs
Remaining 99%: mice
● Prior work: 50%● Google, 2011: 97.3%● Google, 2019: 99.2%
Fraction of resources consumed by
21
![Page 22: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/22.jpg)
Proprietary + Confidential
x - {CPU, RAM}-hours
Fraction of jobs where: {CPU, RAM}-hours > x
α = 0.77α = 0.69
Extremely heavy tailed
Even more heavy-tailed!
22
![Page 23: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/23.jpg)
Proprietary + Confidential
schedulingSince Google's workload has high C2
23
CellHogs can fill all the resources!
![Page 24: April 2020 · A single Borg trace describes the workload in a Borg cell: {Jobs, tasks}, {alloc sets, alloc instances} arrivals and departures: submit, update, finish scheduling decisions:](https://reader034.fdocuments.us/reader034/viewer/2022051913/6003f82b53e12e04ad224123/html5/thumbnails/24.jpg)
Proprietary + Confidential
● New Borg workload trace:○ 8 cells for month of May 2019○ 2.4TB data accessed via BigQuery○ github.com/google/cluster-data
● Workload and machine utilization have increased
● Disparity between hogs and mice more extreme than any other reported trace○ largest 1% of jobs consume >99% of resources
24