B-Optimal: Resource Allocation Optimization forHigh Workload Applications

Rahul Balakrishnan, Amog KamsettyCS 262 Final Paper

Email: rahulb@berkeley.edu, amogkamsetty@berkeley.edu

Abstract—We propose B-Optimal, a Bayesian Optimizationbased resource allocator for large-scale commercial or personalapplications. Our optimizer automatically determines the optimalresource allocation of CPU Quota, Memory, Disk Bandwidth, andNetwork Bandwidth for different microservices by measuring theimpact of configuration changes on application performance. Wetreat microservices as black-boxes and our end-to-end optimizerrequires very minimal human intervention. While prior resourceallocation systems function well for laptop applications, such asthose running the ELK or MEAN Stack, we design a systemthat supports high traffic web-applications, adding features thatenable horizontal scaling within the optimized application. Wealso provide an analysis on the feasibility, advantages, andshortcomings of Bayesian Optimization relative to other resourceallocation techniques. B-Optimal outperforms prior work withinthe short term producing response latencies that are 25% and66% lower than a gradient descent based optimizer across twodifferent applications, respectively, after ten minutes of runningboth optimizers. Finally, horizontal scaling on B-Optimal pro-duces a 52% improvement from a balanced replica configuration,where every service has an equal number of replicas to fill themachine, and limits latency increase to a factor of 2.5x whenworkload is increased by 4x.

I. INTRODUCTION

A. Motivation

Many companies today like Uber and Netflix deploy large-scale web applications that consist of up to 100s or 1000sof different microservices [1]. These different microserviceseach run in separate containers and communicate with eachother over a virtual network. While this approach allowsfor microservices to be reused across multiple applications,managing the communication among these microservices is avery complex effort. To deal with this complexity, companiesrely on microservice orchestration services like Kubernetesthat take in an application and cluster specification and handlesthe launching of these microservices [2]. Although Kuberneteshandles the management of these microservices, it does notoptimize their performance. With a large number of mi-croservices it is increasingly difficult to analytically determineoptimal values for these variables. As a result, users typicallyoverprovision resources and use heuristic rules to determineplacement, leading to inefficient use of resources [2], [3]. B-Optimal alleviates this issue by using Bayesian Optimizationto automatically find optimal resource allocation of CPUQuota, Disk Bandwidth, Memory, and Network Bandwidth fordifferent microservices. B-Optimal first randomly samples theinput space and then considers neighbors around the best per-

Fig. 1. Diagram of the Apartment application that we evaluate our optimizeron. [4]

forming points, choosing the point with the highest expectedimprovement in application latency according to a Gaussianregression of the points evaluated so far. This automatedend-to-end approach allows microservices to optimally utilizeresources and makes the lives of system operators much easier.

B. Evaluation Metrics

We use two main applications and sample workloads forthese applications to evaluate our optimizer. The goal of ouroptimizer is to minimize the latency, given a fixed or variableworkload, for the following two applications in a timelymanner.

1) MEAN Stack: The first application that we evaluateagainst is a sample ToDo application leveraging the MEANStack. The MEAN stack is a commonly used software bun-dle consisting of three services with distinct functionality:a NodeJS web server, a MongoDB cluster as the databaseservice, and a HAProxy load balancer that routes traffic todifferent node application containers. This service stack aimsto imitate the low volume web development applications thatan individual may use to run his or her blog, personal website,or in our case, a ToDo list. The workload that we used for thisapplication consists of requests to the frontend that traverse allservices in the application.

2) Apartment Application: The second application moreclosely resembles corporate use, serving as an apartmentfinding tool, or a database where multiple users can queryapartments that fit their specifications. The application exposes4 endpoints, two relying on the base web page and sign-inportal, which keeps a database entry on MySQL for everyuser account. The third endpoint uses the PostgreSQL databaseto keep track of the apartments and their specifications. Theother features a LogStash pipeline that connects the Postgres

Fig. 2. Baseline with equally provisioned configuration vs. Greedy transferapproach. Lowest latency seen so far is in ms and Time since start is inseconds.

database to an Elasticsearch cluster, which performs a compu-tationally expensive search on the user’s query and pipes thedata to running Kibana instances. We use this app for scalingand high workload benchmarks within our paper. Figure 1shows a diagram of the apartment application and all of itsvarious microservices. The workload that we used for this appconsists of a mix of reads and writes.

II. PRIOR WORK

We consider various baselines and prior work that wecompare B-Optimal against.

A. Static provisioning

The most naive baseline to compare against is a staticprovisioning of resources to each service. As mentioned pre-viously this basic method is what is most commonly used byoperators today. They simply provision a service pipeline suchthat each service shares a machine with a a minimal numberof other services, and all services on each machine take upan equal portion of all resource type [4]. For the rest of thepaper, we will refer to this configuration as an “equally provi-sioned” configuration. Any further transferring of resourcesis done in an ad-hoc approach where operators manuallyinvestigate resource utilization and use that information toinform any allocation transfer decisions. This strategy resultsin the necessity for a large machine deployment, involvingadditional nodes being added to the cluster, which in turn,incurs additional costs. We simulate this approach by placingthree services per machine and providing approximately thirtypercent of the machine’s resources to each service (Fig 2.)

B. Greedy Transfer

In prior work, Yang et al. [5] discuss a heuristic basedresource transfer approach, stemming from a bulk transfer ofresources from one service to another based on service need.We generalize their scheme to a greedy algorithm of resourcetransfers between services. This greedy approach comprisesof large transfers of resources from less utilized services to

more utilized services, where utilization is measured during theadministration of workload on a service pipeline’s endpoints.Put simply, the algorithm starts at a configuration with eachservice having an equal provision of resources. The algorithmthen runs such that for each resource, the least utilized servicedonates half of its resource allocation to the most utilizedservice, the second least utilized service donates half of itsresources to the second most utilized service, and so on. Oneiteration consists of all service pairs performing one transfer.Iterations are run until performance no longer improves, atwhich point the algorithm is terminated and the best configu-ration is returned. We poll the performance over time of thisgreedy algorithm, displaying the results in Figure 2.

The approach of utilization-based resource transfers indi-cates significant performance improvements over the basebaseline. However, allocating resources to services with higherutilization bears several dangers, specifically, those concerningservice bottlenecks. For example, one service that continu-ally receives requests but is blocked in piping the resultsto another service may experience more utilization than thesecond service. To improve performance, however, one mustallocate more resources to the service with lower utilization,to remove the cause of the blocking. As shown in [4],merely looking at utilization metrics to decide which serviceto allocate more resources to does not necessarily result inbetter application performance. Utilization does not capture thecomplex dependencies between the different services, and thusa better approach would be to make allocation decisions bymeasuring overall application performance as a metric insteadof just utilization statistics.

C. Gradient Descent

Concerning more recent prior works in the field of resourceallocation optimization, we consider Throttlebot, built byChang et. al in 2017 [4]. Throttlebot provides a more restrictiveheuristic that defines the transfer it makes between resourcesmotivated by the principles of gradient descent. It estimatesthe impact certain resources has on the overall applicationperformance by iteratively “throttling” each resource. It thenuses those gradient estimates to guide resource transfer deci-sions for the different services. After many iterations of thisgradient descent algorithm, a steady-state will be reached thathas the optimal resource allocation for all the services. A moredetailed description of one iteration of Throttlebot is describedbelow:

1) For each service s, and for each resource type r, calcu-late sensitivity of application performance to changes ofr for s. This estimates a gradient for each (s, r) tuple.

2) Sort the (s, r) tuples by sensitivity in descending order.3) For each pair of tuples: (si, rk) and (sm, rn), if si and

sm are located on the same machine and rk and rn arethe same resource, transfer δ resources from the lowersensitivity tuple to the higher sensitivity one.

4) Ensure that the same (s, r) tuple does not receive newresources from more than one service in the same

iteration (this prevents resource starvation in minimallysensitive tuples).

5) Reduce step size δ by factor of 1.2In the above steps, sensitivity of each service-resource

tuple refers to the ratio of application performance decreaseto resource decrease, the latter of which is a fixed stressamount determined within the pre-run configuration of theThrottlebot. The sensitivity calculations consists of the mosttime-consuming portions of Throttlebot. The large durationof these calculations is due to the necessity to run S × Titerations where S represents the number of services within thepipeline and T represents the number of resource types thatthe optimizer modifies. Per each of these S × T iterations,a performance benchmark must be run to determine howchanging one resource type, belonging to one service, affectsthat application performance. For an application such as theMEAN Stack, this results in 12 sensitivity calculations, as theapplication contains 3 different services and optimizes over 4different resource types. If each workload run takes an averageof 10 seconds, the runtime of sensitivity calculations for justa single iteration is 120 seconds.

The δ reduction by 1.2 per iteration accounts for a nor-mal practice within gradient descent, which underplays latertransfers such that instead of jumping between various localminima, the algorithm converges smoothly into one minimumand is able to terminate after a certain amount of iterations.The termination condition comprises when an iteration nolonger produces a significant performance gain when there areno feasible transfer conditions.

Overall, Throttlebot performs an iterative gradient descentafter building a sensitivity prior using performance tests foreach service-resource tuple. While Throttlebot eventually con-verges to an optimal resource configuration, the time it takes toreach this optimal point is the main drawback of this approach.

D. Bayesian Optimization

The primary prior work regarding Bayesian Optimizationfor resource configuration is CherryPick by Alipourfard etal. in 2017 [6]. Unlike Throttlebot, CherryPick is used toimprove the performance of an application consisting of asingle service, and thus is determining optimal resource al-locations for that single microservice only, leading to a low-dimensional search space. Our work in B-Optimal is inspiredby the Bayesian Optimizer used in Cherrypick, but makessome key improvements in the algorithm to deal with the high-dimensional search space that Throttlebot optimizes over. Inthe following section we describe how the Bayesian Optimizerworks, and the improvements that B-Optimal makes overCherrypick.

III. DESIGN OF B-OPTIMAL

Our primary motivation in developing our own resourceallocation optimizer is to provide a speedup on the exhaustivesearch that Throttlebot performs within its gradient descentalgorithm, developing B-Optimal to achieve lower short termand steady state latency than its gradient descent counterpart.

As mentioned before, S × T iterations are performed periteration of gradient descent, producing a situation where amajority of the runtime is spent towards calculating a prior,or a distribution from we can infer future evaluated points,for the gradient descent to perform. Thus, we use BayesianOptimization to generate an accumulative prior such thatall inputs combinations need not be exhaustively run beforeoptimization occurs.

A. Bayesian Optimization

Bayesian Optimization is a type of iterative optimizationand like gradient descent, is useful for when knowledge of theinternals of the objective function is limited or unknown [7].However, unlike gradient estimation, which is the techniqueemployed by Throttlebot, Bayesian Optimization does notrequire a large number of samples as it imposes a prior on theobjective function. Bayesian Optimization starts off with aninitial prior of the true objective function f and updates thisprior through samples of f(x) to incrementally get a betterapproximation of f . Thus, there are two primary componentsof Bayesian optimization: a surrogate model f ′ which servesas a model of the true object function f , and an acquisitionfunction that is computed from f ′ and is used to determinewhich point to evaluate next.

1) Surrogate Model: One of the most popular choices fora surrogate model is a Gaussian Process, an example of anon-parametric model. Cherrypick uses Gaussian Processes asa prior, and we use the same for B-Optimal. By imposinga Gaussian prior, this makes the assumption that the finalfunction is a sample from the Gaussian Process. Thus theobjective function is determined by a mean function u(x)and a covariance kernel function k(x1, x2). The optimizationfirst starts by defining a prior over the functions by initiallyrandomly sampling points. Then on each iteration of thealgorithm, more points are observed and used to update theGaussian Process. Over time, the Gaussian Process becomes abetter approximation of the objective function, and in conjunc-tion with the acquisition function, can start proposing moreoptimal sampling points.

The authors of Cherrypick decide to go with a GaussianProcess because its inherent expression of uncertainty allowsBayesian Optimization to perform more quickly and organi-cally than other optimization forms, where uncertainty mustbe artificially construed [6]. Linear and logistic regression,for example, take on a backward-oriented approach, wherea function is fit according to minimizing a loss from thepoints calculated so far. Preventing overfitting is largely atask of art and intuition, where human judgment must comeinto play to determine how representative a sample is offuture points that the model may encounter. This post-factomeans of representing uncertainty also creates issues withindeep learning or reinforcement learning models, which areoften only able to express uncertainty through pre-determinedregression patterns, producing issues that largely line up withthose of regression techniques.

2) Acquisition Function: The acquisition function uses thesurrogate model to propose future sampling points. In our case,these sampling points will be better resource configurationsover time. Given an acquisition function u(x), the point tobe sampled at time t is xt = argmax(u(xt|D1:t−1)) whereD1:t−1 = {(x1, y1)...(xt−1, yt−1}) are previously drawn sam-ples from f so far [7]. The acquisition function that Cherrypickand B-optimal use is Expected Improvement [8]:

EI(x) = E(max(f(x)− f(x′), 0)

where f(x’) is the value of the best sample so far. Usingthe Gaussian Process as f(x), the analytical expression ofExpected Improvement is

EI(x) = (m− µ(x))Φ(Z) + σ(x)φ(Z)

where Z = m−µ(x)σ(x) and Φ and φ are the normal cumulative

distribution function the standard normal probability densityfunction respectively. Notice how expected improvement cal-culations are directly tied to the normal probability distributionfunction, suggesting that these calculations are affected cumu-latively by uncertainty recalculations when new points are run.All in all, this contributes to the idea of a progressive priorfunction, one in which all new points calculated represent theprior so far.

B. Improvements over Prior Work

Here we describe two improvements made to the traditionalBayesian Optimization algorithm employed by Cherrypick.Mainly, we increase the random exploration at the beginning,and we increase the number of points that we sample on eachiteration.

1) Initial Random Exploration: Recall from the previoussection that the initial Gaussian prior needs to be initializedby a starting point. CherryPick started from a balanced startingstate, such as an equally provisioned configuration. While thisprovides a constant starting state for all applications, producingconsistent results for multiple runs of the optimizer, such astarting state produces a uni-modal Gaussian Process, wherethe first point is perceived as the mean upon which the NormalProbability Distribution is built. Such a low dimensionalprocess leads to a consistent but myopic outcome, wherepoints often fixate in one local minimum. Compounded withan acquisition function that finds neighboring points in closeproximity to previously calculated samples, the chosen sam-ples rarely stray from a small fixed range. Such a uni-modalGaussian Process often features high uncertainty estimates, asthe process attempts to fit the entire range space based onsamples that are located very close to one another.

We aim to improve this approach by a procedure of randomsearch. We add a feature where the algorithm evaluates severalquasi-random points before beginning the Gaussian-regressionbased optimization. We use the Sobol sequence as the basisof our random generation; the algorithm uses binary bitmanipulation and a hole-finding heuristic to cover all thecombinations over the entire binary space repeats. The purposeof performing random search before the regression phase

is to produce samples in different portions of the Gaussianspace, such that even if samples are not necessarily the bestperforming inputs, they offer future samples that may convergeto various local minima. More specifically, these additionalsamples reduce the variance of each Gaussian curve producedin the range space when compared to a singular encapsulatingGaussian curve, producing a Gaussian Process dimensionalitythat more closely matches the true dimensionality of a multi-service interdependent environment, whilst also increasingaccuracy of expected improvement estimates.

To reinforce this point, the random search period is likelymore integral for the optimization of complex environmentsthan for the optimization of simple environments due tothe fact that such environments bear high dimensionalityperformance functions that require assessment at various localminima. As such, increasing the number of points sampledduring the random search procedure will generally improvethe performance at which B-Optimal converges, perhaps moreso for complex environments than for simple environments.We perform experiments regarding the number of samples runfor random search preceding the optimization and discuss theimplications for both the MEAN App and the Apartment Appin future sections.

2) Acquisition Function: CherryPick [6] uses an acquisitionfunction that compares points uniformly sampled from a fixedrange from either side of the mean. A narrow neighbor findingalgorithm in combination with a fixed starting configurationwith no pre-sampling produces a very small range of evaluatedpoints, as well as a high uncertainty estimate of expectedimprovement. We instead develop a system that randomlysamples from a range that is a percentage of the point aroundwhich we are finding neighbors. This is motivated by twoprimary principles. The first involves domain. Given the noisepresent within system performance calculations on large-scaleclusters, we desire to find distinct input samples that differfrom previously evaluated samples by more than a hundredthof a CPU Quota Percentage. As such, we base the rangein which we can find new points upon the given inputs,allowing us to achieve notable changes between consecutiveevaluation inputs. The second principle largely involves thelimited coverage of the Gaussian prior. Even with adding arandom search period before the optimization, we still desire aperformant optimization application. As such, we only extendthe random search iteration count to, at most, a multiple often. Uncertainty estimates for such a low number of sampleswill be high, since the number of samples is often the samelevel of magnitude as the count of the optimizing variables,and in turn, the dimensionality of the regression. As such,our choice of randomly sampling from a given range for theacquisition function prevents an over reliance on the highuncertainty gaussian curve produced within the regressionphase; this contrasts with CherryPick’s uniformly sampledneighbor points which adhere strongly to the Gaussian curve.

On the spectrum between Bayesian Optimization and Ran-dom Search, our choice of randomly sampling over a largeinterval shifts the B-Optimal slightly towards the latter. By

adding this feature, we are in part extending the preliminaryrandom search process such that it occurs in a smaller formbetween every iteration of optimization.

While we can also combat Gaussian uncertainty by increas-ing the count of neighbor points to assess before choosing thenext evaluation point, we choose to maintain a neighbor countsimilar to that of Cherrypick, which uses around 10 points, tolimit the runtime overhead to the optimizer.

We benchmark various range intervals within the resultssection of our paper.

C. B-Optimal Algorithm

Putting all of the above information together, the algorithmfor B-Optimal is as follows:

1) Randomly sample m starting resource configurations.These points initialize the starting Gaussian prior, GP.

2) For t = 1, 2, ... repeata) For the top n best points seen so far,{x1...xn}, find the next sampling pointxt = argmaxx(u(x|D1:t−1)) where u is theacquisition function.

b) Obtain a possibly noisy sample yt = f(xt) + εtc) Add the sample to previous samples D1:t ={Dt:t−1, (xt, yt)} and update the GP.

D. Comparison to Gradient Descent

As mentioned in a previous section, the main bottleneck toThrottlebot is the large number of performance evaluations(O(S × T )) that need to run before the gradient descenttransfers occur. Every Resource-Service sensitivity value is noteven guaranteed to be used, since certain transfers don’t occurdue to small sensitivity differences or due to the pruning stagewhich removes repeated receiver tuples. In future versions ofThrottlebot, we are working with the NetSys lab to subsetthe combinations to evaluate both randomly and heuristically;however, the current working version of Throttlebot evaluatesall Resource-Service Tuples exhaustively, and this search isvery time-consuming.

Our goal for the design of B-Optimal was to limit theprior calculation overhead while simultaneously achieving ahigh coverage search of the input space. To fulfill the formerpoint, we elected to proceed with Gaussian Regression, whichfeatures an adjustable prior dependent on new curve centersand iteratively changing variance values. To fulfill our goalsof a high coverage search space, we designed a period ofrandom exploration of the input space as well as an increasedand randomized search for neighbors of the best performingpoints so far.

While our coverage is less than the coverage of an pureiterative gradient descent algorithm, which can continuallytransfer a resource in a certain direction without limitationsfrom Gaussian restraints and neighbor finding, our algorithmresembles gradient descent in that the beginning of the al-gorithm comprises of big transfers, through blind randomsampling of the input space, and then narrows down to smallertransfers, through a neighbor finding algorithm that limits

evaluation to a closed range. However, B-Optimal exceedsThrottlebot in its ability to limit overhead per iteration. Witha prior that is calculated only once per iteration and the use ofregression instead of full application performance evaluationto gauge the expected improvement of neighboring points, B-Optimal is able to optimize the application for more iterationsthan Throttlebot does. With the Mean application for example,B-Optimal achieves about 50 iterations in an hour, comparedto 7 iterations by Throttlebot.

We display and discuss the differences in B-Optimal’sruntime and performance within the results section of thepaper.

IV. B-OPTIMAL AND HORIZONTAL SCALING

Both the CherryPick [6] and Throttlebot [4] papers discussapplications designed for use by individuals or small com-panies who expect low amounts of traffic within their web-sites. Throttlebot sends out 350 requests across 50 threads tobenchmark its MEAN Stack application. While this provides asignificant workload for the type of application that Throttlebotaims to benchmark, we look to optimize for workloads ona level of magnitude higher, more in line with the reported40,000 concurrent search requests received by Google WebServers.

To increase the workload-incurring abilities of a base ap-plication, we look to horizontal scaling. The unpublishedAutoTune Paper by UC Berkeley’s NetSys Lab [9] discussesthe idea of horizontal scaling as a means of combating highlyvariable workloads, specifically ones that momentarily scaleby a factor of ten. We helped develop this architecture withNetSys and made various observations concerning the choiceof deployment tool. The paper uses a Kubernetes cluster tohost its running services, using Kubernetes inbuilt scalingfunctionality to increase a replica count for a service ifthat service exceeded a certain utilization threshold. Whenworkload increased, the Kubernetes deployment more or lessmaintained performance, if the utilization threshold for scalingwas set sufficiently low, resulting in a larger amount of runningservice replicas to adjust for the higher workload.

We align with the AutoTune paper in our goals of adaptinglow workload applications to higher workloads, using hori-zontal scaling to limit performance drops through workloadincreases. However, AutoTune’s scaling tests also demonstratethe dangers of unchecked horizontal scaling. When settingthe scaling utilization threshold to 50%, such that replicaswould be added if any service exceeded 50 percent utilization,within the Kubernetes scaling logic, pod count jumped to 30replicas across all running services within the Mean stack.The authors discuss using a machine autoscaler, that scaledup machine count, to keep up with the demands of a quicklyreplicating application, increasing machine count from two toten m4.large nodes by the time scaling had reached a steadystate. As mentioned with respect to our baseline benchmarks,utilization often paints a warped photo about the applicationin question and can often produce excessive scaling when it is

not necessary, as applications that are scaled may still be bot-tlenecked by the next service within the pipeline. Within ourhorizontal scaling application, we desire to pair performanceoriented resource allocation principles with horizontal scaling,such that replicas are only added to the application when theyimprove performance.

In addition, we align with more recent trends in cost man-agement within AWS and Kubernetes, where a cost maximumis allowed for a running application, based on the total coststhat all machines within that application require. Instead ofproviding a strict cost maximum for pod scaling, we insteadadd a system of a pod penalty, or an additive amount appliedto application performance per pod, such that the resultingreplica configuration balances performance for replica countwith an adjustable parameter.

The ease of use of B-Optimal renders the addition ofoptimization parameters a simple task. We simply add vari-ables representing the replica count of each service to bescaled. Notably, we do not scale the load balancer, as theload balancer features throughout in the order of Gb/s whilerequests are normally less than 1 Kb. With the number ofconcurrent requests we are sending, usually a multiple of 103

multiplied by the number of endpoints, we do not exceedthe throughput limit of the HAProxy balancer used withinthe tested applications. B-Optimal proceeds to optimize overreplica counts in addition to resource configurations and apenalty for pod count is applied at the end to the evaluatedperformance.

We provide benchmarks for our scaled version of B-Optimaloptimization within the results section of our paper.

V. IMPLEMENTATION

A. Application Runner

The application resides on a Quilt cluster, a third partytool that serves as a Kubernetes-like deployment environmentwhere each container runs a Docker image. We chose Quiltover Kubernetes as the application is optimized for lowoverhead management of smaller applications, such as theapplications we were testing within this study. The applicationrunner first performs an SSH action into the machines onwhich containers reside, setting each container provision tomatch with the provided resource configuration.

We elected to use four resource types for our optimization:CPU-Quota, Memory Allocation, Disk bandwidth, and Net-work Bandwidth. CPU-Quota signified the percentage of CPUcycles that were allocated to a service. The first three resourcetypes were set using the linux cgroups function, which appliedchanges to the entire container running a service. The Networkresource type was set using linux’s “tc” application.

The workloads described in the previous section wereadministered using Apache’s ab benchmarking tool, whichdoubled as a performance measuring tool for the application.A certain amount of requests were sent out to all exposedendpoints of an application, with a configurable concurrencymeasure. We used 500 requests per endpoint for apartment-app and 350 for Mean Stack, using request numbers from

Fig. 3. In this figure, we offer results regarding all baseline data points onthe Apartment App. Base Baseline represents an equally partitioned resourceconfiguration, where machine resources are split equally amongst all serviceshosted on a particular machine. This allocation is run for the entire durationof the graph and thus represents static provisioning. BayOpt Baseline refers toour recreation of the CherryPick Bayesian Optimization configurations, whichinvolves no random search and a neighbor-finding algorithm consisting ofuniform sampling of the interval surrounding the best point so far. Finally,Transfer baseline refers to the greedy algorithm described in Section 2, whichconsists of large transfers between the high-low utilization pairs amongstservices.

default Throttlebot configurations. The performance resultswere written to output files within the machines that hostedeach container. Latency information was collected from thesefiles and added up amongst all endpoints. The mapping ofresource configuration to performance result is returned to thedatabase, concluding the iteration.

For performance measures, we simply run a process thatpolls a file that contains the best results so far every 30seconds.

B. Scaling Implementation

As mentioned in the previous section, we add variableswithin our configuration to represent replica scaling for ser-vices. We limit replica scaling to 10 pods, such that thefinalized replica count more likely fits into the starting ma-chine configuration. Per iteration, we redeploy our containerdeployment tool with the necessary amount of replicas and runperformance tests with the provided resource configuration.

VI. RESULTS AND EVALUATION

A. Baseline and Optimizer Results

In this section, we provide comparisons between baselinesand the optimizer results. We also compare optimizer tech-niques, displaying performance over time graphs of GradientDescent and Bayesian Optimization techniques.

Generally, the baselines in Figure 3 align with our motiva-tions to shift from a utilization based model to a performancebased model for resource allocation; shifting heavily acrosslines of utilization, as was performed within the Transfer Base-line test, demonstrates only two improvements in performancedespite five transfers having occurred within the duration ofthe graph. On the other hand, even the BayOpt baseline,

Fig. 4. Throttlebot’s gradient descent performance is compared to the GreedyTransfer Approach to resource optimization for the Apartment App. HereThrottlebot begins at an equally provisioned starting configuration.

our recreation of the CherryPick optimizer, which consistsof performance based heuristics for allocation shifts, showspotential for improvement, as no optimizations are performedafter steady state is reached at approximately the 15% markof the time axis of the graph. While the baseline chart doesnot sufficiently isolate variables to explain BayOpt Baseline’sceased optimization, the BayOpt baseline shows signs of aconservative acquisition function, where the range of pointsevaluated is very low, differing by one or two percentage pointsfrom the best point evaluated so far.

Within Figure 4, we compare Throttlebot to the TransferBenchmark, as both are predicated on resource transfersbetween services. In our benchmark, Throttlebot performs 5transfers using its gradient descent algorithm, 4 of which bringimprovements to the lowest latency witnessed so far. Throughbenchmarked performance gains, Throttlebot demonstrates itsdecaying transfer quota; while the first drop is the most sig-nificant, the subsequent drops demonstrates smaller resourceallocation shifts that produce smaller, albeit still significant,performance gains. Throttlebot’s performance shifts largelyoccur in the same direction, as sensitivity rankings are mostlypreserved even as the range of these rankings grow closer.This matches up with the Transfer baseline, where transferdirections are fixed into static allocation calculations at thebeginning of the experiment. Despite fixed transfer directionsin both algorithms, the performance-heuristic of Throttlebotshows more consistent performance improvement during theduration of the graph, as well as a steady state point 45%lower than that of the greedy algorithm.

Within Figure 5, the Bayesian Optimization Baseline keepsup with B-Optimal for the first half of the experiment, indi-cating strong performance of input points located around theinitial equal provisioning. At 500 seconds, B-Optimal is stillwithin its random search and finds an input configuration thatimproves performance to the level of the Baseline, which isundergoing optimization at that point. At that point, however,the two algorithms are in different portions of the input space,with B-Optimal concerning high CPU-Quota values, high

Fig. 5. B-Optimal is compared to the base optimization tool used by Cherryp-ick on the Apartment App. The BayOpt baseline (Cherrypick) lacks randomsearch and large-interval neighbor finding. In the Baseline experiments, B-Optimal is set to first run an equally provisioned starting configuration beforestarting its seed-based random search such that all baseline benchmarks startat the same configuration.

Fig. 6. We benchmark B-Optimal’s behavior relative to Throttlebot for boththe Apartment App and the MEAN App. Both start at equally provisionedconfigurations and B-Optimal continues into its seed-based random explo-ration after this initial configuration.Apt-App: Both applications reach a point of relative steady state at around2500 seconds and we crop the graph accordingly to better demonstrate theperformance shifts within the two optimizers.Mean: Both optimizers reach steady state at about 3200 seconds.

memory, and low disk, while the Baseline considers middlingCPU-Quota values, high memory, and high disk values for allits services. After about 2000 seconds, B-Optimal begins opti-mization, entering the space of high CPU-Quota, high memory,middling disk, and low network, an input space based onthe previous best run within the random search period. Here,B-Optimal outperforms the Bayesian Optimization Baseline,settling at a steady state value approximately 35% lower thanthe Baseline.

Within Figure 6, equally provisioned configurations pro-vides a relatively slow start for both optimizers, and both startat latencies at approximately twice of their steady state. For theapartment app, during the period of random exploration before500 seconds, B-Optimal enters an input space of high CPUQuota and high memory just by random sampling, reachinga latency value of approximately 30,000 more quickly thanThrottlebot, which must perform consecutive transfers to get tothat portion of the input space (Fig 7). In fact, Throttlebot doesnot reach that latency mark until 2500 seconds. Eventually,both algorithms reach a similar steady state around 2600

Fig. 7. B-Optimal: B-Optimal resource allocation percentages at 1400seconds, captured during the optimization phase; Values already lie close to thesteady state configuration, with high CPU Quota within frontend applicationsand high Memory for database applications.Throttlebot: Throttlebot resource allocation at 1400 seconds after 2 transferphases; Notice that values still lie close the equally provisioned configuration,where most service-resource tuples begin at 33%.

seconds. In considering the steady state resource allocationsfor both optimizers (Fig 8), we see that both algorithmsreach a similar configuration with network allocations beingthe biggest deviation between the two optimizers. Throttleboteventually matches the high database memory allocations andhigh frontend CPU quotas that B-Optimal discovers withinthe random search period, indicating that perhaps those fea-tures most strongly correlate with performance within theapartment-app application. We discuss this notion within thediscussion section.

Within the MEAN application, Node CPU Quota allocationsdominate performance within the 3-service application. WhileB-Optimal immediately discovers this within the randomsearch period, Throttlebot performs several transfers to getto that portion of the input space, eventually converging tosimilar values.

B. Bayesian Optimization Tuning

In this section, we focus on assessing B-Optimal’s perfor-mance under different parameter settings. Specifically, we varythe duration of the period of random search that precedesoptimization, comparing time to convergence and convergencevalues of each duration setting. We additionally consider howB-Optimal’s performance over time is affected by the rangeof neighbor finding intervals.

The first parameter that we vary is the number of randomiterations that we conduct at the very beginning before startingthe bayesian optimization phase. These random points areused to estimate the Gaussian process. The results for theseexperiments on the MEAN and Apartment applications are

Fig. 8. Steady state configurations for B-Optimal and Throttlebot. Configura-tions are relatively similar, with the highest variance occurring within networkallocations.

Fig. 9. Evaluation of B-Optimal on the MEAN application with varyingrandom exploration iterations/samples.

shown in Figures 9 and 10 respectively. In varying the randomexploration iteration count, we initially expected that increas-ing random iteration count for higher dimensional applica-tions, such as the Apartment application, would bring largerproportional improvements than doing the same for lowerdimensional applications due to the increased importance ofcovering large input spaces. While the range of steady statevalues seems to align with this claim, as the Apartment appsports a range of approximately 18% of the highest valueacross all steady values compared to Mean-App’s 8%, theranking of iteration count at steady state is unexpected. Forthe Apartment app, at the point of 800 seconds, “2 iteration”and “10 iteration” have begun optimizing while “15 iteration”is still in the phase of random exploration. “2 iteration”outperforms “15 iteration” by this point, with “10 iteration”

Fig. 10. Evaluation of B-Optimal on the Apartment application with varyingrandom exploration iterations/samples. We crop the vertical axis of apartment-app to better display convergence differences between different randomiteration counts.

settling at the lowest steady state latency value. The graphdemonstrates the possibility of the occurrence that a longexploration period produces worse performance values in theshort term, even with a high dimension application with a largeinput space. The possibility of high performance with lowrandom samples pushes the point that choosing a middlingrandom sample count and running the count over multipleseeds may be an approach that produces the best steady statewithin a short-term span. For long term experiments, a higherrandom iteration count is still recommended, since as long asthe optimizer reaches the optimization phase and concludesthe random exploration phase, the optimizer will perform aswell, if not better, than a run with fewer random iterations.

The second parameter that we vary is the range of neighborsthat we consider during the bayesian optimization phase.The results for this experiment are shown in Figure 11. Thelargest interval from which neighbors can be found produces asteady state value that is 30% better than the smallest interval.This largely manifests as the “1x interval” demonstrating thelargest improvements once the optimization phase starts at400 seconds. Neighbors can be found further from the bestperforming point, producing points with potentially higherexpected and actual improvement from the perceived meanof the Gaussian distribution. The Gaussian Process regressionoften inaccurately assumes an isolated best performing point tobe at a local minima or mean value of a Gaussian distribution,which prevents small-interval neighbor-finding heuristics toescape the previously evaluated point and find the actual localminimum around that portion of the input space.

C. Bootstrapping

In this section, we evaluate a hybrid optimizer, that boot-straps gradient descent with the best configuration from B-Optimal’s random search. We compare this performance to afull run of just B-Optimal. The results are show in Figure 12.

The primary runtime shortcoming of Throttlebot is its neces-sity to start from an equally provisioned resource configuration

Fig. 11. We vary the interval in which neighbors are sampled around best-performing points on the Apartment application. Here the interval representsthe percentage of the input that we can deviate from a input to sample potentialfuture points. Each multiplier is in terms of a percent, where 1x represents 1%of deviation above and below the best performing inputs and .01x representsa .01% percent deviation.

Fig. 12. A full B-Optimal run is shown, overlaid by a bootstrapped gradientdescent run, which starts at the beginning of the optimization phase/end ofrandom exploration phase. The best configuration found from the randomexploration phase is used as the starting configuration for Throttlebot.

and approach the optimal configuration using small, iterativesteps. Here, we do away with this short coming by placingThrottlebot in the portion of the input space in which B-Optimal enters its optimization phase. Then, gradient descentoutperforms the random neighbor finding acquisition functionin its convergence time due to the heuristic, uni-directionalnature of resource transfers to more sensitive resources. Thebootstrapped gradient search consists of the fastest conver-gence out of all the configurations tried in this paper and it isworth exploring this system under additional workloads andseeds in future work.

D. Horizontal Scaling

Finally, in this section, we compare convergence valuesof our horizontal scaling implementation, and compare per-

Fig. 13. We display non-optimized static vs scaled performance underdifferent workload settings. Static refers to fixed and singular replica counts,where only one replica per service is deployed. Scaled refers to servicedeployments that are replicated. “Non-Opt” refers to unoptimized replica andresource configurations, in which equally provisioned configurations are used,while “Opt” refers to those produced by B-Optimal. The default workloadadministered is 250 requests, and all increased workloads are relative tothat baseline. Notably, the Non-opt static deployment did not produce ameasurement for 20x workload, as the ab benchmarking tool consistentlytimed out due to missed responses.

formance values to non-scaling versions, as well as non-optimized scaled versions.

Figure 13 shows the results of the optimizer with variousconfigurations. Static refers to running the workload usinga singular replica per service, with the exception of Node,and without increasing the number of replicas. The replicacount for static is 3 for the Node service and 1 for everythingelse. In the scaled configuration, we allow more pods/replicasto be created. Opt uses configuration information from B-Optimal, while Non-Opt refers to unoptimized replica andresource allocations, in which we used equally provisionedconfigurations.

Scaled and static performances are similar at the 1x work-load, as overhead to distribute the workload amongst variousreplicas cancels out any concurrency increases from the replicacount. When workload scales up to 5x and 20x, the replicateddeployments outperform the static deployments as expected.

Under higher workloads, static latency grows significantlyworse, scaling linearly with the workload increase from 5xto 20x even with optimization. In fact, the non-optimizedstatic version under 20x workload times out, due to missingresponses to requests.

Within Figure 13, B-Optimal improves the 5x Workloadlatency of the scaled deployment by 52% from the 5x Work-load Non-Optimized setting. In addition, an optimized scaledconfiguration under a 20x workload produces a latency thatonly exceeds that of an optimized scaled 5x configuration bya factor of 2.5.

Overall, the scaled optimizer shows promise in curbingperformance drops with increased workloads, as well asimproving performance over an equally provisioned replicaconfiguration.

Fig. 14. Replica count for optimized configurations under 5x and 20xWorkloads

The optimized experiments are run where a fixed penaltyis applied per pod deployed, so the replica configuration ofthese data points reflects this cost function. The total replicacounts for 5x workload and 20x workload were 24 and 32,respectively, and the total replica capacity for the cluster was40, indicating that the cluster was not blindly overprovisionedwith replicas and that the final configuration adhered to thereplica penalty.

VII. DISCUSSION

The baseline performance suggests the intrinsic benefitsof using Bayesian Optimization as a resource allocation op-timizer. Our mock of CherryPick’s optimizer, which servesas the Bayesian baseline, outperforms the static provisioningconfiguration as well as a greedy transfer approach. Howeverthe Bayesian Optimization baseline falls short in the designprinciples we aimed to enact within B-Optimal. Specifically,the optimizer does not sufficiently prune the input space with aperiod of random sampling. Instead, it immediately narrows itsefforts close to an equally provisioned input space. Points nearthis space are sampled during optimization, but are done so ina manner that conforms strictly to an under-defined Bayesianregression and within a range that limits discovery of inputpoints that deviate significantly from the mean. This producespoor long-term performance, where within our experimentsthe baseline Bayesian Optimization ceased optimizing after450 seconds and produced worse performance than Throttlebotfrom approximately 2500 seconds onwards.

In designing B-Optimal, we looked to experiments varyingthe duration of the random sampling period and experimentsvarying neighbor-finding intervals, or the range consideredby the acquisition function. We found that short term steadystate was seed oriented, based on how quickly optimal randomsamples were given to the optimizer during the random explo-ration, and how quickly it subsequently began the optimizationphase. While a middling range of 10 iterations produced thebest results in the short term, we increased the iteration countto 15 for our longer experiments, given that we knew thatthe optimizer would perform as well as 10 iterations if notbetter within the long term. For neighbor intervals, we saw thatlarger neighbor intervals produced better steady states evenwithin the short term, since input points were discovered thatwere closer to local maxima than points that were alreadyevaluated. We tuned B-Optimal according to these findings.

Comparing a tuned B-Optimal to Throttlebot, we saw thatB-Optimal outperformed the latter by 25% and 66% in theApartment Application and MEAN stack, respectively, after1000 seconds and converged to a steady state of 3-10% betterthan Throttlebot after 45 minutes.

Our Bayesian Optimization implementation and tuningmaintained better short term performance than Throttlebot,whilst not sacrificing long term steady state, as was witnessedwithin CherryPick’s high latency steady state. While we ful-filled our goals of better short term and long term performancethan Throttlebot, we must still confirm that Bayesian Opti-mization makes a practical choice over Throttlebot despite itsseeding variability. In other words, Bayesian Optimization’sshort term performance, where it most strongly outperformsThrottlebot, is dependent on the random seed, and therefore,the random sample count. In future examination, a meansof dynamically determining the optimal random explorationcount, based on an inferred steady state, or a means of sam-pling from a normal distribution where samples gather aroundoptimal inputs can be tested and implemented. We will revisitthis idea in the future work section. These developments mayalso improve the performance of the bootstrapped gradientdescent, which constitutes our fastest converging algorithmwith a steady state value below 30,000 ms.

Concerning the horizontal scaling version of B-Optimal,the optimizer’s scaling mode produces better performancewith replicas than without replicas, as expected for higherworkloads. The optimizer also improves scaled performancecompared to an equal-provisioning configuration and limitsscaled performance drops when workload increases as wasalso seen within AutoTune’s horizontal scaling experiments.

VIII. FUTURE WORK

In revisiting the B-Optimal and Throttlebot resource allo-cations in the baseline benchmark, we see that B-Optimalfinds the optimal portion of the input space earlier in theexperiment than Throttlebot. However, using a random seedfor the random search phase renders B-Optimal’s short termadvantage over Throttlebot highly variable. To combat this, wereturn to our principle of tuning our selection of inputs basedon performance-based heuristics. Two potential approachescan further narrow down the inputs B-Optimal considersbefore the optimization phase. Firstly, Throttlebot’s gradientdescent sensitivity data provides a performance based estimateof the gradient of the entire application, elucidating the most-impacted resource-service tuples. Therefore, we can considerinputs where the most-impacted tuples hold more resourcesthan less-impacted tuples that share their machines within therandom exploration phase. The initial allocation can begin asa proportion equivalent to the ratio of the sensitivity estimateswithin one machine. To avoid over defining this heuristic, wecan sample from a normal distribution, where values closest tothe mean, or the proportional provisions, are most common butnot omnipresent. Secondly, we can perform a similar task byperforming Principal Component Analysis on the input spaceafter various random samples and sampling from a normal

distribution based around an initial provision proportionalto the rankings of the dimensions. Here, we would samplerandomly from the input space, perform the PCA algorithm,and then refine the remaining random samples from the createddistribution.

Overall, results suggest that a mix of random explorationand heuristic based optimization reaches a competitive steadystate in the quickest manner. Tuning the random explorationusing a mix of random sampling and dimensional analysiscould further speed up the convergence of such an optimizer,whilst limiting hits to steady state latency.

IX. ACKNOWLEDGEMENTS

We would like to thank the UC Berkeley NetSys Lab forthe inspiration for this project.

REFERENCES

[1] T. Vergilio and M. Ramachandran, “Non-functional requirements for realworld big data systems - an investigation of big data architectures atfacebook, twitter and netflix,” 05 2018.

[2] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg,omega, and kubernetes,” ACM Queue, vol. 14, pp. 70–93, 2016. [Online].Available: http://queue.acm.org/detail.cfm?id=2898444

[3] J. Dean and L. A. Barroso, “The tail at scale,” Communicationsof the ACM, vol. 56, pp. 74–80, 2013. [Online]. Available: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scale/fulltext

[4] M. A. Chang, A. Panda, Y.-C. Tsai, H. Wang, and S. Shenker, “Throttlebot- performance without insight,” 2017.

[5] K. K. Yang, L. C. Tay, and C. C. Sum, “A comparison of stochasticscheduling rules for maximizing project net present value,” EuropeanJournal of Operational Research, vol. 85, no. 2, pp. 327–339,September 1995. [Online]. Available: https://ideas.repec.org/a/eee/ejores/v85y1995i2p327-339.html

[6] O. Alipourfard, H. H. Liu, J. Chen, S. Venkataraman, M. Yu,and M. Zhang, “Cherrypick: Adaptively unearthing the best cloudconfigurations for big data analytics,” in Proceedings of the 14th USENIXConference on Networked Systems Design and Implementation, ser.NSDI’17. Berkeley, CA, USA: USENIX Association, 2017, pp. 469–482.[Online]. Available: http://dl.acm.org/citation.cfm?id=3154630.3154669

[7] M. Krasser, “Bayesian optimization,” 2018. [Online]. Available:http://krasserm.github.io/2018/03/21/bayesian-optimization/

[8] E. Brochu, V. M. Cora, and N. de Freitas, “A tutorial on bayesianoptimization of expensive cost functions, with application to activeuser modeling and hierarchical reinforcement learning,” CoRR, vol.abs/1012.2599, 2010. [Online]. Available: http://arxiv.org/abs/1012.2599

[9] M. A. Chang, A. Panda, Y.-C. Tsai, H. Wang, R. Balakrishnan, andS. Shenker, “Autotune,” 2020.