How to Bid the Cloud

Paper #114, 14 pages

ABSTRACTAmazon’s Elastic Compute Cloud (EC2) uses auction-basedspot pricing to sell spare capacity, allowing users to bid forcloud resources at a highly-reduced rate. Amazon sets thisspot price dynamically and accepts user bids above this price.Jobs with lower bids (including those already running) areinterrupted and must wait for a lower spot price before re-suming. We answer two basic questions faced by providersand users: how will the provider set the spot prices, and whatprices should users bid? Computing users’ bid strategies isparticularly challenging: higher bid prices lessen the chanceof interruptions, reducing the time overhead for recoveringfrom interruptions, but can increase user costs. We addressthese questions and challenges in three steps: (1) model-ing the cloud provider’s behavior to infer the spot price andmatching the model to historically offered prices, (2) de-riving optimal bid strategies for different job requirementsand interruption overheads, and (3) adapting this strategy toMapReduce jobs with master and slave nodes having differ-ent interruption overheads. We run our strategies on EC2 fora variety of job sizes and instance types, finding that spotpricing reduces user costs by 90% with modest increases incompletion time compared to on-demand instances.

1. INTRODUCTIONWith the recent growth in demand for cloud resources,

today’s cloud providers face an increasingly complicatedproblem of allocating resources to different users. Theseresource allocations must take into account both avail-able capacity within datacenter networks as well as thespecific job requirements, which are often specified inbinding service level agreements negotiated with theuser. The large number of users submitting differenttypes of jobs to any given cloud further complicates theproblem, creating a highly dynamic environment as jobsare submitted and completed at different times [4].

1.1 Spot Pricing to Shape User DemandWhile many works have considered the operational

problem of scheduling jobs within a datacenter [10, 14,23, 33], most have taken user demands as a given in-put and focused on operational concerns. Yet price set-

ting gives cloud providers a measure of control over userdemand [8]. However, the majority of today’s pricingplans for cloud resources are variants on usage-basedpricing, in which users pay a static per-unit price (perworkload or per hour) to access cloud resources [21].Usage-based pricing can affect overall demand levels,but it does not even out short-term fluctuations [16].To manage these fluctuations and the consequent ex-cess demand for available datacenter capacity, cloudproviders could introduce more flexible pricing plans inwhich resources are priced according to real-time mar-ket demand [1, 38]. Amazon’s Elastic Compute Cloud(EC2) spot pricing [2] employs this strategy.

Spot pricing creates an auction for available resourcesin which users submit bids for resource instances. Thecloud provider then sets a spot price at different times,depending on the number of bids received from users(i.e., demand) and how many idle resources are avail-able at that time (i.e., supply). User bids above thespot price are accepted, but if a user’s bid falls belowthe prevailing spot price, the user’s launched instanceis terminated until the bid again exceeds the spot price.User jobs can thus face interruptions in the course ofcompletion, which may affect user utility. Two naturalquestions then arise: first, how will the provider set thespot price, and second, what prices should users bid?

We provide the answers to these questions in thiswork. Unlike most works on spot pricing, which con-sider only the provider’s viewpoint [12, 19, 34, 37], weaim to both accurately infer how the provider sets thespot prices and then to develop bidding strategies for theuser. Importantly, we do not seek to design the auctionmechanism used by the provider, only to systematicallymodel and estimate how the provider might set the spotprices in order to develop bid strategies for the user.

1.2 Research Challenges and ContributionsWe can gain a basic statistical understanding of Ama-

zon’s prevailing spot prices by studying the two-monthhistory made available by Amazon [6, 17]. However,explaining the spot prices with a model that relatesthe spot prices to users’ submitted bids allows us togain further insights into the effects of different factors

1

Cloud Provider

User Client

Price Distribution Calculator

Price Monitor

Bid Calculator User Input

(job type, etc.)

Spot Price Calculator

Job Monitor User Bid

Spot Price

Figure 1: Client and cloud provider interaction.

like fully utilizing the available capacity and the perfor-mance that users receive from their bids.

Users bidding for spot resources face a tradeoff: bid-ding lower prices allows them to save money, but canalso lengthen job runtimes by increasing the likelihoodof interruptions. Even “interruptible” jobs can face ex-tra delays when recovering from interruptions, e.g., fromretrieving saved data and restarting the job. These de-lays can increase the job running time and thus its cost.Complicating this tradeoff is the fact that user jobs canhave long runtimes spanning many changes in the spotprice [17]. Users then face two key challenges:

1) Users must predict spot prices in order to optimizetheir bids for a given job, not only in the next timeslot but also for all future timeslots until the jobis completed. We leverage our estimation of theproviders’ behavior for this long-term prediction.

2) Different jobs may have different degrees of inter-ruptibility. Even within a single user job that re-quires many computational nodes, e.g., MapRe-duce’s master/slave node model, different nodescan have different interruptibility requirements.

Figure 1 shows the basic architecture of our client andits interaction with the cloud provider. The client cal-culates the bid price based on two types of inputs: userinputs on job characteristics, and the historical distribu-tion of the prices offered by the cloud provider. A pricemonitor ensures that the price distribution is kept up todate, and a job monitor at the provider tracks whetherthe job is ever outbid. The monitor also restarts users’jobs when the spot price falls below their bids.

We first give relevant background information on Ama-zon’s EC2 offerings and pricing in Section 2 before de-veloping the following results:

A framework for setting the spot prices (Sec-tion 3): We develop a model to understand how theproviders set spot prices, using it to bound users’ qual-ity of service and testing it against empirical data.

Users’ optimal bid strategies (Section 4): Given apredicted spot price distribution, we derive optimal bidstrategies for different degrees of job interruptibility.

Adaptation to MapReduce jobs (Section 5): Weadapt our bid strategies for the master and slave nodesof MapReduce jobs and implement our bidding strategyon a MapReduce job.

Experiment on Amazon EC2 (Section 6): We runour client on a variety of EC2 instances and job types,finding that it substantially lowers users’ costs in ex-change for modestly higher running times.

Our work thus considers spot pricing from the pointof view of both the cloud provider and user, leveragingan understanding of how providers set the spot prices todevelop bidding strategies customized to different userrequirements. We discuss some limitations of our workin Section 7 and related works in Section 8 before con-cluding the paper in Section 9. All proofs can be foundin the Appendix.

2. BACKGROUND

2.1 User Jobs and Instance TypesThroughout this work, we consider Infrastructure-as-

a-service (IaaS) cloud services, which are essentially re-mote virtual machines (VMs) with CPU, memory, andstorage resources [3]. We follow Amazon’s terminologyand use the term “instance” to denote the use of a sin-gle VM. Instances can be divided into several discretetypes, each of which may have different hardware andresource capacities; users generally submit requests, orbids, separately for different instance types.

The simplest types of user jobs require only one in-stance and can be served by placing a single bid requestfor a given instance type. Parallelizable jobs, on theother hand, might require multiple instances running inparallel. The MapReduce framework is a common re-alization of this job structure. MapReduce divides jobfunctions among a master node and several slave nodes.The master node assigns computational tasks to slavenodes and reschedules the tasks whenever a failure ofa slave node occurs. After all the tasks are finished onthe slave nodes, the master node returns a result.

2.2 EC2 PricingAmazon offers three types of pricing for instances:

reserved, on-demand, and spot instances. Reserved in-stances guarantee long-term availability (e.g., over ayear) and on-demand instances offer shorter-term usage(e.g., one hour on an instance). Both reserved and on-demand instances charge fixed usage-based prices. Spotinstances, however, do not guarantee availability; userscan use spot instances only if their bid exceeds the spotprice. Amazon generally updates the spot price everyfive minutes and encourages users to run interruptiblejobs on spot instances.1

Spot instances allow two types of bids: one-time andpersistent. One-time bids are submitted once and then

1Amazon charges users separately for ingress and egressbandwidth to EC2. Since the required bandwidth is de-termined by the user job rather than the instance on whichthe job is run, we do not consider these costs.

2

exit the system once they fall below the current spotprice. Thus, submitting a one-time bid takes the risk ofhaving the job interrupted without completing. Persis-tent bids, however, are resubmitted in each time perioduntil the job finishes or is manually terminated by theuser. One-time bids allow for better control over bidcompletion times (e.g., users may default to on-demandinstances if the jobs are not completed), while persis-tent bids allow the user to submit a bid request andthen simply wait until the job finishes.

3. CLOUD PROVIDER MODELIn this section, we develop an explanatory model for

cloud providers’ offered spot prices. Though the model’sprimary use for users is in predicting the future spotprice distribution for use in Sections 4 and 5, our modeladditionally yields interesting insights into the provider’sbehavior. We first formulate a model for choosing therevenue-maximizing prices in Section 3.1 and then con-sider its effects on pending bids in Section 3.2. In par-ticular, since users’ persistent requests are continuallyre-considered if not satisfied, the providers might expe-rience unsustainably large numbers of pending bids. Weshow that the number of pending bids remains boundedunder reasonable conditions. We validate our model inSection 3.3 by fitting it to two months history of spotprices offered by Amazon.

Let us consider a series of discrete time slots t ∈{1, 2, . . . }. At time slot t, the demand for spot instancesis L(t), and we use π(t) to denote the spot price at timeslot t. We restrict the spot price to not exceed the on-demand price π of the same instance type. We alsoimpose the constraint π(t) ≥ π, where π ≥ 0 representsthe provider’s marginal cost of running a spot instance.

3.1 Revenue MaximizationWhen setting the spot price π(t) in each time slot

t, the cloud provider wishes to maximize its revenueπ(t)N(t), where N(t) is the number of accepted bids(i.e., the system workload) and each successful bidderis charged only the spot price π(t), regardless of the bid(s)he placed.2 Since some studies have suggested thatAmazon does not use revenue-maximizing spot prices[6], we also include an optional capacity utilization termβ log (1 +N(t)), which increases with N(t). This termmodels the fact that the provider incurs a machine on/offcost for idle spot instances, giving it an incentive to ac-cept more bids. We use a strictly concave function topenalize extremely heavy workloads.

At each time slot t, the provider chooses the spotprice π(t) so as to maximize the sum of the utilizationterm and revenue, β log

(1 +N(t)

)+ π(t)N(t), subject

to the constraint that π(t) lie between the minimum

2In practice N is a discrete integer, but for tractable analysiswe take N continuous in our model.

and maximum prices (π ≤ π(t) ≤ π). We denote thisoptimal price by π?(t). In practice, we would generallyemphasize the revenue rather than utilization term bychoosing a small scaling factor β. The provider can keepthe number of accepted bids below its available capacityby adjusting the minimum spot price π.

We now formulate the provider’s optimization prob-lem by using the probability distribution fp to denotethe distribution of bids received by the user. For in-stance, if users’ bid prices follow a uniform distribu-tion, then fp(x) = 1/(π−π). Defining L(t) as the totalnumber of bids submitted, the number of accepted bids

is then N(t) = L(t) π−π(t)π−π , or the fraction of submit-

ted bids that exceed the spot price. The provider thenmaximizes the sum of its revenue and utilization term:

maximizeπ(t)

β log

(1 + L(t)

π − π(t)

π − π

)+π(t)L(t)

π − π(t)

π − πsubject to π ≤ π(t) ≤ π.

(1)

In the rest of the paper, we use a uniform distributionfor fp, as is often used to model distributions of uservaluations for computing services [30,31].

We solve (1) to find that the optimal spot price π?(t)satisfies

L(t) =π − π

π − π?(t)

(β

π − 2π?(t)− 1

). (2)

We can thus solve for the optimal solution to (1):

π?(t) = max

{π,

3

4π +

1

2(π − π)

1

L(t)

−1

4

√(π + 2

(π − π

) 1

L(t)

)2

+ 8β(π − π

) 1

L(t)

}.

(3)More weight on the utilization term (a higher β) leadsto a lower spot price and more accepted bids. Since βwill generally be small, we assume that β ≤ (L(t) +1)(π − 2π) and thus that π?(t) > π (the optimal spotprice is above the minimum) in the rest of the paper.

In the above analysis, we considered user bids in asingle time slot. However, bid resubmission may causethe spot price at time t to affect the prices in futuretime slots. We consider this dependency next.

3.2 Stable Job QueuesThe dynamics of user requests (i.e., bids for an in-

stance) consist of four distinct states: new arrivals,pending, running and finished, as shown in Figure 2.

At the beginning of time slot t, there are L(t) bids forthe spot resource remaining in the system, and Λ(t) newbid arrivals. After the spot price is determined for thistime slot, the N(t) spot requests with the highest bidprices are successfully launched. The new arrivals with

3

Pending

New arrivals

Running Finished

(1-�)N(t)

N(t)

Terminated

�(t)

�N(t)

L(t)

Figure 2: State transitions of spot instances. The solidand dashed arrow lines respectively represent the tran-sitions at the start of the current and next time slots.

lower bid prices are then pended, and some pendingrequests remain pended until the next time slot.

After time slot t, we assume that a portion θN(t) ofthe running instances are finished (including instancesthat have completed their jobs and terminated instanceswith one-time requests), while the other (1 − θ)N(t)instances are still running and will be considered to-gether along with the pended bids as bids at the nexttime slot. If terminated, running instances with per-sistent requests revert to the pending state; hence, therequests in the pending state come from three sources:i) failed new arrivals, ii) failed pending requests, and iii)terminated running instances. The number of requestsin the next time slot, L(t + 1), can thus be written interms of the number of requests L(t) in the previoustime slot, along with Λ(t) new arrivals and θN(t) exit-ing spot instances: L(t+ 1) = L(t)− θN(t) + Λ(t).

We assume that the Λ(t) are independent and iden-tically distributed (i.i.d.), following a distribution fΛ

with expected value λ and variance σ [34]. Note thatN(t) can be written in terms of the spot price π?(t):

N(t) = L(t) π−π?(t)

π−π , so the number of submitted bids

in each time slot satisfies

L(t+ 1) =

(1− θ π − π

?(t)

π − π

)L(t) + Λ(t). (4)

Note that 0 ≤ θ ≤ 1 and π ≤ π?(t) ≤ π ensure apositive value of L(t+ 1). Now, depending on the valueof Λ(t), L(t+ 1) may be larger or smaller than L(t).

If too many user bids are continually re-submitted,the number of submitted bids L(t) can become large,possibly diverging to infinity over long timescales. Toshow that such a scenario does not occur, we define theconditional Lyapunov drift as follows:

∆(t) ,1

2L2(t+ 1)− 1

2L2(t), (5)

or the change in the Lyapunov function 12L

2(t) overone time slot. Taking the conditional expectation, weupper-bound the Lyapunov drift (5):

Proposition 1 (Stability). Suppose Λ(t) followssome distribution whose expected value is λ and vari-ance is σ, and suppose that the spot prices π?(t) are cho-sen according to (3). Then the conditional expectationof the Lyapunov drift is upper bounded: E

(∆(t) |L(t)

)≤

(π − π)λ2/(2θπ) + σ/2− εL(t), where ε = θλπ4(π−π) .

Proposition 1 implies that when (1) is used to computethe spot price, the queuing system is stable in the sensethat the time-averaged queue size at any time t is uni-formly bounded [24]. In fact, the number of requestscan ultimately reach an equilibrium:

Proposition 2. The queue sizes of consecutive timeslots are in equilibrium, i.e., L(t+1) = L(t), if and onlyif the optimal spot prices π?(t) satisfy

π?(t) = h(Λ(t)

)=

1

2

(π − β

1 + 1θΛ(t)

)(6)

We observe from (6) that π?(t) is only in terms of therequest arrivals at that time slot, so π?(t), like Λ(t), isi.i.d. at the equilibrium. In the next section, we supposethat the system has reached this equilibrium and showthat the spot prices offered by Amazon correspond to aPareto distribution of the arrival rates Λ(t).

3.3 Characterizing the Spot PriceLeveraging the results in Section 3.2 (i.e., Proposi-

tion 2), we can derive the probability density function(PDF) of the spot price in terms of fΛ, the distribution(i.e., PDF) of Λ(t).

Proposition 3. The probability density function ofthe spot price is given by:

fπ(π) ' fΛ(h−1(π)) (7)

where h−1(π) = θ(

βπ−2π − 1

)is the inverse function of

the function given in (6).

We can thus use the distribution of the Λ(t) at differ-ent time slots, fΛ, to derive the distribution of the spotprices, fπ. Since we do not know the distribution of thebid arrivals, we instead test different distributions andcompare their spot price predictions to the empiricallyobserved spot prices.

We collected the spot price data for four instancetypes from the 14th of August to the 13th of Octoberin 2014 in the US Eastern region; we limit ourselvesto a two-month dataset since Amazon only providesuser access to spot price history for the previous twomonths. The PDF of these prices is shown by the bluebars in Figure 3. We observe that they approximatelyfollows a power-law or exponential pattern, indicatingthat the arrival process Λ(t) is non-Poisson.3 We thususe a Pareto distribution for Λ(t) to estimate the PDFof the spot prices with (6) and (7).

The PDF of a Pareto distribution is

fΛ(Λ) =αΛαminΛα+1

, for Λ ≥ Λmin,

3This observation is consistent with the heavy-tailed distri-butions that many real-world processes follow [5].

4

0.12 0.14 0.16 0.18 0.20

0.01

0.02

0.03

0.04

0.05

0.06

Spot price

Pro

ba

bili

ty d

en

sity

Empirical PDF

Estimated PDF

(a) c3.4xlarge (β = 0.6, θ =0.02, α = 5)

0.26 0.28 0.30

0.01

0.02

0.03

0.04

0.05

0.06

Spot price

Pro

ba

bili

ty d

en

sity

Empirical PDF

Estimated PDF

(b) c3.8xlarge (β = 1.2, θ =0.02, α = 8)

0.03 0.035 0.04 0.045 0.050

0.05

0.1

Spot price

Pro

ba

bili

ty d

en

sity

Empirical PDF

Estimated PDF

(c) r3.xlarge (β = 0.3, θ =0.02, α = 9.5)

0.03 0.035 0.04 0.045 0.050

0.02

0.04

0.06

0.08

Spot price

Pro

ba

bili

ty d

en

sity

Empirical PDF

Estimated PDF

(d) m1.xlarge (β = 0.3, θ =0.02, α = 5.2)

Figure 3: Fitting the probability density function of Amazon spot prices in the US Eastern region by assuming aPareto distribution for Λ(t). We fit our model to the prices for four different instance types.

Table 1: Key terms and symbols.

Symbol Definition

p User bid price

π Spot price

π On-demand price

π Minimum spot price

L Demand for the spot instance

N Number of launched spot instances

Λ New request arrivals

tk Length of one time slot

T Total job completion time

ts Job execution time (w/o interruptions)

tr Recovery time from an interruption

to Overhead time to run multiple sub-jobs

where Λmin = θ(

βπ−2π − 1

)is derived from the mono-

tonic relation between π?(t) and Λ(t) in (6). In Figure 3,we use this distribution to fit the empirical PDF of thespot prices. The α, β, and θ parameters are chosen tominimize the least-squares divergence between the es-timated and empirical PDFs. Figure 3 shows that theestimated probability density function using a Paretodistribution fits the empirical data well. We note thatα > 1 for all instance types, indicating that the distribu-tion has a finite mean and variance. Thus, Proposition1 holds and the system is stable. The non-negligible es-timates for β indicate that Amazon’s spot prices are setto increase both revenue and spare capacity utilization.

4. USER BIDDING STRATEGYWe now derive users’ bid strategies for jobs running

on a single instance, using the spot price PDF from Sec-tion 3 to predict future spot prices. Though time seriesforecasting may be used instead, we note that users’job runtimes generally exceed one time slot, requiringpredictions far in advance. Since the spot prices’ auto-correlation drops off rapidly with a longer lag time [6],such predictions are likely to be difficult. We discussthis further in Section 7.

In this section, we first consider one-time requests(Section 4.1) and then persistent requests (Section 4.2).In general, we assume that users wish to minimize the

cost of running their jobs.4 Users thus face a trade-off between bidding lower prices and experiencing moreinterruptions, which lead to increased job runtime andcan increase their cost. As in Section 3, we considera series of discrete time slots t and suppose the spotprices π(t) = h

(Λ(t)

), t = 0, 1, . . . are i.i.d. as in

Proposition 2. We use p to denote the user’s bid priceand Fπ to denote the cumulative distribution functionof each spot price π(t), corresponding to the PDF fπ in(7): Fπ(p) gives the probability that p > π(t), i.e., thatthe user’s bid is accepted.

Let us summarize our model’s notation in Table 1.Each job is characterized by its execution time ts, ortime required to complete without interruptions. Weuse T to denote the job’s total completion time, i.e., thelength of time between its submission and the time itfinishes and exits the system. Since jobs with persistentrequests may be periodically interrupted, we supposethat persistent jobs are configured to save their data toa separate volume once interrupted and recover it uponresuming running. Writing and transferring this dataintroduces a delay of tr seconds per interruption. Allprices are assumed to be in units of instance/ time slot,and all times given in units of seconds.

4.1 One-time RequestsSince one-time requests are terminated as soon as the

bid price falls below the spot price, we assume thatusers wish to minimize their expected job cost, subjectto the constraint that the job will complete before beingterminated. To find this expected cost, we first find theexpected price that a user must pay to use an instancein each time slot. If a user’s bid at price p is accepted,this expected price is simply the expected spot price, orthe expected value of all possible spot prices that areless than or equal to p:

E(π |π ≤ p) =

∫ pπxfπ(x)dx∫ p

πfπ(x)dx

=

∫ pπxfπ(x)dx

Fπ(p), (8)

4Including deadlines on the completion times does not mate-rially change our model, but we do not explicitly model thisconstraint. Users with hard job deadlines are more likely touse on-demand instances with guaranteed availability.

5

p

tr tr

Time to recover from interruption: 2tr

Execution time of a job without interruption: ts

Running time of a job on spot instance: TF!(p)

Bid price p

Figure 4: An example of job running times for the spotprices of a r3.xlarge-type instance in the US Easternregion on September 09, 2014.

which monotonically increases with p (cf. Appendix C).Thus, as we would intuitively expect, the user must paymore as his/her bid price increases. The user’s expectedcost for this job is then

Φso(p) = tsE(π |π ≤ p) =ts∫ pπxfπ(x)dx

tkFπ(p), (9)

i.e., the expected spot price, multiplied by the numberof time slots it takes for the job to complete. Since theexpected spot price is monotonically increasing in thebid price, the user can minimize his or her expected costby choosing the lowest possible bid price.

The lowest possible bid price is determined by theconstraint that the job completes before being termi-nated. We calculate this price by first finding the ex-pected amount of time that a job will continue runningwithout being interrupted:

tk

∞∑i=1

iFπ(p)i−1(1− Fπ(p)

)=

tk1− Fπ(p)

. (10)

Here the Fπ(p) terms represent the probability that p >π(t), i.e., the request continues to run, and 1−Fπ(p) theprobability that the job will be terminated. We thusfind that ts ≤ tk/ (1− Fπ(p)): the expected amountof time that a job will keep running must exceed itsexecution time. We can now find the optimal bid price:

Proposition 4. The optimal bid price for a one-time request is

p? = max

{π, F−1

π

(1− tk

ts

)}. (11)

As we would expect, the bid price increases as the num-ber of time slots required to complete the job, ts/tk, in-creases: the job then needs to run for more consecutivetimeslots, which becomes more likely with a higher bid.

4.2 Persistent RequestsWe now consider a job that places a persistent spot

instance request. We begin by finding the total timethat the job runs on the system, given a bid price, and

briefly discuss the implications of a job’s interruptibilitybefore finding the user’s optimal bid price.

Job running time. The job’s total completion timeT comprises two types of time slots: the running time,in which the job’s bid price exceeds the spot price andthe job actually runs on the instance, and the idle time.For the bid price p, the job’s expected running time isTFπ(p) (recall that Fπ(p) denotes the probability thatthe bid price p ≥ π(t), the spot price), and the idletime is then T (1 − Fπ(p)). The running time can befurther split into the execution time, ts, and the addi-tional running time required to recover from interrup-tions. We illustrate the job completion time in Figure 4,in which the bid price of p = 0.0323 is represented byan orange dashed line. The grey double-arrowed linerepresents the total running time. Since the job is in-terrupted twice, the total recovery time is 2tr. Thus,for this example, we have TFπ(0.0323) = 2tr + ts.

To determine the total recovery time, we want to findthe expected number of interruptions, i.e., times t inwhich the job runs on the system but was idle in theprevious time slot (p ≥ π(t), p < π(t−1)). Note that thenumber of interruptions equals half of the total numberof times the job’s state changes between running andidle: each interruption requires both an idle-to-runningand running-to-idle transition. We can thus find the ex-pected total number of transitions and divide it by two.To do so, we define 1π

(π(t)

)as an indicator function:

1π

(π(t)

)= 1 if p ≥ π(t); otherwise, 1π

(π(t)

)= 0. We

then consider(1π

(π(t)

)−1π

(π(t+1)

))2

, which equals

1 if 1π(π(t)

)6= 1π

(π(t+ 1)

)(i.e., a transition happens)

and 0 otherwise. The number of idle-to-running transi-

tions in T/tk time slots is then 12

∑T/tk−1k=0

(1π

(π(t)

)−

1π

(π(t+ 1)

))2

. We take the expectation to obtain

E

12

T/tk−1∑k=0

(1π

(π(t)

)− 1π

(π(t+ 1)

))2

(a)=

T

tk

(E

(1π

(π(t)

))−E

(1π

(π(t)

)1π

(π(t+ 1)

)))=

T

tkFπ(p)

(1− Fπ(p)

),

(12)

where (a) is due to(1π

(π(t)

))2

= 1π

(π(t)

)since the

value of 1π(π(t)

)is either 1 or 0.

We now write the expected running time of the job asthe sum of the recovery and execution times: TFπ(p) =(TtkFπ(p)

(1 − Fπ(p)

)− 1)tr + ts. By simplifying, the

running time becomes

TFπ(p) =ts − tr

1− trtk

(1− Fπ(p)

) , (13)

which decreases with p. As the bid price p increases,

6

0 0.1 0.2 0.3 0.4 0.5 0.6 0.70

0.1

0.2

0.3

0.4

0.5

0.6

tr / t

k

Optim

al bid

price (

$)

c3.4xlargec3.8xlarge

Figure 5: The optimal bid price p? for persistent re-quests increases approximately linearly with recoverytime tr. We consider two different instance types; exe-cution time is fixed as ts = 60s.

the job is less likely to be interrupted and will thereforehave a shorter expected running time.

Job interruptibility. We can use the expected run-ning time (13) to observe the effect of the recovery timeparameter, tr, on a job’s feasibility for spot instances.Intuitively, spot instances are more effective for more“interruptible” jobs that can quickly recover from inter-ruptions. In fact, we find that the job’s running time isfinite only if the recovery time is sufficiently small:

tr <tk

1− Fπ(p). (14)

The result in (14) can be obtained by requiring the de-nominator in (13) to be positive. We note that theupper bound to tr is exactly the expected running timeof a job without interruptions (cf. (10)): intuitively, thetime to recover from an interruption should be smallerthan the expected time on an instance between job in-terruptions. We take (14) as a constraint on the bidprice: if the job recovery time is high, the user shouldbid at a higher bid price in order to ensure that the jobcan complete. However, if the job recovery time is less

than one time slot length, tr < min{

tk1−Fπ(p)

}= tk,

and a spot instance is feasible at any price.The optimal bid price. We can now multiply the

expected running time (13) with the expected spot price(8) to find that the cost of a job with a persistent requestis Φsp(p) = TFπ(p)E(π |π ≤ p). The user’s optimal bidprice then satisfies

minimizep

Φsp(p) = ts−tr1− trtk

(1−Fπ(p)

) ∫ pπxfπ(x)dx

Fπ(p)

subject to Φsp(p) ≤ tsπ, tr < tk1−Fπ(p) , π ≤ p ≤ π.

(15)where the first constraint ensures that the cost of run-ning the spot instance is lower than the cost of runningthe job on an on-demand instance, and the second con-straint ensures that the job is sufficiently interruptible.We use p? to denote the optimal bid price to (15).

We now observe that the expected running time in(13) decreases with the bid price, while the expected

spot price increases with the bid price. We find that theexpected cost Φsp(p) first decreases and then increaseswith the bid price p (cf. Appendix D), thus allowing usto solve for the optimal bid price p?:

Proposition 5. If the probability density function ofthe spot price monotonically decreases, i.e., Fπ(p) isconcave, the optimal bid price to (15) is

p? = ψ−1

(tktr− 1

), (16)

where ψ−1(·) is the inverse function of

ψ(p) = Fπ(p)

( ∫ pπxf(x)dx∫ p

π(p− x)f(x)dx

− 1

).

We can observe from (16) that the optimal bid price de-pends only on the execution time ts: the cost is insteaddetermined by the number of time slots needed for eachrecovery, tr/tk. While the execution time is fixed, alonger recovery time lengthens the total running timeand thus the job cost. In Figure 5, we show that for thedistributions of Fπ that we derived in Section 3.3 (cf.Figure 3), the optimal bid price p? increases approxi-mately linearly tr/tk.

5. BIDDING FOR MAPREDUCE JOBSWe now adapt Section 4’s bid strategies for single in-

stances to parallelized MapReduce jobs, in which usersbid for multiple instances at the same time. We firstconsider running only the slave nodes on spot instancesand then consider running the full job (i.e., both mas-ter and slave nodes) on spot instances.5 Though thejob structure introduces an additional requirement onthe master node’s running time–the master node shouldkeep running as long as the slave nodes are running–weshow that this condition does not change the optimalbid for the master node as long as the job is sufficientlyparallelized (i.e., multiple slave nodes run in parallel fora short amount of time).

5.1 Bidding for Slave Nodes OnlyWe first consider running only the slave nodes of a

MapReduce job on several spot instances in parallel.We suppose that the job is split into M sub-jobs of equalsize, each corresponding to one instance request. Wenow wish to calculate the optimal (i.e., cost-minimizing)bid prices for these sub-jobs; since we assume that allsub-jobs are bidding the same type of spot instance, thebid price should be the same for all of them. We againfind the total cost by multiplying the expected runningtime of the job by the expected spot price.

5While master nodes are often run on on-demand instancesto guarantee they will not be interrupted, our experiments inSection 6 show that with sufficiently high bids, interruptionsare rare even on spot instances.

7

To find the job’s expected running time, we denoteeach sub-job i’s total time in the system as Ti. Sincesplitting a job results in additional overhead, e.g., due tomessage passing between the sub-jobs, we use to to rep-resent a constant additional overhead time from split-ting the job.6 Then the total running time satisfies:

M∑i=1

TiFπ(p) =

M∑i=1

(TiFπ(p)

(1− Fπ(p)

)tk

− 1

)tr+ts+to,

i.e., it is the sum of the recovery, execution, and over-head times. Hence, we can extend the result for a singlepersistent bid in (13) as

M∑i=1

TiFπ(p) =ts + to −Mtr

1− trtk

(1− Fπ(p)

) . (17)

Since all sub-jobs run simultaneously, the overall timeto finish the parallelized job is max

i=1,...,MTiFπ(p). Since

all M sub-jobs are of equal size, we have

min maxi=1,...,M

TiFπ(p) =ts + to −Mtr

M(

1− trtk

(1− Fπ(p)

)) . (18)

Though the job is split into smaller pieces, the over-all running time max

i=1,...,MTiFπ(p) is larger than ts/M ,

or the running time without interruptions or overhead.Thus, distributing the job across M instances shortensthe completion time as compared to a single instanceonly if the overhead time is sufficiently small. Com-paring the completion times in both cases, we find thatusing multiple instances shortens the completion timeif to < (M − 1)tk/(1− Fπ(p)).

As in Section 4, the expected cost of running a job onM simultaneous instances is the sum of each instance’sexpected running time, multiplied by the expected spotprice: Φmp =

∑Mi=1 TiFπ(p)E(π |π ≤ p). The user then

minimizes this cost by solving

minimizep

Φmp(p) = ts+to−Mtr

1− trtk

(1−Fπ(p)

) ∫ pπxfπ(x)dx

Fπ(p)

subject to Φmp(p) ≤ tsπ, π ≤ p ≤ π,(19)

where the first constraint ensures that the cost is lowerthan that of running the job on an on-demand instance.Comparing (19) to bidding for a single persistent re-quest in (15), we see that (19) can be solved like (15)in Proposition 5.

By comparing the costs at the optimal price for mul-tiple bids and a single bid, we find that when the over-head time is sufficiently small (to < (M −1)tr), biddingfor multiple spot instances can both lower the cost andshorten the job’s overall running time. In contrast, run-ning the job on an on-demand instance will reduce therunning time but increase the cost.6This overhead time may depend on M , the (fixed) numberof sub-jobs.

5.2 Bidding for Master and Slave NodesRunning a MapReduce job entirely on spot instances

requires us to treat master and slave nodes separately;for instance, we might bid on different instance types forthe master and slave nodes, since the slave nodes willlikely have higher computing requirements. We thusdevelop two separate bid strategies:

1) Master node: Since the master node has to beavailable at all times to manage slave node failuresand to periodically check the status of tasks at theslave nodes, we do not allow any interruptions forthe master node. We thus place a one-time requestfor a single spot instance, as in Section 4.1.

2) Slave nodes: MapReduce requires many slavenodes to process large jobs, but allows slave nodesto be interrupted. Thus, we place a persistent re-quest for each slave node using the strategy in Sec-tion 5.1. The bid prices for the slave nodes must bedetermined jointly with that of the master node,since the master node’s running time should ex-ceed the slave nodes’. We note from Section 5.1that if many simultaneous bids are submitted, theslave nodes’ running time will decrease, shorteningthe required master node running time.

We can formally describe these strategies in the follow-ing optimization problem:

minimizepv,pm

Φso(pm) + Φmp(pv)

subject to tk1−Fmπ (pm)

≥ 1Fvπ (pv)

(ts+to−Mtr

1− trtk

(1−Fvπ (pv)

) − (M−1)tk1−Fvπ (pv)

),

π ≤ pv ≤ π, π ≤ pm ≤ π.(20)

where pm and pv denote the bid prices of the masternode and slave nodes respectively, and Fmπ (·) and F vπ (·)denote the spot prices’ cumulative distribution func-tions for the master and slave node instance types. Thefirst constraint in (20) ensures that the master noderuns longer than any of the slave nodes (cf. (10) and(18)), where the righthand side of the inequality repre-sents the worst-case completion time of the M parallelsub-jobs. We use p?m and p?v to denote the optimal bidprices for (20)’s optimization problem.

We can solve (20) by noting that, aside from the firstconstraint, pm and pv are independent variables. Thus,we can set their optimal values p?m and p?v respectivelyas the optimal bid prices for a one-time single instancerequest (Proposition 4) and for multiple persistent re-quests (as in (19)). The first constraint is satisfied if theuser submits sufficiently many simultaneous requests forthe slave nodes. In practice, this minimum number ofnodes, which we denote as M?, can be as low as 3 or 4,as we show in Section 6’s experiments.

8

Table 2: EC2 instance types. Sizes are given in(vCPU, memory in GiB, SSD storage in GB).

m3 r3 c3

.xlarge (4, 15, 1x32) (4, 30.5, 1x80) (4, 7.5, 2x40)

.2xlarge (8, 30, 2x80) (8, 61, 1x160) (8, 15, 2x80)

.4xlarge – (16, 122, 1x320) (16, 30, 2x160)

.8xlarge – – (32, 60, 2x320)

6. EXPERIMENTAL RESULTSIn this section, we first examine the optimal bid prices

derived in Section 4 on Amazon EC2 spot instances. Bycomparing the price charged per hour, total job comple-tion time and final cost, we illustrate the tradeoff of us-ing different bid strategies. We then run a MapReduceexample on spot instances to further highlight that ourproposed bid strategy in Section 5 is adaptable to paral-lelized MapReduce jobs and can substantially lower thecost. Table 2 summarizes the instance types that we usein our experiments. The m3, r3, and c3 prefixes denotebalanced, memory-optimized, and compute-optimizedinstances respectively. We test a variety of instancetypes and sizes and repeat each experiment ten times foreach instance type; all performance graphs are shownas averages.7 We do not include the runtime of the bidprice calculations in our measurements since it is un-der one minute, and can thus be easily run within onefive-minute time slot.

6.1 Single-Instance BidsParameter setting: We consider a job that needs

one hour (i.e., ts = 1h) to be executed without inter-ruption. The optimal bid prices of five Amazon spot in-stances (r3.xlarge, r3.2xlarge, r3.4xlarge, c3.4x-large and c3.8xlarge) are listed in Table 3 for a one-time request and persistent requests with recovery timestr = 10s and tr = 30s. We use the spot price history forthe two months immediately prior to the experimentsto calculate these prices.

Experiment setup: To simulate an exact one hourrunning time of a spot instance, we create an AmazonMachine Image (AMI), where a shell script is addedto /etc/rc.local so that a one-hour count-down pro-gram will run once the instance is launched using thisAMI. In addition, this program writes instance launchedtime as a sequence of items into Amazon DynamoDB,from which we can obtain the instance status (first runor restarted from interruption) and simulate a recoverytime if the instance is interrupted. We then place allthe spot requests using this AMI.

Results: We use the optimal bid prices for one-timerequests (Table 3) to bid the associated spot instance atrandom times of the day. None of our experiments were

7To ensure accuracy, we use our bills from Amazon to cal-culate the job costs. Since Amazon does not break the billsinto individual jobs, we do not report the cost of each job.

r3.xlarge r3.2xlarge r3.4xlarge c3.4xlarge c3.8xlarge0

0.5

1

1.5

2

2.5

Instance type

Cost

($)

On−demand instanceAnalytical results for spot instanceOne−time spot instance

Figure 6: One-time spot instance requests substantiallylower user cost compared to on-demand instances (on-demand and bid prices are as in Table 3).

interrupted, verifying that our bid strategy for one-timerequests can ensure reliability. Our bills show that thebid strategy for a one-time single bid can reduce up to91% of user cost compared to running the on-demandinstance. Figure 6 compares the cost of one-time re-quests on spot instances to the cost of on-demand in-stances for the instance types in Table 3. We also com-pare the actual costs to the expected cost from our ana-lytical model; these analytical predictions closely matchthe experimental results.

We now use the results of the one-time requests foreach instance type as a baseline to illustrate that userscan further lower their cost in exchange for somewhatlonger completion times by placing persistent requests.In Figure 7, we plot the percentage difference in perfor-mance between the persistent and one-time bids.

Figure 7(a) shows the percentage difference in pricebetween the one-time and persistent requests for the fiveinstance types. The negative values in this figure illus-trate a lower optimal bid price for persistent requests:persistent requests can be interrupted. As we wouldexpect from Section 4’s results, longer recovery times(tr = 30s rather than 10s) yield higher bid prices, sincea higher bid price ensures that the job has sufficienttime to execute after recovering from interruptions.

Conversely, as shown in Figure 7(b), the completiontime of a one-hour job with a persistent request is longerthan that of a one-time request. The difference in thesecompletion times includes both the recovery time whenthe instance is interrupted and the idle time duringwhich the user’s bid price is lower than the spot price.Interestingly, the job with longer recovery time has ashorter completion time, though it is still longer thanthe completion time with a one-time request. We canobserve from Table 3 that a longer recovery time leadsto a higher optimal bid price, which can consequentlyincrease the likelihood of satisfying the spot price andlead to less idle time.

As shown in Figures 7(a) and 7(b), for the persistentrequests, the price charged per hour is lower but comple-tion time is longer compared to the one-time requests.This makes the final cost underdetermined: Figure 7(c)

9

Table 3: Optimal bid prices for a single-instance job on spot instances.

Instance type π πOne-time bid

Persistent bid

(tr = 10s)

Persistent bid

(tr = 30s)

p? E(π |π < p?) p? T p? T

r3.xlarge $0.35 $0.0321 $0.0374 $0.0331 $0.0332 1.4549h $0.0355 1.1903h

r3.2xlarge $0.70 $0.0646 $0.0795 $0.0669 $0.0661 1.7638h $0.0711 1.2558h

r3.4xlarge $1.40 $0.1286 $0.1430 $0.1304 $0.1327 1.2798h $0.1422 1.0976h

c3.4xlarge $0.84 $0.1281 $0.1669 $0.1324 $0.1322 1.4917h $0.1413 1.2180h

c3.8xlarge $1.68 $0.2561 $0.2903 $0.2604 $0.2648 1.2767h $0.2831 1.1209h

−5

−4

−3

−2

−1

0

Instance type

Price (

% w

.r.t. one−

tim

e r

eque

st)

r3.xlarg

e

r3.2

xlar

ge

r3.4

xlar

ge

c3.4

xlar

ge

c3.8

xlar

ge

Persistent request (tr=10s)

Persistent request (tr=30s)

(a) Expected spot price.

0

20

40

60

80

100

120

Instance type

Tim

e (

% w

.r.t. one−

tim

e r

equ

est)

r3.xlarg

e

r3.2

xlar

ge

r3.4

xlar

ge

c3.4

xlar

ge

c3.8

xlar

ge

Persistent request (tr=10s)

Persistent request (tr=30s)

(b) Job completion time.

−3

−2.5

−2

−1.5

−1

−0.5

0

Instance type

Cost

(% w

.r.t

. one−

tim

e r

equest)

r3.xlarg

e

r3.2

xlar

ge

r3.4

xlar

ge

c3.4

xlar

ge

c3.8

xlar

ge

Persistent request (tr=10s)

Persistent request (tr=30s)

(c) Total job cost.

Figure 7: Persistent requests yield lower cost but longer completion times compared to one-time requests. The bidprices for each instance type are given in Table 3.

Table 4: Optimal bid prices for a MapReduce job.

Master node Slave nodes

Client Instancep?m

Instancep?v M?

Setting Type Type

CS1 c3.xlarge $0.133 m3.2xlarge $0.070 5

CS2 m3.xlarge $0.101 m3.2xlarge $0.070 5

CS3 m3.xlarge $0.102 r3.2xlarge $0.071 3

CS4 r3.xlarge $0.042 m3.2xlarge $0.070 5

CS5 r3.xlarge $0.042 r3.2xlarge $0.071 3

demonstrates that persistent bids lead to lower overallcosts. Smaller recovery times (tr = 10s versus 30s) yieldlower costs due to their lower bid prices; though theircompletion times are higher, the higher completion timeis mostly due to more idle time, for which the user isnot charged, rather than extra time running on the in-stance. Thus, persistent requests can reduce user costbut lengthen the job completion time.

6.2 MapReduce JobsWe now implement our optimal bidding strategy to

run Hadoop MapReduce jobs on the Common CrawlDataset [9] using Amazon Elastic MapReduce (EMR).This dataset, which is hosted on Amazon’s Simple Stor-age Service, maintains an open repository of web crawldata that is accessible to the public for free. In this ex-periment, we run the well-known“Common Crawl WordCount” example that counts the frequency of words ap-pearing on the Common Crawl Corpus.

Parameter setting: We consider a word count jobwith recovery time tr = 30s and overhead time to = 60s.Using the bid strategy proposed in Section 5, we listin Table 4 the optimal bid prices and the number ofslave nodes for five client settings with different instancetypes for the master node and the slave nodes.

Experiment setup: We place a one-time request fora single spot instance for the master node and persistentrequests for multiple spot instances for the slave nodes.Since the master node just distributes the raw data andtracks the status of each task, it does not require high-performance instances; we therefore bid on instanceswith better CPU performance for the slave nodes.

Results: We compare the completion time and costof running MapReduce jobs on on-demand and spot in-stances for the five client settings. We use the opti-mal bid prices listed in Table 4 to place the spot in-stance bids. Using the on-demand instance as a base-line, our bills from Amazon show that the bid strategyfor MapReduce jobs can reduce up to 92.6% of user costwith just a 14.9% increase of completion time.

In Figure 8, we show the completion time and thecost of the five client settings for on-demand and spotinstances. Again, our experimental results closely ap-proximate the analytical results from our model. As wewould expect, the job completion time is longer on thespot instances than on the on-demand instance (Fig-ure 8(a)), while the cost of the job running on the spotinstance is much lower than that on the on-demand in-

10

CS1 CS2 CS3 CS4 CS50

1

2

3

4

5

6

Client settings

Co

mp

letio

n tim

e (

ho

urs

)

On−demand instance

Analytical results for spot instance

Spot instance

(a) Job completion time.

CS1 CS2 CS3 CS4 CS50

5

10

15

Client settings

Co

st

($)

On−demand instanceAnalytical results for spot instanceSpot instance

(b) Job cost.

Figure 8: MapReduce jobs can save about 90% of usercost but have a 15% longer completion time on spotcompared to on-demand instances.

stance (Figure 8(b)).

7. DISCUSSIONLike all work based on analytical models, our work

has some limitations. We discuss four important oneshere and suggest ways to address them in future work.

Provider objectives. We have assumed that thecloud provider optimizes the sum of a utilization termand its revenue. However, in practice the provider canhave other concerns, such as users’ social welfare [26,37]or the electricity costs for running instances. Whileour current formulation matches well with the observedspot prices, including these factors in the provider’s spotprice optimization problem may shed more light on theprovider’s behavior.

Temporal correlations. We assume i.i.d. job ar-rivals that yield similarly i.i.d. spot prices at the equilib-rium. However, empirical studies have found that cloudworkloads exhibit temporal correlation, possibly induc-ing some temporal correlation in the spot prices [15].In fact, a study of the spot prices in 2010 shows thepresence of limited autocorrelation for consecutive timeslots [6]. Incorporating these correlations into users’spot price predictions may improve their bid strategies.

Risk-averseness. We suppose that users choose theirbid prices so as to minimize their expected costs, sub-ject to constraints on the expected runtimes. However,risk-averse users may also wish to minimize the variancein costs and runtimes, so as to ensure that particularlybad outcomes do not occur. For instance, we might cal-culate the worst-case (i.e., maximum) job running timein Sections 4 and 5 and choose the bid prices so as tominimize the resulting worst-case cost.

Collective user behavior. The bidding strategiesthat we have developed in Sections 4 and 5 assume thatan individual user’s bid price will not measurably af-fect the provider’s spot price. While our experiments inSection 6 show that this assumption holds for a singleuser, it may not hold if multiple users begin to optimizetheir bid strategies, which might affect the distributionof the submitted bids. To study this scenario, we canassume that users with a distribution of jobs optimize

their bids and use Section 3’s model to derive the effecton the provider’s offered spot price.

8. RELATED WORKCloud scheduling and pricing. Many works have

considered resource allocation in the cloud from a purelyoperational perspective [10, 14, 23, 33]. Others incorpo-rate pricing considerations, e.g., dynamically allocatingcloud resources, so as to maximize the provider’s rev-enue [12, 19, 34] or social welfare [26, 37]. We constructa similar model but relate it to empirical bid prices anduse it to develop bid strategies for users. Joint user-provider interactions for usage-based pricing are consid-ered in [32], but auction-specific works on both providerand user actions are limited to statistical studies of his-torical spot prices [6, 17].

Auctions and bidding. User bidding strategiesfor cloud auctions are much less studied than providerstrategies. While some works have shown that users canreduce their costs by using spot rather than on-demandinstances [28, 36], they only consider heuristic biddingstrategies for single-instance jobs.

In general, auction frameworks assume that users’bids are determined by their valuation of the auctionedresource. Indeed, many works have studied the prob-lem of designing online auctions to ensure truthful userbids [7, 13, 22, 27, 29], including improvements to Ama-zon’s spot pricing [35]. However, in cloud scenarios,user valuations for the instance in a given time slot de-pend on (unknown) future spot prices. While users mayknow their valuation for completing a job, job inter-ruptions will increase the number of instance-time slotsrequired to complete the job. This consideration differ-entiates our work from other auctions for computing orutility resources, e.g., auctions for smart grid electric-ity [11], secondary spectrum access [18], grid comput-ing [20], or Internet data [25].

9. CONCLUSIONSpot pricing opens up an auction-based market in

which cloud providers can dynamically provision datacenter resources to meet user demand and users candevelop bid strategies that lower their cloud resourcecosts. In this work we first consider providers’ settingof the spot prices, developing a revenue-maximizationmodel for the provider and comparing its results to thehistorical spot prices. We then answer the question ofhow users should bid for cloud resources. Since spot in-stances do not guarantee their availability, we considerthe tradeoff between bidding higher prices to avoid in-terruptions (for one-time requests) and bidding lowerprices to save money (for persistent requests).

To show the validity of this tradeoff and our bidstrategies’ adaptability to specific job requirements, weadapt these bid strategies to MapReduce jobs with mas-

11

ter and slave nodes. Finally, we run our bidding clienton Amazon EC2 to verify that our analytical resultsaccurately approximate the real-time experimental re-sults. We visualize the resulting price per hour, jobcompletion time and final cost for both on-demand andspot instances for a variety of instance types. Our bidstrategies can reduce users’ costs by around 90%, withmodest increases in job completion times. Appropriatebidding strategies thus allow users to significantly re-duce the cost of running their jobs while successfullymanaging job interruptions.

APPENDIXA. PROOF OF PROPOSITION 1

Proof. Substituting (4) into (5), the Lyapunov drift

can be expressed as ∆(t) = 12

((1 − θ π−π

?(t)π−π

)L(t) +

Λ(t))2

− 12L

2(t). We now bound the expectation of this

quantity by:

E(∆(t) |L(t)

)(a)

≤ 12

(− θπ

π−π + 14

(θππ−π

)2)L2(t)

+(1− 1

2θππ−π

)L(t)E

(Λ(t)

)+ 1

2E(Λ2(t)

)(b)= 1

2

(− θπ

π−π + 14

(θππ−π

)2)L2(t)

+(

1− 12θππ−π

)λL(t) + 1

2 (σ + λ2)

≤(1− 1

4θππ−π

)maxL(t)

{− 1

2θππ−πL

2(t) + λL(t)}

− 14θλππ−πL(t) + 1

2 (σ + λ2)(c)= π−π

2θπ λ2 + 1

2σ −14θλππ−πL(t),

where (a) is due to π(t) ≤ 12 π, (b) is due to E(Λ2(t)) =

Var(Λ(t)) +(E(Λ(t))

)2and the equality in (c) holds

when L(t) = π−πθπ λ.

B. PROOF OF PROPOSITION 2

Proof. We first prove that (6) is a necessary condi-tion for L(t+ 1) = L(t). We rewrite (4) as

L(t) =π − π

θ(π − π?(t))Λ(t). (21)

Solving the system of equations (2) and (21) yields (6).We then prove that (6) is a sufficient condition for

L(t+ 1) = L(t). From (4), we can derive

L(t+ 1)− L(t)(a)= −θ

(β

π − 2π?(t)− 1

)+ Λ(t)

(b)= θ

β

π −(π − β

1+ 1θΛ(t)

) − 1

+ Λ(t) = 0,

where (a) and (b) are obtained by respectively substi-tuting (2) and (6). Thus, L(t+ 1) = L(t).

C. PROOF OF PROPOSITION 4

Proof. By taking the first-order derivative of Φso(p)in (9), we have

∂Φso(p)/∂p

= ts(Fπ(p))2

fπ(p)(−∫ pπxfπ(x)dx+ pFπ(p)

).

By letting g(p) = −∫ pπxfπ(x)dx + pFπ(p), we have

∂g(p)/∂p = Fπ(p) > 0. So g(p) increases with p. Com-bining the fact that g(π) = −πfπ(π) + πFπ(π) = 0, thenonnegativity and monotonic increasing of g(p) leads to∂Φso(p)/∂p > 0. Therefore, Φso(p) also increases withp. Minimizing Φso(p) is equivalent to finding the min-imum p in its feasible set. From tk/

(1 − Fπ(p)

), we

have p ≥ F−1π

(1 − tk

ts

)due to the monotonic property

of Fπ(p). Thus, the minimum of the feasible set is thelarger one of π and F−1

π

(1− tk

ts

).

D. PROOF OF PROPOSITION 5

Proof. By taking the first-order derivative of Φ(p)in (15), we have

∂Φsp(p)/∂p

= (ts − tr)fπ(p)(1− trtk )+2 trtk

Fπ(p)((1− trtk )Fπ(p)+ tr

tk

(Fπ(p)

)2)2 g(p),

where

g(p) = −∫ p

π

xfπ(x)dx+ p(1− tr

tk)Fπ(p) + tr

tk

(Fπ(p)

)2(1− tr

tk) + 2 trtkFπ(p)

.

Note that first three terms before g(p) in ∂Φsp(p)/∂p arepositive. To show the positivity of ∂Φsp(p)/∂p, we takefirst-order derivative of g(p) and then have ∂g(p)/∂p =

(1− trtk )Fπ(p)+ trtk

(Fπ(p)

)2((1− trtk )+2 trtk

Fπ(p))2 (

(1− trtk

)+2 trtk

(Fπ(p)−pfπ(p)

)).

Due to the positivity and monotonic decreasing prop-erty of fπ(p) (cf. Figure 3), Fπ(p) is concave and Fπ(p)−pfπ(p) ≥ 0. We then have ∂g(p)/∂p ≥ 0. Thus, g(p)monotonically increases with p, so is ∂Φsp(p)/∂p. Bythe fact that g(π) < 0 and g(π) > 0, ∂Φsp(p)/∂p in-creases monotonically from a negative value to a posi-tive value, i.e., Φsp(p) first decreases and then increaseswith p. Hence, Φsp(p) is minimized when ∂Φsp(p)/∂p =0, i.e., g(p) = 0. Letting g(p) = 0, we thus deduce∫ p

π

xfπ(x)dx = p(1− tr

tk)Fπ(p) + tr

tk

(Fπ(p)

)2(1− tr

tk) + 2 trtkFπ(p)

⇒ ψ(p) = Fπ(p)

( ∫ pπxf(x)dx∫ p

π(p− x)f(x)dx

− 1

)=tktr− 1,

which leads to (16). In addition, Φ(p?) < Φ(π) = (ts −tr)E(π |π ≤ π) ≤ tsπ so the constraints in (15) aresatisfied at optimality.

12

10. REFERENCES[1] Agmon Ben-Yehuda, O., Posener, E.,

Ben-Yehuda, M., Schuster, A., andMu’alem, A. Ginseng: market-driven memoryallocation. In Proceedings of the 10th ACMSIGPLAN/SIGOPS international conference onVirtual execution environments (2014), ACM,pp. 41–52.

[2] Amazon. EC2 spot instance. online at:http://aws.amazon.com/ec2/purchasing-options/spot-instances/(2015).

[3] Amazon. Elastic Compute Cloud (Amazon EC2).online at: http://aws.amazon.com/ec2/ (2015).

[4] Armbrust, M., Fox, A., Griffith, R.,Joseph, A. D., Katz, R., Konwinski, A.,Lee, G., Patterson, D., Rabkin, A., Stoica,I., et al. A view of cloud computing.Communications of the ACM 53, 4 (2010), 50–58.

[5] Barabasi, A.-L. The origin of bursts and heavytails in human dynamics. Nature 435, 7039(2005), 207–211.

[6] Ben-Yehuda, O. A., Ben-Yehuda, M.,Schuster, A., and Tsafrir, D. DeconstructingAamazon EC2 spot instance pricing. ACM Trans.on Economics and Computation 1, 3 (2013), 1–16.

[7] Blum, A., Sandholm, T., and Zinkevich, M.Online algorithms for market clearing. Journal ofthe ACM (JACM) 53, 5 (2006), 845–879.

[8] Buyya, R., Yeo, C. S., and Venugopal, S.Market-oriented cloud computing: Vision, hype,and reality for delivering it services as computingutilities. In Proceedings of the 10th IEEEInternational Conference on High PerformanceComputing and Communications (2008), IEEE,pp. 5–13.

[9] Common Crawl. Common Crawl Corpus. onlineat: http://commoncrawl.org/ (2015).

[10] Dogar, F. R., Karagiannis, T., Ballani, H.,and Rowstron, A. Decentralized task-awarescheduling for data center networks. InProceedings of ACM SIGCOMM (2014).

[11] Fang, X., Misra, S., Xue, G., and Yang, D.Smart gridNthe new and improved power grid: Asurvey. IEEE Communications Surveys &Tutorials 14, 4 (2012), 944–980.

[12] Feng, G., Garg, S., Buyya, R., and Li, W.Revenue maximization using adaptive resourceprovisioning in cloud computing environments. InProceedings of the 2012 ACM/IEEE 13thInternational Conference on Grid Computing(2012), pp. 192–200.

[13] Friedman, E. J., and Parkes, D. C. Pricingwifi at starbucks: issues in online mechanismdesign. In Proceedings of the 4th ACM conferenceon Electronic commerce (2003), ACM,

pp. 240–241.[14] Ghodsi, A., Zaharia, M., Hindman, B.,

Konwinski, A., Shenker, S., and Stoica, I.Dominant resource fairness: Fair allocation ofmultiple resource types. In Proceedings of NSDI(2011), vol. 11, pp. 24–24.

[15] Guenter, B., Jain, N., and Williams, C.Managing cost, performance, and reliabilitytradeoffs for energy-aware server provisioning. InProc. of IEEE INFOCOM (2011), IEEE,pp. 1332–1340.

[16] Ha, S., Sen, S., Joe-Wong, C., Im, Y., andChiang, M. TUBE: time-dependent pricing formobile data. In Proceedings of ACM SIGCOMM(2012).

[17] Javadi, B., Thulasiram, R. K., and Buyya,R. Statistical modeling of spot instance prices inpublic cloud environments. In Proc. of IEEE UCC(2011), pp. 219–228.

[18] Jia, J., Zhang, Q., Zhang, Q., and Liu, M.Revenue generation for truthful spectrum auctionin dynamic spectrum access. In Proc. of ACMMobiHoc (2009), pp. 3–12.

[19] Jin, H., Wang, X., Wu, S., Di, S., and Shi,X. Towards optimized fine-grained pricing of iaascloud platform. IEEE Transactions on CloudComputing PP (2014).

[20] Kang, L., and Parkes, D. C. A decentralizedauction framework to promote efficient resourceallocation in open computational grids. In Proc.Joint Workshop on The Economics of NetworkedSystems and Incentive-Based Computing (2007).

[21] Lampe, U., Hans, R., Seliger, M., andPauly, M. Pricing in infrastructure clouds–ananalytical and empirical examination. InAssociation for Information Systems Conference(2014).

[22] Lavi, R., and Nisan, N. Online ascendingauctions for gradually expiring items. InProceedings of the sixteenth annual ACM-SIAMsymposium on Discrete algorithms (2005), Societyfor Industrial and Applied Mathematics,pp. 1146–1155.

[23] Lee, G., Chun, B.-G., and Katz, R. H.Heterogeneity-aware resource allocation andscheduling in the cloud. Proceedings of HotCloud(2011), 1–5.

[24] Leonardi, E., Mellia, M., Neri, F., andAjmone Marsan, M. Bounds on average delaysand queue size averages and variances ininput-queued cell-based switches. In Proceedingsof IEEE INFOCOM (2001), vol. 2, IEEE,pp. 1095–1103.

[25] MacKie-Mason, J., and Varian, H. Pricingthe Internet. In Public Access to the Internet,

13

B. Kahin and J. Keller, Eds. Prentice-Hall,Englewood Cliffs, NJ, 1995.

[26] Menache, I., Ozdaglar, A., and Shimkin, N.Socially optimal pricing of cloud computingresources. In Proceedings of the 5th InternationalICST Conference on Performance EvaluationMethodologies and Tools (2011), ICST (Institutefor Computer Sciences, Social-Informatics andTelecommunications Engineering), pp. 322–331.

[27] Ng, C., Parkes, D. C., and Seltzer, M.Virtual worlds: Fast and strategyproof auctionsfor dynamic resource allocation. In Proceedings ofthe 4th ACM Conference on Electronic Commerce(2003), ACM, pp. 238–239.

[28] Poola, D., Ramamohanarao, K., andBuyya, R. Fault-tolerant workflow schedulingusing spot instances on clouds. ProcediaComputer Science 29 (2014), 523–533.

[29] Porter, R. Mechanism design for onlinereal-time scheduling. In Proceedings of the 5thACM conference on Electronic commerce (2004),ACM, pp. 61–70.

[30] Ren, S., Park, J., and Van Der Schaar, M.Entry and spectrum sharing scheme selection infemtocell communications markets. IEEE/ACMTransactions on Networking 21, 1 (2013),218–232.

[31] Sen, S., Jin, Y., Guerin, R., andHosanagar, K. Modeling the dynamics ofnetwork technology adoption and the role ofconverters. IEEE/ACM Transactions onNetworking 18, 6 (2010), 1793–1805.

[32] Urgaonkar, B., Kesidis, G., Shanbhag,U. V., and Wang, C. Pricing of service inclouds: optimal response and strategicinteractions. ACM SIGMETRICS PerformanceEvaluation Review 41, 3 (2014), 28–30.

[33] Van den Bossche, R., Vanmechelen, K.,and Broeckhove, J. Cost-optimal scheduling inhybrid iaas clouds for deadline constrainedworkloads. In Proceedings of the IEEEInternational Conference on Cloud Computing(2010), IEEE, pp. 228–235.

[34] Wang, P., Qi, Y., Hui, D., Rao, L., and Liu,X. Present or future: Optimal pricing for spotinstances. In Proc. of IEEE ICDCS (2013).

[35] Wang, Q., Ren, K., and Meng, X. Whencloud meets ebay: Towards effective pricing forcloud computing. In Proc. of IEEE INFOCOM(2012), IEEE, pp. 936–944.

[36] Yi, S., Andrzejak, A., and Kondo, D.Monetary cost-aware checkpointing and migrationon amazon cloud spot instances. IEEE Trans. onServices Computing 5, 4 (2012), 512–524.

[37] Zhang, L., Li, Z., and Wu, C. Dynamicresource provisioning in cloud computing: Arandomized auction approach. In Proc. of IEEEINFOCOM (2014).

[38] Zhou, Y., and Wentzlaff, D. The sharingarchitecture: sub-core configurability for iaasclouds. In Proceedings of the 19th internationalconference on Architectural support forprogramming languages and operating systems(2014), ACM, pp. 559–574.

14