Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=teis20

Download by: [Tsinghua University] Date: 17 September 2015, At: 00:34

Enterprise Information Systems

ISSN: 1751-7575 (Print) 1751-7583 (Online) Journal homepage: http://www.tandfonline.com/loi/teis20

SLA-based optimisation of virtualised resource formulti-tier web applications in cloud data centres

Jing Bi, Haitao Yuan, Ming Tie & Wei Tan

To cite this article: Jing Bi, Haitao Yuan, Ming Tie & Wei Tan (2015) SLA-based optimisationof virtualised resource for multi-tier web applications in cloud data centres, EnterpriseInformation Systems, 9:7, 743-767, DOI: 10.1080/17517575.2013.830342

To link to this article: http://dx.doi.org/10.1080/17517575.2013.830342

Published online: 02 Sep 2013.

Submit your article to this journal

Article views: 136

View related articles

View Crossmark data

Citing articles: 1 View citing articles

SLA-based optimisation of virtualised resource for multi-tier webapplications in cloud data centres

Jing Bia*, Haitao Yuanb, Ming Tiec and Wei Tand

aDepartment of Automation, Tsinghua University, Beijing 100084, China; bDepartment of ComputerScience, City University of Hong Kong, Kowloon, Hong Kong; cBeijing Institute of Near SpaceVehicle’s System Engineering, Beijing 100076, P.R. China; dIBM T. J. Watson Research Center,

Yorktown Heights, NY 10598, USA

(Received 1 October 2012; accepted 27 July 2013)

Dynamic virtualised resource allocation is the key to quality of service assurance formulti-tier web application services in cloud data centre. In this paper, we develop aself-management architecture of cloud data centres with virtualisation mechanism formulti-tier web application services. Based on this architecture, we establish a flexiblehybrid queueing model to determine the amount of virtual machines for each tier ofvirtualised application service environments. Besides, we propose a non-linear con-strained optimisation problem with restrictions defined in service level agreement.Furthermore, we develop a heuristic mixed optimisation algorithm to maximise theprofit of cloud infrastructure providers, and to meet performance requirements fromdifferent clients as well. Finally, we compare the effectiveness of our dynamic alloca-tion strategy with two other allocation strategies. The simulation results show that theproposed resource allocation method is efficient in improving the overall performanceand reducing the resource energy cost.

Keywords: performance optimisation; dynamic resource allocation; virtualisation;cloud data centre; service level agreement (SLA); multi-tier web application

1. Introduction

Recently, large-scale cloud data centres (CDCs) have drawn significant attentions, as theyprovide infrastructure services to host many third-party application services concurrentlyin cloud computing platforms (Iosup et al. 2011; Cao, Zhang, and Tan 2012; Zuo, Zhang,and Tan 2013; Li, Xu, and Wang 2013; Hayes 2008; Lama and Zhou 2012; He and Xu2013). The cloud infrastructure providers (CIPs) and their clients (the third-party applica-tion service providers (ASPs)) negotiate utility to determine the revenues and penalties onthe base of the achieved performance level in the service level agreement (SLA). Theterms of performance metrics to be provided and the constraints which include responsetime, throughput, availability, revenue and penalty, are specified in SLA. The CIPs neednot only to maximise the resource utilisation and profit, but also to ensure the SLArequirements of clients. Therefore, the CIPs should comply with the SLA constraints, andclients pay for CIPs according to the specified SLA.

CDCs provide services to many clients by sharing the IT resources distributed indifferent places. To ensure the efficient use of resources, virtualisation technologies havebeen proposed as the useful solution for realising resources sharing in the CDCs (Reed

*Corresponding author. Email: bijing@tsinghua.edu.cn

Enterprise Information Systems, 2015Vol. 9, No. 7, 743–767, http://dx.doi.org/10.1080/17517575.2013.830342

© 2013 Taylor & Francis

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

et al. 1999; Kusic et al. 2008; Verma, Ahuja, and Neogi 2008; Liu et al. 2009; Li et al.2013). With virtualisation technologies, the CDCs host multiple applications in virtualisedenvironments. Physical resource can be partitioned into multiple isolated virtual machines(VMs), and each VM can support the isolation of applications from the underlyinghardware. However, one main issue of the dynamic resource allocation in cloud environ-ments is the high variability of the workloads, which makes it difficult to estimate theresource requirement in advance. In order to efficiently utilise resources with satisfyingthe SLA under fluctuating workloads, we present a multi-tier architecture based on theautonomic computing techniques in the CDCs.

Nevertheless, there are many open challenges involved in dynamic on-demandresources allocation for each tier of the CDCs. Most of existing methods cannotsufficiently adapt to complex cloud environments. These research efforts usuallyassume the system at the equilibrium state, and employ the method of average valueanalysis which is not sufficiently precise (Kalyvianaki, Charalambous, and Hand2009). Traditional models include the static resources allocation strategy (Stat-RA)and the dynamic resources allocation strategy based on physical machines (DPM-RA).Iqbal, Dailey, and Carrera (2009) have shown that the static allocation policy cannotshift dynamically servers when peak workload occur while meeting contracted SLAguarantees. Urgaonkar et al. (2005) propose physical resources allocation mode, butthe precondition is that available resources are always sufficient in any case. Webelieve that in order to solve the above problems and meet performance requirementsspecified in SLA, IT resources (e.g., CPU, memory, disc and network) should bevirtualised and partitioned among the resident VMs to realise optimal configuration.In this paper, in order to meet clients’ requirements and maximise the total profit of theCIPs, we focus on the problem of virtualised resources allocation for the existingcentralised CDC.

The main contribution of this paper is described as follows. First, we develop anautonomic management architecture based on virtualisation mechanism for multi-tierweb applications in the CDCs to enhance resources allocation. The architecture canguarantee that virtualised application service environments (VASEs) execute effectively,and meet SLA clients’ requirements. Second, we propose a flexible hybrid queue modelbased on the CDCs architecture, and we further fully consider response time, through-put, revenue, cost and penalty specified in SLA. Third, we propose and formulate a non-linear constrained optimisation problem. In order to solve the problem, we adopt aheuristic algorithm, which aims at maximising the total profit of the CIPs. The algorithmcan allocate the virtualised resources according to the workload variations dynamically.Finally, we evaluate the effectiveness of dynamic resources allocation strategy based onVMs (DVM-RA) by comparing with two other existing strategies including Stat-RA andDPM-RA. The simulation result shows that the strategy can increase CPU resourceutilisation by more than 50%.

The remaining of the paper is organised as follows. Section 2 discusses the relatedwork. Section 3 gives the overall VMs-based architecture design of the CDCs. Section 4formulates the allocation problem of virtualised resources. Section 5 proposes the concretealgorithm for the dynamic allocation strategy. In addition, we discuss and compare severaldifferent resources allocation strategies. Section 6 gives the performance evaluation resultsand the corresponding analysis. Finally, Section 7 provides the conclusion of the researchand the direction of future work.

744 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

2. Related works

In this section, we review some prior works related to this paper as follows, and alsopresent the main difference between these researches and ours.

Dynamic resource allocation optimisation has been applied to virtualised multi-tierserver clusters. Compared with the dynamic resource allocation, the static allocationpolicy cannot serve dynamically varying workloads demand while meeting the contractedSLA guarantees (Iqbal, Dailey, and Carrera 2009). Slegers, Mitrani, and Thomas (2009)have shown that a dynamic allocation policy for the changing workloads and shifts serverswhen the peak workload occurs, can lower cost than the static allocation policy.

For example, Urgaonkar et al. (2005) provide important insights on dynamic physicalresources allocation for multi-tier clusters. However, it employs physical resources alloca-tion mode, and assumes that available resources are always sufficient in any case. Lamaand Zhou (2012) propose an efficient server allocation approach based on an end-to-endresource allocation optimisation model. The integration of the optimisation model with itsindependent fuzzy controller provides superior performance in the resource utilisationefficiency and the end-to-end response time guarantee. However, the approach does notprovide trivial heterogeneous server configuration in virtualised multi-tier systems.Besides, it does not consider the total profit in the system and clients’ different perfor-mance requirements. Kalman filters have been adopted to track and control the CPUutilisation in virtualised environments in order to dynamically allocate CPU resources toVMS hosting server applications (Kalyvianaki, Charalambous, and Hand 2009). It pre-sents a new resource management scheme that integrates the Kalman filter with feedbackcontrollers which can continuously detect and self-adapt to the unpredictable changes ofworkload intensity. However, only a multi-tier application in a single physical host hasbeen considered. Zhu et al. (2009) have presented a multi-tier and multi-time scalesolution for the automated capacity and workload management of virtualised systems,but they do not consider the trade-off between performance and system cost. Zhang et al.(2002) present a non-linear integer optimisation model for determining the number ofmachines at each tier in a multi-tier server network. Similar to ours, they profile thecomputing resource of data centres in physical servers’ environment while we profile thecomputing resource of cloud environment in VMs environment. So our approach cansupport a fine-grained resource allocation and management for a virtualised application incloud environment. Additionally, their approach uses a simply open queueing networkmodel at each server, which is less accurate than the hybrid queueing model we used.Moreover, in order to find an optimised initial solution, we propose an optimal method forVMs allocation. The optimal method can provide the efficient CPU utilisation computa-tion of the virtualised applications in each tier.

Furthermore, SLAs have been applied in resource allocation especially when the totalprofit in the system depends on how to meet these SLAs. For example, Cardellini et al.(2011) formulate the ASPs resource management as an optimisation problem and proposeboth reactive and proactive heuristic policies that approximate the optimal solution.However, the proposed resource optimisation problem just focuses on single-tier applica-tions. Goudarzi and Pedram (2011) propose an SLA-based resource allocation problem formulti-tier applications in cloud computing environments. They consider the processing,memory requirement, and communication resources as three dimensions. However, itadopts a single and simple M=M=1 queueing system which cannot reflect the real cases.Different from the above approaches, every virtualised resource in our work is allocated asan allocation unit for dynamic adjustment according to the variation of workload. Besides,

Enterprise Information Systems 745

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

the virtualised resources can shift across different application services, so the flexibility andthe scalability in addressing SLA management are both improved. Wu, Garg, and Buyya(2011) propose resource allocation algorithms for SaaS providers to minimise the infra-structure cost and SLA violations in cloud computing environments. However, the worksfocus on client driven SLA-based profit maximisation resource allocation for SaaS provi-ders. Yarmolenko and Sakellariou (2006) evaluate various SLA-based scheduling heuris-tics on parallel computing resources, and adopt resource (the number of CPU nodes)utilisation and income as the evaluation metrics. Nevertheless, our work focuses on thevirtualised resources dispatching in cloud computing environments.

3. VM-based cloud data centre

3.1. Architecture overview

The architecture of a cloud data centre is shown in Figure 1, which is a multi-tierarchitecture including heterogeneous physical servers clusters and VMs clusters. Theclusters are shared by multiple independent VASEs hosting the third-party web applica-tions. Modern web applications are often distributed across different VM-based servers.

In Figure 1, business processes as a service (BPaaS) or software as a service (SaaS)consumers (the third-party ASPs) sign SLA agreement with the CIPs. In order to max-imise profit, the CIPs should adopt the effective mechanism to meet their clients’ (BPaaSor SaaS consumers) specific requirements. Besides, the CIPs’ final objective is to designthe most cost-effective strategy to manage their available resources.

Figure 1. Cloud data centre architecture.

746 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

In order to allocate virtualised resources reasonably, we design an on-demand dis-patcher (ODD). The ODD adopts a dynamic resource allocation strategy based on VMs todispatch requests in the CDC. To avoid ODD to be the bottle neck of the CDC, VMs-based multiple dispatchers run in ODD. The dispatchers adopt self-adaptive workloadbalancing algorithms. Based on the algorithms, the dispatchers can allocate requests fromdifferent clients to ensure that every admitted request can be executed on a VM in ODD.Furthermore, the dispatchers give full consideration to the factors like the latest workloadand requests allocation of VMs.

In addition, one of the important characteristic of the CDC architecture is the high-efficiency reuse of virtualised service resources. When a client requires resources, it can getcorresponding virtualised service resources from the dynamic VMs pool. Correspondingunoccupied virtualised service resources will be recycled into the resources pool to bereused by other clients. In this way, ODD can make use of virtualised service resourcesefficiently and adjust resources dynamically according to the variation of workloads. In thispaper, we only consider a centralised CDC architecture.

Then, we design a VM manager based on the autonomic management technology inSection 3.2, which provides the information including workload, response time, etc. TheVM manager reports regularly the latest information to the CDC. Then the CDC candecide to turn on/off VMs according to the latest workload of the system. Therefore, ODDcan maximise the total profit of the CIPs as well as meet clients’ performancerequirements.

3.2. Autonomic resource management

Due to the large variation in the workloads of VASEs, it is difficult to estimate theworkload requirements in advance. In addition, it is either infeasible or extremelyinefficient to plan the capacity for the worst case. In order to dynamically allocateresources for multi-tier VASEs, the most common approaches are based on Monitor,Analyse, Plan and Execute (MAPE) control loops (White et al. 2004; Kephart and Chess2003). Autonomic systems adjust their operations according to the changing components,demands or external conditions. The systems can dynamically allocate resources toapplications of different clients on the base of short-term demand estimates. The objectiveis to meet the application requirements while adapting IT architecture to the workloadvariations. The architecture of a high-level autonomic computing is shown in Figure 2.The architecture includes a set of heterogeneous physical servers hosting a VirtualMachine Monitor (VMM). The VMM is shared by multiple independent applicationenvironments. Modern VASEs are usually designed with multiple tiers, which are oftendistributed across different servers. The physical resources of servers are partitionedamong multiple VMs, each of which hosts a single VASE. We choose CPU as the criticalfactor for the resource allocation problem. The resource allocator exploits the VMM APIto dynamically partition CPU capacity among multiple VMs and their hosted VASEs.

In order to maximise the CIPs’ profit while maintaining the response time require-ments of different clients, the autonomic computing architecture provides mechanisms toautomate the configuration of VMs. The architecture generates the run-time allocation ofVMs for the CDC. It includes four components as follows: (1) The dispatcher sets upallocation strategy for each tier of the VASE, and determines resource allocation accordingto the SLA with optimisation algorithms. It determines the processing capacity of VMwith the workload estimation and the response time of different clients to optimise theresource allocation of the overall VASE. (2) The monitor collects the workload and the

Enterprise Information Systems 747

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

performance metric of all running VASEs, such as the arriving rate of requests, theaverage service time, and the CPU utilisation. (3) The analyser receives and analysesthe measurements from the monitor to estimate the future workload. It also receives theresponse time of different clients. (4) The executor initiates the VASEs according to theVM configuration to meet the resource requirements of the different clients. In this paper,we mainly consider the dispatcher component in the autonomic computing architecture.

3.3. Problem formulation

We formulate the problem of virtualised resources allocation in the CDC now. We assumethat N Mi;k-tier VASEs run in the CDC to serve K different classes. K denotes the numberof classes. The capacities of physical servers in each tier are shared by the VMs. There areni; j available VMs in the jth tier of VASE i. The crucial variable is defined as aN � ðMi;k þ 1Þ (including the 0th tier) matrix, ConfigMAT, which denotes the allocationsolution of VMs in each tier. Therefore, ConfigMAT can be formulated as

ConfigMAT ¼

c1;0 c1;1 � � � c1;Mi;k

c2;0 c2;2 � � � c2;Mi;k

..

. ... . .

. ...

cN ;0 cN ;2 � � � cN ;Mi;k

2666437775

where ci;j denotes the number of active VMs in the jth tier of VASE i. If ci;j is equal to 0, itmeans that no active VMs exist in the jth tier of VASE i. In order to control the granularityof VMs allocation, the upper limit of ci;j is set as Ci. Ci means the maximum number ofVMs for VASE i in the CDC. ni; j denotes the maximum available number of VMs in thejth tier of VASE i. We consider the allocation matrix ConfigMAT valid iff

App server DB serverWeb server

Virtual Machine MonitorVMM

Physicalinfrastructure

OS

VASE1

OS

VASE2

OS

VASEn

VM1 VM2 VMn

Executor

Analyser

Dispatcher

(4) New VMconfiguration

(1) SLA and systemparameters

(2) Workloadmonitoring

Virtualmachines

Monitor

Autonomic computing loop (3) Predictedworkload

Figure 2. High-level autonomic computing architecture.

748 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

0<PMi;k

j¼0ci; j � Ci ; "i 2 ½1 ; N �

0 � ci; j � ni; j ; "i 2 ½1 ; N � ; "j 2 ½0 ; Mi;k �

8<: (1)

The constraint (1) shows that the total number of VMs and the number of VMs of thesame tier are both restricted by corresponding physical resources in the CDC.

Here, let SLAi denote the performance class of VASE i specified in SLA. Each VASE ihas its corresponding local profit function definition, Pi

Pi ¼ f ðλi; ci;0; ci;1; . . . ; ci;Mi;k ; SLAiÞThe total profit Pg is the function of every local profit of VASE, therefore the whole

optimisation problem (P1) can be formulated as

max Pg ¼ gðP1;P2; . . . ;PN Þ� �

(2)

The total profit in (2) should be maximised with the constraint of (1). Section 4 willgive the formalisation of problem (P1) in its latter half.

We summarise the main notations used throughout this paper in Table 1 for clarity.

4. System performance model

In this section, in order to maximise the total profit of the CIPs, we first establish thehybrid optimisation performance model for VASEs in the CDC. Then we define andformulate the non-linear constrained optimisation problem for dynamic virtualisedresources optimisation. Furthermore, we present an optimisation-based resource allocationmodel.

4.1. The system model

In order to allocate the CDC resources dynamically according to the requirements ofclients, this paper proposes a dynamic VMs allocation model based on the hybridqueueing networks, as shown in Figure 3. This section mainly focuses on the analysisof online web application services, so the response time is viewed as the main perfor-mance metric in VASEs.

In typical multi-tier software architecture, the cross-tier access is prohibited and notadvisable. Each tier only provides the public interface to the previous tier and sendscorresponding requests to the next tier. However, we adopt the hybrid queueing networkmodel for multi-tier systems, which can trace the actions of every tier in VASEs, such asHTTP, J2EE and Database tier. In Figure 3, Tier 0 shows VM-based dispatchers in theCDC. The rest of tiers indicate multi-tier web applications. We adopt the principle of thepriority-based FCFS (First-Come-First-Served) in the queueing networks. Client sessionsbegin with class k requests arriving in the CDC from an exogenous source with the rate ofλ�i;k . The solid lines represent forward requests while the dashed lines show backwardrequests. Each VM in every tier has two queues which represent forward and backwardrequests. In the trace analysis of actual network business website (Menasce and Bennani2006), the actual network workloads conform to Poisson distribution. So we assume thatthe arriving requests occur in a Poisson process. It also means that the interarrival timeexhibits an exponential distribution, and every interaction of a client with the system is aseparate request which is independent of others.

Enterprise Information Systems 749

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

Incomingrequests

Returnrequests

λi,k,1λi,k,2 λi,k,Mi.k

λ∗i,k

Tier 0 Tier 1 Tier Mi,kTier 2

pi,k,1

pi,k,Mi.k

1–pi,k,1(un) 1–pi,k,2

(un) 1–pi,k,Mi.k

(un)

Dispatcher

Figure 3. Multi-tier system model.

Table 1. Parameters and decision variables.

System parameters

N Number of VASEsMi;k Number of tiers corresponding to the class k requests of VASE iK Number of classesCi Maximum number of VMs of VASE i in the CDCci; j Active number of VMs in the jth tier of VASE ini; j Available number of VMs in the jth tier of VASE ieλi;k; j Arriving rate of the class k requests in the jth tier of VASE iλi;k; j;w Arriving rate of the class k requests in the VM w in the jth tier of VASE iRi;k Server-side response time of the class k requests of VASE ieRi;k; j Overall response time of the class k requests in the jth tier of VASE iμi;k; j;w Serving rate of the class k requests in the VM w in the jth tier of VASE iPi Local profit of VASE iPg Total profit of the CDCbai;k; j;w Unit cost of active VM w in the jth tier of the class k requests of VASE ibsi;k; j;w Unit cost of spare VM w in the jth tier of the class k requests of VASE iAVi;k Availability of VMs for the class k requests of VASE iAVi;k; j Availability of VMs for the class k requests in the jth tier of VASE iFVi;k; j Failure probability of VMs for the class k requests in the jth tier of VASE iΛi;k Aggregate arriving rate of the class k requests in VASE i (equivalent to eλi;k;0)αj Amplification factor of the jth tier, j=1,2, . . . ;Mi;k

pi;k; j Probability of the case in which the class k requests restart from the 0th tierafter completion in the jth tier of VASE i

pðunÞi;k; j Probability of the class k requests that will continue to be servedAi;k; j;w=1 Class k requests can be executed in the jth tier of VASE i, otherwise

Ai;k; j;w= 0di;k Penalty of per rejected class k request of VASE ixi;k Number of rejected class k requests of VASE i

SLA parameters

Ri;k Expected response time of the class k requests of VASE i in SLAui;k Utility function value of the class k requests of VASE iAV i;k Expected VM availability of the class k requests of VASE i in SLA

Decision variables

ci;j Number of VMs allocated in the jth tier of VASE i

750 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

Here, Λi;k denotes the aggregate arriving rate of the class k requests of VASE i,Λi;k = eλi;k;0. pi;k; j denotes the probability of the case in which the class k requests restartfrom the 0th tier after completion in the jth tier of VASE i. pðunÞi;k; j denotes the probability ofthe class k requests that will continue to be served. So, pðunÞi;k;j � pi;k;j denotes the probabilityof the case in which the class k requests continue to arrive in the j+1th tier of VASE i afterthe completion of service in the jth tier. For each tier, a request may result in three possiblecases. First, it returns to the system with probability pi;k; j, which is shown by bold dashedlines. Second, it ends with probability of 1� pðunÞi;k; j and leaves the system in the reverseorder. Third, it continues to arrive in the j+1th tier with probability pðunÞi;k; j � pi;k; j. Inaddition, the real-world multi-tier web applications are more complex because there maybe multiple subsequent interactions between adjacent tiers. To make the proposed modelmore close to the reality, this paper presents amplification factor αj for the jth tier,j = 1,2, . . . ;Mi;k , whose value can be set according to the previous records.

The aggregate arriving rate of the class k requests, denoted by Λi;k, is calculated by

Λi;k ¼ eλi;k;0 ¼ λ�i;k þXMi;k

j¼1

eλi;k; j � αj � pi;k; j (3)

Then, eλi;k;1 ¼ pðunÞi;k;0 � eλi;k;0, eλi;k;2 ¼ pðunÞi;k;1 � pi;k;1� �

� eλi;k;1 � α1, . . . , eλi;k;Mi;k = ðpðunÞi;k;Mi;k�1 �pi;k;Mi;k�1Þ � eλi;k;Mi;k�1 � αMi;k�1, i.e., eλi;k;j ¼ ðpðunÞi;k;j�1 � pi;k;j�1Þ � eλi;k;j�1 � αj�1 and pðunÞi;k;0 ¼ 1,

pi;k;0 ¼ 0, 0 � pðunÞi;k;j�1 � 1, pi;k;Mi;k ¼ pðunÞi;k;Mi;k, "j 2 ½1;Mi;k �.

Let temp ¼ PMi;k

j¼2ðαj � pi;k;j �

Qj�1

q¼1αqðpðunÞi;k;q � pi;k;qÞÞ, then we can calculate the aggregate

arriving rate of the class k requests in VASE i by

eλi;k;0 ¼ λ�i;k1� α1 � pi;k;1 � temp

(4)

Here, we describe the ODD as a M=M=c system model, in which there are cdispatchers for VMs. The service time of each dispatcher conforms to different negativeexponential distribution. And the number of clients’ requests is unlimited. Based on thequeueing network model and the assumption that there are multiple available and unoc-cupied dispatchers for VMs, the ODD can always dispatch requests to VMs-baseddispatchers whose processing speed are the fastest. The effective utilisation rate ofODD is set as 60–80%. In order to establish an approximate model, we sort all hetero-geneous dispatchers for VMs by the serving rate in descending order, denoted byμi;k;0;1; μi;k;0;2; . . . ; μi;k;0;ci;0 . Based on the birth and death state equilibrium equations ofMarkov processes (Hernández-Suárez and Castillo-Chavez 1999), the probability of thecase that s requests exist in the dispatchers of the ODD of VASE i can be obtained by

ps ¼

ðeλi;k;0ÞsQsν¼1

aðνÞ� p0; "1 � s � ci;0

ðeλi;k;0Þsaðci;0Þs�ci;0

Qci;0ν¼1

aðνÞ� p0; "s � ci;0

8>>>>><>>>>>:(5)

Enterprise Information Systems 751

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

where aðνÞ ¼Pνq¼1 μi;k;0;q ¼ μi;k;0;1 þ μi;k;0;2 þ . . .þ μi;k;0;ν, "i 2 ½1;N �, "k 2 ½1;Ki�.

Based on the definition of probability,P1

s¼0 ps= 1. The probability that the dispatchersare the spare state of VASE i in the ODD can be calculated as

p0 ¼� Xci;0�1

s¼0

ðeλi;k;0ÞsQsv¼1

aðνÞ

�þ ðeλi;k;0Þci;0Qci;0

ν¼1aðνÞ

� 1

1� eλi;k;0aðci;0Þ

0BBB@1CCCA

�1

(6)

Based on the Little’s law (McKenna 1988), the average response time of the ODD inVASE i is shown as

Ri;k;0 ¼ðeλi;k;0Þci;0�1 � ðρi;k;0 þ ci;0 � ci;0ρi;k;0Þ

ð1� ρi;k;0Þ2Qci;0ν¼1

aðνÞ� p0 þ 1eλi;k;0 �

Xci;0�1

s¼0

ðs � psÞ (7)

where j = 0, ρi;k;0 ¼eλi;k;0aðci0Þ<1, which denotes the utilisation rate of VM-based dispatchers of

the ODD in VASE i.Then, we establish multiple M=M=1 performance models for each tier in multi-tier

VASE i. We can divide arriving requests into multiple portions by the approach ofembedding Markov chain (Meyer, Smythe, and Walsh 1972). The approach can ensurethat the expected average response time are equal in the same tier. We assume that clients’requests arrive in the VM w at the rate of λi;k; j;w, 1 � w � ci; j, so the following can beobtained by

eλi;k;j ¼ ðpðunÞi;k;j�1 � pi;k;j�1Þ � eλi;k;j�1 � αj�1

Ri;k;j;w ¼ 1μi;k;j;w�λi;k;j;w

λi;k;j;1 � λi;k;j;2 � � � � � λi;k;j;ci;jμi;k;j;1 � μi;k;j;2 � � � � � μi;k;j;ci;j

8>>>>><>>>>>:(8)

We can calculate the value of average response time in each tier of VASE i easily

from (8),1 � j � Mi;k . We assume that eλi;k;j is divided into λi;k;j;1, λi;k;j;2, . . . , λi;k;j;ci;j , such

that λi;k;j;1� λi;k;j;2 � � � � � λi;k;j;ci;j , then eλi;k;j=ðpðunÞi;k;j�1 � pi;k;j�1Þ � eλi;k;j�1 � αj�1. Further-

more, the processing abilities of VMs in the same tier are different, s.t., μi;k;j;1 �μi;k;j;2 � � � � � μi;k;j;ci;j . Besides, the processing abilities of VMs from different tiers are

unequal, s.t., μi;k;1;w � μi;k;2;w � � � � � μi;k;Mi;k ;w (i.e., the processing ability of VMs in the

web tier is less than or equal to that of VMs in the App tier). ρi;k;j=eλi;k;jPci;j

w¼1μi;k;j;w

<1 is the

utilisation rate of resources (e.g., CPU, memory, I/O) allocated to VMs in each tier ofVASE i. The paper mainly considers the utilisation of resources from the point of CPU.When the utilisation rate of CPU exceeds a specific threshold (85% in this paper), theanalyser will trigger dispatcher in the ODD. Then the ODD executes a VM migration toensure the SLA constraints.

752 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

4.2. Optimisation problem formulation

In order to maximise Pg, we present a profit function in this paper. Here, we focus on themultiple online VASEs in the CDC. The arriving rate eλi;k;j denotes the workload intensityof class k requests in the jth tier of VASE i. The server-side response time Ri;k isconsidered as the performance metric. We assume that providers and clients negotiatethe SLA contract before the execution of system. The SLA specifies items about theperformance constraints and the cost as follows:

● Ri;k – Expected response time of the class k requests of VASE i in SLA. If a requestis served in Ri;k , the positive profit contributes for the CIPs, i.e., if Ri;k � Ri;k , SLAi

is the revenue type. Otherwise, SLAi is the penalty type.● AV i;k – Expected availability of VM for the class k requests of VASE i. If

AVi;k � AV i;k , client will accept the services of the CIPs. Otherwise, client willnot accept services.

● Ci – Maximum number of VMs of VASE i in the CDC. IfPMi;k

j¼0 ci;j � Ci, therefused clients’ requests will lead to di;k penalty. Otherwise, the CIPs only provideCi VMs. When the actual number of VMs exceeds the planned upper limit, therefused clients’ requests will not be counted into penalty. This needs clients tocarefully estimate the actual requirements of application services in advance and tomake an appropriate cost plan before the deployment of application services.

● bai;k;j;w – Unit cost of active VM w in the jth tier of the class k requests of VASE i.● bsi;k;j;w – Unit cost of spare VM w in the jth tier of the class k requests of VASE i.

We will describe this profit function in the following part. The local profit function ofevery VASE i is determined by the revenue, the sum of penalty, the loss and cost of VMs.Thus, the local profit function in this paper can be calculated as

PiðEÞ ¼ RevenueðEÞ � PenaltyðEÞ � LossðEÞ � CostðEÞ (9)

Our final objective is to maximise the total profit of the CIPs formulated in Equation(10). Furthermore, the revenue, penalty, loss and VM cost from SLA contract canbe maximised during the interaction of dispatchers. The total profit function can beformulated as

PgðEÞ ¼XNi¼1

XKi

k¼1

fΛi;k � ðð�mi;kÞ � Ri;k þ ui;kÞ � ðdi;k � xi;kÞ � ðLVi;k � ð1� AVi;kÞÞg

�XNi¼1

XKi

k¼1

XMi;k

j¼0

ðXci;jw¼1

bai;k;j;w þXni;j�ci;j

w¼1

bsi;k;j;wÞ(10)

● Λi;k is the arriving rate of the class k requests of VASE i.● mi;k ¼ ui;k

Ri;k> 0 denotes the slope of utility function ui;k .

● ui;kðxÞ ¼ bestVal�xbestVal�worstVal 2 ½0 � � � 1�, where x ¼ Ri;k ; bestVal ¼ 0;worstVal ¼ Ri;k :

● Ri;k is the server-side response time of the class k requests of VASE i, which can beformulated as

Enterprise Information Systems 753

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

Ri;k ¼ 1

Λi;kðeλi;k;0 � eRi;k;0 þ

XMi;k

j¼1

Xci;jw¼1

λi;k;j;w � Ri;k;j;wÞ:

● di;k denotes the penalty of per rejected class k request in VASE i.● xi;k denotes the number of rejected class k requests in VASE i.● AVi;k denotes the availability of VMs for the class k requests in VASE i.

AVi;k ¼YMi;k

j¼0

ð1� FVi;k;jÞ ¼YMi;k

j¼0

AVi;k;j

where FVi;k;j ¼ DTi;k;jUTi;k;jþDTi;k;j

is the failure probability of the jth tier; AVi;k;j ¼UTi;k;j

UTi;k;jþDTi;k;jis the availability of the jth tier; UTi;k;j is the available duration time of

VMs in the jth tier; DTi;k;jis the failure duration time of VMs in the jth tier.● LossðEÞ is the total loss of the whole CIPs, which is denoted as

LossðEÞ ¼XNi¼1

XKi

k¼1

LV ðEÞi;kð1�YMi;k

j¼0

ðAVi;k;jÞÞ

The failure probability of VMs for the class k requests in the jth tier of VASE i isdenoted with FVi;k;j, and the availability is AVi;k;j, s.t. FVi;k;j þ AVi;k;j ¼ 1. Inaddition, we assume that faults inside a VASE will eventually affect all VASEs.The loss brought by the faults of the class k requests in VASE i is denoted withLV ðEÞi;k (E denotes the evaluation period defined in SLA).

● CostðEÞ is the total cost of the CIPs, which can be formulated as

CostðEÞ ¼XNi¼1

XKi

k¼1

XMi;k

j¼0

ðXci;jw¼1

bai;k;j;w þXni;j�ci;j

w¼1

bsi;k;j;wÞ

where a cluster in each tier of VASE i consists of ni;j VMs. In order to serve thearriving requests workload, ci;jVMs are active. In case of the unpredictable outburstof requests workload, ni;j � ci;jVMs are spare. They can be quickly started to meetthe increase of requests workload to ensure the availability of the CDC.

So the problem that we intend to solve can be briefly summarised as follows:

maxλi;k;j;w;μi;k;j;w;ci;j

Pg ¼ gðP1;P2; . . . ;PN Þ

¼XNi¼1

XKi

k¼1

ð�mi;kÞ � eλi;k;0 � eRi;k;0 þXMi;k

j¼1

Xci;jw¼1

λi;k;j;w � Ri;k;j;w

!(

�ðdi;k � xi;kÞ ��LVi;k � ð1� AVi;kÞ

���XNi¼1

XKi

k¼1

XMi;k

j¼0

Xci;jw¼1

bai;k;j;w þXni;j�ci;j

w¼1

bsi;k;j;w

!s.t.

Λi;k ¼ eλi;k;0 ¼Xci;1w¼1

λi;k;1;w;"i; k (11)

754 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

Xci;jw¼1

λi;k;j;w ¼ eλi;k;j;"i; k; j (12)

Ai;k;j;w ¼ 0; if λi;k;j;w ¼ 0;"i; k; j;w (13)

Ai;k;j;w ¼ 1; if λi;k;j;w > 0;"i; k; j;w

AVi;k � AV i;k ; (14)

λi;k;j;w � Λi;k � zi;k;j;w;"i; k; j;w (15)

λi;k;j;w � 0;"i; k; j;w

Ai;k;j;w 2 f0; 1g;"i; k; j;wPNi¼1

PKik¼1 Λi;k � ui;k has been omitted in the above objective function, because it is

independent of decision-making variables. The constraint (11) shows that all workloadsare estimated as the arriving rate of the class k requests in the 0th tier of VASE i. Theconstraint (12) shows that the arriving rate of the class k requests in the jth tier of VASE iis equal to the sum of request arriving rates of VM w in the jth tier of the class k requestsin VASE i. Ai;k;j;w in the constraint (13) denotes the state of VM w, the optional value ofwhich is 0 or 1. When λi;k;j;w > 0, Ai;k;j;w ¼ 1, this means that VM w is in the active stateand the class k requests of VASE i will execute on VM w of the jth tier. Otherwise,Ai;k;j;w ¼ 0, VM w is still in the spare state. The constraint (14) restricts that the avail-ability of VMs should be no less than availability AV i;k specified in SLA. The constraint(15) allows the class k requests of VASE i to be executed on VM w only when the class krequests of the jth tier of VASE i have been allocated to VM w.

4.3. Optimisation-based resource allocation

In order to maximise Pg, we need to find a high-quality initial configuration matrix ofVMs (ConfigMAT0) in the CDC. The number of active VMs are the main variable of theproblem because they affect the performance, and the cost. Meanwhile, λi;k;j;w and μi;k;j;wonly affect the performance, so they are viewed as the subordinate variables. Therefore, inorder to find a suitable initial configuration of VMs, we develop a model based on thegiven workloads, the response time, and the specified VMs capacities. The initial solutionof virtualised resources configuration, ConfigMAT0, can be calculated by the proposedalgorithm.

We assume that VMs are homogeneous in the same tier of VASE i while hetero-geneous in different tiers. There are ci;j active VMs in the jth tier of VASE i, which can beformulated as

ci;j ¼ f ðeλi;k;j; μi;k;j;1; μi;k;j;2; . . . ; μi;k;j;ci;jÞThe number of globally active VMs is Cg;i in VASE i, therefore, the optimisation problem(P2) can be formulated as

minfCg;i ¼ f ðci;0; ci;1; ci;2; . . . ; ci;Mi;k Þg

Enterprise Information Systems 755

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

s.t.

XMi;k

j¼0

eRi;k;j � Ri;k ; (16)

Xci;jw¼1

μi;k;j;w > eλi;k;j;"i 2 ½1;N �;"k 2 ½1;Ki�;"j 2 ½0;Mi;k � (17)

The objective of the problem (P2) is to determine the number of VMs allocated todifferent tiers according to the limit of the specified response time Ri;k of VASE i. Theoutput of the model is the least number of VMs in VASE i, denoted with Cg;i.The constraint (16) states that the sum of the response time of every tier cannot exceedthe limited response time Ri;k specified in the SLA agreement. The constraint (17) statesthat the arriving requests rate in the same tier cannot exceed the total amount of availablevirtualised resources of the tier. In this way, the CDC can serve the requests in time.

Based on the above definitions, we can describe the proposed algorithm for initialoptimal configuration of virtualised resources with three steps.

(1) Given the total request arriving rate of the class k requests in VASE i, λ�i;k , we cancalculate the request arriving rate of each tier, eλi;k;0;eλi;k;1; . . . ;eλi;k;Mi;k .

(2) Then, we can calculate the lower limit number of VMs in each tier, which canensure that the request arriving rate of the jth tier cannot exceed the limit ofserving capacity of jth tier.

(3) Furthermore, we can calculate the total response time of tiers from 0th, 1th, . . . ,Mi;k th in VASE i.

In each iteration, a VM will be allocated to the tier whose response time can reducethe most. So we can get the initial optimal configuration of virtualised resources forVASE i; i 2 ½1;N �, ConfigMAT0.

The real workloads show continuous dynamic variations in the CDC. In order to meetthe requirements of SLA for multi-tier VASEs, the virtualised resources must be allocatedon demand in a dynamic way. Thus, the workload variations of multiple VASEs should bemonitored in real-time. Based on the CDC architecture proposed in Section 3.1, we designand optimise the dynamic allocation of virtualised resources to maximise the total profit ofthe CIPs.

5. Allocation methods

In order to realise the optimal resources utilisation with meeting the different resourcesrequirements of clients, we present a dynamic resources allocation strategy based on VMs(DVM-RA), as is described in Section 5.1. In addition, in order to show the effectivenessof the proposed allocation strategy, we also introduce briefly other existing allocationstrategies.

5.1. Dynamic resources allocation strategy based on virtual machines (DVM-RA)

In this paper, DVM-RA denotes the dynamic virtual machines-based resources allocationstrategy. The objective of problem (P1) is to find an optimal configuration variable matrix

756 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

which maximises the total profit in the next period with some constraints. Since theconstraint optimisation problem is a NP-hard problem (Woeginger 2003), we adopt a localsearch method called the gradient descent method (Burachik et al. 1995). In this method,the VM is allocated to serve whose marginal revenue is the highest. This method proceedsin the direction of the slope of the current point and moves up hill until the objectionfunction converges to a particular fixed point. Although the gradient descent method iscommonly used and easily understood, its search performance depends completely on thedomain structure and the initial solution. So it converges very slowly around the optimalsolution. In addition, the process can only converge to a local optimal solution, thus thismethod may not find the total optimal solution.

In order to find the optimal value, we combine this method with the particularstochastic optimisation methods. In this paper, we adopt the Tabu Search (TS) algorithm(Glover, Kochenberger, and Alidaee 1998). This algorithm adopts a high-quality initialallocation of VMs as the initial solution of the total search process. In every loop, thealgorithm disturbs the current matrix and generates a new allocation of VMs. When thesolutions converge to a particular fixed point, the algorithm calculates the variation ofprofit. Finally, the algorithm can record the best allocation of VMs so far and ensure theconvergence of all the search process.

In order to evaluate the effectiveness of DVM-RA, we also present two other existingresource allocation strategies as follows.

5.2. Static resources allocation strategy (Stat-RA)

In this paper, Stat-RA denotes the static resources allocation strategy. Before applicationservices run, Stat-RA allocates resources of the whole CDC to each VASE according to thepriority of different applications services. The allocation state of resources remains unchangedall the time, so this allocation method cannot be flexible when the workloads vary inevitably.The allocation of resources should be considered carefully before the system runs. It will leadto low utilisation of resources if resources are allocated based on the workloads in the worstcase. However, if resources are allocated based on the workloads of the common situation, theSLA penalty may occur when the workloads peak unexpectedly.

5.3. Dynamic resources allocation strategy based on physical machines (DPM-RA)

In this paper, DPM-RA denotes the dynamic physical machines-based resources allocationstrategy. DPM-RA assumes that every node runs in a certain tier of application services.Multiple application services running in different nodes can be isolated effectively. Theallocation method can the estimate states of the system periodically. DPM-RA can adjustthe physical resources on demand according to the changes of the workloads. When therequests of a particular application service are cancelled, DPM-RA can reuse the corre-sponding node providing this service to meet new requests of application services. Whenthe requests of a particular application service peak, DPM-RA can reuse resources to copewith the requests of the application service. And when the requests decrease, DPM-RA canreallocate the corresponding nodes to other application services dynamically.

6. Performance evaluation

In this section, we evaluate the performance of the self-adaptive virtualised resourcesoptimisation method by concrete experiments. Then we discuss the simulation results of

Enterprise Information Systems 757

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

our experimental evaluation in detail. We adopt the same parameters setting to evaluatethe efficiency of the proposed resources allocation strategy. In addition, we compare ourstrategy with the existing strategies from different aspects including the average responsetime, the variation of accumulative total profit, the system throughput, the CPU resourceutilisation and the performance isolation effect. In order to make the results of theexperiments close to the reality, we carefully specify the values of all the parameters inour experiments. The results related to the proposed method in our experiments are basedon 100 simulation runs. And the standard deviation of the results in our models is lessthan 5%. The experiment results show that based on the fine-grained virtualised resourcesallocation, the proposed method can ensure the expense of the CIPs as low as possible. Inthis way, we can maximise the total profit of the CIPs. Furthermore, we adopt thevirtualisation technologies to isolate different applications, and further improve the per-formance of the CIPs.

6.1. Simulation experiment setup

In the CDCs, the arriving workloads of multi-tier VASEs conform to the Poissondistribution. We adopt two open sourced multi-tier web applications for the virtualisedexperiment platform. The two applications including RUBiS (Amza et al. 2002), an onlineauction application and TPC-W (Cain et al. 2001), an online web application. DifferentVMs supporting corresponding VASEs run in servers. So an instance of RUBiS or TPC-W consists of different kinds of VMs, all of which share processors in servers.

The resources (e.g., CPU, memory, disc, network and I/O) requests of RUBiS aredifferent from that of TPC-W in the variations of the workloads. In this paper, we considerCPU as the representative resource in the resource allocation problem. However, thoughwe focus on the simple setup of the problem without other resources constraints, it isworth noting that our formulation can be easily extended to accommodate otherconstraints.

The application mode of TPC-W is similar to that of RUBiS, where VMs arescheduled to provide corresponding VASEs in every tier as well. We can divide clientsvisiting the two multi-tier VASEs into three different classes. We adopt two requeststraces of clients to generate the workloads for VMs: (1) the workloads based on the webtraces from the record of the 20th day (29 July) in the 1998 Soccer World Cup site(Arlitt and Jin 2000) for RUBiS virtualised application services. (2) The EPA-HTTPweb transmission workloads trace from LBL Repository (Bottomley 2004) for TPC-Wvirtualised application services. We take the samples from every 120 requests, so we canreduce the scale of the workloads records while not affecting the characteristic of thewhole variations.

It is common that typical multi-tier web applications tend to distribute computingtasks to the App tier and to distribute data processing tasks to the DB tier in the back end.So we distribute nodes with higher processing abilities to the last two tiers in oursimulation experiment. In order to evaluate the applicability of the proposed multi-tiermodel in complex cases, we set different values for parameters to describe VASEs, asshown in detail in Tables 2–4.

We update the values used by multi-tier VASEs in the proposed multi-tier model every15 min based on the online measurement of Web and App tiers, and record the value ofevery parameter based on the off line measurement information of DB tier when a cycle of15 min is exceeded. The metrics including response time, maximum throughput rate,availability, average profit and penalty in the SLA agreements of two VASEs are given

758 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

and keep unchanged. We assume that the SLA requirements have been specified betweentwo VASEs and clients. We divide clients into three classes including Class 1, Class 2 andClass 3. And 98% response time (i.e., 98% of requests can be served in the time limit) ofthe three classes of clients do not exceed 0.2, 0.4 and 0.5, respectively. The mean responsetimes are set as 10%, 50% and 40% of the total expected response time, respectively (forexample, the mean response times are 20 msec., 100 msec., and 80 msec in three tiers forClass 1). Here, we assume that the charging model is established based on the clients’QoS requirements. For example, in parameters setting of the above two VASEs, Class 1 isthe most strict in restriction of the response time because it pays the most. However, thecorresponding penalty is also maximum at the same time.

The workloads traces of the two VASEs in 24 hours are shown in Figure 4. In realenvironment, we choose carefully the length of cycle based on the requirements. If thelength is too short, the normal operation of system may be affected easily. But the falsedecision may be made easily if the self-adaptive process is invoked frequently. If thelength is too long, however, the self-adaptive process cannot effectively capture instanta-neous variations of the workloads. In the following experiment, we set the length ofcontrolled cycle as 15 min after considering the whole length of the experiment and thelength of the workloads. In order to avoid the incidental instability of performance whenthe CDC resources (mainly CPU resources) are occupied almost completely, the upperlimit of the CPU utilisation of physical nodes in every tier is set as 85%.

Table 3. Characteristics for TPC-W.

Parameter ODD tier Web tier App tier DB tier

Single VM capacity (req/s) 300 90 150 180VMs number 6 12 8 7pðunÞi;k;j�1(%) – – 0.81 0.85pi;k;j�1(%) – – 0.15 0.17Max throughput (%) 0.85 0.85 0.85 0.85

Table 4. SLA characteristics for RUBiS and TPC-W.

Parameter Class 1 Class 2 Class 3

Goal response time (sec) 0.2 0.4 0.6

Availability (%) 0.999 0.981 0.962

Average revenue ($) 2.4 1.8 1.0

Average penalty ($) 8.0 6.5 5.0

Table 2. Characteristics for RUBiS.

Parameter ODD tier Web tier App tier DB tier

Single VM capacity (req/s) 300 90 150 180VMs number 7 16 10 8pðunÞi;k;j�1(%) – – 0.82 0.89pi;k;j�1(%) – – 0.11 0.16Max throughput (%) 0.85 0.85 0.85 0.85

Enterprise Information Systems 759

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

6.2. Simulation experiments

In this section, we adopt the same parameters setting to evaluate the efficiency of severalresources allocation methods described in Section 5.

At first, we analyse the variation of response time due to different number of VMsdistributed in two VASEs. Figure 5 shows the variation of the average response time for aspecific request in VASE1 (RUBiS) and VASE2 (TPC-W) when we adopt DVM-RAstrategy. As illustrated, the horizontal dense dashed line shows that the expected responsetime of RUBiS is set as Class 2, i.e., the expected response time signed in SLA between

Figure 4. Request arrivings of two VASEs.

Figure 5. Ninety-eight percent response time.

760 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

clients and the CIPs is 0.4 sec. Similarly, the horizontal sparse dashed line shows theexpected response time of TPC-W is set as Class 1, i.e., the expected response time signedin SLA between clients and the CIPs is 0.2 sec. It can be drawn from Figure 5 that theresponse time can still keep stable when the workloads peak, only with few short-livedSLA penalties.

Figure 6 shows the variation of accumulative net earnings of resources with time. Itcan be shown that the profit keeps on increasing in a stable way even though the requestworkloads fluctuate obviously because of DVM-RA. However, when we adopt DPM-RAstrategy, the rising trend is not effectively predicted at 1220–1440 min. Due to penaltieshappening, accumulative net earnings drop significantly. The main reason is that physicalserver-based allocation strategy prevents fine-grained sharing of resources. Similarly, inaccordance with the expected result, the result of Stat-RA method is the worst because itsallocation amount of resources never varies with the trend of the request workloads, so theprofit declines obviously at 1080–1140 min. So the DVM-RA strategy can accuratelyobtain the trace variation of the workloads in advance, and virtualised resources can beadequately shared between two VASEs. In this way, we can ensure the SLAs of differentVASEs and keep the profit of CIPs. In conclusion, the DVM-RA strategy can maximise thetotal profit of CIPs.

The self-adaptive DVM-RA strategy can predict the trend of the workload variationsprecisely. In addition, the availability of virtualised resources is relatively high, and theexcess requests are refused only when the arriving workloads exceed the limit of totalvirtualised resources. When it comes to this, some penalties are generated. For example, inthe CDC, the upper limit of maximum virtualised resources in the Web tier is set as 28. InFigure 7, the total virtualised resources for two VASEs are insufficient to meet arrivingrequests at 960–1140 min, so excess requests are refused to keep response time under thespecified value in SLA. In this way, we can ensure the total profit to be higher.

Due to the difference of profit strategies of three classes, clients of Class 1 (TPC-W)expect to get best services, because they pay most. When the total amount of resources in

Figure 6. Variation of accumulative net earnings of resources with time.

Enterprise Information Systems 761

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

the CDC is sufficient to cope with the arriving requests, the ODD will adjust strategiesabout allocation of virtualised resources as reasonably as possible to meet the SLArequirements of all clients’ requests. However, when the amount of virtualised resourcesis relatively limited and insufficient to cope with the outburst of the arriving requests, theODD will be sensitive to this and decide how to allocate virtualised resources optimallyand refuse excess requests.

In Figure 7, when the total amount of virtualised resources is insufficient to cope witharriving requests and the requests of every class compete for limited resources, DVM-RAchooses to refuse the part of requests of Class 2 (RUBiS). Therefore, the expected SLArequirements of Class 1 (TPC-W) can be ensured at the expense of performance loss ofClass 2 (RUBiS). Figure 8 shows the cost variations of VMs in two VASEs with time.Figure 9 shows the validation results of system throughput between the proposed

Figure 7. Total VMs of two VASEs.

Cos

t of v

irtua

l mac

hine

s ($

)

Time (min)

Figure 8. Distribution of virtual machines cost.

762 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

optimisation model and actual measurement in web, App and DB tier. The results showthat proposed model is close to the actual situation and verify the effectiveness of theproposed optimisation model.

Then we analyse the CPU resource usage for RUBiS application. Here ρi;k;j denotesthe utilisation rate of the CPU resource. A higher ρi;k;j means higher utilisation of resourceand less resource waste. Figure 10 presents the CPU utilisation of the web tier throughoutthe experiment. The CPU resource utilisation can be improved along with the variation of

Web tier (measurement)Web tier (model)App tier (measurement)App tier (model)DB tier (model)DB tier (measurement)

Figure 9. Validation results on system throughput.

Web tier (model)Web tier (measurement)Domain 0 for web tier

Figure 10. Validation results on the web tier CPU resource usage.

Enterprise Information Systems 763

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

the request arriving rates for the web tier. The CPU resource usage of the proposedflexible hybrid model is closer to the actual operating environment especially for heavyworkloads in three-tier VASEs. Therefore, we can not only get the minimum number ofVMs, but also get the best CPU resource utilisation with our model.

In order to evaluate the performance isolation effect when we adopt VMs to organiseand allocate virtualised resources, we conduct a series of experiments in sharing andisolating services environment, respectively. In the experiments, we assume the avail-ability of an application instance in every tier to be equal. We set the length of time as 15min. The result illustrated in Figure 11 presents the accumulated total profit in RUBiSenvironment. It can be clearly shown that the accumulative total profit of shared environ-ment is equal to that of isolated environment when the failure does not happen. However,with the decrease of availability, the performance of shared environment declines morequickly than that of isolated environment. It can also be found that with the increase ofavailability, isolated environment can serve more requests than shared environment. Theresults show that the availability and the performance of the whole system can beimproved by performance isolation.

7. Conclusions and future works

In this paper, first, we propose a cloud data centre architecture with virtualised resourcesand present an autonomic management architecture for the CDC. Then we provide amulti-tier architecture-oriented virtualised application performance model to allocatecomputing resources to each tier based on the clients’ SLA restrictions. We formulatethe non-linear constrained optimisation problem by considering restrictions specified inthe SLA such as response time, throughput, revenue, cost and penalty. To solve theoptimisation problem, we develop a heuristic algorithm to ensure the maximum profitof the CIPs in the CDC. We compare the effectiveness of the proposed dynamic allocationstrategy based on VMs with two other existing ones. Simulation experiments with actual

Figure 11. Accumulated profit.

764 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

clients’ workloads show that the proposed fine-grained and dynamic allocation methodcan obtain about 15–20% higher total profit than other two methods. In addition, themethod can meet the SLA requirements of clients in every application even thoughworkloads vary sharply. Furthermore, we can allocate virtualised resources effectivelyfor various requests from the classes of different clients. Therefore, we can maximise thetotal profit of the CDC together at the expense of reduced operation and maintenance.

In future research, we would like to investigate how the current approach can begeneralised to support different virtualised application services running on heterogeneousand distributed platforms. We also plan comprehensive consideration of other metricsincluding memory, disc, I/O usage.

AcknowledgementsThis work is supported by the National Natural Science Foundation of China (No. 61174169 andNo. 61033005), the National High Technology Research and Development Program of China (863Program, No. 2012AA040915) and the National Key Technology Research and DevelopmentProgram (No. 2012BAF15G00).

ReferencesAmza, C., A. Chanda, A. L. Cox, S. Elnikety, R. Gil, K. Rajamani, W. Zwaenepoel, E. Cecchet, and

J. Marguerite. 2002. “Specification and Implementation of Dynamic Web Site Benchmarks.” InProceeding of 2002 IEEE International Workshop on Workload Characterization, 3–13.Piscataway, NJ: Institute of Electrical and Electronics Engineers Computer Society.

Arlitt, M., and T. Jin. 2000. “Aworkload Characterization Study of the 1998 World Cup Web Site.”Network, NJ: Institute of Electrical and Electronics Engineers Computer Society 14 (3): 30–37.

Bottomley, L. 2004. “Epa-http Single www Server Trace.” Accessed April 18, 2012. http://ita.ee.lbl.gov/html/contrib/EPA-HTTP.html

Burachik, R., L. M. Graña Drummond, A. N. Iusem, and B. F. Svaiter. 1995. “Full Convergence ofthe Steepest Descent Method with Inexact Line Searches.” Optimization 32 (2): 137–146.

Cain, H. W., R. Rajwar, M. Marden, and M. H. Lipasti. 2001. “An Architectural Evaluation of JavaTPC-W.” In Proceedings of the Seventh International Symposium on High-PerformanceComputer Architecture, 229–240. Piscataway, NJ: Institute of Electrical and ElectronicsEngineers Computer Society.

Cao, J., W. Zhang, and W. Tan. 2012. “Dynamic Control of Data Streaming and Processing in aVirtualized Environment.” IEEE Transactions on Automation Science and Engineering 9 (2):365–376.

Cardellini, V., E. Casalicchio, F. L. Presti, and L. Silvestri. 2011. “SLA-Aware ResourceManagement for Application Service Providers in the Cloud.” In Proceedings of the 1stInternational Symposium on Network Cloud Computing and Applications, 20–27. Piscataway,NJ: Institute of Electrical and Electronics Engineers Computer Society.

Glover, F., G. A. Kochenberger, and B. Alidaee. 1998. “Adaptive Memory Tabu Search for BinaryQuadratic Programs.” Management Science 44 (3): 336–345.

Goudarzi, H., and M. Pedram. 2011. “Multi-Dimensional SLA-Based Resource Allocation forMulti-Tier Cloud Computing Systems.” In Proceedings of the 2011 IEEE InternationalConference on Network Cloud Computing, 324–331. Piscataway, NJ: Institute of Electricaland Electronics Engineers Computer Society.

Hayes, B. 2008. “Cloud Computing.” Communications of the ACM 51 (7): 9–11.He, W., and L. D. Xu. 2013. “Integration of Distributed Enterprise Applications: A Survey.” IEEE

Transactions on Industrial Informatics. doi:10.1109/TII.2012.2189221.Hernández-Suárez, C. M., and C. Castillo-Chavez. 1999. “A Basic Result on the Integral for

BirthCdeath Markov Processes.” Mathematical Biosciences 161 (2): 95–104.Iosup, A., S. Ostermann, M. N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. J. Epema. 2011.

“Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing.”IEEE Transactions on Parallel and Distributed Systems 22 (6): 931–945.

Enterprise Information Systems 765

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

Iqbal, W., M. Dailey, and D. Carrera. 2009. “SLA-Driven Adaptive Resource Management for WebApplications on a Heterogeneous Compute Cloud.” In Proceedings of the 1st InternationalConference on Cloud Computing, LNCS 5931, 243–253. Berlin: Springer.

Kalyvianaki, E., T. Charalambous, and S. Hand. 2009. “Self-adaptive and Self-Configured CPUResource Provisioning for Virtualized Servers Using Kalman Filters.” In Proceedings of the 6thInternational Conference on Autonomic Computing, 117–126. New York: ACM Press.

Kephart, J. O., and D. M. Chess. 2003. “The Vision of Autonomic Computing.” Computer 36 (1):41–50.

Kusic, D., J. O. Kephart, J. E. Hanson, N. Kandasamy, and G. Jiang. 2008. “Power and PerformanceManagement of Virtualized Computing Environments Via Lookahead Control.” In Proceedingsof the 5th International Conference on Autonomic Computing, 3–12. Piscataway, NJ: Institute ofElectrical and Electronics Engineers Computer Society.

Lama, P., and X. Zhou. 2012. “NINEPIN: Non-Invasive and Energy Efficient Performance Isolationin Virtualized Servers.” In Proceedings of the 2012 42nd Annual IEEE/IFIP InternationalConference on Dependable Systems and Networks, 1–12. Piscataway, NJ: Institute ofElectrical and Electronics Engineers Computer Society.

Li, Q., Z. Wang, W. Li, J. Li, C. Wang, and R. Du. 2013. “Applications Integration in a HybridCloud Computing Environment: Modelling and Platform.” Enterprise Information Systems 7(3): 237–271.

Li, S., L. D. Xu, and X. Wang. 2013. “Compressed Sensing Signal and Data Acquisition in WirelessSensor Networks and Internet of Things.” IEEE Transactions on Industrial Informatics.doi:10.1109/TII.2012.2189222.

Liu, L., H. Wang, X. Liu, X. Jin, W. B. He, Q. B. Wang, and Y. Chen. 2009. “GreenCloud: A NewArchitecture for Green Data Center.” In Proceedings of the 6th International ConferenceIndustry Session on Autonomic Computing and Communications Industry Session, 29–38.Piscataway, NJ: Institute of Electrical and Electronics Engineers Computer Society.

McKenna, J. 1988. “A Generalization of Little’s Law to Moments of Queue Lengths and WaitingTimes in Closed, Product form Queueing Networks.” Analysis and Optimization of Systems 111:1000–1011.

Menasce, D. A., and M. N. Bennani. 2006. “Autonomic Virtualized Environments.” In Proceedingsof the 2006 International Conference on Autonomic and Autonomous Systems, 28–28.Piscataway, NJ: Institute of Electrical and Electronics Engineers Computer Society.

Meyer, P. A., R. T. Smythe, and J. B. Walsh. 1972. “Birth and death of Markov Processes.” InProceedings of the 6th Berkeley Symposium on Mathematical Statistis and Probability, Vol. 3,295–305. Berkeley: University of California Press.

Reed, D., I. Pratt, P. Menage, S. Early, and N. Stratford. 1999. “Xenoservers: AccountableExecution of Untrusted Programs.” In Proceedings of the Seventh Workshop on Hot Topics inOperating Systems, 136–141. Piscataway, NJ: Institute of Electrical and Electronics EngineersComputer Society.

Slegers, J., I. Mitrani, and N. Thomas. 2009. “Static and Dynamic Server Allocation in Systems withOn/Off Sources.” Annals of Operations Research 170 (1): 251–263.

Urgaonkar, B., G. Pacificiy, P. Shenoy, M. Spreitzery, and A. Tantawiy. 2005. “An Analytical Modelfor Multi-Tier Internet Services and Its Applications.” In Proceedings of the 2005 ACMSIGMETRICS International Conference on Measurement and Modeling of Computer Systems,291–302. New York: ACM Press.

Verma, A., P. Ahuja, and A. Neogi. 2008. “pMapper: Power and Migration Cost Aware ApplicationPlacement in Virtualized Systems.” In Proceedings of the 9th ACM/IFIP/USENIX InternationalConference on Middleware, 243–264. Piscataway, NJ: Institute of Electrical and ElectronicsEngineers Computer Society.

White, S. R., J. E. Hanson, I. Whalley, D. M. Chess, and J. O. Kephart. 2004. “An ArchitecturalApproach to Autonomic Computing.” In Proceeding of 1st International Conference onAutonomic Computing, 2–9. Piscataway, NJ: Institute of Electrical and Electronics EngineersComputer Society.

Woeginger, G.J. 2003. “Exact Algorithms for NP-Hard Problems: A Survey.” In CombinatorialOptimization, Lecture Notes in Computer Science, LNCS 2570, 185–207. Berlin: Springer.

Wu, L., S. Garg, and R. Buyya. 2011. “SLA-Based Resource Allocation for Software as a ServiceProvider (SaaS) in Cloud Computing Environments.” In Proceedings of the 11th IEEE/ACM

766 J. Bi et al.

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015

International Symposium on Cluster, Cloud and Grid Computing, 195–204. Piscataway, NJ:Institute of Electrical and Electronics Engineers Computer Society.

Yarmolenko, V., and R. Sakellariou. 2006. “An evaluation of Heuristics for SLA Based Parallel JobScheduling.” In Proceeding of 20th International Conference on Parallel and DistributedProcessing Symposium, 8–15. Piscataway, NJ: Institute of Electrical and ElectronicsEngineers Computer Society.

Zhang, A., P. Santos, D. Beyer, and H. K. Tang. 2002. “Optimal Server Resource Allocation usingan Open Queueing Network Model of Response Time.” In HP Laboratories Technical Report,2002. Palo Alto, CA: HP Laboratories.

Zhu, X., D. Young, B. J. Watson, Z. Wang, J. Rolia, S. Singhal, and B. McKee, et al. 2009. “1000Islands: An Integrated Approach to Resource Management for Virtualized Data Centers.”Cluster Computing 12 (1): 45–57.

Zuo, X., G. Zhang, and W. Tan. 2013. “Self-Adaptive Learning PSO-Based Deadline ConstrainedTask Scheduling for Hybrid IaaS Cloud.” IEEE Transactions on Automation Science andEngineering. doi:10.1109/TASE.2013.2272758.

Enterprise Information Systems 767

Dow

nloa

ded

by [

Tsi

nghu

a U

nive

rsity

] at

00:

34 1

7 Se

ptem

ber

2015