CloudSimNFV: Modeling and Simulation ofEnergy-Efficient NFV in Cloud Data CentersWutong Yang, Minxian Xu, Guozhong Li

University of Electronic Science andTechnology of China

Email: uestcywt@foxmail.com,xmxyt900@gmail.com,

GuozhongLi@hotmail.com

Wenhong TianUniversity of Electronic Science and Technology of China,

BigData Research Center, UESTC, ChinaEmail: tian wenhong@uestc.edu.cn,

Abstract—Network Function Virtualization (NFV) takesadvantage of hardware virtualization to undertake softwareprocessing for various functions, and complements thedrawbacks of traditional network technology. To speedup NFV related research, we need a user friendly andeasy to use research tool, which could support data centersimulation, scheduling algorithms implementation and ex-tension, and provide energy consumption simulation. As acloud simulation toolkit, CloudSim has strong extendibilitythat could be extended to simulate NFV environment.This paper introduces a NFV cloud framework based onCloudSim and an energy consumption model based onmulti-dimensional extension, implementing a toolkit namedClousimNFV to simulate the NFV scenario, proposingseveral scheduling algorithm based on for NFV applica-tions. The toolkit validation and algorithm performancecomparison are also given.

Index Terms—Cloud Computing, Network FunctionVirtualization, Simulation Toolkit, Energy ConsumptionModel, Scheduling Algorithm

I. INTRODUCTION

In traditional data centers, applications are tied tospecific physical servers that are often over-provisionedto deal with upper-bound workload. Such configurationmakes data centers expensive to maintain with wastedenergy and floor space, low resource utilization andsignificant management overhead. With virtualizationtechnology, todays Cloud data centers become moreflexible, secure and provide better support for on-demandallocating [4].

Currently, computation or storage is treated as in-dependent of network, and energy efficiency is notconsidered enough, which leads the service values arenot maximized. The motivations of Network FunctionVirtualization (NFV) comes to satisfy the aim thatputting the network functions in Clouds, combiningvalues (Infrastructure as a service, Compute/Storage,Network infrastructure as a service, etc.), leveraging

NFV infrastructure of other service providers to increaseresiliency, reducing latency and regulatory requirements[7]. Moreover, the ultimate goal for NFV is transformingnetwork architectures by implementing Network Func-tions in Software that can run on commodity hardware[20].

To further foster innovation and development, werequire toolkits that provide a testbed for experimentswith Network Function Virtualization systems within acloud data center. While there isn’t any tool that canfulfill this goal yet. To accelerate the related researchon NFV, we introduce CloudSimNFV that enables tosimulate scenario for NFV.

CloudSimNFV is a newly developed tool built ontop of CloudSim [5]. In this paper, we discuss theinnovation of CloudSimNFV and introduce its detaileddesign implementation. A framework is designed to eval-uate resource management policies applicable for NFVscenario under cloud data center environment. It enablesto simulate cloud data centers, physical machines, virtualmachines and NFV applications. CloudSimNFV can alsoimplement energy consumption model with predefinedloads to monitor the energy cost in data centers. More-over, CloudSimNFV provides portal pages to simplifythe simulation configurations and show the simulationresults.

CloudSimNFV validation is tested with realistic com-munication service providers’ data. The loads com-ing into data centers are periodically fluctuating overtime referring to realistic communication providers data.CloudSimNFV has implemented several scheduling al-gorithms for resource allocation to compare performanceunder several scenarios. The implemented algorithmshave their specific advantages and are possible to beapplied to different NFV scenarios.

The remainder of this paper is as follows. In Section

arX

iv:1

509.

0587

5v1

[cs

.DC

] 1

9 Se

p 20

15

II, we introduce the related work on NFV and why wechoose CloudSim to extend NFV scenario. In SectionIII, we present the requirements for a NFV simulationtool combining the features of NFV scenarios and ap-plications. The CloudSimNFV framework and energyconsumption model are demonstrated in Section IV. Theexperiments and performance comparison are given inSection V. Finally, a conclusion and future work areconcluded in Section VI.

II. RELATED WORK

There have been some research improved the relatedresearch on NFV. Basta et al. [2] proposed the functionsplacement problem in practice, and predicted the waythat applying NFV and SDN to LTE mobile gatewaysto solve the problem. Carella et al. [6] introduced anIP multi-media subsystem based on Network FunctionVirtualization architecture. Bolla et al [3] presented anenhanced framework named DROPv2, which containspower management mechanism to satisfy the energyefficiency of Network Function Virtualization. Raggio etal. [14] introduced EmPOWER as a novel and open plat-form for future tests and research on Network FunctionVirtualization. Udupi et al. [20] emphasized the com-bination of NFV and Openstack platform that a smartscheduler component could help improve the efficiencyof infrastructure. Apart from these mentioned work, aneasy to use and repeatable toolkit for NFV is still needed.

Quite a few cloud simulation tools have been im-plemented to simulate cloud data centers and compareresource scheduling algorithms. A survey paper summa-rized by Tian et al [17] has compared and discussedseveral simulators, like CloudSim [5], GreenCloud [11],iCanCloud [13], CloudSched [18] from their architectureview, modeling view, performance evaluation view andso on. Xu [19] proposed a flexible and light-weightsimulator, namely FlexCloud, for testing cloud resourcescheduling algorithms. All these simulators have theirstrengths and weaknesses. Considering extendibility, im-plemented components (physical machines, virtual ma-chines and task loads) and adopted energy consumptionmodel. CloudSim is the easiest choice to extend andimplement the NFV feature.

III. NETWORK FUNCTION VIRTUALIZATION FORCLOUD DATA CENTERS

Simulation system has huge advantage for researchingmigration policies and energy consumption evaluation inNFV environment, as it need not to be built on complexreal environment and can be sustainable to processsimulation tests for a long period. It could also runexperiments quickly and repeatedly, and provide detailed

results for analysis. As for our algorithm evaluation,we can regard the algorithm as the initial filter thatcould pick up the suitable resource. In addition, withsimulation tools, as the simulation time is quite lessthan realistic tests, much more test cases could be testedfor reference. When we are comparing the efficiency ofalgorithms, we could obtain an approximate result forexpectation before we undertake real tests. Therefore,our main goals focus on repeatability and controllability.We identify the following requirements:1. Enable to simulate the data center core computingelements;2. Enable to simulate variable work loads, which couldfluctuate in real time;3. Enable to simulate different migration policies;4. Enable to simulate different energy consumption mod-els, which could switch to different model and evaluateenergy consumption.5. Enable to simulate Network Function Virtualizationapplications.

IV. CLOUDSIMNFV FRAMEWORK DESIGN

A. CloudSimNFV ArchitectureOur NFV simulation tool, CloudSimNFV is based

on CloudSim platform. Taking advantage of data centersimulation of CloudSim, we extend the specific NFVscenario that we require. We add the energy consumptionmodel and extend scheduling algorithms under NFVscenario. As for the dynamic loads, we can simulatethrough loading the load files. Program codes (whichare written in Java) can provide physical and virtualtopology configurations. Another approach for user inputis a portal that translates requirements into physical andvirtual topology configurations.

Fig.1 shows the overall architecture of CloudSimNFV.Firstly, the user code and scenarios are composed oftwo modules: simulation specification and schedulingpolicy, which is configured by users and defined bydevelopers. The simulation specification module definesthe NFV scenario that simulation would be executedin, user requirements and energy aware characteristic.Scheduling policy module includes implemented and tobe extended scheduling policies, which can be dividedas VM migration policy, VM selection, initial allocationpolicy according to their different scheduling stages.

Besides the description of the user code and scenario,the end-users’ requests, which composes the input work-load for the simulation and is supplied in TXT files.Each type of the workload includes four workload fileswhich are used to simulate cpu workload, memory work-load, storage workload and bandwidth workload sepa-rately. These workload files can form different modes of

2

Fig. 1. CloudSimNFV Architecure

workloads, like compute mode workload (CPU resourceis required most), network mode workload (bandwidthresource is required most), storage model workload(storage resource is required most). Different types ofworkload can simulate diversified NFV working envi-ronment, which help us to test the energy consumptionof each situation.

The lower layer is energy model layer. Users canspecify the energy model according to the actual sce-nario (We will introduce an energy model in SectionIV.B). Corresponding to workload and energy model,the provided scheduling policies, such as VM migrationpolicy, VM selection policy and initial allocation policyare supported by the user code and scenario module.Brokers can be programmed to simulate the behaviorof end-users or data centers. Regarding these policies,a user can either utilize built-in policies or can developtheir own (by extending abstract classes).

VM Services are in charge of managing VMs, bycalculating application execution and packet transmis-sion time between VMs, it also supplies the creation ofcloudlet and the updating of VM state. The next layer,Resource Provisioning, is composed of four modules.

CPU Provisioning is a module to provide CPU resourceto VM, memory Provisioning provides memory resourceto VM, disk Provisioning provides storage resource toVM, and network Provisioning is provision of networkresource to VM. Resource Provisioning depends on theworkload that end-user chooses to use. The next layer,Resource Allocation, containing modules that allocateresources in the bottom layer of the architecture, ac-cording to the VM migration policy and VM selectionpolicy specified by simulator users. These three layersare adopted from the original CloudSim codes.

The bottom layer of this architecture is the cloudresource layer. This layer contains the data center, host,VM and cloudlet components to simulate the differentcomponents of real data centers. Some extension workhas been done in this layer. We extend the cloudlet asNFVlet to simulate the NFV task under NFV scenario(more details are given in IV.C). Physical topology con-figuration is implemented to represent the relationshipbetween these components and can be configured in userinterface.

3

B. Energy Consumption Model

The system energy consumption model is modelingfor the whole system. Many researches have found thatCPU utilization is typically proportional to the overallsystem load, and with realistic tests, a power consump-tion model for blade server is proposed as below [8]:

P = 14.5+0.2Ucpu+(4.5e−8

)Umem+0.003Udisk +(3.1e−8

)Unet(1)

Where Ucpu, Umem, Udisk, Unet are utilization of CPU,memory, storage and network respectively. It can be seenthat other factors have very small impact on total en-ergy consumption. The parameter value could be testedand determined with specific server, such as combininglearning and manual intervention. The parameters maybe different for different servers.

When virtual machine is allocated to physical server,given that the CPU capacity occupied by virtual machineis just the capacity submitted by virtual machine, thenthe increased CPU utilization of physical server aftervirtual machine allocation can be calculated as:

4u =VM.cpu

PM.cpu(2)

where VM.cpu is the CPU capacity of virtual machine,PM.cpu is the CPU capacity of physical machine. Thenthe VM energy consumption can be calculated as:

Evm = (Pmax − Pmin)× (t1 − t0)×VM.cpu

PM.cpu(3)

Energy consumption of physical server could be cal-culated as the turned on energy consumption (whenCPU utilization is 0) adds the virtual machine energyconsumption running on the machine. Epoweron is thepower when server is turned on, Epm is the server energyconsumption:

Epoweron = Pmin × Tpoweron (4)

Epm = Epoweron +

n∑i=1

Evmi (5)

where Tpoweron is the running time of physical server,Evmi is the energy consumption of the i − th virtualmachine, n is the virtual machine number on physicalserver. The total energy consumption of data center:

EDC =

n∑i=1

Epmi (6)

where EDC is the total energy consumption of allphysical servers, Epmi is the energy consumption of thei − th physical machine, n is the sum of all physicalservers.

Fig. 2 shows the energy consumption calculationimplementation structure. The structure is mainly com-posed of the following components:

CloudSim core logic: The original CloudSim corelogic is used to simulate the basic compute elements thatcompose the cloud infrastructure. On CloudSim, physicalhosts can be defined with specific configurations andVMs are placed on the host that meets resource require-ments such as CPU power, memory, network bandwidthand storage size. CloudSim simulates a range of elementsof the cloud architecture, including data center, physicalhost, VM, VM scheduler, workload scheduler, etc.

VM scheduler modules: VmScheduler policy of NFVnot only considers the CPU resource, but also takesall elements into account. VMs are placed in physicalhosts according to VmAllocation policy. During the pe-riod of simulation, hosts migrate VMs using the VMscheduler policy specified by users.And VmSchedulerpolicy would combine cpu resource ,ram resource, diskresource and network resource to decide which VMshould be migrated to a new place. VM owns one ormany cloudlets, and each cloudlet is described with therequired computational power, memory, and storage size.Once the cloudlet is placed in certain VM, it will existuntil cloudlet’s life cycle is over.

Energy aware modules: In order to evaluate the energyconsumption of NFV environment, we developed a Pow-erModelNFV class which makes our energy consumptioncalculation more precisely. It evaluates weighted cpuutilization, ram utilization, disk utilization and networkutilization. All the resources of PowerHost are provi-sioned by RamProvisioner, BwProvisioner and DiskPro-visioner . There are three VM allocation policies we haveappended, these VM allocation policies are inspired byNFV algorithm, ecoCloud algorithm and DRS algorithm.And all the VM allocation policies we developed need toextend the PowerVmAllocationPolicyMigrationAbstractclass.

C. Load Generation

Energy consumption tests under simulation environ-ment are built on data center simulator CloudSim. Inthe CloudSim version after 3.0, Dynamic Voltage Fre-quency Scale (DVFS)[9] technique has been added intoCloudSim. With the combination of CloudSim + DVFS,the model elements include physical machine (PM),virtual machine (VM) and cloudlet. Cloudlet is regardedas the tasks coming into system, which in our model,we transform cloudlet as NFV application as NFVlet.The physical machine modeling and virtual machinemodeling are as same as the modeling in CloudSim.

4

Fig. 2. CloudSimNFV Energy Consumption Model ImplementationStructure

We implement the aforementioned energy consump-tion model in previous section, then define the resourcethat PM may provide, VM and NFVlet may require. Ansimple example is presented below:PM : 1000MIPSVM1 : 100MIPSNFV let1 : 150MIVM2 : 20MIPSNFV let2 : 80MI

Considering this example, with composition of dif-ferent VM and NFVlet, various types of CPU loadscould be generated. Such as C1 = [VM1, NFV let1]C2 = [VM1, NFV let2] C3 = [VM2, NFV let1]C4 = [VM2, NFV let2]

C1 could produce 1.5s duration 10% CPU load,C2 could produce 0.8s duration 10% CPU load, C3could produce 7.5s 2% CPU load, C4 could produce4s duration 2% CPU load. Besides that, more variouscompositions could produce more types of loads withvarious durations and CPU loads, simulating resourcesand tasks under NFV scenario.

The above modeling is for CPU intensive tasks, whichis mainly taking CPU (computational) loads into con-sideration. In CloudSim energy consumption model, itmainly considers CPU loads. In our model, we extendto the CloudSim to capture memory, IO, network loadsby adding the respective parameters and methods.

The loads can be predefined in workload configurationfiles. The workload configuration files can be divided asfour types based on different resource type, includingCPU workload file, memory workload file, disk storageworkload file and bandwidth workload file. Each file iscomposed of many lines, each line represents a timeinterval and includes a 0-100 numeric value, which

TABLE ICOMPOSITIONS WITH DISTRIBUTIONS FOR LOADS GENERATION

Task length distribution Task number Application TypeUniform Distribution 100 CPU IntensiveUniform Distribution 100 I/O IntensiveUniform Distribution 100 HybridNormal Distribution units 1,000 CPU IntensiveNormal Distribution 1,000 I/O IntensiveNormal Distribution 1,000 HybridPoission Distribution 10,000 CPU IntensivePoission Distribution 10,000 I/O IntensivePoission Distribution 10,000 Hybrid

represents the load in this time interval. Like in the CPUload file, in line 1, the value is 40, which means in thefirst time interval, the CPU load is 40%. By readingthis file, the NFV task generator would generate theNFVlet with 40% CPU loads in the first time interval.The NFVlet with more types of loads could also begenerated in the same way.

The loads could also be generated with different distri-butions to control the number and length of tasks. TableI lists some possible compositions with distributions togenerate loads:

After the loads are generated as NFVlets, NFVletswould be allocated to VM to form loads for VM, thento PM (Host), and finally forms the loads in DataCen-ter. Fig. 3 shows the load generation architecture ofCloudSimNFV.

Fig. 3. CloudSimNFV Load Generation Architecture

D. User Interface

To simplify the user configuration for simulation,CloudSimNFV provides user portal to help user config-ure related parameters. Fig. 4 shows the portal page toset simulation parameters, including VM number, hostnumber, output parameters, the algorithms for compari-son (further description in Section V), load type for tests,and etc. Users can edit the parameters as they want, thenclicking the Start Button to start simulation.

5

Start

ParametersVM number

Load Type

Host number

OutputOutput to file

Output path

VM allocation policyVM selection policy

Fig. 4. Simulation Configuration Interface

Fig. 5. Simulation Results Interface

Fig. 5 demonstrates the corresponding simulation re-sults after starting the configured simulation. The resultsshow the basic simulation information and compared theenergy consumption values.

V. VALIDATION AND PERFORMANCE EVALUATION

A. Validation

Validation of CloudSimNFV is essential when itcomes to persuasion of simulation. To fulfill this goal,we design and implement a set of experiments to validateby comparing CloudSimNFV and VMware under sameworkload and duration. VMware is a virtualization plat-form that enables to manage virtual machines on physicalmachine. VMware could produce more realistic data andhas been equipped with energy efficiency managementtechniques and resource scheduling strategies. In ourexperiments, we setup environment that are composite ofphysical nodes that host VMware tool to manage virtualmachine allocation and monitor energy consumption.Then our goal is to compare the energy consumptionunder different durations between hosts in CloudSim-NFV and VMware, so that we can validate the accuracyof CloudSimNFV.

1) Environment Setup: To model the scenario that hasloads fluctuation, we adopted a tool named stress-ng [16]to generate load for the virtual machines in VMware totrace the energy consumption tendency. We also imple-mented the same load type in the way that described inearlier section IV.C. Under this environment, we setupthree physical nodes (two nodes are with 12 cores and48GB memory, another is with 8 cores and 4GB memory,one core capacity is equivalent to Intel Xeon CPUE5-2620 capacity), and each node can support severalvirtual machines (virtual machines are configured as onetype of 4cores with 4GB or 2cores with 2GB, ). Inaddition, we created both physical machines and virtualmachines with the same types in CloudSimNFV. Hence,the loads, virtual machines and physical machines havethe corresponding elements both in realistic environmentVMware and simulation tool CloudSimNFV.

Fig. 6. Simulation Results Interface

2) Validation Results: Fig. 6 demonstrates the ob-tained energy consumption in VMware and CloudSim-NFV under the configured experiment in the earliersection. In this scenario, the total observed durationis 6 hours and we have varied observed periods, like0 − 0.5hr, 0 − 1hr, . . . , 0 − 6hr to trace the energyconsumption fluctuation precisely. In this scenario, thedifferences between VMware and CloudSimNFV arealways less than 5%. This slight loss of accuracy is re-sulted from bandwidth energy consumption. We are stillworking on modeling the bandwidth energy consumptionprecisely considering data transferring.

B. Energy Consumption Performance Evaluation

In this paper, we adopt the Amazon EC2 configurationof VMs and PMs as shown in Table II and III. Note thatone compute unit (CU) has equivalent CPU capacity ofa 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor[1].

In this section, we provide simulation results for com-paring 3 different energy efficient scheduling algorithms

6

TABLE IVLOADS IN ONE DAY’S TRACE

Time Period (hour in one day)Load Type 0-2 2-6 6-8 8-12 12-14 14-18 18-23 23-0

ComputationalIntensive Loads

CPU Load 30% 10% 30% 70% 60% 50% 90% 50%Memory Load 30% 20% 30% 40% 40% 40% 50% 40%Storage Load 20% 20% 20% 20% 20% 20% 20% 20%Bandwidth Load 20% 20% 20% 20% 20% 20% 20% 20%

TABLE VENERGY CONSUMPTION AND MIGRATION TIMES COMPARISON

Test Case Configuration Comparison Index DRS EcoCloud NFV

Case 1 100HOSTS,100VMS

Energy Consumption (KW*h) 50.7217 45.2440 34.7679Migration Times 216 427 176

Case 2 100HOSTS,150VMS

Energy Consumption (KW*h) 74.0505 66.1416 52.5940Migration Times 244 120 242

Case 3 200HOSTS,100VMS

Energy Consumption (KW*h) 48.2157 45.2463 35.0701Migration Times 298 240 312

TABLE II8 TYPES OF VIRTUAL MACHINES (VMS) IN AMAZON EC2

Compute Units Memory Storage VM Type1 units 1.7GB 160GB 1-1(1)4 units 7.5GB 850GB 1-2(2)8 units 15GB 1690GB 1-3(3)6.5 units 17.1GB 420GB 2-1(4)13 units 34.2GB 850GB 2-2(5)26 units 68.4GB 1690GB 2-3(6)5 units 1.7GB 350GB 3-1(7)20 units 7GB 1690GB 3-2(8)

TABLE III3 TYPES OF PHYSICAL MACHINES (PMS) SUGGESTED

PM Pool Type Compute Units Memory StorageType 1 16 units 30GB 3380GBType 2 52 units 136GB 3380GBType 3 40 units 14GB 3380GB

for NFV scenario. The compared algorithms are asfollowing:

1. DRS (Dynamic Resource Scheduling) algorithm[10]: In the initial allocation stage, the algorithm alwaysallocates the VM to PM with the lowest load (typicallythe load means cpu utilization). DRS algorithm prede-fines a threshold that controls system imbalance level.During the migration stage, when the system imbalancelevel g surpasses the predefined threshold, the migrationalgorithm would allocate VM to another PM that coulddecrease the g value to be below threshold.

2. EcoCloud algorithm [12]: The ecoCloud algorithmadopts a probabilistic function to select the most suitablePM to allocate VM. This algorithm would calculate a

probabilistic value for each PM based PM’s utilizationu. Both utilization upper threshold Ta and lower boundTb are predefined. The probability of PM to be allocatedis calculated as fa(u) = 1

Mpup(Ta − u), 0 ≤ u ≤ Ta,

and Mp =pp

(p+1)(p+1)T(p+1)a , where p can be set as 2, 3, 4

, when utilization of PM surpasses the upper thresholdor less than the lower bound, the migration algorithmcould reallocate VM to the PM with highest probabilityaccording to the probabilistic function.

3. NFV algorithm: The NFV algorithm also adoptsa probabilistic function to select the most suitable PMto allocate VM. The probabilistic function is F =−f(x;α,β)

3 + 1, where −f(x;α, β) is Beta distribution,which equals to xα−1(1−x)β−1∫ 1

0ua(1−u)β−1du

. The predefined upperand lower bound are also set, when utilization of PMis not between upper threshold and lower bound, VMmigration would be triggered to make the PM utilizationfalls into the predefined interval. The migration processis also referring to the probabilistic function, in whichthe migrated VM would be reallocated to the PM withhighest probability.

Table IV presents the computational type loads in oneday’s trace that we adopt for tests. This load trace isaccording to communication provider that the loads haspeak (hour 8-12) and nadir (hour 0-2), which is quitesuitable for NFV scenario.

The NFVlets are generated with the loads in TableIV. Then NFVlets are allocated to VMs and VMs areallocated to PM according the scheduling algorithms.We vary the host number and VM number, and comparethe energy consumption and migration times in Table V.From the table, we can notice that under this scenario,

7

NFV algorithm has the best energy-saving effects. Wealso obtain comparison results of the I/O intensive typeloads, loads generated with other distributions (as in-dicated in Table I). With page limitation, we omit thedetailed comparison for other load generation types.

VI. CONCLUSIONS AND FUTURE WORK

As complementary of Software-Defined-Network,Network Function Virtualization aims to leverage stan-dard IT virtualization technology to consolidate variousnetwork equipment types onto industry standard equip-ments. Network Function Virtualization can be appliedto any data plane packet processing and control planefunction in fixed and mobile network infrastructures.

Considering the complexity of realistic testbed, it’sdifficult to test all cases under real environment. Simula-tion tool could accelerate the related research on NFV bytesting more cases and various situations. We propose anovel and repeatable simulation tool CloudSimNFV forNFV scenario based on CloudSim, aiming to improvethe related work on NFV.

We describe our framework design and its compo-nents in detail. Energy consumption model and loadsgeneration approaches are also discussed. Validationexperiments illustrate the accuracy of CloudSimNFV.Performance evaluations show the energy efficient al-gorithms performance under NFV scenario.

We would also like to extend CloudSimNFV withmore scheduling algorithms for more various scenariosin the future. In realistic data NFV data centers, con-straints may exist between different servers. In the laterversions of CloudSimNFV, the customized constraintswould also be supported.

ACKNOWLEDGEMENT

This research is partially supported by China Na-tional Science Foundation (CNSF) with project ID61450110440.

REFERENCES

[1] Amazon, Amazon Elastic Compute Cloud,http://aws.amazon.com/ec2/, 2013.

[2] Basta A, Kellerer W, Hoffmann M, et al. Applying NFV andSDN to LTE mobile core gateways, the functions placementproblem[C] Proceedings of the 4th workshop on All thingscellular: operations, applications, and challenges. ACM, 2014:33-38.

[3] Bolla R, Lombardo C, Bruschi R, et al. DROPv2: energyefficiency through network function virtualization[J]. Network,IEEE, 2014, 28(2): 26-32.

[4] Buyya R, Yeo C, Venugopal S, Broberg J, and Brandic I, Cloudcomputing and emerging it platforms: Vision, hype, and realityfor delivering computing as the 5th utility, Future Generationcomputer systems, vol. 25, no. 6, pp. 599C616, 2009.

[5] Calheiros R, Ranjan R,Beloglazov A, et al., CloudSim: A Toolkitfor Modeling and Simulation of Cloud Computing Environmentsand Evaluation of Resource Provisioning Algorithms, Software:Practice and Experience, vol.41, no.1, pp.23-50, 2011

[6] Carella G, Corici M, Crosta P, et al. Cloudified IP MultimediaSubsystem (IMS) for Network Function Virtualization (NFV)-based architectures[C]//Computers and Communication (ISCC),2014 IEEE Symposium on. IEEE, 2014: 1-6.

[7] Chiosi M, Clarke D, Willis P, et al. Network functions virtual-isation introductory white paper[C]//SDN and OpenFlow WorldCongress. 2012.

[8] Economou D, Rivoire S, Kozyrakis C, et al. Full-system poweranalysis and modeling for server environments[C]. InternationalSymposium on Computer Architecture-IEEE, 200

[9] Gurout T, Monteil T, Da Costa G, et al. Energy-aware simulationwith DVFS[J]. Simulation Modelling Practice and Theory, 2013,39: 76-91.

[10] Gulati A, Shanmuganathan G, Holler A, Ahmad I, Cloud-scaleresource management: challenges and techniques, VMware Tech-nical Journal, 2011.

[11] Kliazovich D, Bouvry P, Khan S, Greencloud: A packet-levelsimulator of energy-aware cloud computing data centers. IEEEConference on GlobalTelecommunications, pp.1-5, 2010.

[12] Mastroianni C, Meo M, Papuzzo G. Probabilistic consolidation ofvirtual machines in self-organizing cloud data centers[J]. CloudComputing, IEEE Transactions on, 2013, 1(2): 215-228.

[13] Nunez A, Vazquez-Poletti J, Caminero A, et al., iCanCloud: AFlexible and Scalable Cloud Infrastructure Simulator, Journal ofGrid Computing 10:185, C209, 2012

[14] Riggio R, Rasheed T, Granelli F. Empower: A testbed for networkfunction virtualization research and experimentation[C], FutureNetworks and Services (SDN4FNS), 2013 IEEE SDN for. IEEE,2013: 1-5.

[15] Soares J, Dias M, Carapinha J, et al. Cloud4nfv: A platformfor virtual network functions[C] Cloud Networking (CloudNet),2014 IEEE 3rd International Conference on. IEEE, 2014: 288-293.

[16] Stress-NG–Tool to load and stress your Ubuntu system,http://www.ubuntugeek.com/stress-ng-tool-to-load-and-stress-your-ubuntu-system.html, 2015

[17] Tian W, Xu M, Chen A, et al. Open-source simulators forCloud computing: Comparative study and challenging issues[J].Simulation Modelling Practice and Theory, 2015.

[18] Tian W, Zhao Y, Xu M, et al. A toolkit for modeling andsimulation of real-time virtual machine allocation in a cloud datacenter[J]. Automation Science and Engineering, IEEE Transac-tions on, 2015, 12(1): 153-161.

[19] Xu M, Tian W, Wang X, et al. FlexCloud: A Flexible andExtendible Simulator for Performance Evaluation of VirtualMachine Allocation[J]. arXiv preprint arXiv:1501.05789, 2015.

[20] Yathiraj U, Debo D, Ramki, Increasing Infrastructure Efficiencyvia Optimized NFV Placement in OpenStack Clouds, OpenStackAtlanta Summit, 2014

8