Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders...

23
Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network Adapter X540 with Intel® Node Manager and VMware* vSphere 5.0* Audience and Purpose This reference architecture outlines a usage of energy management and network technologies as part of planning, provisioning, and optimizing strategies in enterprise and cloud data centers to reduce energy cost and address constrained power situations while realizing 10 Gigabit Ethernet (10GbE) throughput in their environments. This paper will give IT professionals architectural insights with methodical instructions to demonstrate Intel® Manageability products, run end-to-end on VMware* vSphere* 5.0 and 10GbE on Intel® Xeon® processor-based servers. It is intended for data center administrators and enterprise IT professionals who seek energy management solutions to achieve better energy efficiency and power capacity utilization within existing or new data centers. This study also shows the relationship between processor, network throughput, and power management. Our conclusions can inform the IT professional in how understanding the impact of power management on network throughput can bring agility and reduced costs to their environments. Intel® Cloud Builders Guide to Cloud Design and Deployment on Intel® Platforms Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network Adapter X540 with Intel® Node Manager and VMware* vSphere 5.0* January 2013 Intel® Xeon® E5 Processor E5 Family Intel® Ethernet Adapters

Transcript of Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders...

Page 1: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

Intel® Cloud Builders GuideIntel® Xeon® Processor-based ServersPower Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network Adapter X540 with Intel® Node Manager and VMware* vSphere 5.0*

Audience and Purpose This reference architecture outlines a usage of energy management and network technologies as part of planning, provisioning, and optimizing strategies in enterprise and cloud data centers to reduce energy cost and address constrained power situations while realizing 10 Gigabit Ethernet (10GbE) throughput in their environments. This paper will give IT professionals architectural insights with methodical instructions to demonstrate Intel® Manageability products, run end-to-end on VMware* vSphere* 5.0 and 10GbE on Intel® Xeon® processor-based servers. It is intended for data center administrators and enterprise IT professionals who seek energy management solutions to achieve better energy efficiency and power capacity utilization within existing or new data centers. This study also shows the relationship between processor, network throughput, and power management. Our conclusions can inform the IT professional in how understanding the impact of power management on network throughput can bring agility and reduced costs to their environments.

Intel® Cloud Builders Guide to Cloud Design and Deployment on Intel® PlatformsPower Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network Adapter X540 with Intel® Node Manager and VMware* vSphere 5.0*

January 2013

Intel® Xeon® E5 Processor E5 Family Intel® Ethernet Adapters

Page 2: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

2

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Table of ContentsExecutive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Test Configuration and Environment Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Detailed Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Data Center Power Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Details Steps for Baseboard Management Controller (BMC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Details Steps for Intel® Data Center Manager (Intel® DCM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Establishing Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Use Case Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Use Case One . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Use Case Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Use Case Three . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Usage Model Test Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Page 3: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

3

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Executive SummaryCloud data centers require focused planning in regards to density and environmentals. This paper is intended to define the relationship of server power to network throughput in a virtualized environment. To maintain a highly efficient and optimized environment, we’ll limit power to the test environment to show the impact to network traffic to our System under Test (SUT) in a virtualized environment.

Our goal is to create optimized and energy-efficient usage model that can reduce power costs in the data center. Our requirements include the capability to monitor and regulate power in real-time at server, rack, zone, or data center levels. This means the ability to monitor and manage aggregated power consumption within a rack, zone, or data center based on available power and cooling resources.

We’ll use standard network load generation tools to drive the limits of the Intel® Ethernet Converged Network Adapter X540-T2. Our workload profiles are built for maximum throughput to determine how much network capping can be applied before the performance degenerates. The approach is to match actual performance against the lowest power level setting. This showcases the ability to enforce power SLAs across different populations of users with differing workloads. Multiplying this effect in data centers with hundreds or thousands of servers can significantly reduce costs of power and cooling.

To support this evolution, Intel works with end users to create an open data center roadmap of usage models that address key IT pain points for more secure, efficient, and simple cloud architectures built on a foundation of transparency. This paper describes the configuration details for setting up the virtualized scenarios, along with the functionality and performance observed in our lab

testing. Commonly used tools iPerf and NTttcp are used to achieve near line rate performance in most cases. This work is meant to bring new insights on the use of native networking initiators.

IntroductionAt its core, a data center is a centralized repository for storage, management, and dissemination of data. IT and enterprise managers are always challenged to finds efficiencies and cost reduction in regards to their datacenters all while maintaining data integrity.

Virtualization I/O performance is one of the key considerations in IT shops. Virtualization increases efficiencies in the datacenter through it impacts on environmental systems like power, cooling, and spacing. As IT departments look to reduce costs and improve server efficiency, they are turning increasingly towards server virtualization. If you look at the trend of virtualization deployments in the last decade or so, initially it started with consolidation of software development and testing environments and moved on to consolidation of production applications.

High-speed networking has quickly become a hot topic amongst IT professionals. As network transport speeds increase, LAN and storage consolidation becomes achievable and desirable. This is largely due to consolidation of system resources and reduction of network complexities. Regardless of whether this is a traditional or cloud data center, creation of a seamless data transport is of utmost importance to any data center administrator.

Management tools for the data center have become very sophisticated in the granularity to which they control their environments. In addition, more systems can be managed by fewer administrators. Intel® Node Manager is an excellent

example. Intel Node Manager was implemented on Intel® server chipsets starting with Intel® Xeon® processor 5500 series platforms. After significant improvements, Intel® Node Manager 2.0 was introduced in 2012 on server platforms supporting the Intel® Xeon® processor E5 family (presented in this document).

The success of this experiment was yielded from key Intel platform hardware capabilities such as:

• Real-time Server Energy Usage Monitoring, Reporting, and Analysis to get continuous and actual energy usage visibility via agentless monitoring of the servers along with other devices and systems in the enterprise network, data center, and facilities.

• Power Guard Rail and Optimization of Rack Density by imposing power guard to prevent server power consumption from straying beyond a preset limit.

• Disaster Recovery/Business Continuity by applying significantly lower power caps to reduce power consumption and heat generation when unforeseen circumstances like power outage and cooling system failure occurs.

• Intel® QuickData Technology enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput.

• Intel® Integrated I/O and Intel® Data Direct I/O Technology (Intel® DDIO) increases bandwidth, lowers latency, and reduces power consumption through direct communication between the NIC and the processor cache.

• Extended Message Signaled Interrupts (MSI-X) distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance.

Page 4: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

4

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

• Receive Side Coalescing (RSC) aggregates packets from the same TCP/IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing.

• Low Latency Interrupts tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency.

The purpose of this reference architecture is to demonstrate that 10GbE line speed can be achieved while regulating power to the system or systems, to show how to setup and configure different parameters, and to

Figure 1: Test Environment Overview

serve as a reference for end users in deploying these technologies in their virtualized environment.

Test Configuration and Environment OverviewOur physical environment at its core is simple. We use two Intel® Xeon® processor E5-2600 series-based servers running VMware* ESXi* 5.0 u1.0 with five virtual machines (VMs) on each server. Each VM is run from an iSCSI shared volume on an EMC* VNX 5500 storage array. All systems are attached by a Cisco* Nexus 5548 switch and Cisco Nexus* 2232TM Fabric Extension Module (FEX) for the

performance portion of the testing and an Extreme X460 for remote access and hosts management. 10GbE server connections were made with the Intel® Ethernet Converged Network Adapter X540-T2 and 1GbE connection were made with the onboard Intel® Ethernet Server Adapter I350-F2. See Table 1 for details of the systems in this setup.

Page 5: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

5

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Detailed Hardware Configuration

Intel® Xeon® E5 Processor Family Server Under Test (SUT)

• Intel Xeon processor E5-2680 @ 2.70 GHz (2 sockets, 8 cores/socket), SMT enabled, 64Gb RAM, 120GB SSD

• VMware* ESXi* 5.0 build 620860

• VM – 20GB VMDK, 4 vCPU, 4GB RAM, W2K8 R2, 5GB Secondary Storage

• VMware vSphere* Server 5.0 build 5.0.0.16608

• VMware vSphere Client 5.0 build 5.0.0.16964

Data Center Management • Intel® Data Center Manager 3.1.0.7332

Network Configuration • Cisco Nexus 5548 Chassis 32x10GE/Supervisor, Image version 5.1(3)N1(1)

• Cisco Nexus 2232TM FEX 32x10G Base-T

• Intel® Ethernet Converged Network Adapter X540-T2, version: 3.9.13-NAPI

• Intel® Ethernet Server Adapter I350-F2, version: 2.1.11.1

• VMware ESXi Clustered vSphere Distributed Switch (VDS)

• VMware ESXi Native iSCSI Initiator

Storage Configuration • EMC VNX* 5500 Storage System (Unified); Version: 05.31.000.5.709

• 50TB Capacity, 12GB Memory

• EMC Control Station Linux release 3.0 ( NAS 7.0.51)

Cluster and VDS Configuration • Intel® R2300GZ Server (S2600GZ Server Board) Cluster

• PCN_VDS_X540

Application Configuration • IOmeter v.2006.7.27

• # of Managers = 1

• # of Workers = 8

• # of Outstanding I/O 8

• I/O Size = 1MB

• NTttcp v.1.0.0

• iPerf v.2.0.0

Data Collection • CPU utilization, storage I/O, and network throughput captured from “esxtop”

Table 1: Setup and Configuration

Page 6: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

6

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Data Center Power Management

The Baseboard Management Controller (BMC) and detailed steps for Intel® Data Center Manager (Intel® DCM) are the focus of this configuration. Servers, virtual machines, switches, storage, and test programs have to be configured but are outside the scope of this paper. The BMC architecture is shown in Figure 2 below is configured as described below.

1. In the BIOS, configure BMC network settings with static or DHCP option as desired, and provide the BMC hostname. We used DHCP. Note the BMC hostname or IP address.

2. Note the user name and password of the BMC user with administrator privileges. Either use “root” user ensuring ‘administrator’ privileges are granted, or add another user.

3. Install the operating system, applications, and workloads as desired.

4. Visual Data Center connects to the servers out of band with BMC hostname and log in credentials. There is no agent installed on the server.

Figure 2

Page 7: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

7

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Detailed Steps for Baseboard Management Controller (BMC)

1. Press ‘F2’ to get into the BIOS on server startup navigate to the “Server Management Tab” and select “BMC LAN Configuration.”

2. Select the IP source to configure a Dynamic or Static for the management IP address.

Page 8: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

8

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

3. Select the “BMC DHCP Host Name” to configure a unique hostname for the management IP.

4. Select the “User ID” to ensure that a login name exists.

Page 9: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

9

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

5. Select the “User Password” to set a password for management access.

6. Configuration for Intel DCM power manageability is complete.

7. (Optional Step) If there is a Remote Management Module (RMM) installed on this system, the user can open an Internet browser and type in the BMC IP address noted above; in this example http://10.18.223.123. This will open a login window to the server management interface. Login with the default username/password setup in your documentation.

Page 10: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

10

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

8. (Optional Step) A successful login will take the user to the WebUI.

Detailed Steps for Intel® Data Center Manager (Intel® DCM)

1. Login to the Intel DCM via HTTP or directly through the application executable on the server desktop. ‘RT+Click’ in open below “Desktop Hierarchy” to add the servers to be managed.

Page 11: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

11

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

2. Add the server management details that were setup in the BMC:

a. Entity Name

b. Type: Server

c. Connection Name

d. BMC IP Address

e. BMC Username/Password

f. Derated Power or the graph threshold

Page 12: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

12

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

3. Intel DCM now is configured to take basic trending.

4. For this use case, we will need to group the servers. ‘RT+Click’ in open below “Desktop Hierarchy” to add a group.

Page 13: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

13

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

5. Name the group and give it a type. Types can be room, row, rack, or data center. Click 'save' to continue.

6. Simply drag the server or servers into the group.

Page 14: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

14

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

7. ‘RT+Click’ the new group to add a policy. At the “Policy Lists” window, select New to add a new policy.

8. At the “Policy Details” window, set the Policy Type, Policy Entity, Policy Threshold, and Policy Reserve. Click ‘save’ to return to the “Policies List” window. Enable the policy and click close to return to Intel DCM.

Page 15: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

15

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

9. The power cap and average server groups power are now clearly seen. Note that these values do not reflect any load.

Establishing Thresholds

Thresholds are guidelines that administrators can use to establish common boundaries for compute monitoring. The reader should understand that the power and temperature thresholds used in this experiment were created to push the limits of network and storage throughputs and should not be used as a guideline. Users recreating these experiments should develop their own methods to determine the power and temperature thresholds as each data center is unique and generalizations should not be used.

Tests Subtests: Network Subtests: Storage + Network

1. Baseline No Power Cap

2. Power Restriction: 425W

3. Power Restriction: 350W

4. Power Restriction: 300W

5. Power Restriction: 275W

1x VM 10Gb Receive (Rx)

1x VM 10Gb Transmit (Tx)

1x VM 10Gb Bidirectional (Bx)

5x VM 10Gb Receive (Rx)

5x VM 10Gb Transmit (Tx)

5x VM 10Gb Bidirectional (Bx)

5x VM 10Gb Storage Read

5x VM 10Gb Storage Write

1x VM 10Gb Rx + Read

1x VM 10Gb Tx + Write

5x VM 10Gb Rx + Read

5x VM 10Gb Tx + Write

Page 16: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

16

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Use Case OverviewIT organizations (including Intel IT) face significant data center power and cooling challenges. So, companies seek alternative approaches that focus on more efficient use of existing data center power. Power optimization of the workloads is one such approach to achieve power efficiency.

Power optimization an understanding of the workload profile and a performance loss target, if any, it is not to be exceeded. Developers perform a series of experiments to characterize how much capping can be applied before the

performance target is hit. Afterwards, during normal operations, the applications engineer sets power capping targets based on the prior measurements. The system is now said to be “optimized,” because the impact of the application of these caps is now known.

Use Case One: Monitoring Power, Temperature, and Events of a Device or Group

The most basic use case of monitoring gives a data baseline to evaluate the systems in the Intel DCM group. It gives enough data to make conscious decision

for power limiting. This proves very useful as data center administrators are expected to have information on rack density, thermal footprint, and overall power consumption on a measured basis across their data centers. These data points can be aggregated to make critical decisions on server placement, manage ways to optimize cooling, and also maximize the usage of power circuits throughout the room. The figure below shows the server group running at idle for the majority of the time and some moderate workload, all while sustaining a 425W static power cap.

Figure 3: Server Group at Idle with 425W Static Power Cap

Page 17: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

17

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Use Case Two: Creating Power Optimized Policies to Understand Power Efficiencies

At the heart of this experiment, power policies are created in Intel Node Manager that restrict or power cap the server group in a linear fashion to understand the impact to network throughput. This is done to identify power optimization points and power efficiencies. While these policies can be set static or dynamic, all policies were static. The initial baseline tests were done without power restrictions with the following tests restricting power incrementally. In all, five power cap tests were run with up to twelve subtests per power cap test. The example below shows a 300W power cap test with all 12 subtests.

Figure 4: 300W Power Cap with 12 Subtests

Use Case Three: Varied vCPU and Memory Configurations to Understand Power Efficiencies

Each test group was run with VMs in two differing configuration. The first set of tests was run on one and five VM runs, each VM configured with four vCPU and 4GB memory. The second set of tests was run on one and five VM runs, each VM configured with six vCPU and 12GB memory. These test groupings were derived to see the impact of power restrictions to VMs with varying power needs.

Tests Subtests: Network Subtests: Storage + Network

6. Baseline No Power Cap

7. Power Restriction: 425W

8. Power Restriction: 350W

9. Power Restriction: 300W

10. Power Restriction: 275W

1x VM 10Gb Receive (Rx)

1x VM 10Gb Transmit (Tx)

1x VM 10Gb Bidirectional (Bx)

5x VM 10Gb Receive (Rx)

5x VM 10Gb Transmit (Tx)

5x VM 10Gb Bidirectional (Bx)

5x VM 10Gb Storage Read

5x VM 10Gb Storage Write

1x VM 10Gb Rx + Read

1x VM 10Gb Tx + Write

5x VM 10Gb Rx + Read

5x VM 10Gb Tx + Write

Page 18: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

18

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Usage Model Test Results and AnalysisAs mention above, this reference architecture has three primary use cases; baseline power, optimized policies, and varied processor and memory. At completion there were 120 tests runs. Upon analysis of each test group, it was decided to focus on the five VM receive and transmit tests in both four vCPU and six vCPU groups. This first set of graphs below show the actual ESXTOP VMNIC network throughput and the host CPU trending for each power cap test.

Figure 5: Graphed Results of Five VM Tests; Four vCPU/4GB RAM and Six vCPU/12GB RAM

Page 19: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

19

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

While this paper focuses on the series of five VM tests, a baseline of one VM test was needed. This next set of graphs below show the average network throughput and average CPU utilization for both one VM and five VM tests in both four vCPU and six vCPU configurations.

Figure 6: 1x VM Average Network Throughput and CPU Utilization; Four vCPU/4GB RAM and Six vCPU/12GB RAM

Page 20: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

20

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Figure 7: 5x VM Average Network Throughput and CPU Utilization; Four vCPU/4GB RAM and Six vCPU/12GB RAM

The third set of graphs below show the relative variance from baseline for the five VM test groups. Each power cap group’s (425W, 350W, 300W, and 275W) average network throughput and CPU utilization was divided by the baseline average. Just to note, the axis value of 100 is actually is really the value of the baseline i.e. the 425W network throughput value for Rx 4x4 shows the difference between 9.2Gbs and 9.1Gbs, the average value of network throughput of the 425W test. The same is done for CPU utilization.

Figure 8: Efficiency Variances to Baseline

Page 21: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

21

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Analysis

Starting with the first set of data charts, we see all five VM test achieved line-speed at a very steady state in both network throughput and CPU utilization. We see from comparison of the second set of charts that we achieve a better efficiency in scale between the one VM baseline test and the five VM tests overall.

We note that for this two server group, 300W is the teetering point. Our workload began to take longer to complete with a power cap for less or equal to 300W with the 275W test time extending up to 15 percent. Additionally, at a power cap of less than 300W, we begin to see network throughput thrash and loss. We saw no significant difference in the impact to transmit or receive network throughput at power cap settings greater than 300W noting a variance of approximately 2 percent loss of network throughput in the greater than or equal to 300W tests. We didn’t see value in getting more granular within 25W.

As expected, CPU utilization was more noticeable throughout the power cap tests due to the direct relationship between electric power and processor power. We also noted that VM network throughput on the VMs with higher vCPU and memory (6 vCPU and 12 GB RAM) assignments saw a larger impact by the power cap. Yet with the linear stepping down of power states for each power cap test, workloads at greater to or equal to 300W were minimally impacted. Additionally we saw the CPU states become less steady at less than 300W power cap tests.

Our baseline for the two server group was 520W. In power cap steps for 425W, 350W, and 300W, all network throughput test reduction was within 2 percent max total. This translates to a reduction of overall power to the group by 43 percent yielding a less than a 2

percent network throughput loss and a 35 percent reduction to overall average CPU utilization.

ConclusionsWith companies consistently focused on lowering total cost of ownership (TCO) while still meeting customer demands for increased capability, the benefit of tuning your environment is crucial. The test results noted in this paper are just one way to optimize your datacenters to meet operational efficiencies. Intel components are becoming faster, power-efficient, and manageable offering administrators more control of their enterprise environments while meeting or exceeding computing requirements.

We believe our test environment is scalable, manageable, and efficient. Setting power limits removes the high-cost power peaks in data center racks and the results of these tests were able to reduce power significantly while keeping network throughput line-rates.

More Intel/VMware reference architectures are available at the Intel® Cloud Builders Web site at www.intel.com/cloudbuilders.

Page 22: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

22

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

References• Data Center Energy Efficiency with Intel® Power Management Technologies - IT@Intel Brief, February 2010

• 8Gb Classic Fibre Channel vs. 10GbE Unified Networking in a Virtualized Environment – Intel Cloud Builders Reference Architecture 2011: http://www.intelcloudbuilders.com/docs/Intel_Cloud_Builders_Fibre_Channel_Unified_Networking_March_2012.pdf

• Data Center Infrastructure and Energy Management with 3-D Visualization with Visual Data Center* – Intel Cloud Builders Reference Architecture 2012 - http://www.intelcloudbuilders.com/docs/Intel_Cloud_Builders_VDC_March_2012.pdf

• Data Center Energy Management with Dell, Intel, and ZZNode* – Intel Cloud Builders Reference Architecture 2011 - http://www.intelcloudbuilders.com/docs/Intel_Cloud_Builders_Dell_ZZNode.pdf

• Data Center Energy Management Dell, Intel, and JouleX – Intel Cloud Builders Reference Architecture 2011 - http://www.intelcloudbuilders.com/docs/Intel_Cloud_Builders_Dell_JouleX_Data_Center_Energy_March_2012.pdf

• Microsoft System Center Virtual Machine Manager Self-Service Portal* 2.0 with Policy-Based Power Management – Intel Cloud Builders Reference Architecture 2011 - http://www.intelcloudbuilders.com/docs/ICB_Microsoft_VMMSSP_power_mgmt_FINAL_July2011.pdf

• What’s new in VMware vSphere 5.0 - Performance: http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-50-Performance-Technical-Whitepaper.pdf

• What’s new in VMware vSphere 5.0 – Networking: http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-50-Networking-Technical-Whitepaper.pdf

• Intel Cloud Builders Reference Architecture Library: www.intelcloudbuilders.com/library

• Iometer: Website: http://www.iometer.org

• Intel Reference Site: http://www.intel.com/network/connectivity/resources/technologies/10_gigabit_ethernet.html

• Unified Networking Reference Site: http://www.datacenterknowledge.com/archives/2011/01/28/intel-offers-unified-networking-technology/

• Intel® Virtualization Technology for Connectivity http://www.intel.com/network/connectivity/solutions/virtualization.html

• Improving Network Performance in Multi-Core Systems http://www.intel.com/content/www/us/en/ethernet-controllers/improving-network-performance-in-multi-core-systems-paper.html

• Intel Reference Site: http://newsroom.intel.com/community/intel_newsroom/blog/2011/01/27/intel-simplifies-the-data-center

• Intel Product Site: http://www.intel.com/technology/comms/unified_networking/index.html

• DCB: http://www.definethecloud.net/data-center-bridging

• Wikipedia Reference Site: http://en.wikipedia.org/wiki/10GB_Ethernet

• Wikipedia Reference Site: http://en.wikipedia.org/wiki/Data_Center_Bridging

Page 23: Intel® Cloud Builders Guide to Design and Deployment on Intel® … · Intel® Cloud Builders Guide Intel® Xeon® Processor-based Servers Power Capping Impact to the Network: 10GbE

Intel® Cloud Builders Guide: Power Capping Impact to the Network: 10GbE Intel® Ethernet Converged Network

Disclaimers∆ Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See www.intel.com/

products/processor_number for details.INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL

PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or by visiting Intel’s Web site at www.intel.com.

Copyright © 2013 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon inside, and Intel Intelligent Power Node Manager are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others.