Where IT perceptions are reality

Technology Brief

Flexible Blade Server I/O

Featuring

HP ProLiant Gen8 Servers

HP Virtual Connect

HP 10GbE FlexFabric Adapters

(provided by Emulex)

HP Fibre Channel Server Adapters

(provided by Emulex)

Copyright 2013© IT Brand Pulse. All rights reserved.Copyright 2013© IT Brand Pulse. All rights reserved. Document # APP2013011 v16 August, 2013Document # APP2013011 v16 August, 2013

Document # APP2013011 v16 August, 2013 Page 2 of 12

Heterogeneous Workloads

Virtualization Comes of Age

The maturation of virtualization platforms, highlighted by the

recent releases of VMware vSphere 5.1 and Windows Server 2012

Hyper-V, has sharpened the focus of data center management on

increasing virtual machine (VM) density and virtualizing Tier-1

application workloads to maximize their virtualization ROI.

This technology examines the ability of Cisco UCS and HP

BladeSystems to provide useable I/O bandwidth and I/O flexibility

that are needed to support the growth of VM density and

heterogeneous workloads.

VM Density Driving Need for Useable I/O Bandwidth

IT Brand Pulse surveys indicate that IT pros project VM density will

almost double from 13 VMs/server to 24 VMs/server within the

next 24 months—driving a need for more useable

I/O bandwidth.

Heterogeneous Workloads Driving Need for Flexibility

More VMs/server means more diversity in the applications the

server I/O system must support. This drives the need for flexibility

in how blade server I/O bandwidth and policies are provisioned to

different applications.

Converged Infrastructures Demand I/O Flexibility

The convergence of compute, storage and networking into a

shared resource pool is gaining acceptance and deployment in

enterprise data centers. By definition, this sharing of resources by

multiple applications and lines of business is also driving the need

to have a flexible, high throughput and low latency blade

enclosure I/O.

The average number of VMs per

server in my environment:

What I need most to increase

the density of VMs per physical

servers is more:

41.1% of IT professionals surveyed said “server virtualization” was the

application most driving the adoption of 10GbE in their data center. 41.1%

Document # APP2013011 v16 August, 2013 Page 3 of 12

Anatomy of Blade Server Chassis I/O

LANLAN--onon--Motherboard is an Ethernet adapter which includes the first Motherboard is an Ethernet adapter which includes the first

network ports configured on a server.network ports configured on a server. LOM

Ethernet and Fibre Channel uplinks

to LANs and SANs

Ethernet and Fibre

Channel

downlinks to mid-

plane and server

adapters

Embedded

switches and/or

pass-through

modules

Mid-plane

Ethernet LAN-on-

Motherboard

(LOM) adapters

Ethernet or Fibre

Channel

Mezzanine

Adapters

Blade server chassis 16 blade servers and 2 switches in chassis

1 LOM adapter on each server and 1 mezzanine adapter on each server

Application Performance Depends on a Healthy Network

Every blade server chassis has an entire network embedded inside to carry east-west traffic between servers,

and north-south traffic to top-of-rack, end-of-row, and core switches upstream. The I/O performance of

heterogeneous applications running on virtualized blade servers can differ significantly based on:

The ability of some blade server networks to scale without over-subscription

Providing native support for multiple network protocols

Delivering network traffic with low latencies by minimizing route hops

Document # APP2013011 v16 August, 2013 Page 4 of 12

Useable I/O Bandwidth

The ratio of the switching bandwidth to the aggregate The ratio of the switching bandwidth to the aggregate

bandwidth of all ingress (incoming) ports. High ratios bandwidth of all ingress (incoming) ports. High ratios

can result in latency and reduce scalability.can result in latency and reduce scalability. Oversubscription

Oversubscription Reduces Available Bandwidth

Oversubscription occurs when the I/O capacity of the adapter ports connected to a switch port exceeds the

capacity of the switch port. The oversubscription ratio is the sum of the capacity of the adapter ports divided

by the capacity of the switch port.

Oversubscription also results from the imbalance between outgoing bandwidth and incoming bandwidth

available from a blade server chassis to top-of-rack, end-of-row and core switches. Severe oversubscription

reduces useable bandwidth, increases network latency, slows user response time, and makes it difficult to

deliver deterministic application performance.

Blade server designers maximize useable bandwidth and minimize oversubscription. Not all blade server

architectures are optimized for these considerations.

Multi-Tiered Networks Increase Applications Latency

As data traverses a network fabric, each network tier in the path adds delays, negatively impacting

applications performance. As a result, designers also architect for minimal tiering to reduce these latencies.

Cisco UCS Hierarchical Network Architecture

The Cisco UCS design supports 8 blade servers per chassis and defaults to a hierarchical, multi-tiered, “north-

south” networking architecture in Cisco’s preferred End Host Mode. This mode is enabled by creating a

logical system where server-to-server traffic for hosts in the same fabric, travels outside the chassis to the

Fabric Interconnect.

This mode requires two fabric interconnects and two fabric extenders — with each extender providing up to

160Gb/s of total bandwidth for on active-active configuration. A fully loaded blade with a VIC 1240 quad-port

10GbE adapter plus a quad-port mezzanine extender requires 80Gb/s for a total of 640Gb/s with an 8 blade

enclosure—resulting in up to 4:1 oversubscription and reduced available bandwidth.

Oversubscription is exacerbated when the servers are on different fabrics, requiring server-to-server traffic to

travel north of the Fabric interconnects to the end of row aggregation switch. This architecture prevents

network loops but also prevents server-to-server traffic flow between interconnects, instead forcing it to

travel all the way to the end of row switch—resulting in the fabric interconnects become a north-south

bottleneck and up to 10:1 oversubscription, significantly limiting useable I/O bandwidth.

Document # APP2013011 v16 August, 2013 Page 5 of 12

Applications Latency

Latency is the time between the start and completion

of one action measured in microseconds (µs) . Latency

Cisco UCS Network Architecture Latency

The scenarios described also illustrate the additional network tiers elements built into the UCS architecture

which can result in increasing total latency with a commensurate reduction in applications performance and

unpredictable applications behavior. A calculation of network latency is examined in greater detail later in

this report.

End of Row Switch (Two)

16 UCS blade servers in 2 chassis

Fabric Interconnect

Fabric Extender

Single and Multi-fabric Networking with Cisco UCS

North-south traffic with fabric interconnects and end of row Switches is required for single or

multi-fabric traffic flow

Fabric 1 Fabric 2

Oversubscription

Document # APP2013011 v16 August, 2013 Page 6 of 12

HP ProLiant Gen8 Blade Server

HP’s “wireHP’s “wire--once” interconnect solution for cloud and once” interconnect solution for cloud and

virtualized data centers delivers flatter networks and re-virtualized data centers delivers flatter networks and re-

duced cable costs.duced cable costs. Virtual Connect

The Ultimate in Useable Bandwidth

HP ProLiant Gen8 blade servers offer the flexibility of network architectures optimized for server-to-server

“east-west” traffic or “north-south” traffic in/outbound from the blade enclosure. This is accomplished via

configuration options for the HP FlexFabric server network adapters and HP Virtual Connect FlexFabric

modules.

If the application places demands for server-to-server “east-west” traffic within the enclosure, an active-

standby configuration is utilized with the upstream network switch connected to a single port on each HP

Virtual Connect FlexFabric module. However, if the application demands predominantly server-to-core

“north-south” traffic, an active-active configuration enables two uplink ports on the network switch,

increasing overall bandwidth and reducing oversubscription.

A c7000 enclosure with 16 blades, each with two 10GbE FlexFabric adapters and two mezzanine 10GbE

adapters requires 960Gb/sec total bandwidth. The enclosure’s available internal mid plane bandwidth of

7.2Tb/sec, about 7x the required bandwidth – results in zero oversubscription for east-west traffic flow.

For traditional north-south traffic, HP Virtual Connect FlexFabric modules with 16 downlink and 8 uplink ports

results in 2:1 oversubscription, similar or lower than Cisco’s, depending on configurations.

The 7.2 Tb/s useable bandwidth between device bays and interconnect bays allows server-to-server traffic to stay

within a single c7000 enclosure

Document # APP2013011 v16 August, 2013 Page 7 of 12

HP ProLiant Gen8 Blade Server

High performance 10GbE ports are now available for

blade servers in LAN-on-Motherboard (LOM) and mezza-

nine adapter form factors. 10GbE

Network Latency Comparison

The HP Virtual Connect FlexFabric architecture flexibly supports both single and separate fabric traffic pattern

implementations. However, configurations architected with separate fabrics address the enterprise

requirements for resiliency through redundancy.

The HP solution delivers lower latencies for both configurations, based on the calculations from

specifications provided by HP and Cisco.

HP Virtual Connect Network Architecture delivers data with more than 50% lower latency

Total Data Latencies For HP and Cisco Architectures—Lower is better

Document # APP2013011 v16 August, 2013 Page 8 of 12

I/O Flexibility

Support for Only One Network Wire Reduces I/O Flexibility

Based on IT Brand Pulse surveys, 40% of IT organizations are not converging with FCoE. For the 40% of IT

professionals who have been too busy to look at FCoE, or who say they have no plans to converge their LANs

and SANs, parallel Ethernet and Fibre Channel infrastructure will be deployed.

The modular design of blade servers make them inherently flexible. But not all blade server platforms are

equal when it comes to hosting multiple heterogeneous virtualized workloads and delivering I/O flexibility.

While some blade server designs accommodate Ethernet and native Fibre Channel connectivity, the Cisco

UCS design only supports Ethernet connectivity.

40% of IT Professionals are not converging their networking infrastructure

to Fibre Channel over Ethernet. 40%

Wanted: Parallel Ethernet & Fibre Channel Networks

in 2013, the prevalent data center network architecture remains a parallel network architecture, including a mix of specialized NIC, iSCSI, and

Fibre Channel host adapters, as well as Ethernet and Fibre Channel switched fabrics. Cisco UCS blade servers support only Ethernet

connectivity. Adoption of FCoE technology is required to access installed Fibre Channel resources.

Document # APP2013011 v16 August, 2013 Page 9 of 12

HP ProLiant Gen8 Blade Server The Ultimate in I/O Flexibility

HP ProLiant Gen8 blade servers are designed for I/O flexibility with a choice of FlexFabric converged

networking or parallel Ethernet and Fibre Channel networks. The ProLiant Gen8 blade servers are also fully

compliant with Windows Server 2012 Virtual Fibre Channel—an innovation that will play an important role in

the virtualization of Tier-1 workloads with Microsoft Hyper-V.

Choice of I/O Convergence or Divergence meets the needs of

heterogeneous workloads on a single blade platform. Flexibility

HP BladeSystem c7000 enclosure with ProLiant Gen8 blade servers

HP Virtual Connect FlexFabric

10Gb/24-port module supports

connectivity to native Fibre Channel 3PAR

storage at a lower cost than using Fibre

Channel switches

HP Virtual Connect FlexFabric 10Gb/24-port module supports LAN, NAS, iSCSI and FCoE connectivity

Native Fibre Channel server adapter

Over 12 million ports shipped on this stack

Complete enterprise OS support including Solaris

Ethernet LAN on Motherboard (LOM)

Supports LAN, NAS, iSCSI and FCoE connectivity

HP FlexFabric Ready

HP 659818-B21 Mezzanine

FC Adapter

HP FlexFabric 10Gb 2-port 554FLB Adapter

Document # APP2013011 v16 August, 2013 Page 10 of 12

Fibre Channel without Switches

Direct Attached Storage Reduces CAPEX and OPEX

Complementing the HP ProLiant blade server’s flexibility of Fibre Channel (FC) connectivity capability, HP’s

Flat SAN technology enables enterprise class direct attached storage to HP 3PAR Storage Systems without

requiring a Storage Area Network (SAN) fabric. This scalable solution provides connectivity for up to 192 FC

ports and 192 Petabytes of storage capacity. Benefits of this architecture include:

Reduced CAPEX on SAN fabric switches and associated software licenses

Reduced ongoing management and support OPEX of multiple touch points

Latency reduction of up to 55% by removing the fabric switching layer

Up to 2.5x faster Fibre Channel storage provisioning

These benefits accrue while maintaining the flexibility to simultaneously include fabric-attached storage for

traditional SAN connectivity from the same Virtual Connect module.

Simultaneous Direct Attached & SAN-Attached Fibre Channel Storage

HP 3PAR Storage

FC Switch

HP ProLiant Gen8 Blade Servers in a C7000 Enclosure

HP StorageWorks EVA Storage

Choice of Direct-Attached or Fabric-Attached or Simultaneous

Direct and Fabric-Attached Fibre Channel Storage Connectivity Flexibility

Document # APP2013011 v16 August, 2013 Page 11 of 12

Advantage — HP ProLiant Gen8

Emulex Fibre Channel technology is proven with a 15 year history of Emulex Fibre Channel technology is proven with a 15 year history of

deployment of 12 million ports in missiondeployment of 12 million ports in mission--critical environments.critical environments. Proven

Conclusions

For IT organizations who want to scale-up the density of their heterogeneous VM workloads, HP ProLiant

Gen8 blade servers offer more useable bandwidth and superior I/O flexibility.

Feature HP BL460C blade server

with 554FLB or 554M Cisco B200 M3 blade

server with VIC1240

Servers per Chassis 16 8

Servers per Zero Over subscription Traffic Domain 16 0

Over subscription None to 2:1 over subscrip-

tion 4:1 to 10:1 over subscrip-

tion

Single System Latencies 1.5 to 3.0 µsec 3.2 to 7.4 µsec

Support for native Fibre Channel & 10GbE Yes No

Hardware offload

Fibre Channel over Ethernet (FCoE) Yes Yes

iSCSI Yes No – software only

TCP offload engine (TOE) Yes Yes

Document # APP2013011 v16 August, 2013 Page 12 of 12

Resources

Related Links To learn more about the companies, technologies and products mentioned in this report, visit the following web pages:

HP FlexFabric Adapters Provided by Emulex HP BladeSystem HP Virtual Connect Technology HP Virtual Connect Traffic Flow HP BladeSystem and Cisco UCS Comparison Cisco Fabric Extender Cisco UCS Adapters Cisco UCS Ethernet Switching Modes HP Direct Connection Flat SAN Storage IT Brand Pulse

About the Author

Rahul Shah, Director, IT Brand Pulse Labs Rahul Shah has over 20 years of experience in senior engineering and product manage-ment positions with semiconductor, storage networking and IP networking manufacturers including QLogic and Lantronix. At IT Brand Pulse, Rahul is responsible for managing the delivery of technical services ranging from hands-on testing to product launch collateral. You can contact Rahul at rahul.shah@itbrandpulse.com.

SFP+ Copper Latency Substantiation UCS Fabric Interconnect Latency Nexus 5548 Switch Latency UCS Fabric Expander Latency HP Virtual Connect Latency