visnja

Fibre Channel over Ethernet and Nexus 5000

© 2008 Cisco Systems, Inc. All rights reserved. [email protected]

Višnja Milovanović

[email protected]

Systems Engineer

FCoE

© 2008 Cisco Systems, Inc. All rights reserved. 2

10GbE Drivers in the Datacenter

Multi-Core CPU architectures allowing bigger and multiple workloads on the same machine

Server virtualization driving the need for more bandwidth per


Server virtualization driving the need for more bandwidth per server due to server consolidation

Growing need for network storage driving the demand for higher network bandwidth to the server

Multi-Core CPUs and Server Virtualization driving the demand for higher bandwidth network connections

Merging the requirements

� LAN/IP

Must be Ethernet

Losing some frames is good for TCP

� Storage

Must follow the Fibre Channel model


Must follow the Fibre Channel model

Losing frames is not an option

� IPC

Don’t care of the underlying network, provided that

It is cheap

It is low latency

It supports APIs like OFED, RDS, MPI, etc. . .

Can Ethernet be lossless?

� Yes, with 802.3x Pause

For short distances equivalent to FC credits

� But… when customers turn it on, the results are confusing

Standard allows for asymmetric implementations


Standard allows for asymmetric implementations

Vendor implementations are inconsistent

� Anyhow Pause is not enough

It applies to the whole link

� Cisco proposes Priority Flow Control

One pause per IEEE 802.1p priority code point

Public domain (no standard activity yet)

Embraced by many network gear vendors

What is Fibre Channel over Ethernet?

� From a Fibre Channel standpoint it’s

FC connectivity over a new type of cable called… an Ethernet cloud

� From an Ethernet standpoints it’s

Yet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!


but… a challenging one!

� And technically…

FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric

FCoE Enablers

� 10Gbps Ethernet

� Lossless Ethernet

Matches the lossless behavior guaranteed in FC by B2B credits

� Ethernet jumbo frames


Eth

ern

et

He

ad

er

FC

oE

He

ad

er

FC

He

ad

er

FC Payload

CR

C

EO

F

FC

S

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

Max FC frame payload = 2112 bytes

Feature / Standard Benefit

Priority Flow Control (PFC)IEEE 802.1Qbb

Enable multiple traffic types to share a common Ethernet link without interfering with each other

Bandwidth Management IEEE 802.1Qaz

Enable consistent management of QoS at the network level by providing consistent scheduling

Evolving Ethernet


Congestion Management IEEE 802.1Qau

End-to-end congestion management for L2 network (future)

Data Center Bridging Exchange Protocol (DCBX)

Management protocol for enhanced Ethernet capabilities

L2 Multipath forUnicast and Multicast

Increase bandwidth, multiple active paths. No spanning tree (future)

Enabling Differentiated Services in an Ethernet Fabric

Data Center Bridging Capability Exchange Protocol (DCBCXP)

Devices need to discover the edge of the enhanced Ethernet cloud:

Each edge switch needs to learn that it is connected to a legacy switch.

Servers need to learn whether or not they are connected to Enhanced Ethernet device.

DCBX discovery = DCE

CNA

CNA

DCBX discovery =

Classical Ethernet

DCBX discovery =

Classical EthernetStandard 10 GigE NIC

C6500

Fibre Channel= No


Within the Enhanced Ethernet cloud, devices need to discover the capabilities of its peers.

DCBX utilizes the link-layer discovery protocol (LLDP) and handles local operational configuration for each feature

Fibre Channel= No

DCBX used

DCBX Overview

Auto-negotiation of capability and configuration

� Priority Flow Control capability and all associated CoS values enabled with PFC

� FCoE capability

� Allows one link peer to push config to other link peer

FCoE CoS value

� Logical Link Up/down signaling of Ethernet and FC


� DCBX negotiation failures will result in vfc not coming up

� Per-priority-pause not enabled on CoS values with PFC config

� Link partners can choose supported features and willingness to accept configuration from peer.

� Is supported on in point to point configuration

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf

http://www.ieee802.org/1/files/public/docs2008/

Priority Flow Control

Priority based Flow ControlPriority based Flow Control


• Enables lossless behavior for each class of service

• PAUSE sent per priority when buffers limit exceeded

Priority-based Flow Control (PFC)

� PAUSE functionality per Ethernet priority

IEEE 802.1Q defines 8 priorities

Traffic classes are mapped to different priorities:

no traffic interference

Storage traffic may be paused while IP traffic is being forwarded

Or, vice versa


Requires independent resources per priority (buffers)

� High level of industry support

Cisco distributed proposal

Standard Track in IEEE 802.1Qbb

Ethertype = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q tag

16 3 1 12 bits

Priority based bandwidth management

Priority based Bandwidth Management

Priority based Bandwidth Management


• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth

• 802.1Qaz Enhanced Transmission

Distinct Managementand Control Planes

Single Data Plan

MDSMDS

SAN BSAN AvPC

LAN

NexusNexus

L2MP: Port Channel with vPC


10 GbE/FCoE/DataCenter Ethernet

Consolidated I/OServers to Access Layer

Data Center Ethernet / FCoE Links

Cisco Nexus 5000

10 GbE

Fibre Channel

Virtual Port-channels Preserves ExistingDual SANEnvironments

Distinct Managementand Control Planes

Single Data Plan

MDSMDS

SAN BSAN A

Preserves ExistingDual SAN

VPC

Virtual Port-channels

LAN

NexusNexus

L2MP: vPC at the access layer


Consolidated I/OServers to Access Layer

Data Centre Ethernet / FCoE Links

Cisco Nexus 5000

Dual SANEnvironments

10 GbE/FCoE/DataCenter Ethernet

10 GbE

Fibre Channel

Virtual Port-channels

Active/Active

VPC

L2MP: Example of a Fat Tree

� All links are forwarding

� IS-IS computes the forwarding tables


L2MPBackbone

Data Center Access ArchitectureVirtualized Access Switch

LAN

10GE - DCE

SAN A SAN B� Virtualized Access Layer

Architectural flexibility

1GE to 10GE transition

� Consolidated and Consistent Operations

� Virtualization Optimized


1GE

10GE - DCE

Nexus 1000v

� Virtualization Optimized

VN-Link

� Network Fabric

vPC

Unified Fabric (FCoE)

Lossless Ethernet (DCE)

L2 Multipathing (DCE)Virtualized Access Switch

Unified I/O or I/O Consolidation


I/O Consolidation

FC HBA

FC HBA

NIC

FC Traffic

FC Traffic

Enet Traffic

FC HBA

FC HBA

NIC

FC HBA

FC HBA

NIC

Today:� Parallel LAN/SAN Infrastructure

� Inefficient use of Network Infrastructure

� 5+ connections per server – higher adapter and cabling costs

Unified I/O Use Case


NIC

NIC

Enet Traffic

Enet Traffic

NIC

NIC

NIC

NIC

Adds downstream port costs; cap-ex and op-ex

Each connection adds additional points of failure in the fabric

Power and cooling

� Longer lead time for server provisioning

� Multiple fault domains – complex diagnostics

� Management complexity – firmware, driver-patching, versioning

Management

SAN BSAN ALAN

Today:

� Aggregation/Core switches


FC HBA

FC HBA

NIC


Ethernet

FC

� Access – Top of the Rack switches

� Servers

NIC

NIC

FCoE I/O Consolidation BenefitFewer HBA/NIC’s per Server

FCoE CNA

FC HBA

FC HBA

NIC

FC Traffic

FC Traffic

Enet Traffic

FCoE

&


Customers purchase fewer NIC’s and HBA’s

FCoE CNANIC

NIC

Enet Traffic

Enet Traffic

Enet@

10GE

FCoE Converged Network Adaptor

Management

SAN BSAN ALAN

Today


Unified I/O Phase 1Unified I/O Phase 1� Reduction of server adapters

� Fewer Cables

� Simplification of access layer & cabling

� Gateway free implementation - fits in installed base of existing LAN and SAN

� L2 Multipathing Access – Distribution


FCoE

Ethernet

FC

FCoE Switch

� Lower TCO

� Investment Protection (LANs and SANs)

� Consistent Operational Model

� One set of ToR Switches

DCE Fabric w/ FCoE

Unified I/O Phase 2

� Elimination of parallel network infrastructure

� L2/L3 Multipathing end to end

� Faster infrastructure provisioning


Management

StorageArrays

Unified I/O Phase 2


provisioning

� Lower TCO

� Disk array access via DCE or Native FCFCoE

Switch

DCE and FCoE

FC

Unified I/O Phase 3

� Datacenter wide Unified Fabric for LAN and SAN

� L2/L3 Multipathing end to end

� Consistent network policies across datacenter

Management

SAN BSAN ALAN

Today:

Management

SAN BSAN ALAN

Unified I/O Phase 1 (Mid 2008)


Management

DCE Fabric w/ FCoE

Unified I/O Phase 3

StorageArrays


across datacenter

� Lower TCO

DCE and FCoE

Ethernet

FC

DCE and FCoE

Ethernet

FC

DCE and FCoE

Ethernet cabling


Evolution of Ethernet Physical MediaRole of Transport in Enabling these Technologies!

100Mb 1Gb 10Gb

UTP Cat 5 UTP Cat 5

SFP Fiber

10Mb

UTP Cat 3

Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s

X2

SFP+ Cu (BER better than 10 )

SFP+ Fiber

-18


CableTransceiver

Latency (link)Power

(each side)DistanceTechnology

Twinax ~0.1µµµµs~0.1W10mSFP+ CUCopper

MM 62.5µµµµmMM 50µµµµm

~01W82m300m

SFP+ SRshort reach

MM OM2MM OM3

~01W10m100m

SFP+ USRultra short reach

Cat6Cat6a/7Cat6a/7

2.5µµµµs2.5µµµµs1.5µµµµs

~8W~8W~4W

55m100m30m

10GBASE-T

Cat 6/6a/7

iSCSI SRPFCP FCP FCP FCP

SCSI Layer

Operating System / Applications

Encapsulation technologies


Ethernet E. Eth

IP

TCP

iSCSI

IB

SRP

IP

TCP

FCIP

FCP

IP

TCP

iFCP

FCP

FCoE

FCP

FC

FCP

1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps

FCP

SCSI Layer

OS / Applications

Encapsulation technologies

� FCP layer is untouched

� Allows same management tools for Fibre Channel

� Allows same Fibre Channel drivers

� Allows same Multipathing software


E. Ethernet

FCoE

FCP

1, 10 . . . Gbps

� Allows same Multipathing software

� Simplifies certifications with OSMs

� Evolution rather than Revolution

FCoE and iSCSI

� Extension of FC (10G)

� Same Lossless and Performance

� More Oriented towards 1GE

� Applicable for SMB

� New Management

iSCSIiSCSIFCoEFCoE


and Performance Characteristics as FC

� Targeted for Enterprise

� Same Management Model as FC

� Minimal Operational Impact

� New Management Model

� Limited Native Target Support

� Requires Stateful Gateway

FCoE AdvantagesFCoE is managed like FC at initiator, target, and switch level

Same Operational Model

Same Operational Model

Easy to Understand

Easy to Understand

Completely based on the

FC model

Same host-to-switch and switch-to-

switch behavior of FCAligned with the Aligned with the


Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management

Same Managementand Security ModelsSame Managementand Security Models

switch behavior of FC

E.g., in order delivery or

FSPF load balancing

WWNs, FC-IDs, hard/soft

zoning, DNS, RSCN

Aligned with the FC-BB-4 Model,

Standardizedin FC-BB-5

Aligned with the FC-BB-4 Model,

Standardizedin FC-BB-5

FCoE architecture


Terminology

� ENode (FCoE Node): a Fibre Channel HBA and a FCM implemented within an Ethernet NIC

� FCM (FCoE Mapper): the function performing the encapsulation and de-capsulation of FC frames in Ethernet frames


Ethernet frames

� FCF (FCoE Forwarder): a Fibre Channel switch implementing one or more FCMs to attach to an ethernet network

ENode Architecture

FC Node

FCM FCM…


� One MAC address per FCoE Mapper

Enet

port

Enet

port…

Fibre Channel Forwarder Architecture

FCport

FCport

FCport

FC

FC Switch

FCM


Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Ethernet BridgeFCport

� One Universal MAC address per FCMapper

FCoE MAC addresses

� VE_Ports and VF_Ports always use MAC addresses derived from the switch pool

� VN_Ports may use two types of MAC addresses:

SPMA (Server Provided MAC Addresses)

FPMA (Fabric Provided MAC Addresses)


FPMA (Fabric Provided MAC Addresses)

� MAC Addresses are negotiated in FIP

� Initial deployment will use FPMA only

Initial Login Flow ladder

ENode FCoE Switch

VLANDiscovery

FIP:FCoEInitialization

VLANDiscovery

FCFDiscovery

FCFDiscovery


FLOGI/FDISC FLOGI/FDISC Accept

FC CommandFC Commandresponses

Initialization Protocol

FCOEProtocol

Discovery Discovery

Some terminology

� Port types

VF_port / VN_port

VE_port

F_port trunking / channeling

� Device types


Fibre Channel Forwarder (FCF)

FIP snooping bridge

� Switching modes

FC switching

NPV (NPIV gateway)

FIP snooping

FCoE Forwarding (VE_ports)

� FCoE frames have:

MAC addresses (hop-by-hop)

FC addresses (end-to-end)


Ethernet linkFC Fabric

FC Domain 7 FC Domain 3MAC AFCID 7.1.1 FCID 1.1.1

MAC C

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

Ethernet link

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC BSrce. = MAC A

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC CSrce. = MAC B

FC link

FC Domain 1MAC B

VE_port VE_port VF_port VN_port

FCoE Forwarding (FIP snooping)

Ethernet linkFC Fabric

Ethernet link

FC Storage

FC link

VF_port VN_port


FC Fabric

FC Domain 7 FC Domain 1MAC AFCID 7.1.1 FCID 1.1.1

MAC C

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

FCoE Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC CSrce. = MAC A

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC CSrce. = MAC A

FC link

No domainMAC B

Nexus Switching


Data Center Access EvolutionVirtual Access


DC Virtual

Access

Nexus 7010 10-Slot Chassis

� First chassis in Nexus 7000 product family

� Optimized for data center environments

� High density

256 10G interfaces per system

� High performance

1.2Tbps system bandwidth at initial release


1.2Tbps system bandwidth at initial release

80Gbps per slot

60Mpps per slot

� Future proof

Initial fabric provides up to 4.1Tbps

Product family scaleable to 15+Tbps

40/100G and Unified Fabric ready

33.1-38”(84-96.5cm)

17.3” (43.9cm)

21 RU36.5”

(92.7cm)

Nexus 5000 FamilyMulti-Protocol Server Switch

Industry’s First I/O Consolidation Virtualization Fabric for Enterprise Data Center

Industry’s First I/O Consolidation Virtualization Fabric for Enterprise Data Center

SwitchFamily

56-Port L2 Switch

• 40 Ports 10GE fixed

• 2 Expansion Modules

28-Port L2 Switch

• 20 Ports 10GE fixed

• 1 Expansion Module

NX5010 NX5020


OS

Cisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OS

FC + Ethernet

• 4 Ports 10GE

• 4 Ports 1/2/4G FC

CNA

• 2 Port 10GE/FCoE

Mgmt

ExpansionModules Ethernet

• 6 Ports 10GE

Partners

Fibre Channel

• 8 Ports 1/2/4G FC

Server Adapter

• FCoE SW stack

All 10GE switch/module ports are FCoE/Data Center Ethernet capableAll 10GE switch/module ports are FCoE/Data Center Ethernet capable

Front and Rear Panels

All 10GE Ports Are FCoE Capable!Nx5020Nx5020

N+1 Redundant FansDual Redundant


Expansion

Modules

Cables Connect in the Rear for Ease of Server WiringCables Connect in the Rear for Ease of Server Wiring

Power

EntryBase 10GE

10/100/1000

Out of Band ManagementConsole

N+1 Redundant FansDual Redundant

Power Supplies

Nexus 2000 Fabric ExtenderVirtual Chassis

Nexus 5000Virtualized chassis

+

Nexus 5000

=


The Nexus 2000 Fabric Extender (FEX) acts as a remote linecardfor the Nexus 5000, retaining all centralized management and configurationon the Nexus 5000, transforming it into a Virtualized Chassis

+Nexus 2000 Fabric Extender

=

Data Center Access ArchitectureN5K/N2K Advantages – Flexible Cabling

� Cisco Nexus Fabric Extender (FEX) and Nexus 5000 provide a Flexible Access Solution

� Migration to ToR for 10GE servers or selective 1GE server racks if required (mix of ToR and EoR)

� Mixed cabling environment (optimized as required)

� Flexible support for Future Requirements


Combination of EoR and ToR cabling

Nexus 5000/2000 Mixed ToR & EoR

. . .

Cisco Nexus 1000V Architecture

VMW ESX

Server 1

VMware vSwitch

VMW ESX

Server 2

VMware vSwitch

VMW ESX

Server 3

VMware vSwitch

VM #1

VM #4

VM #3

VM #2

VM #5

VM #8

VM #7

VM #6

VM #9

VM #12

VM #11

VM #10

VEM VEM VEMNexus 1000V


Virtual Supervisor Module (VSM)

� Virtual or Physical appliance running Cisco OS (supports HA)

� Performs management, monitoring, & configuration

� Tight integration with VMware Virtual Center

Virtual Ethernet Module (VEM)� Enables advanced networking

capability on the hypervisor

� Hypervisor dependent code

� Provides each VM with dedicated “switch port”

� Collection of VEMs = 1 Distributed Switch

Virtual Center

VMW ESX VMW ESX VMW ESX

Nexus 1000V

VSM

Cisco VN-LinkIncrease Operational Efficiency

ServerServer

VM #5

VM #8

VM #7

VM #6

VM #4

VM #3

VM #2

VM #1

Policy-Based VM Connectivity

Non-DisruptiveOperational Model

Mobility of Network & Security Properties

Cisco VN-Link—Virtual Network Link


VMW ESXVMW ESX

Cisco VN-Link

#5 #8#7#6#4#3#2#1

Network Benefits

� Unifies network mgmt and ops

� Improves operational security

� Enhances VM network features

� Ensures policy persistence

� Enables VM-level visibility

Virtual Center

Server Benefits

� Maintains existing VM mgmt

� Reduces deployment time

� Improves scalability

� Reduces operational workload

� Enables VM-level visibility

Thank you!


visnja

Documents

Transcript of visnja