Cisco Expo 2011 - Talk 2 Cisco · © 2010 Cisco and/or its affiliates. All rights reserved. Cisco...

Cisco Confidential 1© 2010 Cisco and/or its affiliates. All rights reserved.

Cisco Expo 2011

Unified Fabric with FCoE

Jaromír Pilař

Consulting Systems Engineer

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

• Unified Fabric in Data Center

Basic principles and technology enablers

• Fiber Channel over Ethernet

Encapsulation, FCoE and DCB

Standardization

• Unified Fabric Deployment

Single-hop and Multi-hop scenarios

Configuration details

• Conclusions


Unified FabricBasic principles and technology enablers


Processor

Memory

LA

N

Sto

rag

e

IPC

Processor

Memory

IPC: Inter-Process Communication

I/O Subsystem

LA

N

Sto

rag

e

IPC

• Single network instead of three

I/O Subsystem


FC TrafficFC HBA

• Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs

• Limited number of interfaces for Blade Servers

• Standardized and reduced cabling

All Traffic

Goes over

10 GE

CNA

CNA

FC TrafficFC HBA

NIC Enet Traffic

NIC Enet Traffic

NIC Enet Traffic

HCA IPC Traffic

IPC TrafficHCA


LAN/IP

Must be Ethernet

Too much

investment

Too many

applications that

assume Ethernet

Must follow the

Fibre Channel

model

Losing frames is

not an option

StorageIPC(Inter-Process

Communication)

Low latency

required

Support APIs like

OFED, RDS, MPI,

sockets

Major implication: Lossless behavior required in Ethernet environment

Major reason for frame losses: congestion


• Yes, with Ethernet PAUSE Frame

PAUSESTOP

Ethernet Link

Switch A Switch B

Queue Full

Defined in IEEE 802.3—Annex 31B

The PAUSE operation is used to inhibit transmission of data frames for a specified period of time

Ethernet PAUSE transforms Ethernet into a lossless fabric


• Inconsistent implementations

Standard allows for asymmetric implementations

Easy to fix

• PAUSE applies to the whole links

Single mechanism for all traffic classes

• This may cause ―traffic interference‖

e.g., Storage traffic paused due to a congestion on IP traffic


• a.k.a. PPP (Per Priority Pause)

• PFC enables PAUSE functionality per Ethernet priority

IEEE 802.1Q defines eight priorities

Traffic classes are mapped to different priorities:

No traffic interference

IP traffic may be paused while storage traffic is being forwarded

Or, vice versa

Requires independent resources per priority (buffers)

• High level of industry support

Cisco distributed proposal

Standard track in IEEE 802.1Qbb

EtherType = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q Tag

16 3 1 12 Bits


Eight

Priorities

Switch A Switch B

Transmit Queues

Ethernet Link

Receive Queues

One

Two

Three

Four

Five

Seven

Eight

Six

One

Two

Three

Four

Five

Seven

Eight

SixSTOP PAUSE


• Hop-by-hop negotiation for:

Priority Flow Control (PFC)

Bandwidth management

Applications

Logical link-down

• Based on LLDP (Link Level Discovery Protocol)

Added reliable transport

• Allows either full configuration or configuration checking

Link partners can choose supported features and willingness to accept configuration from peer


• IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism

• Products typically implement some form of Deficit Weighted Round Robin (DWRR)

Configuration and interworking is problematic

• Proposal for HW-efficient, two-level DWRR with strict priority support

Consistent behavior and configuration across network elements

• Standard track in IEEE 802.1Qaz


Priorities Are

Assigned to

Individual

Traffic Classes

Priority

Groups

Priority Groups

Are Then

Scheduled

First Level of Scheduling

Inside Each Group

Final Link

Behavior

LAN

SAN

IPC


Fiber Channel over Ethernet


Eth

ern

et

He

ad

er

FC

oE

He

ad

er

FC

Hea

der

FC Payload

CR

C

EO

F

FC

S

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

• 10Gbps Ethernet

• Lossless Ethernet

Matches the lossless behavior guaranteed in FC by B2B credits

• Ethernet jumbo frames

Max FC frame payload = 2112 bytes

Total max frame size = 2180 bytes


• Mapping of FC Frames over Ethernet

• Enables FC to Run on a Lossless Ethernet

FCoE

Fibre

Channel

Traffic

Ethernet

Eth

ern

et

Head

er

FC

oE

Head

er

FC

Head

er

FC Payload CR

C

EO

F

FC

S

FCoE is standardized by the

same organization that develops

the Fibre Channel standard

Standardized via FC-BB-5

June 2009


• Standards are like operating systems - they add features to previous versions

• Different versions (e.g. FC-BB-4, FC-BB-5, FC-BB-6) have different features

• FC-BB-5 fully defined the way totransport Fibre Channel over Ethernet

FC-BB-6 is working on adding features and functionality


• Mapping of FC Frames over Ethernet

• Enables FC to Run on a Lossless Ethernet

• Priority Flow Control IEEE 802.1Qbb creates lossless Ethernet with classes of service

• Bandwidth Management IEEE 802.1Qaz allows flexible bandwidth sharing for LAN and SAN

• Data Center Bridging Exchange Protocol IEEE 802.1Qaz standardized device to device communication on resources

FCoE IEEE DCB

Fibre

Channel

Traffic

Ethernet

Eth

ern

et

Head

er

FC

oE

Head

er

FC

Head

er

FC Payload CR

C

EO

F

FC

S


QCN Operates At a Different Level Than FCoE

• QCN is a core-to-edge protocol to deal with persistent congestion situations in a Layer 2 network

H1

H2

H3

Congestion

Traffic

QCN message

QCN message

DA: H3

SA: H1

DA: H3

SA: H2

DA: H1

SA: H3

DA: H2

SA: H3

When congestion is detectedthe core switch samples some frames, swaps their MAC addresses, and sends notifications backward


Therefore, QCN is Useless for FCoE

DA: FCF-MAC(A)

SA: FPMA(H2)

Encaps. FC frame

D_ID = FC-ID(T2)

S_ID = FC-ID(H2)

DA: FPMA(T2)

SA: FCF-MAC(C)

Encaps. FC frame

D_ID = FC-ID(T2)

S_ID = FC-ID(H2)

DA: FCF-MAC(B)

SA: FCF-MAC(A)

Encaps. FC frame

D_ID = FC-ID(T2)

S_ID = FC-ID(H2)

DA: FCF-MAC(C)

SA: FCF-MAC(B)

Encaps. FC frame

D_ID = FC-ID(T2)

S_ID = FC-ID(H2)

H1

H2

T1

T2

FCF A FCF B FCF C


The two protocols have:

• Two different Ethertypes

• Two different frame formats

• Both are defined in FC-BB-5

FCoE itself

• Is the data plane protocol

• It is used to carry most of the FC frames and all the SCSI traffic

• Uses Fabric Assigned MAC address (dynamic)

FIP (FCoE Initialization Protocol)

• It is the control plane protocol

• It is used to discover the FC entities connected to an Ethernet cloud

• It is also used to login to and logout from the FC fabric

• Uses unique BIA on CNA for MAC

http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html


FCoE Is Fibre Channel at the Host and Switch Level

Same Operational Model

Same Techniques ofTraffic Management

Same Managementand Security Models

Easy to Understand

Completely Based

on the FC Model

Same Host-to-Switch and

Switch-to-Switch Behavior

of FC

e.g., in Order Delivery or

FSPF Load Balancing

WWNs, FC-IDs, Hard/Soft

Zoning, DNS, RSCN


All Standards for FCoE Are Technically Stable

PFC

ETS

DCBX

Inv Dev Appr Pub

Technically Stable

FC-BB-5

Inv Dev Appr Pub

Inv Dev Appr Pub

Inv Dev Appr Pub

Technically stable in October, 2008

Completed in June 2009

Published in May, 2010

Completed in July 2010, awaiting publication

Completed in July 2010 (completing Approval Phase 3)

Completed in July 2010 (completing Approval Phase 3)

DCB


Unified Fabric DeploymentSingle and Multiple Hop Scenarios


• FCF (Fibre Channel Forwarder) is the Fibre Channel switching element inside an FCoE switch

Fibre Channel logins (FLOGIs) happens at the FCF

Consumes a Domain ID

• FCoE encap/decap happens within the FCF

Forwarding based on FC information

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Ethernet Bridge

FC

port

FC

port

FC

port

FC

port

FCF

FCoE SwitchFC Domain ID : 15


VE_Port

VF_Port

VF_Port

VE_Port

VN_Port

VN_Port

FibreChannel over Ethernet Switch

E_NPV

SwitchVF_Port VNP_PortFCF

Switch

End

Node

End

Node

FCoE Switch : FCF

**Available NOW

**Available NOW **Planned


?

?

?

?

???

??

?

??

Switch Switch

Switch

?

T2

I5

I4I3I2

I1

I0

T1T0

Switch Switch

Switch

DNS FSPF

ZoneRSCN DNS

FSPFZone

RSCN

DNS

Zone

FSPF

RSCN

• Ethernet/IP

Bandwidth and services are separate layers, offered by separate entities

• Fibre Channel

Bandwidth and services are collapsed, offered by the fabric

• Unified Fabric design has to incorporate the super-set of requirements

QoS – Lossless ‗and’ Lossfull Fabrics

High Availability – Highly redundant network topology ‘and’ redundant fabrics

Bandwidth – FC fan-in and oversubscription ratios ‘and’ Ethernet/IP oversubscription

Security – FC controls (zoning, port security, …) ‘and’ IP controls (CISF, ACL, …)

Manageability and visibility – Hop by hop visibility for FC ‘and’ Ethernet/IP


• Where is it beneficial to use unified wire / unified dedicated wire / unified devices / unified technology within the Ethernet network?

At the edge of the fabric the volume of end nodes allows for a greater degree of sharing for LAN and SAN

Under-utilized links are prevalent at the access layer (especially with 10G) where combining multiple traffic types on a unified wire makes sense

Is there business case in the aggregation/core of the network to justify running Unified Wires?

• LAN and SAN HA models are very different (and not fully compatible) – so which one wins in the event of a conflict??

• FC and FCoE are prone to HOLB in the network and therefore we are limited in the physical topologies we can build

• Targets are attached to the SAN core/Storage Edge of the SAN, but where do we attach targets in an FCoE network? Into the Aggregation or Core layer? Or is an Ethernet ―storage edge‖ required??

• Where is it more beneficial to deploy two cores – SAN and LAN over a ―unified core‖ topology


Logical IsolationPhysical Isolation

Separate VLANs and VSANs are used to create multiple fabrics on the same devices

Separate Physical Networks are used for each fabric

Isolation at the…

Switch Level

VDC Level

Wire Level

Somewhere In Between

L3

L2

FC

Ethernet

A B

iSCSI FC FC

L2

L3

Core

Aggregation

Access

Virtual Port-

Channel (VPC)


• Servers and FCoE targets are directly connected to the Nexus 5000 over 10Gig FCoE

• Nexus 5000 operates as the FCF

• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000 access layer

Direct Attached Topology

Enhanced Ethernet and FCoE

Ethernet LAN

Native Fibre Channel

SAN A SAN B

FIP enabled CNAs

vPC

FIP or Pre-FIP enabled CNAs

FCoE Targets

Ethernet/LAN

Nexus 5000FCF

Nexus 5000FCF


• Blade servers connect to Nexus 4000 over 10Gig FCoE

Nexus 4000 is a FIP-Snooping Bridge

• Nexus 4000 connects to Nexus 5000 over 10Gig FCoE

Nexus 5000 operates as the FCF

• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000


Ethernet LAN


SAN A SAN B

Nexus 5000FCF

FCoE Targets

Blade Chassis

Nexus 4000: FIP Snooping Bridge

CNA mezzanine cards

Nexus 5000FCF

Ethernet/LAN


• Servers connect to Nexus 2232 over 10Gig FCoE

Server connections to the Nexus 2232 can be Active/Standy or over a vPC

• Nexus 2232 is single homed to upstream Nexus 5000

FEX 2232 can be connected with individual links or a port-channel

Maximum distance between Nexus 5000 and Nexus 2232 is 300 m


Ethernet LAN


Nexus 5000FCF

Nexus 5000FCF

vPC

Nexus 2232 Nexus 2232

Ethernet/LAN CoreSAN A SAN B

FIP enabled CNAs


With NX-OS 5.0(2)N2(1), VE_Portsare supported on/between the Nexus 5000 and Nexus 5500

Distance supported is up to 3 km

VE_Ports are run between switches acting as Fibre Channel Forwarders (FCFs)

VE_Ports are bound to the underlying 10G infrastructure

VE_Ports can be bound to a single 10GE port

VE_Ports can be bound to a port-channel interface consisting of multiple 10GE links


Ethernet LAN


Nexus 5000FCF

Nexus 5000FCF

vPC

Ethernet/LAN CoreSAN A SAN B

FIP enabled CNAs

Nexus 5000FCF

Nexus 5000FCF

VN

VF

VE

VE


Servers, FCoE attached Storage

• Multi-hop edge/core/edge topology

• Core SAN switches supporting FCoE

• N7K with DCB/FCoE line cards

• MDS with FCoE line cards (Sup2A)

• Edge FC switches supporting either

• N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)

• N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)

• Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy

N7K or MDS FCoE enabled Fabric

Switches

FC Attached Storage

Servers

VE

Edge FCFSwitch Mode

VE

Edge Switch in E-NPV

Mode

VF

VNPVE

VE

Nexus 7000 FCoE

support, MDS FCoE

module and E-NPV

planned for Q2CY2011


Unified Fabric DeploymentConfiguration Details


Enable feature fcoe DCB Switch

Create VSAN

Enable FC uplinks if they are available

pod1-n5k-1# configure terminal

pod1-n5k-1(config)# feature fcoe

pod1-n5k-1(config)# vsan database

pod1-n5k-1(config-vsan-db)# vsan 44

pod1-n5k-1(config-vsan-db)# exit

pod1-n5k-1(config-if)# interface san-port-channel 200

pod1-n5k-1(config-if)# switchport trunk allowed vsan 1

pod1-n5k-1(config-if)# switchport trunk allowed vsan add 44

pod1-n5k-1(config-if)# interface fc 2/1-2

pod1-n5k-1(config-if)# channel-group 200 force

fc2/1 fc2/2 added to port-channel 1 and disabled

please do the same operation on the switch at the other end of the channel, then do "no

shutdown" at both ends to bring them up

pod1-n5k-1(config-if)# no shutdown

pod1-n5k-1(config-if)# interface san-port-channel 200

pod1-n5k-1(config-if)# no shut


Map VSAN to VLAN DCB Switch

Create vfc interface and bind it to interface

Move interface vfc to VSAN (also configure zoning etc.)

pod1-n5k-1 (config)# vlan 100

pod1-n5k-1 (config-vlan)# fcoe vsan 44

pod1-n5k-1 (config-if)# interface vfc 1

pod1-n5k-1 (config-if)# bind interface Ethernet1/1

pod1-n5k-1 (config-if)# no shut

pod1-n5k-1 (config-vsan-db)#vsan database

pod1-n5k-1 (config-vsan-db)#vsan 44 interface vfc 1

Configure Ethernet interfacepod1-n5k-1 (config)# interface ethernet 1/1

pod1-n5k-1 (config-if)# switchport mode trunk

pod1-n5k-1 (config-if)# spanning-tree port type edge trunk

Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs,

concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause

temporary bridging loops. Use with CAUTION

pod1-n5k-1 (config-if)# switchport trunk allowed vlan 1,100


VLAN 10,30

VLAN 10,20

Each FCoE VLAN and VSAN count as a VLAN HW resource – therefore a VLAN/VSAN mapping accounts for TWO VLAN resources

FCoE VLANs are treated differently than native Ethernet VLANs: no flooding, broadcast, MAC learning, etc.

BEST PRACTICE: use different FCoE VLANs/VSANs for SAN A and SAN B

The FCoE VLAN must not be configured as a native VLAN

Unified Wires connecting to HOSTS must be configured as trunk ports and STP edge ports

Remember: STP does not run on FCoE vlans between FCFs (VE_Ports)

! VLAN 20 is dedicated for VSAN 2 FCoE traffic

(config)# vlan 20

(config-vlan)# fcoe vsan 2

VSAN 2

STP Edge Trunk

Fabric A Fabric BLAN Fabric

Nexus 5000

FCF

Nexus 5000

FCF

VSAN 3


• vPC with FCoE are ONLY supported

between hosts and N5k or N5k/2232

pairs…AND they must follow specific

rules

A ‗vfc‘ interface can only be associated with a single-port port-channel

While the port-channel configurations are the same on N5K-1 and N5K-2, the FCoE VLANs are different

• FCoE VLANs are ‘not’ carried on the

vPC peer-link (automatically pruned)

FCoE and FIP ethertypes are ‘not’ forwarded over the vPC peer link either

• vPC carrying FCoE between two FCF‘s

is NOT supported

• vPC with FCoE from host to N7k is NOT

supported at FCS Direct Attach vPC Topology

VLAN 10,30

VLAN 10,20

STP Edge Trunk

VLAN 10 ONLY HERE!

Fabric A Fabric BLAN Fabric

Nexus 5000

FCF-ANexus 5000

FCF-B

vPC contains only 2 X

10GE links – one to each

Nexus 5X00


Once feature fcoe is configured, 2 classes are made by default (up to NX-OS 5.0(2)N1(1))

DCB Switch

DCB CNA Adapter

class-fcoe is configured to be no-drop with an MTU of 2240

Best Practice - use the default COS value of 3 for FCoE/no-drop traffic

Can be changed through QOS class-map configuration

policy-map type queuing default-in-policy

class type queuing class-fcoe

bandwidth percent 50

class type queuing class-default

bandwidth percent 50

qos-group 1

q-size: 76800, HW MTU: 2400 (2240 configured)

drop-type: no-drop, xon: 128, xoff: 240

dc11-5020-3# sh class-map type qos

class-map type qos class-fcoe

match cos 3

class-map type qos class-default

match any


Once ‗feature fcoe‘ is configured, 2 user classes are created by default (up to NX-OS 5.0(2)N1(1))

By default, each class is given 50% of the available bandwidth

1Gig FC HBAs

1Gig Ethernet NICs

Traditional Server

Defaults: FCoE and Ethernet each receive 50%

Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs)

dc11-5020-3# show queuing int eth 1/39

Interface Ethernet1/39 TX Queuing

qos-group sched-type oper-bandwidth

0 WRR 50

1 WRR 50


http://download.intel.com/technology/eedc/dcb_cep_spec.pdf

http://www.ieee802.org/1/files/public/docs2008/

Auto-negotiation of capability and configuration

Priority Flow Control capability and associated CoS values

Allows one link peer to push config to other link peer

Link partners can choose supported features and willingness to accept

Discovers lossless Ethernet Capabilities

Responsible for Logical Link Up/Down signaling of Ethernet and FC

DCBX negotiation failures will result in:

Per-priority-pause not enabled on CoS values with PFC configuration

vfc not coming up – when DCBX is being used in FCoEenvironment

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf




Conclusions


• FCoE integrates with today‘s Fibre Channel SANs

• FCoE enables ―Unified Technology‖Enables LAN and SAN traffic to share wires/devices/adapters for access layer TCO benefits

• FCoE is based on EthernetLeverages Ethernet technology, investment, market presence, scaling capability

• FCoE invites more user choiceAligns vendors from storage + network markets (e.g. volume NIC suppliers)

Benefit is more choice, better assurance of technology supply, price

• FCoE enables FC to become more accessible FCoE going on motherboards = less cost and complexity vs. FC NICs

O/S vendors will adopt with native FCoE stacks – less cost and complexity

• Because vendors can build better products with FCoE


Q & A

Thank you.

Cisco Expo 2011 - Talk 2 Cisco · © 2010 Cisco and/or its affiliates. All rights reserved. Cisco...

Documents

Transcript of Cisco Expo 2011 - Talk 2 Cisco · © 2010 Cisco and/or its affiliates. All rights reserved. Cisco...