Cisco Confidential 1© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Expo 2011
Unified Fabric with FCoE
Jaromír Pilař
Consulting Systems Engineer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• Unified Fabric in Data Center
Basic principles and technology enablers
• Fiber Channel over Ethernet
Encapsulation, FCoE and DCB
Standardization
• Unified Fabric Deployment
Single-hop and Multi-hop scenarios
Configuration details
• Conclusions
Cisco Confidential 3© 2010 Cisco and/or its affiliates. All rights reserved.
Unified FabricBasic principles and technology enablers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Processor
Memory
LA
N
Sto
rag
e
IPC
Processor
Memory
IPC: Inter-Process Communication
I/O Subsystem
LA
N
Sto
rag
e
IPC
• Single network instead of three
I/O Subsystem
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
FC TrafficFC HBA
• Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs
• Limited number of interfaces for Blade Servers
• Standardized and reduced cabling
All Traffic
Goes over
10 GE
CNA
CNA
FC TrafficFC HBA
NIC Enet Traffic
NIC Enet Traffic
NIC Enet Traffic
HCA IPC Traffic
IPC TrafficHCA
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
LAN/IP
Must be Ethernet
Too much
investment
Too many
applications that
assume Ethernet
Must follow the
Fibre Channel
model
Losing frames is
not an option
StorageIPC(Inter-Process
Communication)
Low latency
required
Support APIs like
OFED, RDS, MPI,
sockets
Major implication: Lossless behavior required in Ethernet environment
Major reason for frame losses: congestion
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
• Yes, with Ethernet PAUSE Frame
PAUSESTOP
Ethernet Link
Switch A Switch B
Queue Full
Defined in IEEE 802.3—Annex 31B
The PAUSE operation is used to inhibit transmission of data frames for a specified period of time
Ethernet PAUSE transforms Ethernet into a lossless fabric
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
• Inconsistent implementations
Standard allows for asymmetric implementations
Easy to fix
• PAUSE applies to the whole links
Single mechanism for all traffic classes
• This may cause ―traffic interference‖
e.g., Storage traffic paused due to a congestion on IP traffic
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
• a.k.a. PPP (Per Priority Pause)
• PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities
Traffic classes are mapped to different priorities:
No traffic interference
IP traffic may be paused while storage traffic is being forwarded
Or, vice versa
Requires independent resources per priority (buffers)
• High level of industry support
Cisco distributed proposal
Standard track in IEEE 802.1Qbb
EtherType = IEEE 802.1Q Priority CFI VLAN ID
IEEE 802.1Q Tag
16 3 1 12 Bits
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Eight
Priorities
Switch A Switch B
Transmit Queues
Ethernet Link
Receive Queues
One
Two
Three
Four
Five
Seven
Eight
Six
One
Two
Three
Four
Five
Seven
Eight
SixSTOP PAUSE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
• Hop-by-hop negotiation for:
Priority Flow Control (PFC)
Bandwidth management
Applications
Logical link-down
• Based on LLDP (Link Level Discovery Protocol)
Added reliable transport
• Allows either full configuration or configuration checking
Link partners can choose supported features and willingness to accept configuration from peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
• IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism
• Products typically implement some form of Deficit Weighted Round Robin (DWRR)
Configuration and interworking is problematic
• Proposal for HW-efficient, two-level DWRR with strict priority support
Consistent behavior and configuration across network elements
• Standard track in IEEE 802.1Qaz
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Priorities Are
Assigned to
Individual
Traffic Classes
Priority
Groups
Priority Groups
Are Then
Scheduled
First Level of Scheduling
Inside Each Group
Final Link
Behavior
LAN
SAN
IPC
Cisco Confidential 14© 2010 Cisco and/or its affiliates. All rights reserved.
Fiber Channel over Ethernet
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
Eth
ern
et
He
ad
er
FC
oE
He
ad
er
FC
Hea
der
FC Payload
CR
C
EO
F
FC
S
Same as a physical FC frame
Control information: version, ordered sets (SOF, EOF)
Normal ethernet frame, ethertype = FCoE
• 10Gbps Ethernet
• Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits
• Ethernet jumbo frames
Max FC frame payload = 2112 bytes
Total max frame size = 2180 bytes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
• Mapping of FC Frames over Ethernet
• Enables FC to Run on a Lossless Ethernet
FCoE
Fibre
Channel
Traffic
Ethernet
Eth
ern
et
Head
er
FC
oE
Head
er
FC
Head
er
FC Payload CR
C
EO
F
FC
S
FCoE is standardized by the
same organization that develops
the Fibre Channel standard
Standardized via FC-BB-5
June 2009
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
• Standards are like operating systems - they add features to previous versions
• Different versions (e.g. FC-BB-4, FC-BB-5, FC-BB-6) have different features
• FC-BB-5 fully defined the way totransport Fibre Channel over Ethernet
FC-BB-6 is working on adding features and functionality
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• Mapping of FC Frames over Ethernet
• Enables FC to Run on a Lossless Ethernet
• Priority Flow Control IEEE 802.1Qbb creates lossless Ethernet with classes of service
• Bandwidth Management IEEE 802.1Qaz allows flexible bandwidth sharing for LAN and SAN
• Data Center Bridging Exchange Protocol IEEE 802.1Qaz standardized device to device communication on resources
FCoE IEEE DCB
Fibre
Channel
Traffic
Ethernet
Eth
ern
et
Head
er
FC
oE
Head
er
FC
Head
er
FC Payload CR
C
EO
F
FC
S
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
QCN Operates At a Different Level Than FCoE
• QCN is a core-to-edge protocol to deal with persistent congestion situations in a Layer 2 network
H1
H2
H3
Congestion
Traffic
QCN message
QCN message
DA: H3
SA: H1
DA: H3
SA: H2
DA: H1
SA: H3
DA: H2
SA: H3
When congestion is detectedthe core switch samples some frames, swaps their MAC addresses, and sends notifications backward
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Therefore, QCN is Useless for FCoE
DA: FCF-MAC(A)
SA: FPMA(H2)
Encaps. FC frame
D_ID = FC-ID(T2)
S_ID = FC-ID(H2)
DA: FPMA(T2)
SA: FCF-MAC(C)
Encaps. FC frame
D_ID = FC-ID(T2)
S_ID = FC-ID(H2)
DA: FCF-MAC(B)
SA: FCF-MAC(A)
Encaps. FC frame
D_ID = FC-ID(T2)
S_ID = FC-ID(H2)
DA: FCF-MAC(C)
SA: FCF-MAC(B)
Encaps. FC frame
D_ID = FC-ID(T2)
S_ID = FC-ID(H2)
H1
H2
T1
T2
FCF A FCF B FCF C
Cisco Confidential 21© 2010 Cisco and/or its affiliates. All rights reserved.
The two protocols have:
• Two different Ethertypes
• Two different frame formats
• Both are defined in FC-BB-5
FCoE itself
• Is the data plane protocol
• It is used to carry most of the FC frames and all the SCSI traffic
• Uses Fabric Assigned MAC address (dynamic)
FIP (FCoE Initialization Protocol)
• It is the control plane protocol
• It is used to discover the FC entities connected to an Ethernet cloud
• It is also used to login to and logout from the FC fabric
• Uses unique BIA on CNA for MAC
http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
FCoE Is Fibre Channel at the Host and Switch Level
Same Operational Model
Same Techniques ofTraffic Management
Same Managementand Security Models
Easy to Understand
Completely Based
on the FC Model
Same Host-to-Switch and
Switch-to-Switch Behavior
of FC
e.g., in Order Delivery or
FSPF Load Balancing
WWNs, FC-IDs, Hard/Soft
Zoning, DNS, RSCN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
All Standards for FCoE Are Technically Stable
PFC
ETS
DCBX
Inv Dev Appr Pub
Technically Stable
FC-BB-5
Inv Dev Appr Pub
Inv Dev Appr Pub
Inv Dev Appr Pub
Technically stable in October, 2008
Completed in June 2009
Published in May, 2010
Completed in July 2010, awaiting publication
Completed in July 2010 (completing Approval Phase 3)
Completed in July 2010 (completing Approval Phase 3)
DCB
Cisco Confidential 24© 2010 Cisco and/or its affiliates. All rights reserved.
Unified Fabric DeploymentSingle and Multiple Hop Scenarios
Cisco Confidential 25© 2010 Cisco and/or its affiliates. All rights reserved.
• FCF (Fibre Channel Forwarder) is the Fibre Channel switching element inside an FCoE switch
Fibre Channel logins (FLOGIs) happens at the FCF
Consumes a Domain ID
• FCoE encap/decap happens within the FCF
Forwarding based on FC information
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Ethernet Bridge
FC
port
FC
port
FC
port
FC
port
FCF
FCoE SwitchFC Domain ID : 15
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
VE_Port
VF_Port
VF_Port
VE_Port
VN_Port
VN_Port
FibreChannel over Ethernet Switch
E_NPV
SwitchVF_Port VNP_PortFCF
Switch
End
Node
End
Node
FCoE Switch : FCF
**Available NOW
**Available NOW **Planned
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
?
?
?
?
???
??
?
??
Switch Switch
Switch
?
T2
I5
I4I3I2
I1
I0
T1T0
Switch Switch
Switch
DNS FSPF
ZoneRSCN DNS
FSPFZone
RSCN
DNS
Zone
FSPF
RSCN
• Ethernet/IP
Bandwidth and services are separate layers, offered by separate entities
• Fibre Channel
Bandwidth and services are collapsed, offered by the fabric
• Unified Fabric design has to incorporate the super-set of requirements
QoS – Lossless ‗and’ Lossfull Fabrics
High Availability – Highly redundant network topology ‘and’ redundant fabrics
Bandwidth – FC fan-in and oversubscription ratios ‘and’ Ethernet/IP oversubscription
Security – FC controls (zoning, port security, …) ‘and’ IP controls (CISF, ACL, …)
Manageability and visibility – Hop by hop visibility for FC ‘and’ Ethernet/IP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• Where is it beneficial to use unified wire / unified dedicated wire / unified devices / unified technology within the Ethernet network?
At the edge of the fabric the volume of end nodes allows for a greater degree of sharing for LAN and SAN
Under-utilized links are prevalent at the access layer (especially with 10G) where combining multiple traffic types on a unified wire makes sense
Is there business case in the aggregation/core of the network to justify running Unified Wires?
• LAN and SAN HA models are very different (and not fully compatible) – so which one wins in the event of a conflict??
• FC and FCoE are prone to HOLB in the network and therefore we are limited in the physical topologies we can build
• Targets are attached to the SAN core/Storage Edge of the SAN, but where do we attach targets in an FCoE network? Into the Aggregation or Core layer? Or is an Ethernet ―storage edge‖ required??
• Where is it more beneficial to deploy two cores – SAN and LAN over a ―unified core‖ topology
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
Logical IsolationPhysical Isolation
Separate VLANs and VSANs are used to create multiple fabrics on the same devices
Separate Physical Networks are used for each fabric
Isolation at the…
Switch Level
VDC Level
Wire Level
Somewhere In Between
L3
L2
FC
Ethernet
A B
iSCSI FC FC
L2
L3
Core
Aggregation
Access
Virtual Port-
Channel (VPC)
Cisco Confidential 30© 2010 Cisco and/or its affiliates. All rights reserved.
• Servers and FCoE targets are directly connected to the Nexus 5000 over 10Gig FCoE
• Nexus 5000 operates as the FCF
• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000 access layer
Direct Attached Topology
Enhanced Ethernet and FCoE
Ethernet LAN
Native Fibre Channel
SAN A SAN B
FIP enabled CNAs
vPC
FIP or Pre-FIP enabled CNAs
FCoE Targets
Ethernet/LAN
Nexus 5000FCF
Nexus 5000FCF
Cisco Confidential 32© 2010 Cisco and/or its affiliates. All rights reserved.
• Blade servers connect to Nexus 4000 over 10Gig FCoE
Nexus 4000 is a FIP-Snooping Bridge
• Nexus 4000 connects to Nexus 5000 over 10Gig FCoE
Nexus 5000 operates as the FCF
• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000
Enhanced Ethernet and FCoE
Ethernet LAN
Native Fibre Channel
SAN A SAN B
Nexus 5000FCF
FCoE Targets
Blade Chassis
Nexus 4000: FIP Snooping Bridge
CNA mezzanine cards
Nexus 5000FCF
Ethernet/LAN
Cisco Confidential 33© 2010 Cisco and/or its affiliates. All rights reserved.
• Servers connect to Nexus 2232 over 10Gig FCoE
Server connections to the Nexus 2232 can be Active/Standy or over a vPC
• Nexus 2232 is single homed to upstream Nexus 5000
FEX 2232 can be connected with individual links or a port-channel
Maximum distance between Nexus 5000 and Nexus 2232 is 300 m
Enhanced Ethernet and FCoE
Ethernet LAN
Native Fibre Channel
Nexus 5000FCF
Nexus 5000FCF
vPC
Nexus 2232 Nexus 2232
Ethernet/LAN CoreSAN A SAN B
FIP enabled CNAs
Cisco Confidential 34© 2010 Cisco and/or its affiliates. All rights reserved.
With NX-OS 5.0(2)N2(1), VE_Portsare supported on/between the Nexus 5000 and Nexus 5500
Distance supported is up to 3 km
VE_Ports are run between switches acting as Fibre Channel Forwarders (FCFs)
VE_Ports are bound to the underlying 10G infrastructure
VE_Ports can be bound to a single 10GE port
VE_Ports can be bound to a port-channel interface consisting of multiple 10GE links
Enhanced Ethernet and FCoE
Ethernet LAN
Native Fibre Channel
Nexus 5000FCF
Nexus 5000FCF
vPC
Ethernet/LAN CoreSAN A SAN B
FIP enabled CNAs
Nexus 5000FCF
Nexus 5000FCF
VN
VF
VE
VE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Servers, FCoE attached Storage
• Multi-hop edge/core/edge topology
• Core SAN switches supporting FCoE
• N7K with DCB/FCoE line cards
• MDS with FCoE line cards (Sup2A)
• Edge FC switches supporting either
• N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)
• N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)
• Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy
N7K or MDS FCoE enabled Fabric
Switches
FC Attached Storage
Servers
VE
Edge FCFSwitch Mode
VE
Edge Switch in E-NPV
Mode
VF
VNPVE
VE
Nexus 7000 FCoE
support, MDS FCoE
module and E-NPV
planned for Q2CY2011
Cisco Confidential 36© 2010 Cisco and/or its affiliates. All rights reserved.
Unified Fabric DeploymentConfiguration Details
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Enable feature fcoe DCB Switch
Create VSAN
Enable FC uplinks if they are available
pod1-n5k-1# configure terminal
pod1-n5k-1(config)# feature fcoe
pod1-n5k-1(config)# vsan database
pod1-n5k-1(config-vsan-db)# vsan 44
pod1-n5k-1(config-vsan-db)# exit
pod1-n5k-1(config-if)# interface san-port-channel 200
pod1-n5k-1(config-if)# switchport trunk allowed vsan 1
pod1-n5k-1(config-if)# switchport trunk allowed vsan add 44
pod1-n5k-1(config-if)# interface fc 2/1-2
pod1-n5k-1(config-if)# channel-group 200 force
fc2/1 fc2/2 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the channel, then do "no
shutdown" at both ends to bring them up
pod1-n5k-1(config-if)# no shutdown
pod1-n5k-1(config-if)# interface san-port-channel 200
pod1-n5k-1(config-if)# no shut
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Map VSAN to VLAN DCB Switch
Create vfc interface and bind it to interface
Move interface vfc to VSAN (also configure zoning etc.)
pod1-n5k-1 (config)# vlan 100
pod1-n5k-1 (config-vlan)# fcoe vsan 44
pod1-n5k-1 (config-if)# interface vfc 1
pod1-n5k-1 (config-if)# bind interface Ethernet1/1
pod1-n5k-1 (config-if)# no shut
pod1-n5k-1 (config-vsan-db)#vsan database
pod1-n5k-1 (config-vsan-db)#vsan 44 interface vfc 1
Configure Ethernet interfacepod1-n5k-1 (config)# interface ethernet 1/1
pod1-n5k-1 (config-if)# switchport mode trunk
pod1-n5k-1 (config-if)# spanning-tree port type edge trunk
Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs,
concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause
temporary bridging loops. Use with CAUTION
pod1-n5k-1 (config-if)# switchport trunk allowed vlan 1,100
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
VLAN 10,30
VLAN 10,20
Each FCoE VLAN and VSAN count as a VLAN HW resource – therefore a VLAN/VSAN mapping accounts for TWO VLAN resources
FCoE VLANs are treated differently than native Ethernet VLANs: no flooding, broadcast, MAC learning, etc.
BEST PRACTICE: use different FCoE VLANs/VSANs for SAN A and SAN B
The FCoE VLAN must not be configured as a native VLAN
Unified Wires connecting to HOSTS must be configured as trunk ports and STP edge ports
Remember: STP does not run on FCoE vlans between FCFs (VE_Ports)
! VLAN 20 is dedicated for VSAN 2 FCoE traffic
(config)# vlan 20
(config-vlan)# fcoe vsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric BLAN Fabric
Nexus 5000
FCF
Nexus 5000
FCF
VSAN 3
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
• vPC with FCoE are ONLY supported
between hosts and N5k or N5k/2232
pairs…AND they must follow specific
rules
A ‗vfc‘ interface can only be associated with a single-port port-channel
While the port-channel configurations are the same on N5K-1 and N5K-2, the FCoE VLANs are different
• FCoE VLANs are ‘not’ carried on the
vPC peer-link (automatically pruned)
FCoE and FIP ethertypes are ‘not’ forwarded over the vPC peer link either
• vPC carrying FCoE between two FCF‘s
is NOT supported
• vPC with FCoE from host to N7k is NOT
supported at FCS Direct Attach vPC Topology
VLAN 10,30
VLAN 10,20
STP Edge Trunk
VLAN 10 ONLY HERE!
Fabric A Fabric BLAN Fabric
Nexus 5000
FCF-ANexus 5000
FCF-B
vPC contains only 2 X
10GE links – one to each
Nexus 5X00
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
Once feature fcoe is configured, 2 classes are made by default (up to NX-OS 5.0(2)N1(1))
DCB Switch
DCB CNA Adapter
class-fcoe is configured to be no-drop with an MTU of 2240
Best Practice - use the default COS value of 3 for FCoE/no-drop traffic
Can be changed through QOS class-map configuration
policy-map type queuing default-in-policy
class type queuing class-fcoe
bandwidth percent 50
class type queuing class-default
bandwidth percent 50
qos-group 1
q-size: 76800, HW MTU: 2400 (2240 configured)
drop-type: no-drop, xon: 128, xoff: 240
dc11-5020-3# sh class-map type qos
class-map type qos class-fcoe
match cos 3
class-map type qos class-default
match any
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
Once ‗feature fcoe‘ is configured, 2 user classes are created by default (up to NX-OS 5.0(2)N1(1))
By default, each class is given 50% of the available bandwidth
1Gig FC HBAs
1Gig Ethernet NICs
Traditional Server
Defaults: FCoE and Ethernet each receive 50%
Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs)
dc11-5020-3# show queuing int eth 1/39
Interface Ethernet1/39 TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 50
1 WRR 50
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
http://download.intel.com/technology/eedc/dcb_cep_spec.pdf
http://www.ieee802.org/1/files/public/docs2008/
Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values
Allows one link peer to push config to other link peer
Link partners can choose supported features and willingness to accept
Discovers lossless Ethernet Capabilities
Responsible for Logical Link Up/Down signaling of Ethernet and FC
DCBX negotiation failures will result in:
Per-priority-pause not enabled on CoS values with PFC configuration
vfc not coming up – when DCBX is being used in FCoEenvironment
Cisco Confidential 44© 2010 Cisco and/or its affiliates. All rights reserved.
Conclusions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
• FCoE integrates with today‘s Fibre Channel SANs
• FCoE enables ―Unified Technology‖Enables LAN and SAN traffic to share wires/devices/adapters for access layer TCO benefits
• FCoE is based on EthernetLeverages Ethernet technology, investment, market presence, scaling capability
• FCoE invites more user choiceAligns vendors from storage + network markets (e.g. volume NIC suppliers)
Benefit is more choice, better assurance of technology supply, price
• FCoE enables FC to become more accessible FCoE going on motherboards = less cost and complexity vs. FC NICs
O/S vendors will adopt with native FCoE stacks – less cost and complexity
• Because vendors can build better products with FCoE
Cisco Confidential 46© 2010 Cisco and/or its affiliates. All rights reserved.
Q & A
Thank you.
Top Related