Fibre Channel over Ethernet (FCoE) - SNIA · PDF fileFCoE Fabrics must be built with FCoE...
Transcript of Fibre Channel over Ethernet (FCoE) - SNIA · PDF fileFCoE Fabrics must be built with FCoE...
Fibre Channel over Ethernet (FCoE)
John L Hufferd, Brocade
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 2
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced without modificationThe SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 3
Abstract
A new concept is currently moving through the Fibre Channel (T11) standards committee called Fire Channel over Ethernet (FCoE). The FCoE standard will specify the encapsulation of Fibre Channel frames into Ethernet Frames and the amalgamation of these technologies into a network fabric that can support Fibre Channel protocols and other protocols such as TCP/IP, UDP/IP etc.The tutorial will show the Fundamentals of the FCoE concept and describe how it might be exploited in a Data Center environment.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 4
Agenda
Introduction
Goals & Requirements
Consolidation
Architecture
Discovery & Link Instantiation
Topologies
Scenarios
Summary
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 5
IntroductionThis presentation provides an overview of Fibre Channel over Ethernet (FCoE)
One should think about FCoE as placing the FC protocol on a new physical link
Ethernet links instead of physical FC linksBut it is still Fibre Channel
The protocol is being defined in the INCITS Fibre Channel (T11) technical committee
Many details of the protocol still need to be defined but the significant major issues have been resolved
Target T11 Standards completion is the 2H08
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 6
The Origins of Data Center Ethernet
The technology has evolved continuously, showing a great ability to adapt to new technologies and increasing business requirements
Increasing Scalability, Feature, Function
Incr
easi
ng P
erfo
rman
ce Integration of Carrier-grade features
Incremental Protocol Enhancements
Logical Partitioning
Evolution from shared media to dedicated media
1973
2008-2009 CEE
Introduction of Ethernet
10Mbps
100Mbps
1Gbps
10Gbps(Converged Enhanced Ethernet)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 7
Goals/Requirements
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 8
FCoE Goals/Requirements (part 1)
FCoE Fabrics must be built with FCoE Switches Switches with Ethernet ports that provide FCoE capabilities and services – Called, in the standard, an FC Forwarder FCFSwitches that include the functions of traditional FC switchesStandard Ethernet switches may also exist in the fabric but switches with FCoE capabilities are required
FCoE fabrics must operate seamlessly with real FC Fabrics
FC services must operate identically on FCoE fabrics and Fibre Channel fabrics
FCoE must support all Fibre Channel advanced features (e.g. virtual fabrics, IFR, security, etc.) transparently
FCoE is NOT a replacement for FCIP or iFCPFCIP & iFCP use TCP/IPFCIP/iFCP is for inter-switch links beyond the Data Center
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 9
Integrated Ethernet & FCoE switch (FCF) with FC connectionsSome implementations may combine the features and capabilities of an Ethernet Switch with the features and capabilities of a FC switch
• Support Ethernet and IP standards for switching, pathing and routing
• Support FC standards for switching, pathing and routing
• Support current and enhanced Ethernet Standards
• Adapt between FCoE and FC
An FCoE Port (N_Port, F_Port or E_Port)
• Has the same function as in FC
• But is layered on top of Ethernet
• Called VN_Port, VF_Port and VE_Port
Because many logical (virtual)Ports can share one physical port
Note:FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocolNon-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as STP (Spanning Tree Protocol) and MSTP (Multiple STP)
See additional FSPF and STP explanation in Appendix
FCFCFC
CEE Ethernet Ports (with IP & FCoE VF_Port &
VE_Port capabilities)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 10
Connections to a Combo FCoE Switch
Applications
TCPUDPIP
SCSI
Fibre Channel
FCoE
Lossless Ethernet MAC (CEE)
• Fibre Channel is carried over lossless Ethernet as a L3 protocol
Combo Lossless Ethernet (CEE) Switchwith FCoE Switch (FCF) capabilities
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Port capabilities
IP address 123.45.67.89
iSCSI
SCSI
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 11
FCoE Goals/Requirements (part 2)
FCoE requires specific Ethernet extensions to be implementedLossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE) configurations are requiredJumboframe support is required (not a standard, but widely available)
Deployments of FCoE should utilize the advances in Ethernet currently being discussed in IEEE 802.1, specifically:
Priority-based Flow Control (PFC)Enhanced Transmission Selection (ETS)DCB Capability eXchange Protocol (DCB CXP)
These 802.1 advances are important for Consolidated Flows (Messaging, Clustering and Storage)This set of functions has been called Data Center Ethernet, or CEE –
Converged Enhanced Ethernet (intended for a Data Center Environment)
FCoE should require no changes to FC software
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 12
Consolidation
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 13
Dramatic reduction in adapter, switch ports and cabling4-6 cables to 2 cables per server
Seamless connection to the installed based of existing SANs and LANsRequires high performance lightweight frame mappers vs. heavy weight gateways
FCoE has no need to terminate and re-initiate a SCSI connection(e.g. iSCSI to FC)
Effective sharing of high bandwidth links
High End 10GE Server & NIC/HBA Consolidation
Today
With CEEOS3 DB Server
OS2 App Server
OS1 Web Server
Messaging
MPIRDMA
FC HBA
OS3 DB Server
OS2 App Server
OS1 Web Server
IB/Ethernet Cluster
Hyper Visor(vmWare, Xen, etc)
E-HBA(CEE)
Data CenterCEE Network
EthernetTCP/IP
FC SAN
Data CenterCEE Network
E-HBA(CEE)
E-HBA(CEE)
•NIC•TCP acceleration.•MPI, RDMAover Ethernet
•FCoE
OS3 DB Server
OS2 App Server
OS1 Web Server
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 14
Architecture
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 15
FC Encapsulation Into Network Packets (2 FCoE Related Packet types)
IEEE 802.3Layers
FC Levels(Unchanged)
FCoE Mapping
FC-4
FC-3FC-2
FC-1
FC-0
FC-4
FC-3FC-2
MACPHY
Frame Check Sequence(CRC)
Protocol control information: Version, SOF, EOF, etc. FC Imbedded Frames: Same as in Physical FC
EthernetHeader FCSFCoE
HeaderFC Header SCSI Commands/Data
Ethertype “FCoE”
Fibre Channel over Ethernet (Fibre Channel over Ethernet (FCoEFCoE) Packets) Packets
Protocol control information: Version, Op-codes, etc. Discovery and Login/Logout Parameters
EthernetHeader FCSFIP
HeaderDescriptors
Ethertype “FIP”
FCoE Initialization Protocol (FCoE Initialization Protocol (FIPFIP) Packets) Packets
Ethernet Header provides things needed for the physical network, including “Ethertype”
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 16
FIP Protocol and FCoE Protocol
Discovery PhaseFCFs Discovery each other and form a FabricENodes and FCFs Discover
Potential VN_Port VF_Port pairingCapabilities of Potential pairing
Login PhaseENodes chose among discovered FCFs’ PortsCreates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC LinkTwo allowed alternatives for the ENode MAC Addressing
Fabric Provided MAC Addresses (FPMA)Server Provided MAC Addresses (SPMA)Chosen by FCF (FPMA & SPMA Described latter)
Uses: FLOGI, FLOGI ACC, LOGO, …
Data Transfer PhasePLOGI/PRLIAll other FC protocol frames (ELS, FC4 ULPs. etc.)
FCoE Initialization Protocol (FIP)
FCoE Protocol
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 17
FIP Operation FormatWord 31-24 23-16 15-8 7-0
0 Destination MAC Address (6 Bytes)
12 Source MAC Address (6 Bytes)
3 ET=FIP (16 bits) Ver (4b) Reserved (12 bits)
4 FIP Operation Code5 Reserved FIP subcode Descriptor List Length
6 Flags F P
S P
S F
PAD to minimum length or mini-Jumbo length
…
n
n+1 Ethernet FCS
Optional IEEE 802.1q4 ByteTag goes here
Descriptor listvariesIn size
Ethernet framesizeIs 64Bytes to 2220Bytes
FIP Operation Code Reserved FIP SubCode
Descriptor List Length Flags FP SP S F
Descriptor List
Solicited bit FCF bitCapability Bits (SPMA or FPMA)
See Appendix for Descriptor list items
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 18
FC’s Encapsulation in Ethernet (FCoE)Word 31-24 23-16 15-8 7-0
0 Destination MAC Address (6 Bytes)
12 Source MAC Address (6 Bytes)
3 ET=FCoE (16 bits) Ver (4b) Reserved (12 bits)
4 Reserved5 Reserved6 Reserved SOF (8 bits)
7 Encapsulated FC FrameFC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)(including FC-CRC)
…
n
n+1 EOF (8 bits) Reservedn+2 Ethernet FCS
Optional IEEE 802.1q4 ByteTag goes here
This field varies
In size
Ethernet framesizeIs 64Bytes to 2220Bytes
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 19
FCF Model
Link End
Point (LEP)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 20
ENode (HBA) Model
Each ENode (HBA) may have multiple Physical Ethernet PortsEach Physical Port may have multiple Logical VN_Ports
Link End
Point (LEP)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 21
Model of the ENODE with Multiple Logical FC interfaces
FCoEController
Lossless Ethernet MAC Ethernet_Port
FCEntityVN_Port
FCoEEntity
FCoE_LEP
FC-3 /FC-4s
MAC Address of “Burnt-in
MAC
FCEntityVN_Port
FCoEEntity
FCoE_LEP
FC-3 /FC-4s
. . .
• For each logical N_Port (VN_Port) there is one FLOGI and perhaps 100’s of FDISC
• Each VN_Port is seen by the Host as a separate (logical) FC connection
• The number of (logical) FC connections is implementation dependent
• Only one MAC Address is required for the FCoE Controller and the VN_Ports on a single physical MAC (aka Server Provided MAC Address – SPMA)• FCF may chose to specify new MAC addresses for each VN_Port (aka Fabric Provided MAC Address – FPMA)
In this model this is where
FC-2 functions
live
MAC Address of FCoE_LEP(VN_Port)
May or may not be the same as the FCoE controller
In this model this is where the
Encapsulation /De-Encapsulation
functions live
Multiple FC NPIV instances on a single
logical FC Host interface
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 22
The Logical FC Link is defined by a MAC Address pair• A VN_Port MAC Address • A VF_Port MAC Address
For a logical FC link the FCoE Frames are always sent to and received from a specific FCF’s MAC Address• Therefore, pathing to and from the FC driver is always defined by the MAC Address of the partner FCF’s VF_Port
Multiple Logical FC connections via a single Ethernet MAC
Examples of single MACs with connections to two different FCFs Switch
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 23
Functions of an FCoE Initiator ASIC
NIC Function
FCoE
function
FCFunction
Host PCIe interface
ExternalPort
FCoE
ASIC
• Has a Normal NIC interface (A) to the Host
• Has one or more Normal FC interfaces (B,C) to the Host
• FCoE functions not seen by the Host
• FCoE functions perform the Encapsulation and De- encapsulation
• The FCoE function Instantiates a Logical FC N_Port, called a VN_Port
Lossless Ethernet
MAC
A B C
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 24
HBA with Multiple Logical FC Interfaces
• MAY have one “burnt-in MAC address for both IP and FCoE/FIP packets
Or
• MAY have different “Burnt- in” MACs for IP and FCoE/FIP packets
• Used to separate HW based FCoE from other Ethernet Traffic
• Most NICs come with several “Burnt-in” MAC Addresses
• The FCoE controller will perform the FIP functions and will instantiate new VN_Ports as FCoE Link End Point (LEP)• With the same MAC address as the FCoE Controller (SPMA)• Or with a new MAC address specified by the FCF (FPMA)
FCoE
Controller
FC-3 /FC
-4sFCEntity
VN_Port
FCoE
EntityFC
oE_LEP
Lossless Ethernet MACEthernet_Port
NIC
Etype=FCoE
or FIP?
FCoE Chip
MAC Address of “Burnt-in
MAC
FCoE Function
FC-3 /FC
-4s
FCEntity
VN_Port
FCoE
EntityFC
oE_LEP
MAC Address of “Burnt-in
MAC
A B C
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 25
FIP (FCoE Initiation Protocol) Discovery and Link Instantiation
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 26
ENode to FCF Logical Links
FCfabric
H1
H2 ClassicalEthernet
LAN
FCF B
LosslessEthernetSwitch
FCF A
VN_Ports,VN_Port_Names
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
VF_Port,VF_Port_Name
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
FCF-MAC(A)
FCF-MAC(B)
MAC(H1)
MAC(H2)
FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF xMAC(y): A MAC address of a Lossless Ethernet port of ENode y
Note: The FCFs must discover each other and create a Fabric before the ENode Discover FCFs
An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FC Link with an FCF (VN_Port VF_Port) before an Normal FC frame flow
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 27
Initial Login Flow ladder (2 Phases)
End-Node FCF
Discovery Multicastto “ALL FCFs”Unicast FCF MAC Address et.al. in Jumbo Frame
FLOGI
FLOGI ACC with the FCF’s chosen VN_Port MAC address as a descriptor value
FC Command(Using the FCF selected MAC Address as the SA) FC Command responses
(using the command’s SA as a DA)
Discovery Phase
Login Phase
Normal FC Processing
FCoE Initialization Protocol
FCOE ProtocolSee Appendix for more details in Discovery Consideration and Actions
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 28
Subsequent Login Flow ladder (1 Phase)
End-Node FCF
FLOGI
Login Phase
Normal FC Processing
FCoE Initialization Protocol
FCOE ProtocolSee Appendix for more details in Discovery Consideration and Actions
FC Command(Using the FCF selected MAC Address as the SA)
FC Command responses(using the command’s SA as a DA)
FLOGI ACC with the FCF’s chosen VN_Port MAC address as a descriptor value
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 29
Topologies
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 30
FCoE Switch & FC Fabric
FCFCFCFCoE Switch
An FCoE Switch may connect to a normal FC switchVia the FC E-Port Ethernet
FC
Note: FCoE servers and storage will probably use an FCoE HBA (or chip set)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 31
Multiple FCoE Switch (FCF) Topologies
EthernetFC
FCFCFCLossless Ethernet Fabric
FCFCFCLossless Ethernet
Fabric
FCoE Switch
A Lossless Ethernet Fabric can be made up of Combo FCoE SwitchesLossless Ethernet switches configured into a Lossless Ethernet
Fabric can Front the FCoE Switch
FCFCFC
FCoESwitch
FCoESwitch
FCoESwitch
Lossless Ethernet
FCoE Switches deployed at the edges of the Lossless Ethernet FabricFCoE Switches connected via VE_Ports and Lossless Ethernet
FCFCFC
FCoESwitch
FCoESwitch
LosslessEthernet
1
LosslessEthernet
2
LosslessEthernet
4
FCoESwitch
LosslessEthernet
3
A VE_Port in an FCF may connect to other VE_Port in another FCFAnd an FCF FC E_Port may connect to an FC switch E_Port
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 32
Status of Current Data Center Networking
There can be 3 different networks
System Area Network (SyAN)Used for Clustering/Low Latency
Storage area Network (SAN)Used to Access to Storage
LAN/WAN External NetworksUsed for General messagingUsed for Client-Server MessagingUsed for NAS
Often divided into at least 3 management domains
Data Center Server (clustering) NetworkData Center Storage NetworkOutfacing (IP) Network
LAN/WANs– Messaging– NAS
Data Center
Remote Offices
System Area Network (SyAN)•Clustering Fabric•InfiniBand•Myrinet•Ethernet•CTC
Data CenterServer & Storage Network
Management GroupOutfacing (IP) Network
Management Group
LAN/WAN •Messaging•NAS
Storage Area Network- Fibre Channel
Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 33
Current FabricsRemoteRemoteOfficesOffices
Clustering Network
There is a FC Storage Network, a Clustering Network & an “Outfacing” NetworkIP Network Management Group
FC
FC
FC
Ficon
Ficon StorageController
Mainframe
Data Center Server & Storage Network Management GroupFile Storage Arrays (NAS)
iSCSIStorage
FCNetwork
Ficon
(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
Outfacing IP Network
Local & Remote Local & Remote Business CampusBusiness Campus
Focus: Low Latency & High
Bandwidth
Focus: Protection, Bandwidth/Congestion
Management
FC Link
EthernetLink
Ficon Link
Clustering Network
Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 34
View of the Future Data Center Networks
Remote Remote OfficesOffices
Data Center Server & Storage Network
Management Group
Outfacing (IP) NetworkManagement Group
Storage & System Storage & System Area NetworkArea Network••Logically Single FabricLogically Single Fabric
(with FCoE)(with FCoE)
Other IP NetsOther IP Nets(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
The The Server/Storage NetworksServer/Storage Networks will become a Consolidated will become a Consolidated FabricFabric
Managed by the Data Center Managed by the Data Center System and Storage System and Storage Management GroupManagement Group
–– Includes Storage and Includes Storage and clustering provisioningclustering provisioning
–– SAN and SyAN managed SAN and SyAN managed as a single fabricas a single fabric
Focus: High Bandwidth and Focus: High Bandwidth and Low LatencyLow Latency
The Outfacing (The Outfacing (IP) Management IP) Management GroupGroup remains the sameremains the same
Focus: Protection, Bandwidth Focus: Protection, Bandwidth and Congestion Managementand Congestion Management
Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 35
Phase in: FC & High Performance Lossless Ethernet Unify into a Data Center Fabric
Remote Remote OfficesOffices
FC
FC
FCoE
FC
(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
Outfacing IP Network
Ficon
iSCSIStorage
Ficon
Ficon StorageController
Mainframe
File Storage Arrays (NAS)
• FCoE permits intermixing of multiple Connection types/protocols• Clustering messaging, General Messaging, and Storage
• The DataCenter Fabric will “Trunk” to the “Outfacing” Network (including iSCSI sys)• Some Customers may want keep a mixed environment on-going
DataCenterFabric
EthernetSW
EthernetSW
FC & EthernetSW Blades
FC Link
EthernetLink
Ficon Link
Business Campus
with iSCSI connections
Including iSCSI Gateways
Note: with multiple Data Centers there may also be interconnects
with DWDM, FCIP/iFCP, etc.
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 36
Scenarios
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 37
Scenario 1: FCoE & IP Flows
FCFCFC
LosslessEthernet
ClassicalEthernet
FCoESwitch
Internet
FCoESwitches
LosslessEthernet
FCoE Flows
IP Flows
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 38
Scenario 2: FCoE Right & Wrong
FCFCFC
LosslessEthernet
ClassicalEthernet
FCoESwitch
Internet
FCoESwitches
LosslessEthernet
FCoE Flows
Invalid FCoE Flows
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 39
Summary
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 40
Summary
FCoE is a simple, efficient mechanism for encapsulating Fibre Channel in Ethernet frames
FCoE is being standardized in INCITS Fibre Channel (T11) technical committee
Target completion is 2H08
Maximum benefit of Fibre Channel is achieved:Evolutionary model of FC Switches and FC SANsEmphasis placed on capitalizing on the benefits of Converged Enhanced Ethernet (CEE)
Being discussed in the IEEE 802.1 standards working group
Thank You!
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 42
Q&A / Feedback
Please send any questions or comments on this presentation to SNIA: [email protected]
For additional information refer to http://www.t11.org/fcoe
Many thanks to the following individuals for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti Howard Goldstein Walter DeyRobert Snively Suresh Vobbilisetty Silvano GaiJoe Pelissier John Hufferd
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 43
Appendix
Additional RequirementsFCoE Relation to ISO LayersFlowsAdditional TopologiesFSPF and STPFIP Considerations and ActionsFIP DescriptorsPause vs. BB_Credit
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 44
FCoE Goals/Requirements (part 3)
FCoE must be a direct mapping of Fibre Channel over an Ethernet networkFCoE must be layered on top of Ethernet
FSPF used to route FCoE packetsEthernet Spanning Tree (STP), MSTP, etc, is at a layer below
FCoE to allow an evolutionary approach towards consolidation of fabrics
The Fibre Channel N_Port, F_Port and E-Port constructs must be retained
With FCoE, ports may be connected with Logical Ethernet Links– May pass through Ethernet switches– Identified by pairs of end point MAC addresses
Physical Ethernet Links can replace physical FC Links Physical Ethernet Links can carry all Ethernet traffic, including FCoE, but combined traffic needs the CEE capabilities
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 45
FCoE Goals/Requirements (part 4)
“Combo FCoE Switches” may be built that support normal Ethernet traffic, FCoE traffic , & FC trafficThe FCoE solutions should appear as a Fibre Channel to a Fibre Channel experienced customerFCoE should keep the Fibre Channel operations independent from Ethernet forwarding
Keeps management /Troubleshooting simpleCommon physical structures, different logical structures
Based on Ethertype (Ethertype = FCoE)
Storage Management should be unchanged
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 46
FCoE Relation to ISO Layers
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 47
Flows
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 48
Logical Fabric Topology
FCFCFC
FCoESwitch
FCoESwitch
FCoESwitch
LosslessEthernet
LosslessEthernet
A
H1
H2
H3H4 S2
EthernetDestination
& Source
EncapsulatedFC Frame
D_ID
S_ID
FCoE-A MACFCoE-H2 MAC
FC_ID for S1FC_ID for H2
FC_ID for S1FC_ID for H2
FC_ID for S1FC_ID for H2
FC_ID for S1FC_ID for H2
FCoE-B MACFCoE-A MAC
FCoE-C MACFCoE-B MAC
FCoE-S1 MACFCoE-C MAC
EthernetFC
LogicalTransaction Path
An FCoE Switch receives FCoE frames addressed to its FC-MAC address and forwards them based on the D_ID of the encapsulated FC frame
An FCoE Switch rewrites the SA and DA of an FCoE frame
LosslessEthernet
LosslessEthernet
S1
Path #1 Path #2 Path #3 Path #4
#1#2
#3
#4B
C
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 49
Single Ethernet Fabric with FCoE Switches
FCFCFCFCoE
SwitchFCoE
SwitchLossless Ethernet
AC
H1
H2
H3
S1
S2
FCoE-A MACEthernet
Destination& Source FCoE-H2 MAC
EncapsulatedFC Frame
D_ID
S_ID
FC_ID for S1FC_ID for H2
FC_ID for S1FC_ID for H2
FC_ID for S1FC_ID for H2
FCoE-C MACFCoE-A MAC
FCoE-S1 MACFCoE-C MAC
Ethernet Switch
Ethernet Switch
Ethernet Switch
Ethernet Switch
Ethernet Switch
EthernetFC
LogicalTransaction Path
Path #1 Path #2 Path #3
#1
#2 #3
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 50
FC Host to FCoE Storage
FCFCFCFCoE
Switch
FCoESwitch
Lossless EthernetH1
H2
H3
S1
S3
FCoE-S2 MACEthernet
Destination& Source FCoE-A MAC
EncapsulatedFC Frame
D_ID
S_ID
FC_ID for S2FC_ID for H5
FC_ID for S2FC_ID for H5
FC_ID for S2FC_ID for H5
FCoE-A MACFCoE-C MAC
Ethernet Switch
Ethernet Switch
Ethernet Switch
Ethernet Switch
H5S2
Ethernet Switch
EthernetFC
LogicalTransaction Path
Path #3 Path #2
Path #1
#1
#2
#3
CA
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 51
Additional Topologies
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 52
Additional Topologies (1)
FCoESwitch (A)
FCoESwitch (B)
FCoESwitch (E)
FCoESwitch (F)
LosslessEthernet Switch
LosslessEthernetSwitch
FCFCFC
Example of Topologies with Rack Mount servers
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 53
Additional Topologies (2)
FCoESwitch (B)
FCoESwitch (E)
FCoESwitch (F)
LosslessEthernet Switch
LosslessEthernetSwitch
FCFCFC
FCoESwitch (A)
Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) SupportExample of Topologies with Blade Servers
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 54
FSPF and STP
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 55
FSPF & STP Concepts with FCoE (basic)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 56
FSPF & STP Concepts with FCoE (interconnected)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 57
Equivalent FC topology
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 58
FIP Considerations and Actions
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 59
FCoE Discovery Considerations
The Discovery phase of the FCoE Initialization Protocol (FIP) uses two types of messages, Solicitations and AdvertisementsThe FIP Discovery Phase helps define the FCF Ports that are available for the Link instantiation Phase The ENodes discover the FCF ports that can become VF_Ports and FCFs discover other FCF ports that can become VE_Ports
ENodes Solicit (via Multicast) Advertisements from FCFs while specifying their capabilitiesIn response FCF’s Advertise their availability and capabilities back to the ENodesFCF’s Multicast their existent to other FCF
The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBA and/or FCF) “FCoE Controllers”
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 60
FCF Discovery ActionsFCF supporting VE_Ports:
Discovers other VE_Port capable FCF-MACs, connected to the same Lossless Ethernet segment, by:
Transmitting a multicast Solicitation to ‘All-FCFs’(with the FCF bit set to one)
Receives back Jumbo Unicast Advertisements from VE capable MACs
To verify the support of Ethernet Jumbo frames in the pathIn response to receipt of a Multicast to “All-FCFs” from another FCF
Has its own VE capable MACs sends their own Jumbo Unicast Advertisements
Instantiates VE_Port to VE_Port connections and Exchanges FC ELP (Extended Link Protocol) and Fabric configuration (using Ethertype=FCoE) with the other FCFs VE_Port capable MACs
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 61
ENode ActionsWhen an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs connected to the same Lossless Ethernet segment
Transmits a multicast Solicitation to ‘All-FCFs’ (with the FCF bit set to zero)Receives Jumbo unicast Advertisements from select compatible VF_Port capable FCF-MACsStores the discovered FCF-MACs in an FCF list
When an ENode receives an Advertisement that a new FCF is available, it may send a unicast Solicitation to it and receive a Jumbo unicast Advertisement in reply
To verify the support of Ethernet Jumbo frames in the pathMay then perform FLOGIs (with Ethertype-FIP) to a vendor specific subset of the FCF-MACs in the FCF list
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 62
FIP Descriptors
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 63
FIP Descriptors (1)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 64
FIP Descriptors (2)
Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 65
Pause vs. BB_CreditBoth mechanisms are used to avoid dropping frames
With different trade-offs
The Pause mechanism requires at least the (2 x RTT x bandwidth) product on a link as buffer space
But allows Buffer handling in an arbitrary wayWell suited for networks with limited (bandwidth x delay) product (e.g. within the data center)
The Pause frame is handled by the MAC layerSimilar to the R_RDY handling by the FC-1 level
The BB_Credit mechanism prevents loosing frames over any link
But links go under-utilized if link credits (& buffers) are < that needed for (RTT x BW)Requires buffer handling in maximum frame size units