FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

FC/FCoE Topologies, Protocols and Limitations

Erik Smith Consulting Technologist – Connectrix BU

Abstract

An in-depth discussion of FC and FCoE protocols focusing on

– topologies currently supported

– topologies under development

– known issues

Review of current EMC SAN best practices and reasons behind them

Describe technical benefits and limitations of both FC and FCoE

Describe currently supported FC and FCoE topologies and EMC-recommended best practices

Discuss known limitations with FC and FCoE

Agenda

FC or Ethernet

FC vs. FCoE vs. iSCSI

Supported Topologies

Best Practices

Futures

FC or Ethernet

Fibre Channel

32G ??

FC or Ethernet

Fibre Channel

Ethernet

32G ??

10G 40/100G

FC or Ethernet

If you’re asking yourself this question, you’re not alone

Fibre Channel

Ethernet

32G ??

10G 40/100G

FC or Ethernet You don’t really need to decide right now

Physical Fibre Channel

Physical Ethernet

Today Time Line

Physical Ethernet

Host Storage Network

Today Time Line

Physical Ethernet

FC or Ethernet Summary

Ethernet wins eventually – There’s still plenty of time to decide what this

means to you

Migrating to Ethernet does not equal rip and replace all FC

– Evolutionary versus Revolutionary

Agenda

FC or Ethernet

FC vs. FCoE vs. iSCSI – FC/FCoE vs. iSCSI

– FC vs. FCoE

Best Practices

Futures

FC/FCoE vs. iSCSI

It’s more about what’s right for your environment and less about which protocol is better

FC/FCoE vs. iSCSI

0 5 10 15 20 25

Cut through

Store and Forward

Latency (usec)

FC/FCoE vs. iSCSI

0 1000 2000 3000 4000 5000 6000 7000

Cut through

Store and Forward

Latency (usec)

FC/FCoE vs. iSCSI

With block I/O, uncongested network latency is practically a rounding error

0 1000 2000 3000 4000 5000 6000 7000

Cut through

Store and Forward

Latency (usec)

>=325x

FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Zone 1

Zone 2

Fabric (FC or DCB)

Zones are created by grouping the WWPNs of the host interface and storage interface into a “zone”. The set of zones created are put into a “zone set”

and activated on the fabric.

FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Zone 1

Zone 2

Zone 3

Zone 4

Zone 5

Zone 6Zone 7

Zone 8

Zone 9

Zone 10

Zone 11

Zone 12

Fabric (FC or DCB)

The number of zones in the fabric should always be greater than, or equal to, the number of initiators in

the fabric.

FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

With iSCSI, each host needs to be individually and manually pointed at a storage port by specifying

either an IP Address, IQN, or both.

FC/FCoE vs. iSCSI

FC vs. FCoE vs. iSCSI provisioning steps…

FC/FCoE vs. iSCSI

host …on

network

storage Total

FC/FCoE vs. iSCSI

OS ...on

host …on

network

storage Total

Windows

VMware

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

VMware

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 7 FCoE 7 iSCSI 7

VMware

FC 7 FCoE 7 iSCSI 7

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 7 FCoE 7 iSCSI 7

VMware

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 FCoE 7 iSCSI 7

VMware

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 7

VMware

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7

FC 1 5 7 13 FCoE 7 iSCSI 14 7

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7 7

FC 1 5 7 13 FCoE 7 iSCSI 14 7 7

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7 7

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7 7 33

FC 1 5 7 13 FCoE 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7 7 37

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 2 7 iSCSI 19 7 7 33

FC 1 5 7 13 FCoE 1 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 7 iSCSI 23 7 7 37

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 2 37 7 iSCSI 19 7 7 33

FC 1 5 7 13 FCoE 1 37 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 37 7 iSCSI 23 7 7 37

FC/FCoE vs. iSCSI

OS Protocol ...on

host …on

network

storage Total

Windows

FC 2 5 7 14 FCoE 2 37 7 46 iSCSI 19 7 7 33

FC 1 5 7 13 FCoE 1 37 7 45 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 37 7 45 iSCSI 23 7 7 37

FC/FCoE vs. iSCSI

Management during runtime – FC/FCoE perform some amount of self

documentation due to FC Login and Name Server registrations

FC/FCoE vs. iSCSI Summary

It’s more about what’s right for your environment and less about which protocol is better

– Network/Network stack latency are not currently the best place to focus on to reduce response time

– EMC believes ▪ Network-centric is better suited for larger

environments

▪ End-node-centric is better suited for smaller environments

FC vs. FCoE

Essentially the same in terms of – Network-centric

– Similar management tools

– Same multipathing software (for iSCSI as well)

– Similar basic port types ▪ N_Ports / F_Ports vs. VN_Ports and VF_Ports

▪ E_Ports vs. VE_Ports

– Same scalability limits ▪ Number of domains

▪ Number of N_Ports / VN_Ports

▪ Number of hops

FC vs. FCoE

Completely different transports – Physical FC versus Physical Ethernet

– Point-to-point links cannot be assumed with FCoE

– FCoE uses PFC instead of BB_Credit ▪ Distance implications

FC vs. FCoE

Completely different transports – Point-to-point links cannot be assumed

FC vs. FCoE

Must support FIP Snooping

FC vs. FCoE

VLAN 1

VLAN 2

FC vs. FCoE

Completely different transports – Point to point links cannot be assumed

DO NOT DO THIS! Only one Fabric per VLAN

FC vs. FCoE

Completely different transports – FCoE uses PFC instead of BB_Credit

▪ Distance implications

FC vs. FCoE

0 Waiting for

FC vs. FCoE

Buffer reaches High water mark

FC vs. FCoE

Effective bandwidth

Distance

1k 5k Max Max+x

FC vs. FCoE Summary

FCoE is FC – Management tools

– Basic concepts

– Multipathing

– Scalability limits

Transport use has implications – Just because you can, doesn’t mean you should

▪ One fabric per VLAN

▪ Pay attention to topologies that are being created

– Do not use FCoE for distances that exceed the maximum supported by the hardware vendor

Agenda

FC or Ethernet

Supported Topologies – General Guidelines

– Vendor-Specific

Best Practices

Futures

Supported Topologies General Guidelines

FC – Maximum 5 hops

– Maximum 55 domains

– Maximum 6000 N_Ports

– Maximum 10,000 N_Ports per L3 SAN

– Avoid heterogeneous FC-SW interop (please) ▪ Use NPIV if possible

Supported Topologies General Guidelines

FCoE – All FC guidelines apply (including multi-hop)

– Heterogeneous FC-SW interop is not supported

– When possible, connect directly to an FCF

– When not possible, use a FIP Snooping Bridge

Supported Topologies Vendor-Specific

There are at least 100 possible topologies – Refer to either of the EMC FCoE TechBooks

Supported Topologies Vendor-Specific – Cisco – End Device Connectivity

N2k FC

FCoE Host

N7k MDS

N2k FC

FCoE Host

N7k MDS

N2k FC

FCoE Host

N7k MDS

N2k FC

FCoE Host

N7k MDS

N2k FC

FCoE Host

N7k MDS

Supported Topologies Vendor-Specific – Cisco – Network Connectivity

FC Fabric

Vendor-Specific – Cisco – FIP Snooping Bridge Support

Supported Topologies Cisco Summary

Over 100 supported Cisco topologies are described in the FCoE TechBook

Extensive support for multi-hop FCoE

Support for FIP Snooping Bridges

Connectivity to existing FC SAN does not require the use of an FC router

Supported Topologies Vendor-Specific – Brocade – End Device Connectivity

FCoE Host

Supported Topologies Vendor-Specific – Brocade – Network Connectivity

FC FCoE

8000 FC

Fabric

FC FCoE

8000 FC

Fabric

FC FCoE

8000 FC

Fabric

Supported Topologies Vendor-Specific – Brocade – FIP Snooping Bridge

Brocade

Supported Topologies Brocade Summary

A number of supported Brocade topologies are described in the FCoE TechBook

Support for multi-hop FCoE in VCS only

No support for FIP Snooping Bridges

Connectivity to existing FC SAN requires the use of an FC router

Agenda

FC or Ethernet

Best Practices

Futures

Best Practices

Maximum hops

Maximum N_Ports

Single Initiator Zoning

Monitor for bit errors

Best Practice – Max Hops

Best practice – Do not exceed 5 hops

Reason – Concern about data corruption

– Concern about fabric segmentation in certain circumstances

4 - Layer 2 Ethernet Hops Example:

4 Ethernet switches

3 FC Hops Example:

4 FC switches

Queue hold time – Length of time a frame is held before it’s discarded

– Typically between 500ms and 2 seconds

Switches contain queues – Used for buffering

– These queues have a “hold time” value associated with them

R_A_TOV >= (max hops * Hold time) * 2

R_A_TOV – Maximum life span of a frame x 2 (FC-FS)

– Typically 10 seconds

Sw mod

Ethernet (Row) FC SAN

Blade Server

Sw mod

Blade Server

Sw mod

Blade Server

Sw mod

Blade Server

Sw mod

Blade Server

Sw mod

StorageBlade Server

Best Practice – Max Hops Summary

Do not exceed 5 hops – If you must, ensure that the configuration will

never create a condition where frames older than ½ of R_A_TOV will exist ▪ IOW, you need to know the hold time for every element

▪ Consider error conditions and failures

Links between any network elements that contain buffers must be counted as a FC Hop

– This includes FSBs and NPIV Gateways

Best Practice – Max N_Ports

Best practice – The number of N_Ports should not exceed 6000

Reason – Originally

▪ Testing revealed fabric segmentation due to HLO timeout between switches with around 1000 N_Ports

Best Practice – Max N_Ports Summary

Do not exceed the maximum number of N_Ports supported

– Results will be unpredictable

– In extreme cases, fabric segmentation can result

Best Practice – Single Initiator Zoning

Best practice – Each zone should only contain one initiator and

the targets it will access

Reason – Originally

▪ Concern about initiators logging in to other initiators

– Today ▪ Limits the number of name server queries that are sent

to the switch

▪ Related to Max N_Ports

▪ Recommended still due to its use in test configurations

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

Without zoning, the response would include information about 15 other N_Ports.

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

For each N_Port returned in the NS Response, host will query the Name Server for additional information. If a host only

needs to access 1 Target this means 14 extra NS queries per N_Port.

Initiator login with Single Initiator Zoning

Initiator login without Zoning

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

Host 2

Host 3

Host 4

Host 5

Host 6

Fabric (FC or DCB)

Host will attempt PLOGI/PRLI with all N_Ports and perform report LUNs with all Targets.

Single Initiator Zoning Summary

Use single Initiator / single target Zoning if at all possible

If not possible or practical – e.g., some VMware and RecoverPoint

environments

– Learn to watch for the signs ▪ Randomly missing N_Port logins during power on or

after link events

▪ Elongated boot times

– Target Driven Zoning will help when available!

Monitor for Bit Errors

Best practice – Monitor for bit errors

Reason – VERY bad for performance

– FC – Buffer loss

– FCoE – Bit errors can cause unintentional data distribution ▪ An I/O convergence issue not an FCoE one

Agenda

FC or Ethernet

Best Practices

Futures – TDZ

– Network Virtualization

Futures — Target Driven Zoning (TDZ)

• TDZ is an EMC initiative aimed at automating the provisioning of networked storage

• TDZ allows for the SAN to automatically configure zoning based on information provided to it by a storage port

• TDZ makes use of Peer Zoning

– Approved in FC-GS-6 and FC-SW-6

Futures – Network Virtualization

• Network Virtualization

– Generic term being used to describe “Overlay Networks”

– Encapsulations being discussed to support this functionality are

• NVGRE

• VXLAN

• STT

– Could be very disruptive to today’s SANs

Futures

• Join us for more information!

• Birds-of-a-Feather: Storage Networking for the Future

– Time: Tuesday 1:30p

– Room: Marcello 4401 A

In Summary

Physical FC will be around for a while

Migrating to Ethernet – Can be done at any time

– Does not require rip and replace

Follow Best practices – Unless you have a compelling reason not to

– Understand the risks

The EMC FCoE Tech books – contain hundreds of supported topologies; and

– detailed installation instructions

Additional information

Please check out my blog www.brasstacksblog.typepad.com

Additional information

Or follow me on twitter!

@ErikSmith7

Provide Feedback & Win!

125 attendees will receive $100 iTunes gift cards. To enter the raffle, simply complete:

– 5 sessions surveys

– The conference survey

Download the EMC World Conference App to learn more: emcworld.com/app

FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

Technology

Transcript of FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

SII14 FCoE Introduction

Industry Standards for the Exponential Growth of Data ... · Upcoming Standards – FCoE update • FC-BB-6 is under way defining the next generation of FCoE • Three new topologies

Fibre Channel over Ethernet (FCoE)...Ethernet (FCoE) One should think about FCoE as placing the FC protocol on a new physical link New Lossless Ethernet links instead of physical FC

FCoE iSCSI

Brocade Fabric OS FCoE User Guide, 8.2 › en-us › collaterals › unauth › ...Supported FCoE Topologies FCoE-to-FC Direct-Attach Topology In this topology, an FCoE host (for example,

UCS Storage Integration, Technologies, and Topologies · PDF fileFCoE Uplink Port is a “Border Port ... UCS FC Uplink Port 1 Pin Group 2: UCS FC Uplink Port 4. UCS to Brocade FC

Deployment of FCoE with Dell FC Flex IOM · 2018-08-24 · 5 Dell Networking FC Flex IOM: Deployment of FCoE with Dell FC Flex IOM, Brocade FC switches, and Dell Compellent Storage

FCoE and FCoCEE - IBM Redbooks · 2 FCoE and FCoCEE Rather, FCoE is an enhancement that expands FC into the Ethernet by combining two leading-edge technologies (FC and the Ethernet).

Cisco Fcoe Npv

FCoE over TRILL draft-mme-trill-fcoe-00 Informational .

Fibre Channel over Ethernet (FCoE) - Storage Networking ......Ethernet (FCoE) One should think about FCoE as placing the FC protocol on a new physical link New Lossless Ethernet links

Fibre Channel over Ethernet (FCoE) Concepts Concepts.pdf1 Fibre Channel over Ethernet (FCoE) Concepts Slide 2 What is FCoE? • Fibre Channel over Ethernet (FCoE) is the transport

Brocade Evaluator Group Fc vs Fcoe Rt

Fibre Channel Technologies “Current & Future” · Channel over Ethernet tunnels FC through Ethernet and is compatible with all existing Fibre Channel fabric environments. FCoE

Brocade Fabric OS FCoE Configuration Guide, 8.2 › ... › dam › common › fos-820-fcoe.pdfSupported FCoE topologies FCoE-to-FC direct-attach topology In this topology, there can

UF FCoE Final

Fibre-channel: The Ultimate Interface for Flash Storage · PDF file2005: 4Gb FC 2008: 8Gb FC 2012: 16Gb FC 2009: FCOE Arbitrated Loop Fabric Services Virtualization NPIV . Converged

Configure Fcoe Nexus5k

FCoE Concepts

Introducing dynamic FCoE