Extended Distance Technologies

Extended Distance Technologies

Version 1.1

• Distance Extension Technologies Overview

• Distance Extension Considerations

• Distance Extension Solutions

Mugdha KulkarniDavid HughesEric PunDaniel GandanegaraVinay Jonnakuti

Extended Distance Technologies TechBook2

Copyright © 2011 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date regulatory document for your product line, go to the Technical Documentation and Advisories section on EMC Powerlink.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

Part number H8079.1

Contents

Preface.............................................................................................................................. 7

Chapter 1 Extended Distance OverviewEarly implementations of SAN environments ............................. 14DWDM ............................................................................................... 15CWDM................................................................................................ 19

Differences between DWDM and CWDM............................. 19SONET................................................................................................ 21GbE...................................................................................................... 23TCP/IP ............................................................................................... 24

TCP terminology........................................................................ 24TCP error recovery .................................................................... 28Network congestion .................................................................. 31Internet Protocol security (IPsec) ............................................ 32

Chapter 2 Distance Extension ConsiderationsLink speed.......................................................................................... 36Data buffering and flow control ..................................................... 37

Fibre Channel ............................................................................. 37Maximum supported distance per Fibre Channel BB_Credit guidelines................................................................. 38Buffer-to-buffer credit information......................................... 42

TCP/IP window................................................................................ 53Active and passive devices.............................................................. 54

Buffer-to-buffer local termination ........................................... 54SRDF with SiRT.......................................................................... 56Fast write/ write acceleration.................................................. 58SiRT with distance vendor write acceleration....................... 59

Extended Distance Technologies TechBook 3

Contents

Link initialization ...................................................................... 60FC SONET/GbE/IP ......................................................................... 61Network stability and error recovery ............................................ 62

Chapter 3 IP-Based Distance Extension SolutionsNetwork design best practices........................................................ 64

Network conditions impact on effective throughput .......... 64EMC-Brocade distance extension solutions.................................. 66

Brocade 7500............................................................................... 67Brocade 7800............................................................................... 69

Configuring IPsec ............................................................................. 78Fast Write and tape pipelining........................................................ 80

Supported configurations......................................................... 81EMC-Cisco MDS distance extension solution .............................. 84

Supported configurations......................................................... 84Symmetrix setup........................................................................ 85VNX setup .................................................................................. 85CLARiiON setup ....................................................................... 85References ................................................................................... 85

EMC-Brocade M Series distance extension solution ................... 86Supported configurations......................................................... 86Implementation best practices................................................. 88Configuration of the local SAN ID and iFCP gateway ........ 89Symmetrix setup........................................................................ 89CLARiiON setup ....................................................................... 90Settings on Brocade/ Brocade M Series/ Cisco/QLogic switches....................................................................................... 90Additional documentation....................................................... 91

EMC-QLogic distance extension solution..................................... 92Supported configurations......................................................... 92Scalability.................................................................................... 93Best practices .............................................................................. 94SmartWrite ................................................................................. 94References ................................................................................... 95

Summary............................................................................................ 96

Glossary ......................................................................................................................... 99

Index.............................................................................................................................. 121


Title Page

Figures

1 DWDM example ............................................................................................. 152 Fibre Channel link extension ........................................................................ 173 STS-1 organization ......................................................................................... 224 Slow start and congestion avoidance .......................................................... 305 Fast retransmit ................................................................................................ 316 BB_Credit mechanism ................................................................................... 387 Flow control managed by Fibre Channel switch (without buffering

from distance extension devices) ...................................................................558 Flow control (with buffering from distance extension devices) .............. 569 Normal write command process .................................................................. 5710 SRDF SiRT ....................................................................................................... 5811 Write command with SiRT ........................................................................... 5912 All F_Ports will benefit .................................................................................. 6013 Link initialization (More than 100 ms R_T_TOV) ..................................... 6114 Brocade 7500 configuration example .......................................................... 6915 Basic overview of Trunking components ................................................... 7116 Single tunnel, Fastwrite and Tape Pipelining enabled ............................. 7417 Multiple tunnels to multiple ports, Fastwrite, and Tape Pipelining

enabled on a per-tunnel/per-port basis........................................................7418 Single tunnel, Fast Write and tape pipelining enabled ............................. 8219 Multiple tunnels to multiple ports ............................................................... 8320 Cisco MDS 9000 distance extension example ............................................. 8421 Brocade M Series in an SRDF, MirrorView, or SAN Copy

environment, example 1 ..................................................................................8722 Brocade M Series in an SRDF, MirrorView, or SAN Copy

environment, example 2 ..................................................................................8723 Brocade M Series in an SRDF, MirrorView, or SAN Copy

environment, example 3 ..................................................................................8824 SANbox 6142 Intelligent Router ................................................................... 93


Figures


Preface

This EMC Engineering TechBook provides a basic understanding of distance extension technologies and information to consider when working with extended distance. IP-based distance extension solutions are also included.

E-Lab would like to thank all the contributors to this document, including EMC engineers, EMC field personnel, and partners. Your contributions are invaluable.

As part of an effort to improve and enhance the performance and capabilities of its product lines, EMC periodically releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all versions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, please contact your EMC representative.

Audience This TechBook is intended for EMC field personnel, including technology consultants, and for the storage architect, administrator, and operator involved in acquiring, managing, operating, or designing a networked storage environment that contains EMC and host devices.

EMC Support Matrixand E-Lab

InteroperabilityNavigator

For the most up-to-date information, always consult the EMC Support Matrix (ESM), available through E-Lab Interoperability Navigator (ELN), at: http://elabnavigator.EMC.com, under the PDFs and Guides tab.

The EMC Support Matrix links within this document will take you to Powerlink where you are asked to log in to the E-Lab Interoperability Navigator. Instructions on how to best use the ELN (tutorial, queries, wizards) are provided below this Log in window. If you are


http://elabnavigator.EMC.com


https://elabnavigator.emc.com


8

Preface

unfamiliar with finding information on this site, please read these instructions before proceeding any further.

Under the PDFs and Guides tab resides a collection of printable resources for reference or download. All of the matrices, including the ESM (which does not include most software), are subsets of the E-Lab Interoperability Navigator database. Included under this tab are:

◆ The EMC Support Matrix, a complete guide to interoperable, and supportable, configurations.

◆ Subset matrices for specific storage families, server families, operating systems or software products.

◆ Host connectivity guides for complete, authoritative information on how to configure hosts effectively for various storage environments.

Under the PDFs and Guides tab, consult the Internet Protocol pdf under the "Miscellaneous" heading for EMC's policies and requirements for the EMC Support Matrix.

Relateddocumentation

Related documents include:

◆ The former EMC Networked Storage Topology Guide has been divided into several TechBooks and reference manuals. The following documents, including this one, are available through the E-Lab Interoperability Navigator, Topology Resource Center tab, at http://elabnavigator.EMC.com.

These documents are also available at the following location:

http://www.emc.com/products/interoperability/topology-resource-center.htm

• Backup and Recovery in a SAN TechBook

• Building Secure SANs TechBook

• Fibre Channel over Ethernet (FCoE): Data Center Bridging (DCB) Concepts and Protocols TechBook

• Fibre Channel SAN Topologies TechBook

• iSCSI SAN Topologies TechBook

• Networked Storage Concepts and Protocols TechBook

• Networking for Storage Virtualization and RecoverPoint TechBook

• WAN Optimization Controller Technologies TechBook

• EMC Connectrix SAN Products Data Reference Manual

Extended Distance Technologies TechBook




Preface

• Legacy SAN Technologies Reference Manual

• Non-EMC SAN Products Data Reference Manual

◆ EMC Support Matrix, available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com >PDFs and Guides

◆ RSA security solutions documentation, which can be found at http://RSA.com > Content Library

All of the following documentation and release notes can be found at http://Powerlink.EMC.com. From the toolbar, select Support > Technical Documentation and Advisories, then choose the appropriate Hardware/Platforms, Software, or Host Connectivity/HBAs documentation links.

Hardware documents and release notes include those on:

◆ Connectrix B series ◆ Connectrix M series ◆ Connectrix MDS (release notes only)◆ VNX series◆ CLARiiON ◆ Celerra ◆ Symmetrix

Software documents include those on:

◆ EMC Ionix ControlCenter ◆ RecoverPoint ◆ Invista ◆ TimeFinder ◆ PowerPath

The following E-Lab documentation is also available:

◆ Host Connectivity Guides◆ HBA Guides

For Cisco and Brocade documentation, refer to the vendor’s website.

◆ http://cisco.com

◆ http://brocade.com




http://RSA.com

http://powerlink.emc.com

10

Preface

Authors of thisTechBook

This TechBook was authored by Mugdha Kulkarni, Eric Pun, David Hughes, Daniel Gandanegara, and Vinay Jonnakuti, with contributions from the following EMC employees: Kieran Desmond, Ger Halligan, and Ron Stern, along with other EMC engineers, EMC field personnel, and partners.

Mugdha Kulkarni is a Senior Systems Integration Engineer and has been with EMC for over 6 years. For the past 6 years, Mugdha has worked in the E-Lab qualifying Symmetrix and VNX series releases. Mugdha is also involved in the technical evaluation of Fibre Channel over Ethernet (FCoE) products, including the CNA and FCoE switches.

David Hughes is a Principal Systems Integration Engineer and has been with EMC for over 15 years. For the past 5 years, David has worked in EMC E-Lab qualifying blade servers, FC/FCIP switch hardware and firmware, FC-to-iSCSI SAN Routers, EMC VNX series and CLARiiON storage systems, WAN Optimization Controllers and EMC's VPLEX. Prior to working in the E-Lab, David was an EMC Level II Technical Support subject matter expert for Brocade products, providing support across all EMC-supported FC switches and host connectivity. David also spent time working in EMC's IT and Manufacturing departments.

Eric Pun is a Senior Systems Integration Engineer and has been with EMC for 11 years. For the past several years, Eric has worked in E-lab qualifying interoperability between Fibre Channel switched hardware and distance extension products. The distance extension technology includes DWDM, CWDM, OTN, FC-SONET, FC-GbE, FC-SCTP, and WAN Optimization products. Eric has been a contributor to various E-Lab documentation, including the SRDF Connectivity Guide.

Daniel Gandanegara is a Senior Systems Integration Engineer and has been with EMC E-Lab for over 2 years, qualifying distance extension products and their integration with FC switches and EMC storage solutions. Prior to joining EMC, Daniel worked for other technology companies, including Hewlett-Packard and A*STAR Data Storage Institute.

Vinay Jonnakuti is a Systems Integration Engineer and has been with EMC's E-Lab for over 3 years in the storage environment. Vinay qualifies WAN-Optimization appliances with SRDF (GigE/FCIP), SAN-Copy, MirrorView, and RecoverPoint. Vinay also qualifies Brocade , Cisco FCIP, Fibre Channel, and iSCSI with the Symmetrix storage platform.


Preface

Conventions used inthis document

EMC uses the following conventions for special notices:

CAUTION!CAUTION, used with the safety alert symbol, indicates a hazardous situation which, if not avoided, could result in minor or moderate injury.

IMPORTANT!An important notice contains information essential to software or hardware operation.

Note: A note presents information that is important, but not hazard-related.

Typographical conventionsEMC uses the following type style conventions in this document.

Normal Used in running (nonprocedural) text for:• Names of interface elements (such as names of windows,

dialog boxes, buttons, fields, and menus)• Names of resources, attributes, pools, Boolean expressions,

buttons, DQL statements, keywords, clauses, environment variables, functions, utilities

• URLs, pathnames, filenames, directory names, computer names, filenames, links, groups, service keys, file systems, notifications

Bold Used in running (nonprocedural) text for:• Names of commands, daemons, options, programs,

processes, services, applications, utilities, kernels, notifications, system calls, man pages

Bold (cont.) Used in procedures for:• Names of interface elements (such as names of windows,

dialog boxes, buttons, fields, and menus)• What user specifically selects, clicks, presses, or types

Italic Used in all text (including procedures) for:• Full titles of publications referenced in text• Emphasis (for example a new term)• Variables

Courier Used for:• System output, such as an error message or script • URLs, complete paths, filenames, prompts, and syntax when

shown outside of running text


12

Preface

Where to get help EMC support, product, and licensing information can be obtained as follows.

Product information — For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:

http://Powerlink.EMC.com

Technical support — For technical support, go to Powerlink and choose Support. On the Support page, you will see several options, including one for making a service request. Note that to open a service request, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or with questions about your account.

We'd like to hear from you!

Your feedback on our TechBooks is important to us! We want our books to be as helpful and relevant as possible, so please feel free to send us your comments, opinions and thoughts on this or any other TechBook:

[email protected]

Courier bold Used for:• Specific user input (such as commands)

Courier italic Used in procedures for:• Variables on command line• User input variables

< > Angle brackets enclose parameter or variable values supplied by the user

[ ] Square brackets enclose optional values

| Vertical bar indicates alternate selections - the bar means “or”

{ } Braces indicate content that you must specify (that is, x or y or z)

... Ellipses indicate nonessential information omitted from the example


http://Powerlink.EMC.com

1

To comprehend the distance extension solutions for Storage Area Networks it is important to understand and recall the challenges when implementing SAN connectivity over remote distances. The following information is provided in this chapter:

◆ Early implementations of SAN environments............................... 14◆ DWDM................................................................................................. 15◆ CWDM................................................................................................. 19◆ SONET ................................................................................................. 21◆ GbE....................................................................................................... 23◆ TCP/IP................................................................................................. 24

Note: Refer to the “FCIP configuration” section in the WAN Optimization Controller Technologies TechBook, located at http://elabnavigator.EMC.com, Topology Resource Center tab, for more details on Brocade and Cisco FCIP configuration information.

Note: Refer to the “FCIP configuration and setup” section in the WAN Optimization Controller Technologies TechBook, located at http://elabnavigator.EMC.com, Topology Resource Center tab, for a distance extension case study using FCIP.

Extended DistanceOverview

Extended Distance Overview 13



14

Extended Distance Overview

Early implementations of SAN environmentsTo increase a single port between two Fibre Channel switches separated by a large geographical distance, every two strands (transmit, receive) of optical fiber cable were required to be physically added by the distance provider. The customer would generally incur expensive construction, service, and maintenance costs when adding a bulk of fiber cables intended to satisfy current E_Port connectivity requirements while allowing future growth potential and redundancy against accidental fiber breaks. Existing fibers that were used for Ethernet implementations could not be shared and required separate dedicated channels per protocol. The challenges involved with this process would stem anywhere from mandatory to extraneous costs associated with fiber cable maintenance.

In addition to costs, there were physical hardware limitations to achieving connectivity between (at least) two geographically separated sites. Fibre Channel optics installed on the Fibre Channel switch were at the mercy of the limited optical output transmission power. Even with repeater technology, distortion of the optical wavelength transmitted by the optics can occur over several hops.

The Fibre Channel switches provided limitations as well. Link initialization and flow control were solely controlled by the Fibre Channel switches. The Fibre Channel standard would actually dictate the thresholds in regards to supporting large distances through optical connectivity and the obtainable bandwidth between two Fibre Channel ports.

To finalize the list of challenges that SAN environments had to overcome, each Fibre Channel switch provider had its own non-standard and standard ways of implementing their native environments. This may deviate from the mass interpretation of the Fibre Channel standards.



DWDMDense Wavelength Division Multiplexing (DWDM) is a process in which different channels of data are carried at different wavelengths over one pair of fiber-optic links. This is in contrast with a conventional fiber-optic system in which just one channel is carried over a single wavelength traveling through a single fiber.

Using DWDM, several separate wavelengths (or channels) of data can be multiplexed into a multicolored light stream transmitted on a single optical fiber (dark fiber). This technique to transmit several independent data streams over a single fiber link is an approach to opening up the conventional optical fiber bandwidth by breaking it up into many channels, each at a different optical wavelength (a different color of light). Each wavelength can carry a signal at any bit rate less than an upper limit defined by the electronics, typically up to several gigabits per second.

Different data formats being transmitted at different data rates can be transmitted together. Specifically, IP data, ESCON SRDF®, Fibre Channel SRDF, SONET data, and ATM data can all be traveling at the same time within the optical fiber.

DWDM systems are independent of protocol or format, and no performance impacts are introduced by the system itself.

Figure 1 illustrates the DWDM technology concept:

Figure 1 DWDM example

DWDM 15

16


For EMC® customers it means that multiple SRDF® channels and Fibre Channel Inter Switch Links (ISL) can be transferred over one pair of fiber links along with traditional network traffic. This is especially important where fiber links are at a premium. For example, a customer may be leasing fiber, so the more traffic they can run over a single link, the more cost effective the solution.

With today's technology, the capacity of a single pair of fiber strands is virtually unlimited. The limitation comes from the DWDM itself. Optical-to-electrical transfers for switching and channel protection are required and limit the input traffic per channel.

Available DWDM topologies include point-to-point and ring configurations with protected and unprotected schemas. DWDM technology can also be used to tie two or more metro area data centers together as one virtual data center.

DWDM systems can multiplex and de-multiplex a large amount of channel quantities. Each channel is allocated its own specific wavelength (lambda) band assignment. Each wavelength band is generally separated by 10 nm spacing(s). As optical technologies improve, separations between each channel may be further reduced enabling more channels to be packed (tighter) onto a single duplex dark fiber.

DWDM has a higher cost associated due to greater channel consolidation, flexibility, utilization of higher quality hardware precision-cooling components (to prevent low frequency signal drift) and the capabilities of regenerating, re-amplifying and reshaping (3R) wavelengths assigned to channels to ensure optical connectivity over vast distances.

Varying circuits pack capabilities are also offered in a DWDM environment. DWDM circuit packs / blades can provide the following protocol conversions:

◆ Fibre Channel to SONET

◆ Fibre Channel to Gigabit Ethernet

◆ Fibre Channel to IP

In addition, some circuit packs can enable features such as write acceleration and buffer-to-buffer credit spoofing. To verify the latest supported distance systems and features, refer to the EMC Support Matrix.





Figure 2 shows a general concept of Fibre Channel link extension using DWDM.

Figure 2 Fibre Channel link extension

Note: All components are randomly selected and do not reflect a specific setup or configuration.

Note: Distance limitation may also be affected by application response time-out values and should consider signal propagation delay over site distance.

The following list provides general envelope guidelines for using DWDM systems:

◆ May be used for ESCON RDF distance extension, with direct connection between EMC Symmetrix® ESCON director ports and DWDM input ports.

◆ May be used for ISL extension of Fibre Channel switched fabrics. (E-Lab™ Navigator describes switch compatibility.)

◆ Fabric topology guidelines are provided per Fibre Channel switch topology documentation.

LocalDWDM FC switch

Server

RemoteDWDM

d4 d2 d1 d3 d5

FC switch

d1 = DWDM signal over dark fiber medium.d2 and d3 = Local ISL connections between switches and DWDM input. Can be SM or MM depending on DWDM and switch interfaces or local distance requirements.d4 and d5 = Local storage or server connections into the fabric.

StorageStorage

DWDM 17


18


◆ Direct connections between host HBA or Symmetrix Fibre Channel director to a DWDM port are not supported. E-Lab Navigator contains specific DWDM distance and topology guidelines.

◆ As a general approach, two distances need to be measured. The shorter of the two is the maximum distance to be supported in the site.

For differences between DWDM and CWDM, refer to “Differences between DWDM and CWDM” on page 19.





CWDMCoarse Wave Division Multiplexing (CWDM), like DWDM, uses similar processes of multiplexing and de-multiplexing different channels by assigning different wavelengths to each channel. CWDM is intended to consolidate environments containing a low number of channels at a reduced cost.

CWDM contains 20 nm separations between each assigned channel wavelength. CWDM technology generally uses cost-effective hardware components that require a reduced amount of precision-cooling components usually dominant in DWDM solutions due to the wider separations. With CWDM technology the number of channel wavelengths to be packed onto a single fiber is greatly reduced.

CWDM implementations, like DWDM, utilize an optical-to-electrical-to-optical technology where all the channels are multiplexed into a single CWDM device performing the optical-to-electrical-to-optical conversion.

A CWDM connectivity solution can use optics generating a higher wavelength with increased output optical power. Each channel is designated its own specific wavelength by the specific hot-pluggable CWDM GBIC/SFP optic installed on the Fibre Channel Switches. With clean fibers, minimal patch panel connections, and ample optical power, CWDM optics alone can provide connectivity distances of up to 100 km per channel. To complete this solution a passive MUX/DEMUX is required to consolidate multiple channel-wavelengths into a single duplex 9-micron dark fiber.

Differences between DWDM and CWDM

The following are differences between DWDM and CWDM:

◆ Number of channels that are supported per solution.

DWDM systems can support channels ranging from 16 channels or above while CWDM supports 16 channels or below.

◆ CWDM GBIC/SFP optics can be used to increase the wavelength output of a channel (such as, FC-switch optics).

CWDM 19

20


The CWDM GBIC/SFP optics is usually installed in the Fibre Channel switch or client device. The wavelength and optical power enhanced links are then multiplexed and de-multiplexed to and from a single-mode 9-micron dark fiber.

◆ Costs.

Hardware components included with DWDM units are higher in cost due to precision-cooling techniques required to prevent signal drift. DWDM offers greater channel flexibility and capacity.

◆ Configurations can be complex with CWDM.

CWDM requires specific optics for each specific wavelength. Growth for a CWDM environment is limited and difficult to manage when supporting environments growing to larger channel support. More cabling would be required, thereby increasing complexity.

◆ DWDM devices offer circuit packs with numerous features such as, protocol conversions, buffer-to-buffer credit spoofing, write acceleration).



SONETSynchronous Optical NETwork, (SONET), is a standard for optical telecommunications transport, developed by the Exchange Carriers Standards Association for ANSI. SONET defines a technology for carrying different capacity signals through a synchronous optical network. The standard defines a byte-interleaved multiplexed transport occupying the physical layer of the OSI model.

Synchronization is provided by one principal network element with a very stable clock (Stratum 3), which is sourced on its outgoing OC-N signal. This clock is then used by other network elements for their clocks (loop timing).

SONET is useful in a SAN for consolidating multiple low-frequency channels (Client ESCON and 1, 2 Gb Fibre Channel) into a single higher-speed connection. This can reduce DWDM wavelength requirements in an existing SAN infrastructure. It can also allow a distance solution to be provided from any SONET service carrier, saving the expense of running private optical cable over long distances.

The basic SONET building block is an STS-1 (Synchronous Transport Signal), composed of the transport overhead plus a Synchronous Payload Envelope (SPE), totaling 810 bytes. The 27-byte transport overhead is used for operations, administration, maintenance, and provisioning. The remaining bytes make up the SPE, of which an additional nine bytes are path overhead. It is arranged as depicted in Figure 3. Columns 1, 2, and 3 are the transport overhead.

SONET 21

22


Figure 3 STS-1 organization

An STS-1 operates at 51.84 Mb/s, so multiple STS-1s are required to provide the necessary bandwidth for ESCON, Fibre Channel, and Ethernet, as shown in Table 1. Multiply the rate by 95% to obtain the usable bandwidth in an STS-1 (reduction due to overhead bytes).

One OC-48 can carry approximately 2.5 channels of 1 Gb/s traffic, ss shown in Table 1. To achieve higher data rates for client connections, multiple STS-1s are byte-interleaved to create an STS-N. SONET defines this as byte-interleaving three STS-1s into an STS-3, and subsequently interleaving STS-3s.

By definition, each STS is still visible and available for ADD/DROP multiplexing in SONET, although most SAN requirements can be met with less complex point-to-point connections. The addition of DWDM can even further consolidate multiple SONET connections (OC-48), while also providing distance extension.

Table 1 SONET/Synchronous Digital Hierarchy (SDH)

STS Optical carrier Optical carrier rate (Mb/s)

STS-1 OC-1 51.840

STS-3 OC-3 155.520

STS-12 OC-12 622.080

STS-48 OC-48 2488.320

STS-192 OC-192 9953.280



GbEGigabit Ethernet (GbE) is a terminology describing an array of technologies involved in the transmission of Ethernet packets at the rate of 1024 megabits (Mb/s) or 1 gigabit per second. Gigabit Ethernet is specifically designed to surpass the traditional 10/100 Mb/s link speeds. GbE is defined by the IEEE publication 802.3z, which was standardized in June, 1998. This is a physical layer standard following elements of the ANSI Fibre Channel’s physical layer. This standard is one of many additions to the original Ethernet standard (802.3 - Ethernet Frame) published in 1985 by the IEEE organization. The following are nomenclature and characteristics of GbE.

◆ 1000Base-SX is defined as a fiber-optic Gigabit Ethernet standard encompassing the use of multi-mode (50 or 62.5 micron) fiber with 850 nanometer wavelengths. Distances of over 500 meters can be achieved.

◆ 1000Base-Lx is defined as a fiber-optic Gigabit Ethernet standard encompassing the use of single-mode (9 micron) fiber with 1310 nanometer wavelengths. Distances of 10 km or more can be achieved.

◆ Copper coaxial cabling, multi-mode fiber-optic cabling (50 and 62.5 micron) and single-mode (9 micron) cabling are available choices for the 802.3z standard.

◆ GbE is mainly used in distance extension products as the transport layer for protocol such as TCP/IP. However, in some cases the product is based on a vendor-unique protocol.

◆ Distance products using GbE may offer features such as compression, write acceleration, and buffer credit spoofing

GbE 23

24


TCP/IPThe Transmission Control Protocol (TCP) is a connection-oriented transport protocol that guarantees reliable in-order delivery of a stream of bytes between the endpoints of a connection. TCP achieves this by assigning each byte of data a unique sequence number, maintaining timers, acknowledging received data through the use of acknowledgements (ACKs), and retransmission of data if necessary. Once a connection is established between the endpoints data can be transferred. The data stream that passes across the connection is considered a single sequence of eight-bit bytes, each of which is given a sequence number.

This section contains information on the following:

◆ “TCP terminology” on page 24

◆ “TCP error recovery” on page 28

◆ “Network congestion” on page 31

◆ “Internet Protocol security (IPsec)” on page 32

TCP terminology

This section provides information for TCP terminology.

Acknowledgements(ACKs)

The TCP acknowledgement scheme is cumulative as it acknowledges all the data received up until the time the ACK was generated. As TCP segments are not of uniform size and a TCP sender may retransmit more data than what was in a missing segment, ACKs do not acknowledge the received segment, rather they mark the position of the acknowledged data in the stream. The policy of cumulative acknowledgement makes the generation of ACKs easy and any loss of ACKs do not force the sender to retransmit data. The disadvantage is the sender does not receive any detailed information about the data received except the position in the stream of the last byte that has been received.

Delayed ACKs Delayed ACKs allow a TCP receiver to refrain from sending an ACK for each incoming segment. However, a receiver should send an ACK for every second full-sized segment that arrives. Furthermore, the standard mandates a receiver must not withhold an ACK for more than 500 ms. The receivers should not delay ACKs that acknowledge out-of-order segments.



Maximum segmentsize (MSS)

The maximum segment size (MSS) is the maximum amount of data, specified in bytes, that can transmitted in a segment between the two TCP endpoints. The MSS is decided by the endpoints, as they need to agree on the maximum segment they can handle. Deciding on a good MSS is important in a general inter-networking environment because this decision greatly affects performance. It is difficult to choose a good MSS value since a very small MSS means an under-utilized network, whereas a very large MSS means large IP datagrams that may lead to IP fragmentation, greatly hampering the performance. An ideal MSS size would be when the IP datagrams are as large as possible without any fragmentation anywhere along the path from the source to the destination. When TCP sends a segment with the SYN bit set during connection establishment, it can send an optional MSS value up to the outgoing interface’s MTU minus the size of the fixed TCP and IP headers. For example, if the MTU is 1500 (Ethernet standard), the sender can advertise a MSS of 1460 (1500 minus 40).

Maximumtransmission unit

(MTU)

Each network interface has its own MTU that defines the largest packet that it can transmit. The MTU of the media determines the maximum size of the packets that can be transmitted without IP fragmentation.

Retransmission A TCP sender starts a timer when it sends a segment and expects an acknowledgement for the data it sent. If the sender does not receive an acknowledgement for the data before the timer expires, it assumes that the data was lost or corrupted and retransmits the segment. Since the time required for the data to reach the receiver and the acknowledgement to reach the sender is not constant (because of the varying Internet delays), an adaptive retransmission algorithm is used to monitor performance of each connection and conclude a reasonable value for timeout based on the round trip time.

SelectiveAcknowledgement

(SACK)

TCP may experience poor performance when multiple packets are lost from one window of data. With the limited information available from cumulative acknowledgements, a TCP sender can only learn about a single lost packet per round trip time. An aggressive sender could choose to retransmit packets early, but such retransmitted segments may have already been successfully received. The Selective Acknowledgement (SACK) mechanism, combined with a selective repeat retransmission policy, helps to overcome these limitations. The receiving TCP sends back SACK packets to the sender confirming receipt of data and specifies the holes in the data that has been received. The sender can then retransmit only the missing data segments. The selective acknowledgment extension uses two TCP

TCP/IP 25

26


options. The first is an enabling option, SACKpermitted, which may be sent in a SYN segment to indicate that the SACK option can be used once the connection is established. The other is the SACK option itself, which may be sent over an established connection once permission has been given by SACKpermitted.

TCP segment The TCP segments are units of transfer for TCP and used to establish a connection, transfer data, send ACKs, advertise window size and close a connection. Each segment is divided into three parts:

◆ Fixed header of 20 bytes

◆ Optional variable length header, padded out to a multiple of 4 bytes

◆ Data

The maximum possible header size is 60 bytes. The TCP header carries the control information. SOURCE PORT and DESTINATION PORT contain TCP port numbers that identify the application programs at the endpoints. The SEQUENCE NUMBER field identifies the position in the sender’s byte stream of the first byte of attached data, if any, and the ACKNOWLEDGEMENT NUMBER field identifies the number of the byte the source expects to receive next. The ACKNOWLEDGEMENT NUMBER field is valid only if the ACK bit in the CODE BITS field is set. The 6-bit CODE BITS field is used to determine the purpose and contents of the segment. The HLEN field specifies the total length of the fixed plus variable headers of the segment as a number of 32-bit words. TCP software advertises how much data it is willing to receive by specifying its buffer size in the WINDOW field. The CHECKSUM field contains a 16-bit integer checksum used to verify the integrity of the data as well as the TCP header and the header options. The TCP header padding is used to ensure that the TCP header ends and data begins on a 32-bit boundary. The padding is composed of zeros.

TCP window A TCP window is the amount of data a sender can send without waiting for an ACK from the receiver. The TCP window is a flow control mechanism and ensures that no congestion occurs in the network. For example, if a pair of hosts are talking over a TCP connection that has a TCP window size of 64 KB, the sender can only send 64 KB of data and it must stop and wait for an acknowledgement from the receiver that some or all of the data has been received. If the receiver acknowledges that all the data has been received. The sender is free to send another 64 KB. If the sender gets back an acknowledgement from the receiver that it received the first



32 KB (which is likely if the second 32 KB was still in transit or it is lost), then the sender could only send another 32 KB since it cannot have more than 64 KB of unacknowledged data outstanding (the second 32 KB of data plus the third).

The primary reason for the window is congestion control. The whole network connection, which consists of the hosts at both ends, the routers in between, and the actual connections themselves, might have a bottleneck somewhere that can only handle so much data so fast. The TCP window throttles the transmission speed down to a level where congestion and data loss do not occur.

The factors affecting the window size are as follows:

Receiver’s advertised windowThe time taken by the receiver to process the received data and send ACKs may be greater than the sender’s processing time, so it is necessary to control the transmission rate of the sender to prevent it from sending more data than the receiver can handle, thus causing packet loss. TCP introduces flow control by declaring a receive window in each segment header.

Sender’s congestion window The congestion window controls the number of packets a TCP flow has in the network at any time. The congestion window is set using an Additive-Increase, Multiplicative-Decrease (AIMD) mechanism that probes for available bandwidth, dynamically adapting to changing network conditions.

Usable window This is the minimum of the receiver’s advertised window and the sender’s congestion window. It is the actual amount of data the sender is able to transmit. The TCP header uses a 16 bit field to report the receive window size to the sender. Therefore, the largest window that can be used is 2**16 = 65K bytes.

Window scalingThe ordinary TCP header allocates only 16 bits for window advertisement. This limits the maximum window that can be advertised to 64 KB, limiting the throughput. RFC 1323 provides the window scaling option, to be able to advertise windows greater than 64 KB. Both the endpoints must agree to use window scaling during connection establishment.

The window scale extension expands the definition of the TCP window to 32 bits and then uses a scale factor to carry this 32- bit

TCP/IP 27

28


value in the 16-bit Window field of the TCP header (SEG.WND in RFC-793). The scale factor is carried in a new TCP option — Window Scale. This option is sent only in a SYN segment (a segment with the SYN bit on), hence the window scale is fixed in each direction when a connection is opened.

TCP error recoveryIn TCP, each source determines how much capacity is available in the network so it knows how many packets it can safely have in transit. Once a given source has this many packets in transit, it uses the arrival of an ACK as a signal that some of its packets have left the network and it is therefore safe to insert new packets into the network without adding to the level of congestion. TCP uses congestion control algorithms to determine the network capacity. From the congestion control point of view, a TCP connection is in one of the following states.

◆ Slow start: After a connection is established and after a loss is detected by a timeout or by duplicate ACKs.

◆ Fast recovery: After a loss is detected by fast retransmit.

◆ Congestion avoidance: In all other cases. Congestion avoidance and slow start work hand-in-hand. The congestion avoidance algorithm assumes that the chance of a packet being lost due to damage is very small. Therefore, the loss of a packet means there is congestion somewhere in the network between the source and destination. Occurrence of a timeout and the receipt of duplicate ACKs indicates packet loss.

When congestion is detected in the network it is necessary to slow things down, so the slow start algorithm is invoked. Two parameters, the congestion window (cwnd) and a slow start threshold (ssthresh), are maintained for each connection. When a connection is established, both of these parameters are initialized. The cwnd is initialized to one MSS. The ssthresh is used to determine whether the slow start or congestion avoidance algorithm is to be used to control data transmission. The initial value of ssthresh may be arbitrarily high (usually ssthresh is initialized to 65535 bytes), but it may be reduced in response to congestion.

The slow start algorithm is used when cwnd is less than ssthresh, while the congestion avoidance algorithm is used when cwnd is greater than ssthresh. When cwnd and ssthresh are equal, the sender may use either slow start or congestion avoidance.



TCP never transmits more than the minimum of cwnd and the receiver’s advertised window. When a connection is established, or if congestion is detected in the network, TCP is in slow start and the congestion window is initialized to one MSS. Each time an ACK is received, the congestion window is increased by one MSS. The sender starts by transmitting one segment and waiting for its ACK. When that ACK is received, the congestion window is incremented from one to two, and two segments can be sent. When each of those two segments is acknowledged, the congestion window is increased to four, and so on. The window size increases exponentially during slow start as shown in Figure 4 on page 30. When a time-out occurs or a duplicate ACK is received, ssthresh is reset to one half of the current window (that is, the minimum of cwnd and the receiver's advertised window). If the congestion was detected by an occurrence of a timeout the cwnd is set to one MSS.

When an ACK is received for data transmitted the cwnd is increased, but the way it is increased depends on whether TCP is performing slow start or congestion avoidance. If the cwnd is less than or equal to the ssthresh, TCP is in slow start and slow start continues until TCP is halfway to where it was when congestion occurred, then congestion avoidance takes over. Congestion avoidance increments the cwnd by MSS squared divided by cwnd (in bytes) each time an ACK is received, increasing the cwnd linearly as shown in Figure 4. This provides a close approximation to increasing cwnd by, at most, one MSS per RTT.

TCP/IP 29

30


Figure 4 Slow start and congestion avoidance

A TCP receiver generates ACKs on receipt of data segments. The ACK contains the highest contiguous sequence number the receiver expects to receive next. This informs the sender of the in-order data that was received by the receiver. When the receiver receives a segment with a sequence number greater than the sequence number it expected to receive, it detects the out-of-order segment and generates an immediate ACK with the last sequence number it has received in-order (that is, a duplicate ACK). This duplicate ACK is not delayed. Since the sender does not know if this duplicate ACK is a result of a lost packet or an out-of-order delivery, it waits for a small number of duplicate ACKs, assuming that if the packets are only reordered there will be only one or two duplicate ACKs before the reordered segment is received and processed and a new ACK is generated. If three or more duplicate ACKs are received in a row, it implies there has been a packet loss. At that point, the TCP sender retransmits this segment without waiting for the retransmission timer to expire. This is known as fast retransmit ( see Figure 5 on page 31).

cwnd

RTT

Slow start: Exponentialgrowth of cwnd

SYM-001457

ssthresh

Congestion avoidance: Lineargrowth of cwnd



After fast retransmit has sent the supposedly missing segment, the congestion avoidance algorithm is invoked instead of the slow start; this is called fast recovery. Receipt of a duplicate ACK implies that not only is a packet lost, but that there is data still flowing between the two ends of TCP, as the receiver will only generate a duplicate ACK on receipt of another segment. Hence, fast recovery allows high throughput under moderate congestion.

Figure 5 Fast retransmit

Network congestion A network link is said to be congested if contention for it causes queues to build up and packets start getting dropped. The TCP protocol detects these dropped packets and starts retransmitting them, but using aggressive retransmissions to compensate for packet loss tends to keep systems in a state of network congestion even after the initial load has been reduced to a level which would not normally have induced network congestion. In this situation, demand for link bandwidth (and eventually queue space), outstrips what is available. When congestion occurs, all the flows that detect it must reduce their transmission rate. If they do not do so, the network will remain in an unstable state with queues continuing to build up.

Send segments 21 - 26

Receive ACK for 21and 22

Received 3 duplicateACKs expecting 23Retransmit 23

Received ACK for 26expecting 27

23 lost in the network

Received segment 21 and 22send ACK for 21 and 22expecting 23

Received 24 still expecting 23 senda duplicate ACK

Received 25 still expecting 23 senda duplecate ACK

Received 26 still expecting 23 senda duplicate ACK

GEN-000299

TCP/IP 31

32


Internet Protocol security (IPsec)Internet Protocol security (IPsec) is a set of protocols developed by the IETF to support secure exchange of packets in the IP layer. IP Security has been deployed widely to implement Virtual Private Networks (VPNs).

IP security supports two encryption modes:

◆ Transport

◆ Tunnel

Transport mode encrypts only the payload of each packet, but leaves the header untouched. The more secure Tunnel mode encrypts both the header and the payload.

On the receiving side, an IP Security compliant device decrypts each packet. For IP security to work, the sending and receiving devices must share a public key. This is accomplished through a protocol known as Internet Security Association and Key Management Protocol/Oakley (ISAKMP/Oakley), which allows the receiver to obtain a public key and authenticate the sender using digital certificates.

Tunneling and IPsecInternet Protocol security (IPsec) uses cryptographic security to ensure private, secure communications over Internet Protocol networks. IPsec supports network-level data integrity, data confidentiality, data origin authentication and replay protection. It helps secure your SAN against network-based attacks from untrusted computers, attacks that can result in the denial-of-service of applications, services, or the network, data corruption, and data and user credential theft.

By default, when creating an FCIP tunnel, IPsec is disabled.

FCIP tunneling with IPsec enabled will support maximum throughput as follows:

◆ Unidirectional: approximately 104 MB/s

◆ Bidirectional: approximately 90 MB/s

Used to provide greater security in tunneling on an FR4-18i blade or a Brocade SilkWorm 7500 switch, the IPsec feature does not require you to configure separate security for each application that uses TCP/IP. When configuring for IPsec, however, you must ensure that there is



an FR4-18i blade or a Brocade SilkWorm 7500 switch in each end of the FCIP tunnel. IPsec works on FCIP tunnels with or without IP compression (IPComp).

IPsec requires an IPsec license in addition to the FCIP license.

IPsec terminology

AES Advanced Encryption Standard. FIPS 197 endorses the Rijndael encryption algorithm as the approved AES for use by US government organizations and others to protect sensitive information. It replaces DES as the encryption standard.

AES-XCBC Cipher Block Chaining. A key-dependent one-way hash function (MAC) used with AES in conjunction with the Cipher-Block-Chaining mode of operation, suitable for securing messages of varying lengths, such as IP datagrams.

AH Authentication Header. Like ESP, AH provides data integrity, data source authentication, and protection against replay attacks but does not provide confidentiality.

DES Data Encryption Standard is the older encryption algorithm that uses a 56-bit key to encrypt blocks of 64-bit plain text. Because of the relatively shorter key length, it is not a secured algorithm and no longer approved for Federal use.

3DES Triple DES is a more secure variant of DES. It uses three different 56-bit keys to encrypt blocks of 64-bit plain text. The algorithm is FIPS-approved for use by Federal agencies.

ESP Encapsulating Security Payload is the IPsec protocol that provides confidentiality, data integrity, and data source authentication of IP packets, as well as protection against replay attacks.

MD5 Message Digest 5, like SHA-1, is a popular one-way hash function used for authentication and data integrity.

SHA Secure Hash Algorithm, like MD5, is a popular one-way hash function used for authentication and data integrity.

MAC Message Authentication Code is a key-dependent, one-way hash function used for generating and verifying authentication data.

TCP/IP 33

34


HMAC A stronger MAC because it is a keyed hash inside a keyed hash. SA Security association is the collection of security parameters and authenticated keys that are negotiated between IPsec peers.


2

This chapter provides the following information to consider when working with extended distance.

◆ Link speed........................................................................................... 36◆ Data buffering and flow control ...................................................... 37◆ TCP/IP window................................................................................. 53◆ Active and passive devices............................................................... 54◆ FC SONET/GbE/IP........................................................................... 61◆ Network stability and error recovery ............................................. 62

Distance ExtensionConsiderations

Distance Extension Considerations 35

36

Distance Extension Considerations

Link speedLink speed is an important aspect of distance extension configurations. Within the SAN networks link speeds equate to the amount of maximum bandwidth reachable on an E_Port and/or an F_Port. There are a variety of link speeds that are supported in a SAN network. Table 2 compares and contrasts the STS, optical carrier, and Fibre Channel link speed rates.

Table 2 STS-1s and optical carrier rates

STS Optical carrier Optical carrier rate Fibre Channel link speeds

STS-1 OC-1 51.84 Mb/s

STS-3 OC-3 155.52 Mb/s

STS-12 OC-12 622.08 Mb/s

STS-24 OC-24 1244.16 Mb/s 1.0625 Gb/s or 100 MB/s



STS-192 OC-192 9953.28 Mb/s 10.51875 Gb/s or 12.75 Gb/s



Data buffering and flow controlThe following information is discussed in this section:

◆ “Fibre Channel,” next

◆ “Maximum supported distance per Fibre Channel BB_Credit guidelines” on page 38

◆ “Buffer-to-buffer credit information” on page 42

Fibre ChannelFibre Channel uses the BB_Credit (buffer-to-buffer credit) mechanism for hardware-based flow control. This means that a port has the ability to pace the frame flow into its processing buffers. This mechanism eliminates the need of switching hardware to discard frames due to high congestion. EMC testing has shown this mechanism to be extremely effective in its speed and robustness.

BB_Credit management occurs between any two Fibre Channel ports that are connected. For example:

◆ One N_Port and one F_Port

◆ Two E_Ports

◆ Two N_Ports in a point-to-point topology

◆ In Arbitrated Loop different modes

The standard provides a frame-acknowledgement mechanism in which an R_RDY (Receiver Ready) primitive is sent from the receiving port to the transmitting port for every available buffer on the receiving side. The transmitting port maintains a count of free receiver buffers, and will continue to send frames if the count is greater than zero.

The algorithm is as follows:

1. The transmitter's count initializes to the BB_Credit value established when the ports exchange parameters at login.

In an Arbitrated Loop environment the credits are established by the receiving port sending in advance R_RDY primitives after the login to establish the credit.

2. The transmitting port decrements the count per transmitted frame.

Data buffering and flow control 37

38


3. The transmitting port will stop sending frames when the credit reaches zero.

4. When a link reset occurs, the credit values are reestablished to values negotiated upon login.

5. The transmitting port increments the count per R_RDY it receives from the receiving port.

Figure 6 provides a view of the BB_Credit mechanism.

Figure 6 BB_Credit mechanism

As viewed from Port A’s perspective, when a link is established with Port B, BB_Credit information is exchanged. In this case, Port B provided a BB_Credit count of 5 to Port A. For Port A, this means it can transmit up to five Fibre Channel frames without receiving an R_RDY.

Maximum supported distance per Fibre Channel BB_Credit guidelines

In order to achieve maximum utilization of the Fibre Channel link it is highly advisable that both ports, connected on either side of the long haul setup provided by the DWDM, be capable of high BB_Credit counts. Use the following formula to calculate the approximate BB_Credit(s) required for the specific long haul application. To calculate for BB_Credits, use the following formula for calculating the required BB_Credit count:

Port BFrame

R_RDY

5 BB_Credits

Frame

Frame-

-

-

Port A5 BB_Credits

Speed Formula

1 Gb/s BB_Credit = ROUNDUP [2 * one-way distance in km/4] * 1



8 Gb/s BB_Credit=ROUNDUP [2 * one-way distance in km/4] * 8

10 Gb/s BB_Credit=ROUNDUP [2 * one-way distance in km/4] * 12



The factor of 2 in the formulas accounts for the time it takes the light to travel the entire roundtrip distance: frame from transmitter to receiver and R_RDY back to transmitter.

Maximum allowable distance is based on optical power measurements of the site. These measurements should be approved by DWDM and fiber services provider(s). The distance between an ISL ports on a Fibre Channel switch to a DWDM port should be included as part of the total distance (d1+d2+d3). Refer to Figure 2 on page 17.

The following BB_Credit charts will aid in providing estimates in regards to the amount of credits that should be present on the link when factoring Fibre Channel link speeds and link distances between the E_Ports.

Assuming the following is true:

◆ Light propagation in glass is 5 microseconds/km, or 59 seconds/m.

◆ Frame size is 2148 bytes/frame.◆ Fibre Channel bit rate depends on the Fibre Channel speed.

Maximum distances assume 100% utilization of the ISL. If the ISL is not fully utilized, greater distances can be achieved since more BB_Credits become available. For example, for a 2 Gb/s switch port with 120 BB_Credits and with an ISL that is only 50% utilized, the maximum distance is 240 km.


40


Since Brocade’s credit information is provided by ASIC types, review Table 3 to correlate between switch ASIC and model numbers.

Table 3 Brocade switch ASIC and model numbers (page 1 of 2)

Vendor ASIC/Family EMC name Vendor name

Brocade Bloom Connectrix® DS-16B SilkWorm 2800

Bloom Connectrix DS-16B2 Silkworm 3800

Bloom Connectrix DS-32B2 SilkWorm 3900

Bloom Connectrix ED-12000B SilkWorm 12000

Bloom2 Connectrix ED-24000B SilkWorm 24000

Bloom2 Connectrix DS-16B3 Silkworm 3850

Bloom2 Connectrix DS-8B3 SilkWorm 3250

Condor Connectrix DS-4100B Brocade 4100

Condor Connectrix ED-48000B Brocade 48000



Condor 2 Connectrix DS-5100B Brocade 5100

Condor 2 Connectrix ED-DCX-B DCX

Condor 2 Connectrix ED-DCX-4S-B DCX-4S

Goldeneye Connectrix DS-220B SilkWorm 220E

Goldeneye 2 Connectrix DS-300B Brocade 300

Goldeneye 2 Connectrix DS-5300B Brocade 5300



Table 4 provides information on Cisco Fibre Channel ASIC.

Brocade M Series Stitch ED-1032 ED-5000

Viper / Fuji-Shasta DS-16M ES-3016

DS-16M2 ES-3216

DS-32M ES-3032

DS-32M2 ES-3232

ED-64M ED-6064

ED-140M ED-6140

Posideon/Teton N/A ES-4300

DS-24M2 ES-4500

Sanera ED-10000M Intrepid 10000

Pegasus/Teton DS-4400M ES-4400

DS-4700M ES-4700

Table 3 Brocade switch ASIC and model numbers (page 2 of 2)

Vendor ASIC/Family EMC name Vendor name

Table 4 Cisco Fibre Channel ASIC information

Cisco MDS family Hardware (Similar Fibre Channel ASICs are listed in the same cell)

Generation 1 • 16, 32-port 2 G FC• 9216,9216A, 9216i• MPS-14/2• SSM

Generation 2 • 12, 24, 48-port 4 G FC• MSM18/4• 9222i

Generation 2 4-port 10 G FC (DS-X9704)

Generation 2 MDS 9124x

Generation 2 MDS 9134

Generation 3 24, 48, 4/44-port 8G FC

Generation 3 DS 9148


42


Buffer-to-buffer credit informationDetermining sufficient amount of buffer-to-buffer credits is crucial when provisioning Fibre Channel environments prior to utilization. Miscalculating the amount of credits may lead to less than desired performance (such as, buffer-to-buffer credit, starvation, or backpressure).

Credit starvation occurs when the amount of available credits reaches a zero state preventing all forms of Fibre Channel I/O-transmission from occurring. Once this condition is reached a timeout value will be triggered causing the link to reset.

Refer to the next sections for basic credit table for switches and storage arrays for Brocade B Series and Brocade M Series and Cisco.

Brocade credit chart With regards to flow control, Brocade switches support at least two forms of flow control options on the E_Port. VC_RDY and R_RDY flow control are both available options for all Brocade switch types.

For VC_RDY flow control, Brocade switches require an “Extended Fabric Mode” which will require to be activated through license code. Table 5, next, Table 6 on page 43, and Table 7 on page 44, are provided to display the supported distances for an E_Port when activating these modes in a Fibre Channel point-to-point switched fabric environment. These tables are broken down by ASIC type.

Table 5 Bloom and Bloom II ASICs (page 1 of 2)

Mode Description Buffer allocation @ 1 Gb/s

Buffer allocation @ 2 Gb/s

Distance @ 1 Gb/s

Distance @ 2 Gb/s

Earliest Fabric OS release

Extended Fabric license required?

L0 Level 0 static mode; default

5 5 10 Km 5 Km All No

LE Level E Static Mode;

13 19 n/a 10 Km v3.x, v4.x No

L0.5 Level 0.5 static mode

19 34 25 Km 25 Km v3.1.0, v4.1.0, 5.x

Yes

L1 Level 1 static mode

27 54 50 Km 50 Km All Yes

L2 Level 2 static mode

60 65 / 108 for Bloom II

100 Km 60 Km 100 Km for Bloom II

All Yes



LD Dynamic mode; auto detects distance upon initialization

Auto Auto Auto(Max is 200 Km)

Auto(Max is 200 Km)

v3.1.0, v4.1.0, v4.4.0, 5.x – depending on model

Yes

LS Static long distance mode (user specified)

User specified User specified User specified

User specified

v5.1.0 Yes

Table 5 Bloom and Bloom II ASICs (page 2 of 2)

Mode Description Buffer allocation @ 1 Gb/s


Distance @ 1 Gb/s

Distance @ 2 Gb/s



Table 6 Condor ASIC

Mode Buffer allocation @ 1 Gb/s


Buffer Allocation @ 4 Gb/s

Distance @ 1 Gb/s

Distance @ 2 Gb/s

Distance @ 4 Gb/s



L0 5 5 5 10 Km 5 Km 2 Km All No

LE 11 16 26 10 Km 10 Km 10 Km 3.x, 4.x No

L0.5 18 31 56 25 Km 25 Km 25 Km 3.1.0, 4.1.0, 4.x, 5.x

Yes

L1 31 56 106 50 Km 25 Km 50 Km All Yes

L2 56 106 206 100 Km 100 Km 100 Km All Yes

LD Auto Auto Auto Auto (max 500 Km)

Auto (max 250 Km)

Auto (max 100 Km

3.1.0, 4.1.0, 4.x, 5.x – depending on model

Yes

LS User specified

User specified

User specified

User specified (max 500 Km)



5.1.0 Yes


44


d

?

Table 7 Condor 2 ASIC





Distance @ 1 Gb/s

Distance @ 2 Gb/s

Distance @ 4 Gb/s

Distance @ 8 Gb/s


ExtendeFabric license required

L0 8 8 8 8 10 Km 5 Km 2 Km 1 Km 6.0x Yes

LE 11 16 26 46 10 Km 10 Km 10 Km 10 Km 6.0x Yes

LD Auto Auto Auto Auto Auto Auto Auto Auto 6.0x Yes

LS User specified

User specified

User specified

User specified

User specified (Refer to Table 10 on page 46)



User specified(Refer to Table 10 on page 46)

6.0x Yes



d

?

Keep in mind that each Brocade switch family, ASIC, and mode type (such as, L1, L2, LD,and so on) will have unique VC_RDY amounts and characteristics depending on specific fabric configurations. Please refer to the EMC Support Matrix for specific configuration information.

Table 8 Goldeneye ASIC




Distance @ 1 Gb/s

Distance @ 2 Gb/s

Distance @ 4 Gb/s



L0 3 3 3 6 Km 3 Km 1.5 Km All No

LE 11 16 31 10 Km 10 Km 10 Km 3.x, 4.x No

L0.5 18 31 56 25 Km 25 Km 25 Km 5.1.0 Yes

L1 31 56 106 50 Km 50 Km 50 Km 5.1.0 Yes

L2 56 106 n/a 100 Km 100 Km n/a 5.1.0 Yes

LD Auto Auto Auto Auto Auto Auto 5.1.0 Yes

LS User Specified

User Specified

User Specified

User Specified (max 293 Km)



5.1.0 Yes

Table 9 Goldeneye 2 ASIC





Distance @ 1 Gb/s

Distance @ 2 Gb/s

Distance @ 4 Gb/s

Distance @ 8 Gb/s


ExtendeFabric license required

L0 8 8 8 8 10 Km 5 Km 2 Km 1 Km 6.1x Yes

LE 11 16 26 46 10 Km 10 Km 10 Km 10 Km 6.1x Yes

LD Auto Auto Auto Auto Auto Auto Auto Auto 6.1x Yes

LS User specified

User specified

User specified

User specified





6.1x Yes



46


Brocade also supports R_RDY flow control (through Portcfgislmode). Brocade R_RDY mode can be activated when connecting to distance extension devices providing additional Buffer-to-Buffer Credits.

Brocade Extended Fabrics Brocade’s Extended Fabrics is a licensed feature that extends Storage Area Networks (SANs) across longer distances for disaster recovery and business continuance operations by enabling a modified buffering scheme in order to support long distance fibre channel extensions, such as MAN/WAN optical transport devices. This bulletin is suitable for external dissemination.

Configurable distances for Extended Fabrics

Table 10 shows the maximum supported extended distances (in kilometers) that can be configured for one port on a specific switch or blade at different speeds.

Table 10 Configurable distances for Extended Fabrics (page 1 of 2)

Switch/blade model

Maximum distances (km) that can be configured assuming 2112 Byte Frame Size

1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s

300 972 486 243 121

4100/5000 500 250 100 N/A

4900 500 250 100 N/A

5100 3388 1694 847 423

5300 588 294 147 73

5410 1164 582 291 145.5

5424 972 486 243 121.5

5450 940 470 235 117.5

5480 972 486 243 121.5

7500 500 250 100 N/A

7600 500 250 100 N/A

7800 822 410 205 102

VA-40FC 3388 1694 847 423

Brocade Encryption Switch 2784 1392 696 348



Note: The 10 Gb/s FC10-6 blade has two port groups of three ports each. For extended ISLs, all buffers available to a group are used to support one port at up to 100 km.

Refer to the Brocade Fabric OS switch documentation, located at http://powerlink.emc.com, for further details.

Flow control

The Fibre Channel standards specifications (for example, FC-PH and FC-SW) define a method of flow control called R_RDY to manage and control traffic as it flows across data links. Although the standards define how R_RDY flow control should be used, it does not prohibit the use of other vendor unique methods. By default, Brocade switches use Virtual Channel (VC) flow control over E_Port connections within a fabric.

FA4-18 500 250 100 N/A

FC4-16 500 250 100 N/A

FC4-16IP 500 250 100 N/A

FC4-32 500 250 100 N/A

FC4-48 500 250 100 N/A

FC8-16 2589 / 2781 1294 / 1390 647 / 695 323 / 347

FC8-32 2589 / 3277 1294 / 1638 647 / 819 323 /409

FC8-48 2461 / 3149 1230 / 1574 615 / 787 307 / 393

FC10-6 See the note at the end of this table for information about this blade.

FR4-18i 500 250 100 N/A

FS8-18 3208 1604 802 401

FX8-24 2125 1062 531 265

Table 10 Configurable distances for Extended Fabrics (page 2 of 2)

Switch/blade model

Maximum distances (km) that can be configured assuming 2112 Byte Frame Size

1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s


48


VC flow control provides the following advantages over R_RDY:

◆ The ability to differentiate fabric internal traffic from end-to-end device traffic.

In this case, switches generate fabric internal traffic that communicate state information to each other, such as link state information for routing, and device information for Name Service. This type of traffic is given a higher priority so that switches can distribute the most up-to-date information across the fabric even under heavy device traffic.

◆ The ability to differentiate data flows of end-to-end device traffic to avoid head-of-line blocking.

In the case of (2), when there are multiple I/Os multiplexed over a single ISL, by assigning different VCs to different I/Os and giving them the same priority, each I/O can have a fair share of the bandwidth so that a large-size I/O will not consume the whole bandwidth and starve a small-size I/O, thus balance the performance of the different devices communicating across the ISL. To identify a VC between two end-points of a link, VC_RDY is used.

Buffer allocation

When a switch port is configured for Extended Fabrics, additional credit is given to virtual channels that carry class 2 or 3 data traffic. This allows distances between switches to be extended over greater distances while maintaining maximum performance over ISLs. The Brocade Extended Fabrics license allows ISLs to be connected at up to 60 km for 2 Gb/s links and up to 100 km for 1 Gb/s links without degradation of performance.

When Extended Fabrics is enabled on Fabric OS v3.x and v4.x switches, two changes occur:

◆ Additional buffer credits are allocated to certain Virtual Channels on the long distance E_Port, and

◆ ARB(vc) is used as inter-frame gap instead of idles.

The additional buffers allow the E_Port “pipe” to be fully utilized over long distances and the ARB(vc) ordered set is used to notify the receiving switch as to which VC queue the next incoming frame should be placed on. There is a different ARB(vc) primitive for each of the eight possible virtual channels.



MAN/WAN optical transport devices

Vendors of optical transport devices may not be aware of E_Port functionality on Brocade switches, which may cause interoperability issues under certain configurations. Although there are certain workarounds, any vendor wishing to understand this functionality can contact Brocade. All devices tested in the Fabric Aware program are verified to operate under ideal switch configurations.

If the extension devices between the Brocade switches transparently propagate all traffic as is, these ARB(vc)s will not cause any problems. However, recently some transport devices have been introduced that do more than simply pass through the Fibre Channel frames. In some cases, and in some modes, these devices have been shown to have problems processing the ARB(vc) frames resulting in disruption of traffic over the long distance connection.

In these cases there are at least three solutions to this issue:

◆ If the extension device is capable of being configured in a mode which transparently passes Fibre Channel frames, there should be no disruption of traffic due to the ARB(vc) frames.

◆ If the 'fabric.ops.mode.longDistance' bit is set to '1' on all Brocade switches in the fabric, the ARB(vc) primitives will not be sent. The default setting of this parameter is '0'. In order to set this bit the switches will need to be disabled and the bit set using either the configure command in a telnet or serial console window or through a GUI management interface. In the Web Tools GUI this bit can be set by selecting the Admin button from the main screen and then clicking the enable button under Extended Fabrics Mode on the Extended Fabric tab. Despite the label of this button, it does not actually enable/disable Extended Fabrics and, in fact, the only effect this button has to set or unset the fabric.ops.mode.longDistance bit.

Note: This parameter will need to be set on all switches in the fabric, not just the switch that has the long distance connection. Also note that this parameter affects all E_Ports on the switch (long distance or otherwise) by changing the amount of buffer credits allocated to the port.

◆ Since optical transport devices are designed to provide connectivity over long distance, many vendors provide their own method of managing flow control over long distance connections. This can allow FC performance to be maintained at up to hundreds or even thousands of kilometers without degradation.


50


If the vendor supports this type of configuration, Brocade switches can be configured to use standards based R_RDY flow control using the portCfgISLMode CLI command. Extended Fabrics would not be necessary.

Note: The latest updated firmware levels and hardware levels support the combination of both Extended Fabric Modes with R_RDY mode implementation. This allows the customer to bypass the old challenges of configuring the Brocade Fabric environment to its pure native mode.

Refer to the EMC Brocade switch documentation for further details.

Brocade M Seriescredit chart

Brocade M Series supports only R_RDY flow control. Each Brocade M Series Family type switch will have unique credit amounts. Refer to Table 11 for details of the Brocade M Series credit chart.

Table 11 Brocade M Series credit chart (page 1 of 2)

Switch type (EMC/Brocade M Series)

Module / Optic Link speed Number of credits Notes

ED-1032 / ED-5000 Multi-mode, single-mode

1 Gb / 2 Gb 60

DS-16M / ES -3016 Multi-mode, single-mode

1 Gb / 2 Gb 60

DS-16M2 / ES-3216 Multi-mode, single-mode

1 Gb / 2 Gb 60

DS-32M / ES3032 Multi-mode, single-mode

1 Gb / 2 Gb 60


1 Gb / 2 Gb 60

ED-64M / ED-6064 Multi-mode, single-mode

1 Gb / 2 Gb 60

ED-140M / ED-6140 Multi-mode, single-mode

1 Gb / 2 Gb 60

N/A / ES-4300 Multi-mode, single-mode

1 Gb / 2 Gb 12 / 7 12 on the first 4 and 7 on the rest… Credit increases applies to specified quad areas.



Cisco MDS creditchart

Cisco MDS switches only utilizes R_RDY flow control. Table 12 displays the number of BB-credits are available per E_Port.


1 Gb / 2 Gb 12 / 7 12 on the first 4 and 7 on the rest… Credit increases applies to specified quad areas.

ED-10000M / Intepid 10000 Multi-mode, single-mode

1 Gb/2 Gb/10Gb 1373

DS-4400M/ ES-4400 Multi-mode, single-mode

1 Gb / 2 Gb/ 4 Gb

DS-4700M / ES-4700 Multi-mode, single-mode

1 Gb / 2 Gb / 4 Gb

Table 11 Brocade M Series credit chart (page 2 of 2)

Switch type (EMC/Brocade M Series)

Module / Optic Link speed Number of credits Notes

Table 12 Cisco MDS credit chart

Switch Type Blade/Optic Support Link Speed Number of Credits Notes

9509 Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255



9216A Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255

9216i Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255

9120 Multi-mode, single-mode, CWDM 1 Gb / 2 Gb 255 Based on the first quad



52


Symmetrix FibreAdapter credit chart

EMC Symmetrix boards uses R_RDY flow control. Table 13 displays the number of BB-credits available per Fibre Channel Adapter F_Port.

Table 13 Symmetrix Fibre Adapter credit chart

Symmetrix Family Board Type / Optic Link Speed Number of Credits

Symmetrix 5.0 Fibre Adapter / multi-mode 1 Gb / 2 Gb 7





TCP/IP windowA TCP window is the amount of data a sender can send without waiting for an ACK from the receiver. The TCP window is a flow control mechanism and ensures that no congestion occurs in the network. For example, if a pair of hosts are talking over a TCP connection that has a TCP window size of 64 KB (kilobytes), the sender can only send 64 KB of data and then it must stop and wait for an acknowledgment from the receiver that some or all of the data has been received. If the receiver acknowledges that all the data has been received then the sender is free to send another 64 KB. If the sender gets back an acknowledgment from the receiver that it received the first 32 KB (which could happen if the second 32 KB was still in transit or it could happen if the second 32 KB got lost), then the sender could only send another 32 KB since it cannot have more than 64 KB of unacknowledged data outstanding (the second 32 KB of data plus the third).

The primary reason for the window is congestion control. The whole network connection, which consists of the hosts at both ends, the routers in between, and the actual connections themselves, will have a bottleneck somewhere that can only handle so much data so fast. The TCP window throttles the transmission speed down to a level where congestion and data loss do not occur. The factors affecting the window size are as follows:

◆ Receiver’s advertised windowFor more information, refer to “Receiver’s advertised window” on page 27.

◆ Sender’s congestion windowFor more information, refer to “Sender’s congestion window” on page 27.

◆ Usable windowFor more information, refer to “Usable window” on page 27.

◆ Window scalingFor more information, refer to “Window scaling” on page 27.

TCP/IP window 53

54


Active and passive devicesThis section contains the following information:

◆ “Buffer-to-buffer local termination,” next

◆ “SRDF with SiRT” on page 56

◆ “Fast write/ write acceleration” on page 58

◆ “SiRT with distance vendor write acceleration” on page 59

◆ “Link initialization” on page 60

Buffer-to-buffer local terminationIn Fibre Channel, BB_Credits are a method of maintaining the flow control of transmitting Fibre Channel frames. BB_Credits help maintain a balanced flow of I/O transmissions while avoiding underutilization or oversubscription of a Fibre Channel link.

Figure 7 on page 55 shows what the buffering flow control would normally follow without the local termination. This places the burden on the end nodes to maintain and track the BB_Credit flow control on the Fibre Channel link. The flow control distance will be determined by the amount of credits and the link speed that is supported by the end nodes. The end nodes can be an E_Port or F_Port.

BB_Credits are provided by the Fibre Channel switches. The distance extension device is transparent and does not participate in BB_Credit flow control. Link speed, latency, and the amount of available credits will determine the performance characteristics of these configurations.



Figure 7 Flow control managed by Fibre Channel switch (without buffering from distance extension devices)

Determining sufficient amount of BB_Credits is crucial when provisioning Fibre Channel environments prior to utilization. Miscalculating the amount of credits may lead to performance degradation due to credit starvation.

Note: EMC recommends adding 20% margin to calculated BB_Credit values to account for spikes in traffic.

Credit starvation occurs when the number of available credits reaches zero preventing all forms of Fibre Channel transmissions from occurring. Once this condition is reached a timeout value will be triggered causing the link to re-initialize. To avoid this condition, sufficient BB_credits must be available to meet the latency and performance requirements for the particular SRDF deployment.

The standard Fibre Channel flow control and BB_Credit mechanism is adequate for most short-haul deployments. With longer distance deployments however, the Fibre Channel flow control model is not as effective. Additional buffering and WAN-optimized flow control are often needed.

Figure 8 on page 56 shows a configuration where the distance extension devices are providing additional buffering and flow control mechanisms for the purpose of increasing distances between locations. To accomplish this, the Fibre Channel end nodes are

SRDF RF SRDF RF

DISTANCE

NODE

DISTANCE

NODE

Switch

Local flowcontrol

Switch

Local flowcontrol

Local RemoteFlow control managed fromFibre Channel end nodes

Local flow control

Active and passive devices 55

56


provided with immediate R_RDY responses with every "sent" FC-frame. This occurs within the local flow control segments. The distance extension nodes, in turn, implement their own buffering and WAN-optimized flow control.

Figure 8 Flow control (with buffering from distance extension devices)

Refer to the distance extension vendor documentation for detailed information on each vendor’s buffering and flow control implementations.

SRDF with SiRTSingle RoundTrip (SiRT) for Fibre Channel SRDF directors (RFs) was introduced in EMC Enginuity™ 5772 for SRDF/S mode only. It is dynamically enabled for SRDF/S links > 12 Km for block sizes up to 32K in Enginuity 5773 code. SiRT is compatible with Fast Write/Write Acceleration switches and extenders, as it will measure link latency and disable automatically if connected to these devices. As a best practice, it is recommended that either the EMC SiRT feature or the third-party fast write feature should be used. Both should not be enabled simultaneously.

The Fibre Channel SiRT feature for the Fibre Channel director can be set to Off or Automatic. When set to Automatic, this feature will only accelerate write I/Os using criteria based on latency and I/O size.

SRDF RF SRDF RF

DISTANCE

NODE

DISTANCE

NODELocal flow

controlLocal flow

controlDistance flow

control

Switch

Local flowcontrol

Local flowcontrol

Switch

Local flowcontrol

Local flowcontrol

Local Remote



Note: EMC recommends contacting your EMC Customer Service Representative to verify that the setting is enabled if required in your environment.

Figure 9 shows the normal write process without the SiRT feature.

Figure 9 Normal write command process

The intended purpose of this feature is to maintain SRDF/S synchronicity while improving performance by localizing the transfer-ready response to the local RF port, thereby reducing an unnecessary acknowledgement response (trip) over the dark fiber distance (step 2 in Figure 9). Immediate benefits are apparent upon activation in transparent SRDF synchronous distance extension environments.


58


If applicable, multiple SRDF synchronous links can maximize their I/O performance over the network (transparent WDM environment).

In the example shown in Figure 10, RF1 (R1 F_Port) and RF2 (R2 F_Port) are managing the SiRT flow control.

Figure 10 SRDF SiRT

Legend:

Fast write/ write accelerationEMC Connectrix and other third-party products offer single roundtrip for Fibre Channel capabilities (fast write/write acceleration) that can also increase SRDF throughput for direct-attach or Fibre Channel switched fabric configurations over extended distances. It is transparent to SRDF FC links and is used for all SRDF modes to decrease response time (SRDF/S) or improve performance over long distance links (mostly for adaptive copy and SRDF/AR, but also for some SRDF/A configurations).

Red RF-ports with SiRT activated.

Blue A step-by-step of a single write command with SiRT enabled.



Figure 11 shows a write command with fast write features.

Figure 11 Write command with SiRT

For Connectrix or third-party products, refer to the EMC Support Matrix available at http://elabnavigator.EMC.com to verify which of these products are supported for SRDF configurations.

IMPORTANT!Not all products offering this feature are supported with SRDF due to unique write commands utilized by SRDF.

SiRT with distance vendor write acceleration

With this in mind SiRT usage, in combination with the distance extension device-offered write acceleration mode, must be addressed. Essentially for environments where the distance extension device is already servicing write commands on an E_Port level, it is recommended to disable SiRT. Refer to Figure 12 on page 60.





60


Figure 12 All F_Ports will benefit

Legend:

In Figure 12, by enabling the write acceleration feature on the distance extension device, potentially all F_Ports (RF ports, FA ports, tape, etc.) issuing writes traversing across the E_Port attached to the distance extension client port can also take advantage of the throughput benefits from the activated write acceleration feature.

Link initializationFor link initialization of a Fibre Channel port, Fibre Channel specifications state that the maximum tolerable response time for a response is 100 milliseconds roundtrip time. This timeframe coincides with the limited timeframe of the Receiver-Transmitter Timeout Value (R_T_TOV), which is how long an FC port listens for a link response to a link service before an error is detected.

Red RF ports benefiting from distance extension device, write acceleration.

Blue Scope.



FC SONET/GbE/IPDistance devices or circuit packs/blades performing protocol conversions from Fibre Channel to and from an alternate backbone protocol are required to maintain the lowest link initialization timeout value. In contrast to Fibre Channel’s R_T_TOV, the SONET, GbE, and IP implementations can extend well beyond the 100 millisecond roundtrip time. For these environments, the distance extension devices should offer a setting enabling “local initialization” to occur between the “local” Fibre Channel port and the “local” distance extension client port rather than initializing the “local” Fibre Channel port across the actual physical distance to its “remote” Fibre Channel port (Figure 13).

Figure 13 Link initialization (More than 100 ms R_T_TOV)

FC SONET/GbE/IP 61

62


Network stability and error recoveryThis section explains how the following handle error recovery.

CWDM CDWM devices do not participate in error recovery at any level. The device to handle the recovery depends on the level the error occurred. In case of link events, it will be handled by the Fibre Channel ports (switch or storage) across the CWDM link. In case of SCSI level errors, the application (SRDF or MirrorView™) will handle the error recovery. Link bit errors will cause SCSI level errors.

DWDM Error recovery is based on the attach client circuit pack that the Fibre Channel ports attached to. If the Fibre Channel ports attached to a Buffer-to-Buffer credit spoofing circuit then link events will be handled locally with the attached Fibre Channel port. SCSI level errors will be handled by the application. Link bit errors will cause SCSI level errors.

SONET Error recovery is based on the attach client circuit pack that the Fibre Channel ports attached to. If the Fibre Channel ports attached to a Buffer-to-Buffer credit spoofing circuit, link events will be handled locally with the attached Fibre Channel port. SCSI level errors will be handled by the application. Link bit errors will cause SCSI level errors.

GE Error recovery is based on the attach client circuit pack that the Fibre Channel ports attached to. If the Fibre Channel ports attached to a Buffer-to-Buffer credit spoofing circuit then link events will be handled locally with the attached Fibre Channel port. SCSI level errors will be handled by the application. Link bit errors will cause SCSI level errors.

TCP/IP Error recovery will be handled by the TCP/IP distance device (review “TCP/IP” on page 24). If the errors persist and do not provide sufficient quality for the link to recover, the errors will be propagated to the attached Fibre Channel ports.


3

This chapter contains the following information on IP-based distance extension solutions.

◆ Network design best practices ......................................................... 64◆ EMC-Brocade distance extension solutions ................................... 66◆ Configuring IPsec............................................................................... 78◆ Fast Write and tape pipelining ......................................................... 80◆ EMC-Cisco MDS distance extension solution ............................... 84◆ EMC-Brocade M Series distance extension solution..................... 86◆ EMC-QLogic distance extension solution ...................................... 92◆ Summary ............................................................................................. 96

IP-Based DistanceExtension Solutions

IP-Based Distance Extension Solutions 63

64

IP-Based Distance Extension Solutions

Network design best practicesThe network should be dedicated solely to the IP technology being used and other traffic should not be carried over it.

The network must be well-engineered with no packet loss or duplication. This would lead to undesirable retransmission. While planning the network, care must be taken to ensure that the utilized throughput will never exceed the available bandwidth. Oversubscribing available bandwidth will lead to network congestion, which causes dropped packets and leads to TCP slow start. Network congestion must be considered between switches as well as between the switch and the end device.

The MTU must be configured based on the maximum available MTU supported by each component on the network.

Network conditions impact on effective throughput

Table 14 on page 65 demonstrates the impact of network conditions on TCP/IP effective throughput (data provided to the distance extension device by the Fibre Channel devices—the amount of data on the link will be greater due to TCP retransmission).

The distance between the sites has a significant impact on the distance system effective throughput. However, it is a fixed value. Packet loss, on the other hand is not a fixed value and can be relatively high due to TCP recovery mechanism and therefore has a greater impact. When designing the distance extension solution, network conditions must be taken into account to ensure that the effective throughput is sufficient for the solution needs. Over-utilization of the effective throughput will result in errors at the application levels.

Review “TCP/IP” on page 24 for how to maximize effective throughput.



Table 14 Network impact on effective throughput example

Compression Network conditions Effective throughput

enabled 100 ms RTT with 1% packet loss 1 MB/s

100 ms RTT with no packet loss 6 MB/s

50 ms RTT with 1% packet loss 3 MB/s


200 ms RTT with 1% packet loss 800 KB/s

200 ms RTT with no packet loss 3.7 MB/s

disabled 100 ms RTT with 1% packet loss 360 KB/s






Network design best practices 65

66


EMC-Brocade distance extension solutionsThis section discusses:

◆ “Brocade 7500” on page 67

◆ “Brocade 7800” on page 69

The following Brocade terminology is used throughout this section.

Backbone Fabric Routers provide a backbone (BB) Fabric to interconnect routers for more scalable and flexible routed SANs. Each router may have many edge fabric connections, but only one BB fabric. Routers connect to the BB fabric through E_Ports, and all N_Port and NL_ Port connections on a router are part of the BB fabric. With 4 Gb routers, a number of hosts and storage devices may be connected to the BB fabric.

Edge Fabric Fibre Channel fabric connected to a router through an EX_Port (IFL). This is largely the same as any standard Fibre Channel fabric. This is, for the most part, where the hosts and storage are attached.

E_Port A port on an FC switch or router, which connects to another switch or router, forming an ISL. If the devices previously formed separate fabrics, these fabrics merge, putting all fabric services into one distributed image.

EX_Port FC Routers use EX_Ports instead of E_Ports on routed interfaces. To connect a router to a switch, you connect its EX_Port to another switch's E_Port using an appropriate cable. Routers still use E_ or VE_Ports to form a backbone fabric.

IFL The connection between an E_Port and an EX_Port is an "Inter-Fabric Link".

ISL The connection between two E_Ports is an Inter-Switch Link.



Brocade 7500FCIP tunneling enables you to connect one central office to different branch offices using different VE_Ports or VEX_Ports, thereby enabling branch offices to connect with each other without having to merge data center and branch office fabrics.

Fibre Channel frame encapsulation on one VE_Port and the reconstruction of Fibre Channel frames on the other VE_Port is transparent to the initiator and target, but the administration of VE_Ports is different from other Fibre Channel port types.

Fabric OS supports FCIP ISLs between two Brocade switches (Brocade 7500 or 48000 with a FR4-18i blade) or routers.

FCIP also supports:

◆ Configuration and management of GbE ports

◆ Compression and decompression of Fibre Channel frames moving through FCIP tunnels

◆ Statistics gathering on several layers

◆ Traffic shaping that adheres to a rate limit on a per tunnel basis

◆ FCIP tunnel/GbE port event notification

◆ Fibre Channel Router capabilities over VE_Ports

LSAN LSAN Logical SANs are zones which span fabrics. They will traverse at least one EX_Port or VEX_Port. LSANs are how connectivity is configured across routers.

VE_Port An FCIP port on an FC switch will create a "Virtual E_Port". This is physically an IP/Ethernet interface, but each FCIP tunnel "looks" like an FC E_Port to the rest of the fabric.

VEX_Port In addition to supporting virtual E_Ports, Brocade platforms allow the FCIP and FC Router features to be combined, creating a Virtual EX_Port. FC Router features to be combined, creating a Virtual EX_Port.

EMC-Brocade distance extension solutions 67

68


FCIP tunneling introduces the following concepts:

◆ Tunnel

An FCIP tunnel carries Fibre Channel traffic (frames) over IP networks such that the Fibre Channel fabric and all Fibre Channel devices in the fabric are unaware of the IP network’s presence. Fibre Channel frames "tunnel" through IP networks by dividing frames, encapsulating the result in IP packets on entering the tunnel, and then reconstructing them as they leave the tunnel.

◆ VE_Port

Special types of ports, called VE_Ports (virtual E_Port), function somewhat like an E_Port. The link between an VE_Port and a VE_Port is called an interswitch link (ISL). You can configure multiple ISLs from a Brocade 7500 or 48000 with an FR4-18i blade. After you configure the VE_Ports on either two Brocade 7500s or 48000s with the FR4-18i blade, an FCIP connection is established between them. VE_Ports do not prevent fabric merging. Using a VEX_Port is one way to prevent fabrics from merging.

◆ VEX_Port

A VEX_Port enables routing functionality through an FCIP tunnel. VEX_Ports are virtual FC_Ports that are exposed by FCIP tunnels connecting to either the Brocade 7500 or 48000 with a FR4-18i blade; they run interfabric links (IFLs) as EX_Ports to enable Fibre Channel router capability. You can have up to eight VEX_Ports per GbE on the Brocade 48000 with a FR4-18i blade.

◆ GbE

Gigabit Ethernet ports are available on the Brocade 7500 and 48000 with a FR4-18i blade. These ports support FCIP with link speeds up to 1 Gb/s. Each GbE port (ge0, ge1) supports up to eight FCIP tunnels.

Note: You cannot create more than one FCIP tunnel on a given pair of IP address interfaces (local and remote). However, you can create multiple FCIP tunnels on an IP interface so that, minimally, either the local or remote IP interface will be unique and not have any other FCIP tunnel on it. When the GbE port has a valid SFP and is physically connected to any other GbE port, the status output from the switchShow command is online.



Supportedenvironment

Figure 14 shows an example of a Brocade 7500 configuration.

Figure 14 Brocade 7500 configuration example

References For more information, refer to www.brocade.com. For configuration help, refer to the Brocade FOS 5.1 Administration Guide.

Brocade 7800The FX8-24/7800 supports all features and functions associated with FCIP on the FR4-18i/7500 platforms. New FCIP functionality associated with the FX8-24 blade are:

◆ 10 x 1 GbE ports available

◆ 2 x 10 GbE ports available (note that both 10 GbE ports and 1 GbE ports cannot be enabled simultaneously)

◆ 12 x 8 Gb FC ports

IP WANnetwork

Data centerFC SAN

OfficeFC SAN

OfficeFC SAN

OfficeFC SAN

FibreChannelinitiator

FibreChannelinitiator

SilkWorm 7500 SilkWorm 7500

VE_Port

VE_Port VE_Port

VE_Port

SilkWorm 48000with FR4-18i Blade

SilkWorm 48000with FR4-18i Blade

FibreChannel

target

FibreChanneltarget

GEN-000296


http://www.brocade.com

70


◆ FCIP Trunking

◆ IPV6

◆ IPV4

◆ DSCP marking

◆ VEX

New FCIP features supported on the 7800 platform are:

◆ 6 x 1 GbE ports

◆ 16 x 8 Gb FC ports

◆ FCIP Trunking

◆ IPV6

◆ IPV4

◆ DSCP marking

Note: Unlike the FR4-18i/7500, FCIP tunnels in FX8-24/7800 are no longer associated with a specific GbE port.

FCIP TrunkingFCIP Trunking is a new feature which has been introduced with the 7800 and FX 8-24 FOS Release v6.3.x. (Refer to the EMC Support Matrix for the supported FOS v6.3.x versions.)

FCIP Trunking is a method for managing the use of WAN bandwidth and for providing redundant paths over the WAN that can protect against transmission due to WAN failure. Trunking is enabled by creating logical circuits within an FCIP tunnel. A tunnel may have multiple circuits. Each circuit is a connection between a pair of IP addresses that are associated with source and destination end-points of an FCIP tunnel.

Figure 15 on page 71 shows the relationship of trunks and circuits to VE_Ports, FCIP tunnels, and the physical GbE interfaces. FC traffic enters and exits an FCIP tunnel on a VE_Port. Applications on the FC side have no awareness of the existence of the FCIP tunnel. FCIP Trunking routes the FC traffic over FCIP circuits. FCIP circuits route traffic over a WAN using any of the GbE interfaces. An FCIP circuit is a logical connection between two peer switches or blades, so the same construct exists in each peer switch or blade.





Figure 15 Basic overview of Trunking components

TCP Trunking provides the following features:

◆ Load balancing across multiple connections

◆ Failover to remaining connections if a link fails

◆ Lossless Failover

◆ Lossless Link Loss (LLL)—Data in-flight is not lost when a link goes down

◆ Data in-flight will be resen— Same as with TCP

◆ In-Order-Delivery (IOD) after a failover: Data in-flight will be delivered in the correct order— Same as TCP

◆ Works with both FICON and FC: Supports FastWrite, OSTP and FICON Emulation over multiple links

CircuitEach circuit is a connection between a pair of IP addresses that are associated with source and destination end-points of an FCIP tunnel. An Ethernet interface can have one or more FCIP tunnels and circuits. Circuits in a tunnel can use the same or different Ethernet interfaces.

MetricA circuit has a “cost metric”. Lower metric circuits are preferred over higher metric circuits. When there are circuits with different metrics, all traffic goes through the circuits with lowest metric and no traffic goes through circuits with higher metric. If all circuits with the lowest


72


metric fail, circuits with higher metric are used. If all circuits have the same metric, traffic flows on all circuits. The remote end of a tunnel reorders frames to maintain in-order delivery. Load-leveling is automatically done across circuits with the lowest metric.

If a circuit fails, FCIP Trunking tries first to retransmit any pending send traffic over another lowest metric circuit. If no lowest metric circuits are available, then the pending send traffic is retransmitted over any available circuits with the higher metric.

TunnelFCIP tunnels are used to pass Fibre channel I/O through an IP network. FCIP tunnels are built on a physical connection between two peer switches or blades. An FCIP tunnel forms a single logical tunnel from the circuits. A tunnel scales bandwidth with each added circuit, providing lossless recovery during path failures and ensuring in-order frame delivery.

FCIP Tunnels can be formed by using the VE_Ports or VEX_Ports. VE_Ports and VEX_Ports are virtual E_Ports. VE_Ports are used to create interswitch links (ISLs). If VE_Ports are used on both ends of an FCIP tunnel, the fabrics connected by the tunnel are merged.

VEX_Ports enable interfabric links (IFLs). If a VEX_Port is on one end of an FCIP tunnel, the fabrics connected by the tunnel are not merged. The other end of the tunnel must be defined as a VE_Port. VEX_Ports are not used in pairs.

Adaptive Rate LimitingAdaptive Rate Limiting (ARL) is performed on FCIP tunnel connections to change the rate in which the FCIP tunnel transmits data through the TCP connections. ARL uses information from the TCP connections to determine and adjust the rate limit for the FCIP tunnel dynamically. This allows FCIP connections to utilize the maximum available bandwidth while providing a minimum bandwidth guarantee.

ARL applies a minimum and maximum traffic rate and allows the traffic demand and WAN connection quality to dynamically determine the rate. As traffic increases, the rate grows towards the maximum rate. If traffic subsides, the rate reduces towards the minimum. If traffic is flowing error-free over the WAN, the rate grows towards the maximum rate. If TCP reports an increase in retransmissions, the rate reduces towards the minimum.



QoS prioritiesEach FCIP circuit is assigned four TCP connections for managing FC Quality of Service (QoS) priorities over an FCIP tunnel. The priorities are as follows:

◆ F class – F class is the highest priority, and is assigned bandwidth as needed, at the expense of lower priorities, if necessary.

◆ QoS high – The QoS high priority gets at least 50% of the bandwidth.

◆ QoS medium – The QoS medium priority gets at least 30% of the bandwidth.

◆ QoS low – The QoS low priority gets at least 20% of the bandwidth.

Open Systems Tape PipeliningOpen Systems Tape Pipelining (OSTP) can be used to enhance open systems SCSI tape write I/O performance. When the FCIP link is the slowest part of the network, OSTP can provide accelerated speeds for read and write I/O over FCIP tunnels. To use OSTP, you need to enable FCIP Fastwrite and Tape Pipelining.

◆ FCIP Fastwrite accelerates the SCSI write I/Os over FCIP.

◆ Tape Pipelining accelerates SCSI read and write I/Os to sequential devices (such as tape drives) over FCIP, which reduces the number of round-trip times needed to complete the I/O over the IP network and speeds up the process. Each GbE port supports up to 2048 simultaneous accelerated exchanges.

Both sides of an FCIP tunnel must have matching configurations for these features to work. FCIP

Fastwrite and Tape Pipelining are enabled by turning them on during the tunnel configuration process. They are enabled on a per-FCIP tunnel basis.

FCIP Fastwrite and Tape Pipelining configurationsTo help understand the supported configurations, consider the configurations shown in the following two figures. In both cases, there are no multiple equal-cost paths. In Figure 16 on page 74, there is a single tunnel with Fastwrite and Tape Pipelining enabled.


74


Figure 16 Single tunnel, Fastwrite and Tape Pipelining enabled

In Figure 17, there are multiple tunnels, but none of them create a multiple equal-cost path.

Figure 17 Multiple tunnels to multiple ports, Fastwrite, and Tape Pipelining enabled on a per-tunnel/per-port basis



FCIP tunnels and VE_Ports on the 7800 switch

Note: A Brocade 7800 16/6 switch can support eight VE_Ports and Brocade 7800 4/2 can support two FCIP tunnels, and therefore eight FCIP tunnels.

Each FCIP tunnel is associated with a VE port. VE_Ports are numbered from 16 to 23. On the 7800 switch and on FX8-24 blades, VE_Ports do not have to be associated with a particular GbE port.

The full bandwidth provided by the six GbE ports is available to all tunnels. FCIP trunking provides load balancing. Failover capabilities are provided through the use of virtual FCIP circuits. Up to four FCIP circuits may be defined per tunnel. A single circuit cannot exceed 1 Gb/s capacity.

Note: The Open Systems Tape Pipelining is not supported with Brocade 7800 4/2.

FCIP tunnels and VE_Ports on the FX8-24 bladeAn FX8-24 blade can support 20 VE_Ports, and therefore 20 FCIP tunnels. Each FCIP tunnel is associated with a specific VE_Port. On FX8-24 blades, and on the 7800 switch, VE_Ports do not have to be associated with a particular GbE port.

VE_Ports 12 through 21 may use GbE ports ge0 through ge9, or they may use XGE port 1. VE_Ports 22 through 31 can only be used by XGE port 0. The total bandwidth cannot exceed 20 Gb/s.

There are twelve FC ports, numbered 0 through 11. The FC ports can operate at 1, 2, 4, or 8 Gb/s. There are ten GbE ports, number 0 through 9. Ports XGE0 and XGE1 may be configured as 10 GbE ports. The FX8-24 blade provides a maximum of 20 Gb/s of bandwidth for Ethernet connections, and can operate in one of three different modes:

◆ 1 Gb/s mode—You can use all the GbE ports (0 through 9).

◆ 10 Gb/s mode—You can use the XGE0 and XGE1 ports.

◆ Dual mode—You can use GbE ports 0 through 9, and port XGE0.

Note: VEX_Ports are not supported on the FX8-24 blade.

The full bandwidth provided by the ten GbE ports or two 10 GbE ports is available to all tunnels.


76


FCIP trunking provides load balancing. Failover capabilities are provided through the use of virtual FCIP circuits. FCIP tunnels using GbE ports can have up to four FCIP circuits spread across four GbE ports. FCIP tunnels using 10 GbE ports can have up to ten FCIP circuits over one 10 GbE port. A single circuit cannot exceed 1 Gb/s capacity. To create an FCIP tunnel with a capacity of 10 Gb/s over a 10GbE port, you must create an FCIP tunnel with ten FCIP circuits.

Virtual fabrics and the FX8-24 bladeThe FX8-24 FC ports can be part of any logical switch. The GE_Ports and VE_Ports on the FX8-24 blade can be part of any logical switch. GE_Ports and VE_Ports ports may be moved between any two logical switches. Ports do not need to be offline when they are moved. GE_Ports and VE_Ports are independent of each other, so both must be moved in independent steps, and you must clear the configuration on VE_Ports and GE_Ports before moving them between logical switches.

Note: This differs from the FR4-18i blade, where only GE_Ports need to be moved and all the VE_Ports created on that GE_Port are automatically moved. You do not need to delete VE_Port and GbE port configuration information.

The total number of VE_Ports in all the logical switches is equal to the maximum number of VE_Ports on an FX8-24 blade (which is 20) multiplied by the maximum number of FX8-24 blades allowed on a DCX or DCX-4S chassis (which is 2). VEX_Ports are not supported on the FX8-24 blade.

Table 15 compares the Brocade FX 8-24, Brocade 7800 16/6, and Brocade 7800 4/2.

Table 15 Product comparison (page 1 of 2)

Standard features Brocade FX8-24 Brocade 7800 16/6 Brocade 7800 4/2

Supported storage Open systems and mainframe

Open systems and mainframe

Open systems only

8 Gb/s Fibre Channel/FICON Ports 12 16 4

1 GbE ports 10 6 2

10 GbE ports (2) Optional N/A N/A

Maximum FCIP Bandwidth 20 Gb/s 6 Gb/s 2 Gb/s



Maximum number of FCIP tunnels 20 8 2

Maximum bandwidth per FCIP tunnel Up to 10 Gb/s with Optional FCIP Trunking

Up to 4 Gb/s with Optional FCIP Trunking

Up to 2 Gb/s with Optional FCIP Trunking

Integrated Routing Optional Optional Optional

High-performance compression Included Included Included

FCIP Fast Write Included Included Included

Open Systems Tape Pipelining Included Included Not Supported

Storage-Optimized TCP Included Included Included

Brocade DCFM FCIP management Included Included Included

FCIP Quality of Service Brocade DCX (Included) Brocade DCX-4S (Optional)

Optional Optional

FCIP Trunking Optional Optional Optional

Adaptive Rate Limiting Optional Optional Optional

Advanced Accelerator for FICON Optional Optional Not Supported

FICON CUP Optional Optional Not Supported

Table 15 Product comparison (page 2 of 2)

Standard features Brocade FX8-24 Brocade 7800 16/6 Brocade 7800 4/2

Supported storage Open systems and mainframe

Open systems and mainframe

Open systems only


78


Configuring IPsecFor more information on IPsec, refer to the “Internet Protocol security (IPsec)” section in the iSCSI SAN Topologies TechBook, located at http://elabnavigator.EMC.com, Topology Resource Center tab.

IPsec requires predefined configurations for IKE and IPsec. You can enable IPsec only when these configurations are well-defined and properly created in advance.

The following steps provide an overview of the IPsec protocol. All of these steps require that the correct policies have been created. Because policy creation is an independent procedure from FCIP tunnel creation, you must know which IPsec configurations have been created. This ensures that you choose the correct configurations when you enable an IPsec tunnel.

1. Some traffic from an IPsec peer with the lower local IP address initiates the IKE negotiation process.

2. IKE negotiates SAs and authenticates IPsec peers during phase 1 that sets up a secure channel for negotiation of phase 2 (IPsec) SAs.

IKE negotiates SA parameters, setting up matching SAs in the peers. Some of the negotiated SA parameters include encryption and authentication algorithms, Diffie-Hellman group and SA lifetimes.

3. Data is transferred between IPsec peers based on the IPsec parameters and keys stored in the SA database.

4. IPsec tunnel terminates. SA lifetimes terminate through deletion or by timing out.

The first step to configuring IPsec is to create a policy for IKE and a policy for IPsec. Once the policies have been created, you assign the policies when creating the FCIP tunnel.

IKE negotiates SA parameters and authenticates the peer using the preshared key authentication method. Once the two phases of the negotiation are completed successfully, the actual encrypted data transfer can begin.

IPsec policies are managed using the policy command.

You can configure up to 32 IKE and 32 IPsec policies. Policies cannot be modified; they must be deleted and re-created in order to change




the parameters. You can delete and re-create any policy as long as the policy is not being used by an active FCIP tunnel.

Each FCIP tunnel is configured separately and may have the same or different IKE and IPsec policies as any other tunnel. Only one IPsec tunnel can be configured for each GbE port.

Limitations Be aware of the following limitations:

◆ IPv6, NAT, and AH are not supported.

◆ You can only create a single secure tunnel on a port; you cannot create a nonsecure tunnel on the same port as a secure tunnel.

◆ IPsec specific statistics are not supported.

◆ Fast Write and tape pipelining cannot be used in conjunction with secure tunnels.

◆ To change the configuration of a secure tunnel, delete the tunnel and re-create it with the desired options.

◆ Jumbo frames are not supported for IPsec.

◆ There is no RAS message support for IPsec.

◆ Only a single route is supported on an interface with a secure tunnel.

Configuring IPsec 79

80


Fast Write and tape pipeliningIn cases where the FCIP link is the slowest part of the network, and where this affects speed, consider using Fast Write and tape write acceleration (tape pipelining). Fast Write and tape pipelining are two individual features that provide accelerated speeds to FCIP tunnels in some configurations. Because of their similarities, they are both described in this section.

Supported only in Fabric OS 5.2.x andlater, Fast Write accelerates the SCSI write I/Os over FCIP.

Tape pipelining accelerates SCSI write I/Os to sequential devices (such as tape drives) over FCIP. This reduces the number of roundtrip times needed to complete the I/O over the IP network and speeds up the process. In order to use tape pipelining, you must enable Fast Write as well.

Both sides of an FCIP tunnel must have a matching configuration for these features to work.

Compression, Fast Write, and tape pipelining features do not require any predefined configurations like IPsec does. This makes it possible to enable these features when you create the FCIP tunnels by adding optional parameters such as –c, -f, or -t.

Table 16 on page 81 provides a comparison of Fast Write and tape pipelining.



a. Total of 2048 simultaneous exchanges combined for Fast Write and tape pipelining.

Supported configurations

To help understand the supported configurations, review the supported configurations shown in Figure 18 on page 82 and Figure 19 on page 83.

Table 16 Fast Write and tape pipelining comparison

Fast Write Tape pipelining

Does not support multiple equal-cost path configurations.

Does not support multiple equal-cost path configurations or multiple non-equal-cost path configurations. (Refer to “Supported configurations” on page 81.)

Class 3 traffic is accelerated with Fast Write.

Class 3 traffic is accelerated between host and sequential device.

With sequential devices (tape drives), there are 1024 initiator-type (IT) pairs per GbE Port, but 2048 initiator-tape-LUN (ITL) pairs per GbE Port. The ITL pairs are shared among the IT pairs.a

• Example 1:You can have two ITL pairs for each IT pair as long as the target has two LUNs.

• Example 2:If a target has 32 LUNs, you can have 32 ITL pairs for IT pairs. In this case, only 64 IT pairs are associated with ITL pairs. The rest of the IT pairs are not associated to any ITP pairs, so no tape pipelining is performed for those pairs. By default, only Fast Write-based acceleration is performed on the unassociated pairs.

Does not support multiple non-equal-cost path between host and sequential device.

Fast Write and tape pipelining 81

82


In Figure 18, there is a single tunnel with Fast Write and tape pipelining enabled.

Figure 18 Single tunnel, Fast Write and tape pipelining enabled

FC SAN FC SAN

This connectioncan be VE-VE or

VEX-VE

FCIP tunnelFW=1, TA=1

GE 0

GE 1

GE 0

GE 1

Tape2

Tape1

T1

T0

Hn Hn Tn

H2

H1



In Figure 19, there are multiple tunnels, but none of them create a multiple equal-cost path. Fast Write and tape pipelining are enabled on a per-tunnel, per-port basis.

Figure 19 Multiple tunnels to multiple ports

FC SAN

FC SAN

FCIP tunnel 0FW=0, TA=0

These connectionsmust all be VEX-VE

H1

H2

GE 0

GE 1

FC SAN

GE 0

GE 1

GE 0

GE 1

GE 0





H2

H1

H3

H9H8

H7

H6

H5

H4

Hn

H11

SYM-001461

H10

Fast Write and tape pipelining 83

84


EMC-Cisco MDS distance extension solutionThe Cisco MDS 9000 family of switches can be used to link EMC storage devices (Symmetrix, VNX™ series, and CLARiiON®) across IP networks using the FCIP protocol for disaster recovery applications (SRDF and MirrorView) and for data migration (SAN Copy™). The MDS 9000 family supports the Fibre Channel and Gigabit Ethernet protocols.


Figure 20 shows an example of Cisco MDS 9000 distance extension.

Figure 20 Cisco MDS 9000 distance extension example

Note these configuration rules:

◆ Cisco MDS switches can be used as part of a disaster recovery (DR) and/or data migration SAN only.

◆ SRDF, MirrorView, and SAN Copy are the only supported configurations.

◆ Remote host I/O configurations are supported across the FCIP link.

◆ Host I/O across the FCIP link can be supported if the application can tolerate the latency incurred due to the FCIP link

VSAN Blocal SAN traffic

VSAN ASRDF/MV/SC

VSAN ASRDF/MV/SCVSAN A

FCIP

Local data center

VSAN Clocal SAN traffic

Remote data center

Allowed VSANs on FCIP = VSAN A

SRDF, MirrorView, SAN Copy



Note: E-Lab Navigator describes the latest supported configurations and minimum code requirements.

Symmetrix setup Symmetrix SRDF ports should be configured as standard Fibre Channel SRDF ports. In a Fibre Channel environment, the Cisco MDS switch provides all the services of a Fibre Channel switch, similar to those provided by any other Fibre Channel switch.

VNX setup VNX MirrorView ports should be configured as standard Fibre Channel MirrorView ports.

CLARiiON setup CLARiiON MirrorView ports should be configured as standard Fibre Channel MirrorView ports.

ReferencesSearch for the additional documentation and the Cisco MDS Configuration Guide at http://www.cisco.com and select the document relevant to the code running on your box.

EMC-Cisco MDS distance extension solution 85


http://www.cisco.com

86


EMC-Brocade M Series distance extension solutionBrocade M Series Eclipse switches are multiprotocol switches. They support the bridging and routing for Fibre Channel and Gigabit Ethernet. In an FC environment, the switch provides all the FC services similar to those provided by any other FC switch.

The Brocade M Series multiprotocol switches are also used in distance-extension configurations running SRDF or MirrorView over IP, for SAN Copy deployment over IP, or as a gateway device to perform FC-iSCSI translation. On these switches, ports 1 through 8 should be configured as FC ports (F_Ports or R_Ports). Ports 9 to 12 can be configured as HA ports, if you want to have HA at the router. Ports 13 to 16 can be configured as either iFCP (TCP) ports (to cover long distances) or as iSCSI ports. There can be minimum of two HA links and maximum of four HA links between two Eclipse routers (2640). All the iFCP and iSCSI connections should be made on the primary router. HA is not supported on the Eclipse 1620 routers.


Figure 21 on page 87, Figure 22 on page 87, and Figure 23 on page 88 show examples of a Brocade M Series in an SRDF, MirrorView, or SAN Copy environment.



Figure 21 Brocade M Series in an SRDF, MirrorView, or SAN Copy environment, example 1


EMC 2

EMC2

FC SANswitch

McDATA a multi-protocol switch




SRDF, MirrorView,or SAN copy

FC SANswitch

SANisland

SANisland

CLARiiON

Symmetrix

CLARiiON

Symmetrix

IP Cloud

GEN-000444

EMC 2

EMC2

FC SANswitch





SRDF, MirrorView,or SAN copy

FC SANswitch

SANisland

SANisland

CLARiiON

Symmetrix

CLARiiON

Symmetrix

IP Cloud

GEN-000445

EMC-Brocade M Series distance extension solution 87

88



Implementation best practices

Consider the following best practices:

◆ Multi-vendor switches (Brocade, Brocade M Series, Cisco, or QLogic) within a single SAN Island are not supported.

◆ There should be a minimum of two HA links and maximum of four HA links between two Brocade M Series Eclipse switches.

◆ All the iFCP and iSCSI connections should be made on the primary router.

◆ A maximum of four interswitch links (ISLs) between the Brocade M Series Eclipse switch and the Brocade/Brocade M Series/Cisco/ QLogic is supported.

EMC 2EMC 2

McDATA multi-protocol switch

McDATA multi-protocol switch

SAN island 1

CLARiiON Symmetrix

Supported interoperableFC switch

Supported interoperableFC switch

SAN island N

CLARiiONSymmetrix

SRDF,MirrorView,

or SAN Copy

SAN islandN-1

SAN island2

IP cloud

GEN-000446



◆ The Symmetrix (SRDF) or CLARiiON (MirrorView/SAN Copy) ports can be directly connected to the FC ports on the Brocade M Series Eclipse switch, or to Brocade/ Brocade M Series/Cisco/QLogic switches in the local SAN.

Configuration of the local SAN ID and iFCP gateway

The local SAN ID is conceptually similar to the Domain ID used in FC, but is local to the switch and will not affect the 24-bit addresses assigned to any N_Ports connected to the switch. Brocade M Series's Eclipse 2640 Series User Manual provides details for setting up this parameter.

The connection between SAN Islands is carried through the iFCP port (ports 13-16). Each iFCP port on the local site can be connected to iFCP ports on one or more remote sites. The association between the local site and the remote one is done through the iFCP Gateway definition. Refer to Brocade M Series documentation for further details.

Multi-site configuration: One local SAN Island to many remote SAN Islands

When designing a multi-site configuration, note the following restrictions:

◆ The number of remote SAN Islands connected to one iFCP port on the local SAN Island will not exceed eight connections per iFCP port (eight entries per port in the Gateway definition table per iFCP port).

◆ The total number of FC initiator-target pairs, zoned across the iFCP link, will not exceed 16 per iFCP port.

All configuration changes must be saved to flash memory. Changes require a switch reboot to become active.

Symmetrix setup For iFCP configurations, the Symmetrix ports should be configured as standard FC SRDF ports. The Brocade M Series Eclipse switch, similar to any FC switch, provides FC switching services such as zoning, RSCN, name services, and so on, to the SRDF ports connected to it.


90


CLARiiON setup For iFCP configurations, the CLARiiON ports should be configured as standard FC MirrorView or SAN Copy ports. The MirrorView application runs only in synchronous mode.

Settings on Brocade/ Brocade M Series/ Cisco/QLogic switchesWhile setting up the Brocade/ Brocade M Series/Cisco/QLogic switches to work with the Brocade M Series Eclipse switch, the following settings are essential:

◆ The Brocade/Brocade M Series/Cisco/QLogic switches must use the latest code supported by EMC for Interop Solutions. Refer to the EMC Support Matrix, "Edge Fabric Heterogeneity," for more information.

◆ The E_Port on the Brocade must be in Brocade native mode or Brocade M Series mode (to which the Brocade M Series Eclipse switch is connected) must be hard set to the actual port speed and autonegotiation must be disabled.

◆ The Cisco and QLogic switches must be set up to work in the Interop mode (Cisco interop mode 1 and QLogic open fabric).

◆ Once the zoning is defined on the Brocade M Series Eclipse switches, it appears in the Brocade/Brocade M Series/Cisco/ QLogic active configurations as zones beginning with SoIP_xxx. When changes are made through the SANvergence Manager, these zones will be updated accurately. The Brocade, Brocade M Series, Cisco, and QLogic zoning management utilities must not be used to alter these zones.

◆ When zones are created through Brocade M Series Eclipse switches, they are appended to the active zone configurations on the other switches. These zones are not automatically saved to the zoning library on all switches in the SAN. There may be cases where updates can be made using more than one application and this may result in errors. To avoid this, after every zoning change made using Brocade M Series Eclipse switch applications, the updated zone sets should be saved to the zoning libraries. Refer to documentation on Connectrix Manager, EMC Ionix™ ControlCenter®, or VisualSAN® as applicable.




Additional documentationAll Brocade M Series documentation can be found at http://www.brocade.com.

Refer to the Brocade M Series Eclipse User Guide (1620/2640) for additional information regarding:

◆ Command Line Interface reference

◆ SANVergence Manager

◆ Recommended settings


92


EMC-QLogic distance extension solutionThe QLogic iSR-6142 Storage Router is a low cost FC/iSCSI solution designed to enable users to replicate data between FC SANs over a LAN/WAN utilizing iSCSI/GigE as the transport over distance.

The router contains two 1/2 GB/s FC ports and dual 10/100/1000 MB/s iSCSI/GigE ports. The routers interconnect through the dual GigE/iSCSI links allowing the replication data to be transmitted between two end devices. The two routers allow up to 4 FC SANs to be connected as NL_ports (that is, 2 per router) and prevent the SANs from merging into one large SAN.

This router is intended for low to mid-range environments where distance extension and device replication, such as EMC's VNX series and CLARiiON MirrorView software, are essential.


The iSR-6142 Storage Router supports one distinct topology in an EMC environment:

WAN Topology — Interconnecting remote SAN Islands (also known as Remote SAN Island Connectivity).

The SANbox 6142 Intelligent Router supports inter-connecting remote SAN islands. This does not result in the merging of the two end fabrics but will allow communication to occur between two end nodes when correctly configured (Figure 24 on page 93).



Figure 24 SANbox 6142 Intelligent Router

As shown in Figure 24, CX1_SPA1 (CLARiiON MirrorView port) is attached to Fabric_1. CX2_SPA1 (CLARiiON MirrorView port) is attached to Fabric_2. Using the QLogic SANbox 6142 it is possible to establish the communication between the MirrorView ports while maintaining two separate fabrics. the QLogic Sanbox 6142 will create virtual entities on each fabric to represent the remote device. The mechanism to establish the connection is called remotemap. The remotemap is created using the CLI/GUI from either of the routers and is communicated to the remote router over the WAN. This remotemap presents CX1_SPA1 to Fabric_2 and CX2_SPA1 to Fabric_1 as an NL_Port. This NL_Port needs to be zoned local CX N_Port to allow communication between the two arrays over distance.

Scalability The following are scalability guidelines, restrictions, and limitations:

◆ Maximum number of connections = 1024.◆ Maximum number of virtual FC ports = 64 per unit (31 per FC

port with 1 additional dedicated to each FC port for discovery - VP0 and VP1).

◆ Maximum number of concurrent I/Os = 1024 per unit (typically 32 per session).

◆ Maximum number of initiators/targets = 62 per unit (31 per port).

GEN-000288

CLARiiON

CLARiiON

CX1_SPA1

CX2_SPA1

FC SANFABRIC_1

FC SANFABRIC_2

TCP/IPiSCSI network

QLogicSANbox 6142

QLogicSANbox 6142

CX1_SPA2(virtual)CX1_SPA1(virtual)

EMC-QLogic distance extension solution 93

94


Best practicesRequirements for this configuration are as follows:

◆ At least one FC Port of the iSR-6142 should be connected to FC SAN.

◆ iSCSI/ GE Port IP addresses of remote router and iSCSI/GE port IP addresses of local routers must be accessible by each other.

◆ Remote iSR-6142 management port IP address and local SANbox 6142 management port IP address must be accessible by each other.

Recommendations for this configuration are as follows:

◆ Both GigE links are utilized with load balancing enabled.

◆ Compression is enabled over distance.

◆ Smart Writes is enabled.

◆ Windows Scaling is enabled with the recommended Windows Scaling Factor setting.

◆ Header and Data Digest is enabled.

◆ Zone each N_Port that will have a remotemap to both of the router FC ports.

◆ Use WWPN zoning.

SmartWriteWhen connecting SAN over long distances, round trip delays create significant impact to the performance. Typically, data writes involve two or more round trip latencies that result in a significant barrier to the data replication performance. SmartWrite technology is designed to minimize the round trip latency of any write I/O to a single round-trip latency. Benefits realized with this feature key include:

◆ Minimizes round trip delays for any data write operation to a single round trip latency.

◆ Allows load balancing over multiple IP links.

◆ Provides failover and failback between two gigabit ethernet links.



◆ Allows data compression. This is very useful when data round trip latencies between two routers exceed more than 25 ms or long distance link rate is equal or less than 4500 Mb/s (DS-3 line rate).

References

For more information, refer to http://www.QLogic.com.

Please reference the QLogic SANbox 6142 Intelligent Storage Router User Guide for additional information regarding:

◆ Command Line Interface reference

◆ SANsurfer Router Manager GUI

◆ Recommended Windows Scaling Factor determined by latency between routers

◆ Hardware

Additional documentation regarding the QLogic SANbox 6142 Intelligent Storage Router includes:

◆ QLogic SANbox 6142 Quick Start Guide

EMC-QLogic distance extension solution 95

http://www.QLogic.com

96


SummaryTable 17 compares the distance extension solutions features for TCP/IP products.

a. Only one FCIP tunnel can be configured per GigE port if TCP Byte Streaming is enabled.

Table 17 Distance extension comparison table for TCP/IP products

Feature Symmetrix (GigE)

Brocade Cisco MDS Brocade M Series

QLogic

Fast Write n/a yes yes yes yes

Jumbo frames yes yes yes yes no

Encryption no no yes no no

Applications all families of srdf srdf, srdfa, mva, mvs, sancopy

srdf, srdfa, mva, mvs, sancopy, ors

srdf, srdfa, mva, mvs, sancopy

mva,mvs,sancopy

Host I/O n/a yes yes yes no

Protocols tcp fcip fcip ifcp iscsi

Authentication no yes yes no yes

Number of sessions per port

64 8a 1 64 32

Load Balancing yes yes yes no yes

Compression yes yes yes yes yes



Table 18 compares the distance extension solution features for non- TCP/IP products.

Legend:

Table 18 Distance extension comparison table for non TCP/IP products

Distance extension chassis

Client/WDM/Protocol conversion Link speed Features Switch vendor support

Client Side WAN/ side/Line side 1 Gb

2 Gb

4 Gb

10 Gb

BBC CLB WA FEC COM Brocade Cisco Brocade M Series

QLogic

FC-SW

FC-Direct

CWDM DWDM GbE SONET

ADVA FSP2000

X X X X X X X X X

ADVA FSP3000

X X X X X X

Ciena CN2000

X X X X X X X X X X X X X

Ciena CN4200

X X X X X X X X

Cisco ONS 15454

X X X X X X X X X X X X

Cisco ONS 15540

X X X X X X X

Nortel 5200

X X X X X X X X X X

Nortel 3500

X X X X X X X X X X

BBC: BBC spoofing

WA: Write Acceleration

CLB: Channel Load Balancing

WA: Write Acceleration

FEC: Forward Error Correction

COM Compression

Summary 97

98



Glossary

This glossary contains terms related to EMC products and EMC networked storage concepts.

Aaccess control A service that allows or prohibits access to a resource. Storage

management products implement access control to allow or prohibit specific users. Storage platform products implement access control, often called LUN Masking, to allow or prohibit access to volumes by Initiators (HBAs). See also “persistent binding” and “zoning.”

active domain ID The domain ID actively being used by a switch. It is assigned to a switch by the principal switch.

active zone set The active zone set is the zone set definition currently in effect and enforced by the fabric or other entity (for example, the name server). Only one zone set at a time can be active.

agent An autonomous agent is a system situated within (and is part of) an environment that senses that environment, and acts on it over time in pursuit of its own agenda. Storage management software centralizes the control and monitoring of highly distributed storage infrastructure. The centralizing part of the software management system can depend on agents that are installed on the distributed parts of the infrastructure. For example, an agent (software component) can be installed on each of the hosts (servers) in an environment to allow the centralizing software to control and monitor the hosts.


100

Glossary

alarm An SNMP message notifying an operator of a network problem.

any-to-any portconnectivity

A characteristic of a Fibre Channel switch that allows any port on the switch to communicate with any other port on the same switch.

application Application software is a defined subclass of computer software that employs the capabilities of a computer directly to a task that users want to perform. This is in contrast to system software that participates with integration of various capabilities of a computer, and typically does not directly apply these capabilities to performing tasks that benefit users. The term application refers to both the application software and its implementation which often refers to the use of an information processing system. (For example, a payroll application, an airline reservation application, or a network application.) Typically an application is installed “on top of” an operating system like Windows or Linux, and contains a user interface.

application-specificintegrated circuit

(ASIC)

A circuit designed for a specific purpose, such as implementing lower-layer Fibre Channel protocols (FC-1 and FC-0). ASICs contrast with general-purpose devices such as memory chips or microprocessors, which can be used in many different applications.

arbitration The process of selecting one respondent from a collection of several candidates that request service concurrently.

ASIC family Different switch hardware platforms that utilize the same port ASIC can be grouped into collections known as an ASIC family. For example, the Fuji ASIC family which consists of the ED-64M and ED-140M run different microprocessors, but both utilize the same port ASIC to provide Fibre Channel connectivity, and are therefore in the same ASIC family. For inter operability concerns, it is useful to understand to which ASIC family a switch belongs.

ASCII ASCII (American Standard Code for Information Interchange), generally pronounced [aeski], is a character encoding based on the English alphabet. ASCII codes represent text in computers, communications equipment, and other devices that work with text. Most modern character encodings, which support many more characters, have a historical basis in ASCII.

audit log A log containing summaries of actions taken by a Connectrix Management software user that creates an audit trail of changes. Adding, modifying, or deleting user or product administration


Glossary

values, creates a record in the audit log that includes the date and time.

authentication Verification of the identity of a process or person.

Bbackpressure The effect on the environment leading up to the point of restriction.

See “congestion.”

BB_Credit See “buffer-to-buffer credit.”

beaconing Repeated transmission of a beacon light and message until an error is corrected or bypassed. Typically used by a piece of equipment when an individual Field Replaceable Unit (FRU) needs replacement. Beaconing helps the field engineer locate the specific defective component. Some equipment management software systems such as Connectrix Manager offer beaconing capability.

BER See “bit error rate.”

bidirectional In Fibre Channel, the capability to simultaneously communicate at maximum speeds in both directions over a link.

bit error rate Ratio of received bits that contain errors to total of all bits transmitted.

blade server A consolidation of independent servers and switch technology in the same chassis.

blocked port Devices communicating with a blocked port are prevented from logging in to the Fibre Channel switch containing the port or communicating with other devices attached to the switch. A blocked port continuously transmits the off-line sequence (OLS).

bridge A device that provides a translation service between two network segments utilizing different communication protocols. EMC supports and sells bridges that convert iSCSI storage commands from a NIC- attached server to Fibre Channel commands for a storage platform.

broadcast Sends a transmission to all ports in a network. Typically used in IP networks. Not typically used in Fibre Channel networks.


102

Glossary

broadcast frames Data packet, also known as a broadcast packet, whose destination address specifies all computers on a network. See also “multicast.”

buffer Storage area for data in transit. Buffers compensate for differences in link speeds and link congestion between devices.

buffer-to-buffer credit The number of receive buffers allocated by a receiving FC_Port to a transmitting FC_Port. The value is negotiated between Fibre Channel ports during link initialization. Each time a port transmits a frame it decrements this credit value. Each time a port receives an R_Rdy frame it increments this credit value. If the credit value is decremented to zero, the transmitter stops sending any new frames until the receiver has transmitted an R_Rdy frame. Buffer-to-buffer credit is particularly important in SRDF and MirrorView distance extension solutions.

CCall Home A product feature that allows the Connectrix service processor to

automatically dial out to a support center and report system problems. The support center server accepts calls from the Connectrix service processor, logs reported events, and can notify one or more support center representatives. Telephone numbers and other information are configured through the Windows NT dial-up networking application. The Call Home function can be enabled and disabled through the Connectrix Product Manager.

channel With Open Systems, a channel is a point-to-point link that transports data from one point to another on the communication path, typically with high throughput and low latency that is generally required by storage systems. With Mainframe environments, a channel refers to the server-side of the server-storage communication path, analogous to the HBA in Open Systems.

Class 2 Fibre Channelclass of service

In Class 2 service, the fabric and destination N_Ports provide connectionless service with notification of delivery or nondelivery between the two N_Ports. Historically Class 2 service is not widely used in Fibre Channel system.

Class 3 Fibre Channelclass of service

Class 3 service provides a connectionless service without notification of delivery between N_Ports. (This is also known as datagram service.) The transmission and routing of Class 3 frames is the same


Glossary

as for Class 2 frames. Class 3 is the dominant class of communication used in Fibre Channel for moving data between servers and storage and may be referred to as “Ship and pray.”

Class F Fibre Channelclass of service

Class F service is used for all switch-to-switch communication in a multiswitch fabric environment. It is nearly identical to class 2 from a flow control point of view.

community A relationship between an SNMP agent and a set of SNMP managers that defines authentication, access control, and proxy characteristics.

community name A name that represents an SNMP community that the agent software recognizes as a valid source for SNMP requests. An SNMP management program that sends an SNMP request to an agent program must identify the request with a community name that the agent recognizes or the agent discards the message as an authentication failure. The agent counts these failures and reports the count to the manager program upon request, or sends an authentication failure trap message to the manager program.

community profile Information that specifies which management objects are available to what management domain or SNMP community name.

congestion Occurs at the point of restriction. See “backpressure.”

connectionless Non dedicated link. Typically used to describe a link between nodes that allows the switch to forward Class 2 or Class 3 frames as resources (ports) allow. Contrast with the dedicated bandwidth that is required in a Class 1 Fibre Channel Service point-to-point link.

Connectivity Unit A hardware component that contains hardware (and possibly software) that provides Fibre Channel connectivity across a fabric. Connectrix switches are example of Connectivity Units. This is a term popularized by the Fibre Alliance MIB, sometimes abbreviated to connunit.

Connectrixmanagement

software

The software application that implements the management user interface for all managed Fibre Channel products, typically the Connectrix -M product line. Connectrix Management software is a client/server application with the server running on the Connectrix service processor, and clients running remotely or on the service processor.


104

Glossary

Connectrix serviceprocessor

An optional 1U server shipped with the Connectrix -M product line to run the Connectrix Management server software and EMC remote support application software.

Control Unit In mainframe environments, a Control Unit controls access to storage. It is analogous to a Target in Open Systems environments.

core switch Occupies central locations within the interconnections of a fabric. Generally provides the primary data paths across the fabric and the direct connections to storage devices. Connectrix directors are typically installed as core switches, but may be located anywhere in the fabric.

credit A numeric value that relates to the number of available BB_Credits on a Fibre Channel port. See“buffer-to-buffer credit”.

DDASD Direct Access Storage Device.

default Pertaining to an attribute, value, or option that is assumed when none is explicitly specified.

default zone A zone containing all attached devices that are not members of any active zone. Typically the default zone is disabled in a Connectrix M environment which prevents newly installed servers and storage from communicating until they have been provisioned.

Dense WavelengthDivision Multiplexing

(DWDM)

A process that carries different data channels at different wavelengths over one pair of fiber optic links. A conventional fiber-optic system carries only one channel over a single wavelength traveling through a single fiber.

destination ID A field in a Fibre Channel header that specifies the destination address for a frame. The Fibre Channel header also contains a Source ID (SID). The FCID for a port contains both the SID and the DID.

device A piece of equipment, such as a server, switch or storage system.

dialog box A user interface element of a software product typically implemented as a pop-up window containing informational messages and fields for modification. Facilitates a dialog between the user and the application. Dialog box is often used interchangeably with window.


Glossary

DID An acronym used to refer to either Domain ID or Destination ID. This ambiguity can create confusion. As a result E-Lab recommends this acronym be used to apply to Domain ID. Destination ID can be abbreviated to FCID.

director An enterprise-class Fibre Channel switch, such as the Connectrix ED-140M, MDS 9509, or ED-48000B. Directors deliver high availability, failure ride-through, and repair under power to insure maximum uptime for business critical applications. Major assemblies, such as power supplies, fan modules, switch controller cards, switching elements, and port modules, are all hot-swappable.

The term director may also refer to a board-level module in the Symmetrix that provides the interface between host channels (through an associated adapter module in the Symmetrix) and Symmetrix disk devices. (This description is presented here only to clarify a term used in other EMC documents.)

DNS See “domain name service name.”

domain ID A byte-wide field in the three byte Fibre Channel address that uniquely identifies a switch in a fabric. The three fields in a FCID are domain, area, and port. A distinct Domain ID is requested from the principal switch. The principal switch allocates one Domain ID to each switch in the fabric. A user may be able to set a Preferred ID which can be requested of the Principal switch, or set an Insistent Domain ID. If two switches insist on the same DID one or both switches will segment from the fabric.

domain name servicename

Host or node name for a system that is translated to an IP address through a name server. All DNS names have a host name component and, if fully qualified, a domain component, such as host1.abcd.com. In this example, host1 is the host name.

dual-attached host A host that has two (or more) connections to a set of devices.

EE_D_TOV A time-out period within which each data frame in a Fibre Channel

sequence transmits. This avoids time-out errors at the destination Nx_Port. This function facilitates high speed recovery from dropped frames. Typically this value is 2 seconds.


106

Glossary

E_Port Expansion Port, a port type in a Fibre Channel switch that attaches to another E_Port on a second Fibre Channel switch forming an Interswitch Link (ISL). This link typically conforms to the FC-SW standards developed by the T11 committee, but might not support heterogeneous inter operability.

edge switch Occupies the periphery of the fabric, generally providing the direct connections to host servers and management workstations. No two edge switches can be connected by interswitch links (ISLs). Connectrix departmental switches are typically installed as edge switches in a multiswitch fabric, but may be located anywhere in the fabric

Embedded WebServer

A management interface embedded on the switch’s code that offers features similar to (but not as robust as) the Connectrix Manager and Product Manager.

error detect time outvalue

Defines the time the switch waits for an expected response before declaring an error condition. The error detect time out value (E_D_TOV) can be set within a range of two-tenths of a second to one second using the Connectrix switch Product Manager.

error message An indication that an error has been detected. See also “information message” and “warning message.”

Ethernet A baseband LAN that allows multiple station access to the transmission medium at will without prior coordination and which avoids or resolves contention.

event log A record of significant events that have occurred on a Connectrix switch, such as FRU failures, degraded operation, and port problems.

expansionport See “E_Port.”

explicit fabric login In order to join a fabric, an Nport must login to the fabric (an operation referred to as an FLOGI). Typically this is an explicit operation performed by the Nport communicating with the F_port of the switch, and is called an explicit fabric login. Some legacy Fibre Channel ports do not perform explicit login, and switch vendors perform login for ports creating an implicit login. Typically logins are explicit.


Glossary

FFA Fibre Adapter, another name for a Symmetrix Fibre Channel director.

F_Port Fabric Port, a port type on a Fibre Channel switch. An F_Port attaches to an N_Port through a point-to-point full-duplex link connection. A G_Port automatically becomes an F_port or an E-Port depending on the port initialization process.

fabric One or more switching devices that interconnect Fibre Channel N_Ports, and route Fibre Channel frames based on destination IDs in the frame headers. A fabric provides discovery, path provisioning, and state change management services for a Fibre Channel environment.

fabric element Any active switch or director in the fabric.

fabric login Process used by N_Ports to establish their operating parameters including class of service, speed, and buffer-to-buffer credit value.

fabric port A port type (F_Port) on a Fibre Channel switch that attaches to an N_Port through a point-to-point full-duplex link connection. An N_Port is typically a host (HBA) or a storage device like Symmetrix, VNX series, or CLARiiON.

fabric shortest pathfirst (FSPF)

A routing algorithm implemented by Fibre Channel switches in a fabric. The algorithm seeks to minimize the number of hops traversed as a Fibre Channel frame travels from its source to its destination.

fabric tree A hierarchical list in Connectrix Manager of all fabrics currently known to the Connectrix service processor. The tree includes all members of the fabrics, listed by WWN or nickname.

failover The process of detecting a failure on an active Connectrix switch FRU and the automatic transition of functions to a backup FRU.

fan-in/fan-out Term used to describe the server:storage ratio, where a graphic representation of a 1:n (fan-in) or n:1 (fan-out) logical topology looks like a hand-held fan, with the wide end toward n. By convention fan-out refers to the number of server ports that share a single storage port. Fan-out consolidates a large number of server ports on a fewer number of storage ports. Fan-in refers to the number of storage ports that a single server port uses. Fan-in enlarges the storage capacity used by a server. A fan-in or fan-out rate is often referred to as just the


108

Glossary

n part of the ratio; For example, a 16:1 fan-out is also called a fan-out rate of 16, in this case 16 server ports are sharing a single storage port.

FCP See “Fibre Channel Protocol.”

FC-SW The Fibre Channel fabric standard. The standard is developed by the T11 organization whose documentation can be found at T11.org. EMC actively participates in T11. T11 is a committee within the InterNational Committee for Information Technology (INCITS).

fiber optics The branch of optical technology concerned with the transmission of radiant power through fibers made of transparent materials such as glass, fused silica, and plastic.

Either a single discrete fiber or a non spatially aligned fiber bundle can be used for each information channel. Such fibers are often called optical fibers to differentiate them from fibers used in non-communication applications.

fibre A general term used to cover all physical media types supported by the Fibre Channel specification, such as optical fiber, twisted pair, and coaxial cable.

Fibre Channel The general name of an integrated set of ANSI standards that define new protocols for flexible information transfer. Logically, Fibre Channel is a high-performance serial data channel.

Fibre ChannelProtocol

A standard Fibre Channel FC-4 level protocol used to run SCSI over Fibre Channel.

Fibre Channel switchmodules

The embedded switch modules in the back plane of the blade server. See “blade server” on page 101.

firmware The program code (embedded software) that resides and executes on a connectivity device, such as a Connectrix switch, a Symmetrix Fibre Channel director, or a host bus adapter (HBA).

F_Port Fabric Port, a physical interface within the fabric. An F_Port attaches to an N_Port through a point-to-point full-duplex link connection.

frame A set of fields making up a unit of transmission. Each field is made of bytes. The typical Fibre Channel frame consists of fields: Start-of-frame, header, data-field, CRC, end-of-frame. The maximum frame size is 2148 bytes.


Glossary

frame header Control information placed before the data-field when encapsulating data for network transmission. The header provides the source and destination IDs of the frame.

FRU Field-replaceable unit, a hardware component that can be replaced as an entire unit. The Connectrix switch Product Manager can display status for the FRUs installed in the unit.

FSPF Fabric Shortest Path First, an algorithm used for routing traffic. This means that, between the source and destination, only the paths that have the least amount of physical hops will be used for frame delivery.

Ggateway address In TCP/IP, a device that connects two systems that use the same

or different protocols.

gigabyte (GB) A unit of measure for storage size, loosely one billion (109) bytes. One gigabyte actually equals 1,073,741,824 bytes.

G_Port A port type on a Fibre Channel switch capable of acting either as an F_Port or an E_Port, depending on the port type at the other end of the link.

GUI Graphical user interface.

HHBA See “host bus adapter.”

hexadecimal Pertaining to a numbering system with base of 16; valid numbers use the digits 0 through 9 and characters A through F (which represent the numbers 10 through 15).

high availability A performance feature characterized by hardware component redundancy and hot-swappability (enabling non-disruptive maintenance). High-availability systems maximize system uptime while providing superior reliability, availability, and serviceability.

hop A hop refers to the number of InterSwitch Links (ISLs) a Fibre Channel frame must traverse to go from its source to its destination.


110

Glossary

Good design practice encourages three hops or less to minimize congestion and performance management complexities.

host bus adapter A bus card in a host system that allows the host system to connect to the storage system. Typically the HBA communicates with the host over a PCI or PCI Express bus and has a single Fibre Channel link to the fabric. The HBA contains an embedded microprocessor with on board firmware, one or more ASICs, and a Small Form Factor Pluggable module (SFP) to connect to the Fibre Channel link.

II/O See “input/output.”

in-band management Transmission of monitoring and control functions over the Fibre Channel interface. You can also perform these functions out-of-band typically by use of the ethernet to manage Fibre Channel devices.

information message A message telling a user that a function is performing normally or has completed normally. User acknowledgement might or might not be required, depending on the message. See also “error message” and “warning message.”

input/output (1) Pertaining to a device whose parts can perform an input process and an output process at the same time. (2) Pertaining to a functional unit or channel involved in an input process, output process, or both (concurrently or not), and to the data involved in such a process. (3) Pertaining to input, output, or both.

interface (1) A shared boundary between two functional units, defined by functional characteristics, signal characteristics, or other characteristics as appropriate. The concept includes the specification of the connection of two devices having different functions. (2) Hardware, software, or both, that links systems, programs, or devices.

Internet Protocol See “IP.”

interoperability The ability to communicate, execute programs, or transfer data between various functional units over a network. Also refers to a Fibre Channel fabric that contains switches from more than one vendor.


Glossary

interswitch link (ISL) Interswitch link, a physical E_Port connection between any two switches in a Fibre Channel fabric. An ISL forms a hop in a fabric.

IP Internet Protocol, the TCP/IP standard protocol that defines the datagram as the unit of information passed across an internet and provides the basis for connectionless, best-effort packet delivery service. IP includes the ICMP control and error message protocol as an integral part.

IP address A unique string of numbers that identifies a device on a network. The address consists of four groups (quadrants) of numbers delimited by periods. (This is called dotted-decimal notation.) All resources on the network must have an IP address. A valid IP address is in the form nnn.nnn.nnn.nnn, where each nnn is a decimal in the range 0 to 255.

ISL Interswitch link, a physical E_Port connection between any two switches in a Fibre Channel fabric.

Kkilobyte (K) A unit of measure for storage size, loosely one thousand bytes. One

kilobyte actually equals 1,024 bytes.

Llaser A device that produces optical radiation using a population inversion

to provide light amplification by stimulated emission of radiation and (generally) an optical resonant cavity to provide positive feedback. Laser radiation can be highly coherent temporally, spatially, or both.

LED Light-emitting diode.

link The physical connection between two devices on a switched fabric.

link incident A problem detected on a fiber-optic link; for example, loss of light, or invalid sequences.

load balancing The ability to distribute traffic over all network ports that are the same distance from the destination address by assigning different paths to different messages. Increases effective network bandwidth. EMC PowerPath software provides load-balancing services for server IO.


112

Glossary

logical volume A named unit of storage consisting of a logically contiguous set of disk sectors.

Logical Unit Number(LUN)

A number, assigned to a storage volume, that (in combination with the storage device node's World Wide Port Name (WWPN)) represents a unique identifier for a logical volume on a storage area network.

MMAC address Media Access Control address, the hardware address of a device

connected to a shared network.

managed product A hardware product that can be managed using the Connectrix Product Manager. For example, a Connectrix switch is a managed product.

management session Exists when a user logs in to the Connectrix Management software and successfully connects to the product server. The user must specify the network address of the product server at login time.

media The disk surface on which data is stored.

media access control See “MAC address.”

megabyte (MB) A unit of measure for storage size, loosely one million (106) bytes. One megabyte actually equals 1,048,576 bytes.

MIB Management Information Base, a related set of objects (variables) containing information about a managed device and accessed through SNMP from a network management station.

multicast Multicast is used when multiple copies of data are to be sent to designated, multiple, destinations.

multiswitch fabric Fibre Channel fabric created by linking more than one switch or director together to allow communication. See also “ISL.”

multiswitch linking Port-to-port connections between two switches.

Nname server (dNS) A service known as the distributed Name Server provided by a Fibre

Channel fabric that provides device discovery, path provisioning, and


Glossary

state change notification services to the N_Ports in the fabric. The service is implemented in a distributed fashion, for example, each switch in a fabric participates in providing the service. The service is addressed by the N_Ports through a Well Known Address.

network address A name or address that identifies a managed product, such as a Connectrix switch, or a Connectrix service processor on a TCP/IP network. The network address can be either an IP address in dotted decimal notation, or a Domain Name Service (DNS) name as administered on a customer network. All DNS names have a host name component and (if fully qualified) a domain component, such as host1.emc.com. In this example, host1 is the host name and EMC.com is the domain component.

nickname A user-defined name representing a specific WWxN, typically used in a Connectrix -M management environment. The analog in the Connectrix -B and MDS environments is alias.

node The point at which one or more functional units connect to the network.

N_Port Node Port, a Fibre Channel port implemented by an end device (node) that can attach to an F_Port or directly to another N_Port through a point-to-point link connection. HBAs and storage systems implement N_Ports that connect to the fabric.

NVRAM Nonvolatile random access memory.

Ooffline sequence

(OLS)The OLS Primitive Sequence is transmitted to indicate that the FC_Port transmitting the Sequence is:

a. initiating the Link Initialization Protocol

b. receiving and recognizing NOS

c. or entering the offline state

OLS See “offline sequence (OLS)”.

operating mode Regulates what other types of switches can share a multiswitch fabric with the switch under consideration.


114

Glossary

operating system Software that controls the execution of programs and that may provide such services as resource allocation, scheduling, input/output control, and data management. Although operating systems are predominantly software, partial hardware implementations are possible.

optical cable A fiber, multiple fibers, or a fiber bundle in a structure built to meet optical, mechanical, and environmental specifications.

OS See “operating system.”

out-of-bandmanagement

Transmission of monitoring/control functions outside of the Fibre Channel interface, typically over ethernet.

oversubscription The ratio of bandwidth required to bandwidth available. When all ports, associated pair-wise, in any random fashion, cannot sustain full duplex at full line-rate, the switch is oversubscribed.

Pparameter A characteristic element with a variable value that is given a constant

value for a specified application. Also, a user-specified value for an item in a menu; a value that the system provides when a menu is interpreted; data passed between programs or procedures.

password (1) A value used in authentication or a value used to establish membership in a group having specific privileges. (2) A unique string of characters known to the computer system and to a user who must specify it to gain full or limited access to a system and to the information stored within it.

path In a network, any route between any two nodes.

persistent binding Use of server-level access control configuration information to persistently bind a server device name to a specific Fibre Channel storage volume or logical unit number, through a specific HBA and storage port WWN. The address of a persistently bound device does not shift if a storage target fails to recover during a power cycle. This function is the responsibility of the HBA device driver.

port (1) An access point for data entry or exit. (2) A receptacle on a device to which a cable for another device is attached.


Glossary

port card Field replaceable hardware component that provides the connection for fiber cables and performs specific device-dependent logic functions.

port name A symbolic name that the user defines for a particular port through the Product Manager.

preferred domain ID An ID configured by the fabric administrator. During the fabric build process a switch requests permission from the principal switch to use its preferred domain ID. The principal switch can deny this request by providing an alternate domain ID only if there is a conflict for the requested Domain ID. Typically a principal switch grants the non-principal switch its requested Preferred Domain ID.

principal switch In a multiswitch fabric, the switch that allocates domain IDs to itself and to all other switches in the fabric. There is always one principal switch in a fabric. If a switch is not connected to any other switches, it acts as its own principal switch.

principle downstreamISL

The ISL to which each switch will forward frames originating from the principal switch.

principle ISL The principal ISL is the ISL that frames destined to, or coming from, the principal switch in the fabric will use. An example is an RDI frame.

principle upstream ISL The ISL to which each switch will forward frames destined for the principal switch. The principal switch does not have any upstream ISLs.

product (1) Connectivity Product, a generic name for a switch, director, or any other Fibre Channel product. (2) Managed Product, a generic hardware product that can be managed by the Product Manager (a Connectrix switch is a managed product). Note distinction from the definition for “device.”

Product Manager A software component of Connectrix Manager software such as a Connectrix switch product manager, that implements the management user interface for a specific product. When a product instance is opened from the Connectrix Manager software products view, the corresponding product manager is invoked. The product manager is also known as an Element Manager.


116

Glossary

product name A user configurable identifier assigned to a Managed Product. Typically, this name is stored on the product itself. For a Connectrix switch, the Product Name can also be accessed by an SNMP Manager as the System Name. The Product Name should align with the host name component of a Network Address.

products view The top-level display in the Connectrix Management software user interface that displays icons of Managed Products.

protocol (1) A set of semantic and syntactic rules that determines the behavior of functional units in achieving communication. (2) A specification for the format and relative timing of information exchanged between communicating parties.

RR_A_TOV See “resource allocation time out value.”

remote access link The ability to communicate with a data processing facility through a remote data link.

remote notification The system can be programmed to notify remote sites of certain classes of events.

remote userworkstation

A workstation, such as a PC, using Connectrix Management software and Product Manager software that can access the Connectrix service processor over a LAN connection. A user at a remote workstation can perform all of the management and monitoring tasks available to a local user on the Connectrix service processor.

resource allocationtime out value

A value used to time-out operations that depend on a maximum time that an exchange can be delayed in a fabric and still be delivered. The resource allocation time-out value of (R_A_TOV) can be set within a range of two-tenths of a second to 120 seconds using the Connectrix switch product manager. The typical value is 10 seconds.

SSAN See “storage area network (SAN).”

segmentation A non-connection between two switches. Numerous reasons exist for an operational ISL to segment, including interop mode incompatibility, zoning conflicts, and domain overlaps.


Glossary

segmented E_Port E_Port that has ceased to function as an E_Port within a multiswitch fabric due to an incompatibility between the fabrics that it joins.

service processor See “Connectrix service processor.”

session See “management session.”

single attached host A host that only has a single connection to a set of devices.

small form factorpluggable (SFP)

An optical module implementing a shortwave or long wave optical transceiver.

SMTP Simple Mail Transfer Protocol, a TCP/IP protocol that allows users to create, send, and receive text messages. SMTP protocols specify how messages are passed across a link from one system to another. They do not specify how the mail application accepts, presents or stores the mail.

SNMP Simple Network Management Protocol, a TCP/IP protocol that generally uses the User Datagram Protocol (UDP) to exchange messages between a management information base (MIB) and a management client residing on a network.

storage area network(SAN)

A network linking servers or workstations to disk arrays, tape backup systems, and other devices, typically over Fibre Channel and consisting of multiple fabrics.

subnet mask Used by a computer to determine whether another computer with which it needs to communicate is located on a local or remote network. The network mask depends upon the class of networks to which the computer is connecting. The mask indicates which digits to look at in a longer network address and allows the router to avoid handling the entire address. Subnet masking allows routers to move the packets more quickly. Typically, a subnet may represent all the machines at one geographic location, in one building, or on the same local area network.

switch priority Value configured into each switch in a fabric that determines its relative likelihood of becoming the fabric’s principal switch.


118

Glossary

TTCP/IP Transmission Control Protocol/Internet Protocol. TCP/IP refers to

the protocols that are used on the Internet and most computer networks. TCP refers to the Transport layer that provides flow control and connection services. IP refers to the Internet Protocol level where addressing and routing are implemented.

toggle To change the state of a feature/function that has only two states. For example, if a feature/function is enabled, toggling changes the state to disabled.

topology Logical and/or physical arrangement of switches on a network.

trap An asynchronous (unsolicited) notification of an event originating on an SNMP-managed device and directed to a centralized SNMP Network Management Station.

Uunblocked port Devices communicating with an unblocked port can log in to a

Connectrix switch or a similar product and communicate with devices attached to any other unblocked port if the devices are in the same zone.

Unicast Unicast routing provides one or more optimal path(s) between any of two switches that make up the fabric. (This is used to send a single copy of the data to designated destinations.)

upper layer protocol(ULP)

The protocol user of FC-4 including IPI, SCSI, IP, and SBCCS. In a device driver ULP typically refers to the operations that are managed by the class level of the driver, not the port level.

URL Uniform Resource Locater, the addressing system used by the World Wide Web. It describes the location of a file or server anywhere on the Internet.

Vvirtual switch A Fibre Channel switch function that allows users to subdivide a

physical switch into multiple virtual switches. Each virtual switch consists of a subset of ports on the physical switch, and has all the properties of a Fibre Channel switch. Multiple virtual switches can be connected through ISL to form a virtual fabric or VSAN.


Glossary

virtual storage areanetwork (VSAN)

An allocation of switch ports that can span multiple physical switches, and forms a virtual fabric. A single physical switch can sometimes host more than one VSAN.

volume A general term referring to an addressable logically contiguous storage space providing block I/Oservices.

VSAN Virtual Storage Area Network.

Wwarning message An indication that a possible error has been detected. See also “error

message” and “information message.”

World Wide Name(WWN)

A unique identifier, even on global networks. The WWN is a 64-bit number (XX:XX:XX:XX:XX:XX:XX:XX). The WWN contains an OUI which uniquely determines the equipment manufacturer. OUIs are administered by the Institute of Electronic and Electrical Engineers (IEEE). The Fibre Channel environment uses two types of WWNs; a World Wide Node Name (WWNN) and a World Wide Port Name (WWPN). Typically the WWPN is used for zoning (path provisioning function).

Zzone An information object implemented by the distributed Nameserver

(dNS) of a Fibre Channel switch. A zone contains a set of members which are permitted to discover and communicate with one another. The members can be identified by a WWPN or port ID. EMC recommends the use of WWPNs in zone management.

zone set An information object implemented by the distributed Nameserver (dNS) of a Fibre Channel switch. A Zone Set contains a set of Zones. A Zone Set is activated against a fabric, and only one Zone Set can be active in a fabric.

zonie A storage administrator who spends a large percentage of his workday zoning a Fibre Channel network and provisioning storage.

zoning Zoning allows an administrator to group several devices by function or by location. All devices connected to a connectivity product, such as a Connectrix switch, may be configured into one or more zones.


120

Glossary


Index

Aactive and passive devices 54

BBB_Credit

guidelines 38buffer-to-buffer

local termination 54

CCisco MDS 9000 84Congestion

network 31credit starvation 42CWDM 19

DData buffering and flow control 37devices

active and passive 54distance extension 35

technologies 35DWDM 15

FFast Write 80FCIP

with Cisco MDS 9000 family 84Fibre Channel

and BB_Credit 37BB_Credit guidelines 38

SiRT 56flow control and data buffering 37

GGbE (Gigabit Ethernet) 23

IInternet Protocol Security (IPsec) 32IPsec

and tunneling 32configuring 78

IPsec (Internet Protocol security) 32IPsec

terminology 33iSCSI

technology 34

Llink initialization 60link speed 36

MMDS 9000 84

NNetwork congestion 31

Ppassive and active devices 54


122

Index

SSiRT

with SRDF 56SiRT (Single roundtrip) 56SmartWrite 94solution 86SONET 21

Ttape pipelining 80TCP

error recovery 28terminology 24

TCP/IP 24, 53


Extended Distance Technologies

Technology

Transcript of Extended Distance Technologies