SAN Extension Design and Solutions

Design and SolutionsSAN ExtensionParesh Gupta, Technical Marketing Engineer, Cisco

Mark Allen, Manager Technical Marketing, Cisco

January 2017

2© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Resilience

Safety

Restoration

Business Continuity

Industry wide Ecosystems

Infrastructure

Planning

Competitive EdgeRedundancy16 member PortChannel

Disaster Recovery

Incident Management

Cloud Storage Protection

Customer SatisfactionStability

Restoration

Crisis Management

Investment Protection

RiskWhy

SAN Extension


Hope for the best, plan for the worst

- Lee Child


Recovery Point and Recovery Time Objective

Time

DisasterStrikes

Recovery PointLast Point Where

Data in Usable State

Recovery TimeSystem Recovered

and Operational

How far back? How long to recover?

Shorter RPO/RTO�̶ Higher $$$�̶ Replication�̶ Hot standby systems

Longer RPO/RTO�̶Lower $$$�̶Tape backup/restore�̶Cold standby systems


SANExtension

Minimize RPOData should be in sync before & after disaster

Minimize RTORecovery should be quick

Investment ProtectionInvestment for more than a decade

Choice of ProtocolFC or FCoE or FCIP

Acceptable LatencyLatency within acceptable limit ofReplication or Backup application

DistanceHow far is the recovery site?

$$

SecurityLink Encryption for security of data in motion

CostDesign within the budget

High AvailabilityIncreased availability

Considerations for SAN Extension Design

SAN Extension Design Options


Cisco Multi-Protocol Product Portfolio

12+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM)

Cisco MDS9700

48x16G Line-Rate FC

LAN/SAN SAN COMPUTE

Cisco UCS C-Series

Rack Servers

Cisco UCS B-SeriesBlade Servers

Cisco UCS 6300 Series FI

Cisco UCS 6200 UP

Cisco Nexus 9000

Cisco Nexus 7000

Cisco Nexus 5600

Cisco Nexus 5500

CiscoNexus 3000

CiscoNexus 2000

24 x40GFCoE

Cisco MDS9250i

Cisco MDS 9148S

48x10G Line-Rate FCoE

Cisco MDS9396S

Nexus 5672UP-16G

16G FC: Nexus 2348UPQ

16G FC, 40G FCIP

Consistent and SimplifiedFeatures, Management, and Programmability


Cisco MDS 9000 Switch Family

9RU

MDS 9710 MDS 97064 module slotsUp to 192 ports

14RU

26RU

MDS 97188 module slots

Up to 384 ports

FCIP SAN Extension

16 module slotsUp to 768 ports

MDS 9148S MDS 9396S MDS 9250i

48 x 16G FC

48 x 10Gbps FCoE

24 x 40 Gbps FCoE

24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE

SAN Directors Director Modules

Fabric Switches


Typical SAN Design

• Dual fabric design : 2 Fibre Channel connections from Server to Storage• Multipath software provides high availability• Separate Access and Replication fabrics• Dual fabrics maintained over SAN extension

Replication FabricReplication FabricAccess Fabric

“B” Fabric

“A” FabricDC

InterconnectNetwork

SiteA SiteB

MDS

MDS

MDS

MDS

MDS

MDS


Introducing Virtual SAN (VSAN)

• Dual fabrics (E.g., yellow VSAN and red VSAN) over distance• Inter Switch Link (ISL) carry multiple VSANs (known as trunking)• Each VSAN maintains it’s own fabric services

• FSPF: ‘Fabric Shortest Path first’ for route calculation• Name server, zoning database, etc.

Replication VSANReplication VSANAccess VSAN

“B” Fabric

“A” FabricDC

InterconnectNetwork

SiteA SiteB

MDS

MDS

MDS

MDS

VSANs – Increased redundancy, scalability and reduced cost


High Availability (HA) replication design

• Client based protection by• PortChannel• Storage arrays• Rerouting by FSPF

• Network based Protection by • Optical protection schemes


“B” Fabric

“A” Fabric

SiteA SiteB

MDS

MDS

MDS

MDS


Link HA via PortChannel

• Multi-protocol support : FC, FCoE or FCIP• Up to 16 members in a port channel• Increased Resilience and availability

• Single logical link• No FSPF re-calculations when members go down

• Route member links over diverse geographic paths


“B” Fabric

“A” Fabric

SiteA SiteB

MDS

MDS

MDS

MDS


Extending optical FC SAN : B2B credit requirement

Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps

512 Bytes 2 BB/km 4 BB/km 8 BB/km 16 BB/km 24 BB/km 32 BB/km

1024 Bytes 1 BB/km 2 BB/km 4 BB/km 8 BB/km 12 BB/km 16 BB/km

2112 Bytes 0.5 BB/km 1 BB/km 2 BB/km 4 BB/km 6 BB/km 8 BB/km

B2B credit requirement increases with

Distance Speed Frame size

B2B

cre

dit

requ

irem

ent


SAN Extension over Dark Fiber


“B” Fabric

“A” Fabric

SiteA SiteB

• 1/2/4/8/10/16 Gbps FC• Distance : limited due to optics and fiber cable

• SW or LW (10 KM) or ER (40KM) optics. OM1, OM2, OM3 or OM4 cables

• Client protection only : PortChannel / Storage arrays / Rerouting by FSPF• Loss of path reduces bandwidth of only one fabric by 50%

• Cost: Low


SAN Extension over CWDM Network

• Colored CWDM SFPs (8G FC) used in FC switches (no transponder required)• Distance : limited due to optics and fiber cable and dB loss in MUX (max 40 KM)• Client protection only : PortChannel / Storage arrays / Rerouting by FSPF

• Loss of path reduces bandwidth of both “A” and “B” fabrics by 50%• No topology change (no FSPF recalculation)

• Cost: Fair


“B” Fabric

“A” Fabric

SiteA SiteB

MUX

MUX

MUX

MUX

MDS

MDS

MDS

MDS


Dense Wavelength Division Multiplexing (DWDM)• Up to 32 channels per fiber• Longer Distance than CWDM : Use of Erbium-Doped Fiber Amplifiers (EDFA)• Multi Protocol Capability for data center to data center connectivity

• 1, 2, 4, 8, 10 or 16 Gbps FC, FICON, GigE, 10GigE, ESCON, IBM GDPS

• Client Protection : PortChannel / Storage arrays / Rerouting by FSPF • As well as Network Protection : Splitter / Line card

Optical Splitter Protection

Protected Lambda

Optical Splitter

Working Lambda

MDS MDS

Linecard or Y-Cable Protection

Y-cable

MDS MDS

Single transponder required Dual transponders required, more expensive

Protects against fiber breaks Protects against fiber breaks and Line card failure


SAN Extension over DWDM Network – Option 1

• Optical network sharing by both fabrics • Client protection via PortChannel – Recommended

• Single fiber cut will not affect fabric• Loss of path reduces bandwidth of both “A” and “B” fabrics by 50%

• Cost : High


“B” Fabric

“A” Fabric

SiteA SiteB

DWDM Ring

MDS

MDS

MDS

MDS


SAN Extension over DWDM Network – Option 2

• Dedicated optical network per fabric • Client protection via PortChannel – Recommended

• Single fiber cut will not affect fabric• Loss of path reduces bandwidth of only one fabric by 50%

• Cost : High+


“B” Fabric

“A” Fabric

SiteA SiteB

DWDM Ring

DWDM Ring

MDS

MDS

MDS

MDS


SAN Extension Technology Options

Limited by Optics (Power Budget)Dark Fiber1/2/4/8/10/16G FC, 10GE FCoE

CWDM1/2/4/8G FC, 10GE FCoE

DWDM1/2/4/8G FC, 10GE FCoE

SONET/SDH1/2/4G FC

Data Center Campus Metro Regional National

Increasing Distance

Sync

Sync

Sync

Limited by Optics (Power Budget)

Limited by B2B_CreditsOpt

ical

Async

Global

Sync

ProtectionClient Network

Cost

Cost

Cost

Cost


Fibre Channel over Internet Protocol (FCIP)

SiteB FC SANSiteA FC SAN

IP Network

FCIP TunnelMDS MDS

Single FSPF routing domain

• IETF standard for Linking Fibre Channel SANs over IP (RFCs 3821 & 3643)• Point-to-point tunnel between FCIP link end-points


FCIP Frame Details

• Segmentation and reassembly at default MTU of 1500 B (performance hit)• Recommendation: End to end IP MTU of 2300 bytes• All Cisco FCIP products support jumbo frames

+ +

FCIPHeader

EthernetHeader

IPHeader

TCPHeader

TCPOpts FC Frame

Ethernet

CRC3214 20 20 12 28 4

94

EISLHdr

SO

F

4 8

2172

VSAN Routing for TE port

optHdr0-16

RTTM is constantly measured for Round Trip Time

2270=

4


SAN Extension over FCIP

• Client protection via PortChannel – Recommended• Portchannel individual FCIP links to separate Ethernet switches/routers• Each WAN link carries two FCIP tunnels• Global reach : Reliable delivery by TCP, No B2B credit requirement on FCIP link

IP network


“B” Fabric

“A” Fabric

SiteA SiteB

MDS

MDS

MDS

MDS


SAN Extension Technology Options

Limited by Optics (Power Budget)Dark Fiber1/2/4/8/10/16G FC, 10GE FCoE

CWDM1/2/4/8G FC, 10GE FCoE

DWDM1/2/4/8G FC, 10GE FCoE

SONET/SDH1/2/4G FC

Data Center Campus Metro Regional National

Increasing Distance

Sync

Sync

Sync

Limited by Optics (Power Budget)

Limited by B2B_CreditsOpt

ical

Async

Global

Sync

ProtectionClient Network

Cost

Cost

Cost

Cost

23

Async (WAN)MDS9000 FCIPGE, 10GE IP

Sync (Metro Eth)Cost


Native FCoE SAN ExtensionFCDCB/FCoEEthernetFCoE


“B” Fabric

“A” Fabric

SiteA SiteB

Nexus

Nexus

Nexus

• FCoE SAN Extension uses same design principles as FC or FCIP• Separate VLANs/VSANs for Host and replication traffic• Multiple geographical diverse paths• Client (PortChannel) or Network Protection (DWDM or SONET/SDH)

• Distance depends on underlying media and Buffer (instead of B2B credits)• Typical FCoE SAN Extension rely on FC or FCIP to transport data over distance


Native FCoE SAN ExtensionFCDCB/FCoEEthernetFCoE

• FCoE Attached disk replication uses FC or FCIP transport network• SAN Extension design criteria based on FC or FCIP interconnect network


“B” Fabric

“A” Fabric

SiteA SiteBNexus

MDS 9250i

MDS 9700

Nexus

MDS 9250i

MDS 9700

SAN Extension – Advanced Features


Link Layer Security

IPNetworkDWDM

Name: XYZSSN: 1234567890Amount: $123,456Status: Gold

@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@

FC TrustSec IPSec



@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@

Primary DC

Secondary DC

Backup DC

• Hardware supported, no additional latency• DH-CHAP used for peer authentication• Encryption: AES 128 bit key

Fibre Channel TrustSec• Hardware support, no additional latency• Encryption: AES (128 or 256 bit key), DES

(56 bit), 3DES (168 bit)

IPSec


Application I/O Acceleration• Distance impacts performance of disk replication and tape backups• Latency due to distance is compounded by multiple round trips per command

28

I/O Accelerator (IOA)

disk and tape over FC or FCIP

Write Accelerationdisk over FCIP (FCIP-WA)

Tape Acceleration tape over FCIP (FCIP-TA)

Solution

MDS 9250i24/10 SAN Extension Module (SEM)


FCIP Data Compression• Compression increases link data capacity or reduce consumed bandwidth• Two compression modes available

• Auto – Optimizes Compression based on bandwidth and data rate (Recommended)• Mode2 – Deflate based compression algorithm

• Data Compressibility is data stream dependent• All nulls or ones → high compression (>30:1) • Random data (e.g., encrypted) → low compression (~1:1)• “Typical” rate is 4:1 (MDS 9250i and 24/10 SEM), but may vary considerably

• Application throughput is the most important factor

Supported in HW on MDS 9250i and 24/10 SEM without any additional latency

SAN Extension – Expert Adviceby Mark Allen,Manager, Storage Technical Marketing


SAN Extension Best PracticesDo not leave FCIP configuration to default values

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 320

5

10

15

20

25

30

35

40

45

Exponential “Slow Start”(increase 2x pkts per RTT)

LossLoss

# Round Trip Times (RTT)

Packets S

ent per Round Trip Low Throughput During

This Period

Linear “Congestion Avoidance”

(MDS +2/cwnd per ack)(TCP +1/cwnd per ack)

Congestion Window Halved on Packet Loss; Retransmission

Signals Congestion…Slow Start Threshold Adjusted

Traditional TCPMDS TCP implementation


SAN Extension Best Practices

• In built Shaper sends at a rate consumable by the downstream path• Immediately sends at “minimum-bandwidth” rate (avoids early stages of traditional

slow start)• Ramps up to “maximum-bandwidth” rate (using usual slow start and congestion

avoidance methods)

Configure TCP max and min bandwidth

Dedicated link Shared link with other FCIP tunnel

Shared link with non-storage traffic

max= path bandwidth (BW) max = allocated for this tunnel max = (link BW – other traffic BW)

min = 95% of max min = 80-95% of max min = 80-95% of max

switch(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 900 round-trip-time-ms 10



• TCP window size is directly proportional to RTT• Set correct RTT to fully utilize WAN pipe• Do not leave to default (1 ms)• Use ‘ping’

• OR ‘ips measure-rtt’ (preferred) to determine RTT

Configure correct value for Round Trip Time (RTT)

MDS9000# ping 192.168.20.2PING 192.168.20.2 (192.168.20.2) 56(84) bytes of data.64 bytes from 192.168.20.2: icmp_seq=1 ttl=254 time=0.740 ms64 bytes from 192.168.20.2: icmp_seq=2 ttl=254 time=0.621 ms64 bytes from 192.168.20.2: icmp_seq=3 ttl=254 time=17.8 ms

MDS9000# ips measure-rtt 192.168.20.2 interface ipStorage 1/2Round trip time is 111 micro seconds (0.11 milli seconds)



• Set proper IP DSCP values to prioritize FCIP traffic

Apply proper QoS policies when sharing link between storage and other traffic


FCIP Capacity Planning

FCIP LinkIP MAN / WAN

SiteA SiteB

MDS MDS

• Multiple parameters must be tuned to keep the WAN pipe full• TCP Parameters (Window size, max BW, Round trip time, SACK…) • Outstanding I/Os• Transfer size

• Standard traffic generating tools (like IOmeter) can be used• Requires test hosts and target


SAN Extension Tuner (SET)

• Lightweight tool (Only SCSI Read and Writes) integrated with NxOS on MDS• Allows you to configure an unused iSCSI interface as a FC Initiator and Target• Generates custom traffic and reports parameters

• I/O per second, Throughput, Round Trip Time, Compression ratio

• Configured by CLI or GUI (Cisco DCNM aka Fabric Manager)

FCIP LinkIP MAN / WAN

SiteA SiteBIPStorage1/1

IPStorage1/2 IPStorage1/2

IPStorage1/1

MDS MDS


Top 3 pitfalls to avoid1. Do not over-complicate the design2. Understand the QoS policy. Storage traffic must be subjected to tcp-max-bw, not

the typical traffic policing3. Understand when to use Inter-VSAN Routing


What Cisco has done really wellInvestment Protection

48 x 16G FC

48 x 10Gbps FCoE

24 x 40 Gbps FCoE

24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE

• All current (and future) modules can be used in any slot without any restrictions on MDS 9700

• Full FCIP backward interoperability is maintained

• 24/10 SEM module can be connected to MDS 9250i, MDS 9222i and SSN-16 module (for MDS 9500)

• Protects your investment for more than a decade

• Not mandatory to upgrade remote locations just because you upgraded the primary location

MDS 9250i

16 port GigE Storage Services Node (SSN-16)

MDS 9222i

FCIP

Summary


Advanced FeaturesExtended Credits FCIP WACompression

Encryption

FCIP TAIOA QoSSET

FCIP TuningTCP max and min bandwidth

Round Trip Time (RTT)

SAN Extension for Business Continuity and Disaster Recovery

SAN Extension Design and Solutions

Technology

Transcript of SAN Extension Design and Solutions