11/05/07 1TDC564-08

TDC 564TDC 564Local Area Networks Local Area Networks

Lecture 8Lecture 8IP-based Storage Area IP-based Storage Area

NetworkNetwork

11/05/07 2TDC564-08

Course OutlineCourse Outline IP over FC (RFC 2625)IP over FC (RFC 2625) IP-SANIP-SAN

iSCCI (RFC 3720)iSCCI (RFC 3720) IP and FC-SAN InterworkingIP and FC-SAN Interworking

FC Encapsulation (RFC 3643)FC Encapsulation (RFC 3643)FCIP (RFC 3821) – FC over IPFCIP (RFC 3821) – FC over IPiFCP (RFC 4172)iFCP (RFC 4172)

11/05/07 3TDC564-08

RFC 2625 – IP and ARP RFC 2625 – IP and ARP over FCover FC

FC supports multiple higher layer FC supports multiple higher layer protocols, and SCSI is the most widely protocols, and SCSI is the most widely used one.used one.

Needs/Applications of IP over FC:Needs/Applications of IP over FC: Access data in SAN from IP-based serversAccess data in SAN from IP-based servers interworking between NAS and SANinterworking between NAS and SAN

RFC 2625 addresses two issues.RFC 2625 addresses two issues. A scheme to encapsulate IP and ARP packets A scheme to encapsulate IP and ARP packets

inside the FC frame (as the FC payload)inside the FC frame (as the FC payload) A procedure to resolve the address mappingA procedure to resolve the address mapping

11/05/07 4TDC564-08

IP over FC (RFC 2625)IP over FC (RFC 2625)App-1: accessing SAN from IP-based App-1: accessing SAN from IP-based

serversservers

SAN

DataIPL2

PHY

IPL2

PHY

IPRFC 2625FC FC

IPRFC 2625FC

Data

SAN

FC/IP Gateway

FC-basedStorage Device

11/05/07 5TDC564-08

IP over FC (RFC 2625)IP over FC (RFC 2625)(App-2: interworking between SAN and (App-2: interworking between SAN and

NAS)NAS) SAN

DataIPL2

PHY

IPL2

PHY

IPRFC 2625FC FC

IPRFC 2625FC

L2PHY

IPData

IPL2

PHY

SAN

FC/IP Gateway

FC/IP Gateway

NAS-basedStorage Device

11/05/07 6TDC564-08

FC Address Resolution FC Address Resolution Three steps schemesThree steps schemes

ARP - IP to MACARP - IP to MAC MAC => World-Wide Port Name (WW_PN)MAC => World-Wide Port Name (WW_PN) FC-ARP - WW_PN => Port_IDFC-ARP - WW_PN => Port_ID

Optional Header is used for Network Header by Optional Header is used for Network Header by RFC 2625RFC 2625

D_NAA (4)Dst. Network Address (60) S_NAA

(4)Src. Network Address (60) 16 bytes

NAA: Network Address Authority

unused (12 bits)IEEE 802.3 MAC Address (48 bits)

Format of Network Header

Format of Network Address (WW_PN)

11/05/07 7TDC564-08

RFC 2625: IP and ARP over RFC 2625: IP and ARP over FC (1)FC (1)

ARP Request (broadcast)- Sender HW (MAC) Address = HW(A)- Sender IP Address = IP(A)- Target HW (MAC) Address = ??- Target IP Address IP(B)

A

B C D

FC switch

FC SAN

11/05/07 8TDC564-08

RFC 2625: IP and ARP over RFC 2625: IP and ARP over FC (2)FC (2)

ARP Reply (unicast):- Sender HW (MAC) Address = HW(B)- Sender IP Address = IP(B)- Target HW (MAC) Address = HW(A)- Target IP Address IP(A)

A

B C D

11/05/07 9TDC564-08

RFC 2625: IP and ARP over RFC 2625: IP and ARP over FC (3)FC (3)

FC-ARP Request:- Requester N_Port_Name (WW_PN)- Requester N_Port_ID- Responder N_Port_Name (WW_PN)- Responder N_Port_ID = ?? A

B C D

unused (12 bits)IEEE 802.3 MAC Address (48 bits)WW_PN

11/05/07 10TDC564-08

RFC 2625: IP and ARP RFC 2625: IP and ARP over FC (4)over FC (4)

FC-ARP Reply:- Requester N_Port_Name- Requester N_Port_ID- Responder N_Port_Name- Responder N_Port_ID

A

B C D

Advantages of IP for SANAdvantages of IP for SAN

11/05/07 11TDC564-08

IP Network CapabilitiesIP Network Capabilities

11/05/07 12TDC564-08

IP-SAN ProtocolsIP-SAN Protocols

11/05/07 13TDC564-08

11/05/07 14TDC564-08

IP - SANIP - SAN

FC -SAN

IP

IP

Storage Devices

11/05/07 15TDC564-08

IP storage networking – carrying storage traffic IP storage networking – carrying storage traffic over IPover IP

Uses TCP, a reliable transport for deliveryUses TCP, a reliable transport for delivery Can be used for local data center and long haul Can be used for local data center and long haul

applicationsapplications Two primary IETF protocols/standards:Two primary IETF protocols/standards:

iSCSIiSCSI – Internet SCSI – allows block storage to be – Internet SCSI – allows block storage to be accessedaccessed over a TCP/IP network as though it were locally over a TCP/IP network as though it were locally attachedattached

FCIPFCIP – Fibre-Channel-over-IP – used to tunnel Fibre – Fibre-Channel-over-IP – used to tunnel Fibre Channel frames over TCP/IP connectionsChannel frames over TCP/IP connections

IP - Storage Area IP - Storage Area Network (SAN)Network (SAN)

IPIP TCPTCP iSCSIiSCSI SCSISCSI DataData

IPIP TCPTCP FCIPFCIP FCFC SCSISCSI DataData

11/05/07 16TDC564-08

Internet SCSI (iSCSI) Internet SCSI (iSCSI) iSCSI is a proposed industry standard that allows SCSI iSCSI is a proposed industry standard that allows SCSI

block I/O protocols (commands, sequences, and block I/O protocols (commands, sequences, and attributes) to be sent over a network using the popular attributes) to be sent over a network using the popular TCP/IP protocol. TCP/IP protocol.

A way to access storage across an IP network as A way to access storage across an IP network as though it was locally attached.though it was locally attached.

Transports SCSI protocol commands and data across Transports SCSI protocol commands and data across an IP networkan IP network

Cisco and IBM co-authored original iSCSI protocol Cisco and IBM co-authored original iSCSI protocol draftdraft

iSCSI Protocol is a standard maintained by the IETFiSCSI Protocol is a standard maintained by the IETF IP Storage (IPS) Working GroupIP Storage (IPS) Working Group RFC 3720RFC 3720

iSCSIiSCSI

11/05/07 17TDC564-08

iSCSI over TCP/IPiSCSI over TCP/IP

11/05/07 18TDC564-08

Command SCSI

Command Data Command Data

TCP Header TCP Payload

11/05/07 19TDC564-08

iSCSI ApplicationsiSCSI Applications Connectivity : iSCSI can be used for DAS Connectivity : iSCSI can be used for DAS

or SAN connections. or SAN connections. iSCSI capable devices could be placed on iSCSI capable devices could be placed on

an existing LAN (shared with other an existing LAN (shared with other applications) in a similar way to NAS applications) in a similar way to NAS devices.devices.

iSCSI capable devices could be attached iSCSI capable devices could be attached to a LAN which is dedicated to storage I/O to a LAN which is dedicated to storage I/O (in other words an IP SAN), or even on a (in other words an IP SAN), or even on a LAN connected to only one processor (like LAN connected to only one processor (like a DAS). a DAS).

11/05/07 20TDC564-08

iSCSI Protocol StackiSCSI Protocol Stack

DataSCSIiSCSITCPIP

EthernetPHY

DriverTCPIP

EthernetPHY

Driver

DataSCSIiSCSI

TCPIP

EthernetPHY

Driver

DataSCSI

iSCSIAdapter

Card

HostProcessing

Host Bus Adapter (HBA)

Software iSCSIiSCSI TCP Offload

Engine(TOE)

iSCSI Host Bus Adapter

(HBA)

iSCSI Performance iSCSI Performance ConsiderationConsideration

11/05/07 21TDC564-08

iSCSI BenefitiSCSI Benefit

11/05/07 22TDC564-08

Host Bus Adapter (Fibre Host Bus Adapter (Fibre Channel)Channel)

11/05/07 23TDC564-08

QLogic PCI-EXPRESS 4Gb 1PT Fibre Host Bus Adapter – STD FW•  PCI-Express • •  1 Port (Fibre Channel)•  Data Rate: 4Gb •  Standard Firmware

Price Quote: $951Source: http://www.cdw.com

11/05/07 23TDC564-08

Host Bus Adapter (iSCSI)Host Bus Adapter (iSCSI)

11/05/07 24TDC564-08

The QLE4060C iSCSI Host Bus Adapter (HBA) provides connectivity to SANs over Ethernet and TCP/IP network infrastructures for PCI Express bus-based servers. Using a widely deployed and familiar networking technology, iSCSI can lower the total cost of ownership to better manage and support storage networking.

Source: http://www.cdw.orgPrice quote: $815

10/100/1000BaseTPCI

iSCSI StorageiSCSI Storage

11/05/07 25TDC564-08

Promise Technology VTrak M-Class iSCSI RAIDPrice quote: $4,929Source: //www.cdw.com

Network: Gigabit Ethernet (2)Protocol: TCP/IP, iSCSICapacity: 16-drive bay (500G – 1TB per drive)

11/05/07 26TDC564-08

iSCSI iSCSI ApplicationApplicationData

SCSIiSCSITCPIP

Layer2PHY

iSCSITCPIP

Layer2PHYIP

SCSIFCP

FC 0-2

SCSIFCP

FC 0-2

Data

Cisco iSCSI Configuration Example

iSCSIGateway

Server

StorageDevice

Multi-TCP SessionsMulti-TCP Sessions

11/05/07 27TDC564-08

HBA: Multi-Path I/OHBA: Multi-Path I/O

11/05/07 28TDC564-08

LUN: Logical [Storage] Units

HBA iSCSI vs. Software iSCSI

No performance advantage of using iSCSI HBA Throughput Round trip time

Significantly higher cost of iSCSI HBA $800 vs. $100 (Server NIC)

What are the advantages of iSCSI HBA? Answer: CPU Utilization (off-load)

Reference: http://download.intel.com/network/connectivity/products/iSCSIcomp0407.pdf

11/05/07 29TDC564-08

Is IP-SAN similar Is IP-SAN similar to NAS?to NAS?

What are the What are the advantages, if any, of advantages, if any, of IP-SAN vs. NAS?IP-SAN vs. NAS?

11/05/07 TDC564-08 30

Performance Analysis: iSCSI vs. NAS

(software based – no HBA)

http://technomagesinc.com/pdf/ip_paper.pdf

iSCSI NFS

11/05/07 31TDC564-08

Sequential Read/Write Tests

Conclusion: 1. Comparable performance in character read/write and block write2. Significant advantage of iSCSI in block read (20-25%)

11/05/07 32TDC564-08

Small Files Read/Write

11/05/07 33TDC564-08

IOGen Test (Emulation of Database)

11/05/07 34TDC564-08

11/05/07 35TDC564-08

FC-SAN vs. iSCSIFC-SAN vs. iSCSI Since the iSCSI appliance attaches to the existing Ethernet Since the iSCSI appliance attaches to the existing Ethernet

network, NAS and iSCSI are very similar in network network, NAS and iSCSI are very similar in network architecturearchitecture However, the performance would be significantly different.However, the performance would be significantly different.

Both iSCSI and SAN use Block I/O to transport data, whereas Both iSCSI and SAN use Block I/O to transport data, whereas NAS uses File I/O. NAS uses File I/O.

SAN offers better performance (c.f. NAS), but is more SAN offers better performance (c.f. NAS), but is more expensive and requires a higher skill set to implement. iSCSI expensive and requires a higher skill set to implement. iSCSI and NAS offer better pricing and skills may already be in and NAS offer better pricing and skills may already be in place to implement them. place to implement them.

Both SAN and iSCSI offer the performance benefit of Block Both SAN and iSCSI offer the performance benefit of Block I/O.I/O.

Case Study: FC-SAN vs. iSCSI

http://www.promise.com/marketing/whitepaper/file/2_MClass-iSCSI%20SCSI%20and%20Fibre-10-07-

05.pdf11/05/07 36TDC564-08

4G bps

11/05/07 37TDC564-08

11/05/07 38TDC564-08

FC over IP FC over IP • The encapsulation of Fibre Channel frames into IP

packets and tunneled through an existing TCP/IP network infrastructure, in order to connect remote islands of FC-based SANs

FC SAN IP FC SAN

IPIP TCPTCP FCIPFCIP FCFC SCSISCSI DataData

FC Frame Encapsulation (RFC 3643)

Describes the common Fibre Channel (FC) frame encapsulation format and a procedure for the measurement and calculation of frame transit time through the IP network

Encapsulation HeaderSOFFC Frame BodyEOF

11/05/07 39TDC564-08

FC Encapsulation Header

Q: Why is time stamp needed in the FC encapsulation header?

11/05/07 40TDC564-08

FC Encapsulat

ion

11/05/07 41TDC564-08

FCIP (RFC 3821)Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric

11/05/07 42TDC564-08

FCIP Protocol Stacks

11/05/07 43TDC564-08

11/05/07 44TDC564-08

FC over IP FC over IP IP SAN

DataSCSI

FC 0-2 FC 0-2

FCIPTCPIPL2

PHY

IPL2

PHY

TCPIPL2

PHY

FC 0-2

DataSCSI

FC 0-2FCP FCP

SAN

an IP tunnel for FC-based SANApplication: interconnect SAN over IP-WAN.

FCIPFCP FCP

FCIP Configuration Example

11/05/07 45TDC564-08

11/05/07 46TDC564-08

Internet Fiber Channel Internet Fiber Channel Protocol (iFCP)Protocol (iFCP)

iFCP – iFCP – RFC 4172 (Sept. 2005)(Sept. 2005) iFCP is a protocol that uses the FCP iFCP is a protocol that uses the FCP

over IP networks to interconnect over IP networks to interconnect FC-based storage devices.FC-based storage devices.

It is considered a It is considered a gatewaygateway protocol, protocol, while FCIP is considered a while FCIP is considered a tunnel tunnel protocol.protocol.

iFCP (RFC 4172) iFCP specifies an architecture and a gateway-to-

gateway protocol for the implementation of fibre channel fabric functionality over an IP network.

This functionality is provided through TCP protocols for fibre channel frame transport and the distributed fabric services specified by the fibre channel standards.

The architecture enables internetworking of fibre channel devices through gateway-accessed regions with the fault isolation properties of autonomous systems and the scalability of the IP network

11/05/07 47TDC564-08

iFCP Network Example

11/05/07 48TDC564-08

11/05/07 49TDC564-08

iFCP (RFC 4172) iFCP (RFC 4172) IP

DataSCSI

FC 0-2

FC 0-2

FCP

TCPIPL2

PHY

IPL2

PHY

TCPIPL2

PHY

FC 0-2

DataSCSI

FC 0-2

FCP FCP

Gateway approachApplication: interconnect SAN over IP-WAN.

FCPiFCPiFCP

iFCP GW iFCP GW

F-port

F-port

11/05/07 49TDC564-08

iFCPiFCP

11/05/07 50TDC564-08

FCIP vs. iFCPFCIP vs. iFCP

11/05/07 51TDC564-08

http://whatis.techtarget.com/magPrintFriendly/0,293813,sid35_gci1096482,00.html

FCIP vs. iFCP

McDATA

11/05/07 52TDC564-08

11/05/07 53TDC564-08

SAN Performance SAN Performance ComparisonComparison

http://www.netapp.com/library/tr/3423.pdf

11/05/07 54TDC564-08

SummarySummary Needs for large storage – continual growthNeeds for large storage – continual growth

10109 9 (G) => 10(G) => 101212 (T) => 10 (T) => 101515 (P) => 10 (P) => 1018 (E) ……. (E) ……. From dedicated solution to network-based From dedicated solution to network-based

solutionsolution DAS => NAS => SAN => IP-SANDAS => NAS => SAN => IP-SAN

Convergence of SAN and IP-LAN/WANConvergence of SAN and IP-LAN/WAN It is an IP world!It is an IP world!

SCSI is the protocol for block data SCSI is the protocol for block data transmissiontransmission SCSI over FCSCSI over FC SCSI over IP (iSCSI)SCSI over IP (iSCSI)

FC and IP interworking protocolsFC and IP interworking protocols IP over FCIP over FC FC over IP (FCIP) and iFCPFC over IP (FCIP) and iFCP