Post on 18-Nov-2014
description
Storage NetworkingThe Path to Performance
Howard Goldstein, President HGAI
Slide 2 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Session Outline
• What’s Important?• Performance Elements, Storage Devices Matter• Bit Rate, Bandwidth, Latency & Throughput• Circuit Switching vs. Packet Switching• Spraying, Striping, Ports & PHYs, Load Sharing• TCP/IP Offload Engines• Virtualization• FC Fabric Gateways & Routers• Virtual Fabrics• FCoE• Summary
Slide 3 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Information
Repository
Access
Processing
Sharing Information
Price Performance
ScalabilityReliability
What’s Important?
Slide 4 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Performance Elements
Year Storage Network Penalty1992 10 MBPS .1 MBPS 100-11994 20 MBPS 1 MBPS 20-11996 40 MBPS 10 MBPS 4-11998 100 MBPS 100 MBPS 1-1
Wires to Disk Wires to NetworkSCSI 10bT shared
Fast-Wide SCSI FDDI sharedUltra SCSI 100bT switched
Fibre Channel Gigabit Switched
IP / Ethernet networks are fast enough for storage
Slide 5 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Performance Myths
• FC at 4 Gbps is four times the Ethernet bit rate at 1 Gbps so FC must be 4x faster than iSCSI
• TCP/IP protocols is much less efficient than FC, so iSCSI must be much slower than FC
• TOEs are required when using iSCSI to reduce server CPU utilization consumed by TCP/IP overhead
• All “Enterprise” applications require high performance
Slide 6 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Disk Technology
PlatterSectorTrack Head
Slide 7 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Disk Technology
PlatterSectorTrack Head
Slide 8 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Disk Technology
PlatterSector Track HeadActuator Arm
Slide 9 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Disk Technology
PlatterSector Track HeadActuator ArmCylinder
SCSI/ATA Performance may depend on• Revolutions per minute, Seek time, • Latency, Data transfer time• Caching mechanisms• Application I/O Types – Random vs. SequentialSSD changes the SCSI/ATA performance game!
Slide 10 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Cache Types
• Read Caching– Recently Used Cache: Data hot spots– Read-Ahead Cache: Sequential pre-fetch
• Write Caching– Write-Through Cache: Direct to non-volatile memory.
Optimized for integrity– Write-Back (Write-Behind) Cache: Faster response to
applications. Optimized for performance
Slide 11 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Technical Performance Drivers
• Bit Rate• Bandwidth• Latency• Throughput
Slide 12 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Bit Rate, Bandwidth, & Throughput
Slide 13 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Parallel vs. Serial
Does increasing bit rate increase speed?
A recent student metaphor:
Slide 14 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Parallel vs. SerialSerial Data Transfers
Clock RecoveryCircuit
Serial Data InClocked Data
Recovered Clock
Skew causes misalignment at the receiver
All bits are aligned at transmitting device
Parallel Data Transfers
Example of Cross Talk
Electromagnetic Coupling
Example of Signal Skew
Serial protocol:• Eliminates Signal Skew• Eliminates Cross-talk• Simplifies Interconnect• Allows Higher Data Rate• Point to Point
Source: Maxtor
Slide 15 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What About Speed?
• Signal Speed for light or voltage is the same!– 299,792,458 Meters per Second
• .03 Meters per Nanosecond (ns)
– 186,282.397 Miles per Second • 1 Foot per Nanosecond (ns)
Slide 16 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Which is faster?
• Copper • Fiber
Slide 17 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Which is faster?
• Copper
• Fiber
Copper CAT5 approximately with 4.72 ns/m
Fiber approximately 5 ns/m
Media Propagation Delay
Slide 18 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The Rise and Fall
• Copper
• Fiber
The Rise Time
The Fall Time
The Bit Time
The Rise Time
The Fall Time
The Bit Time
Slide 19 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Bit Rate – Serial Scaling
Does increasingbit rate increasespeed?
Slide 20 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Serial Link Bit Rate Scaling
40 Bits = Tword
1 Bit
1 Bit @ 1.5 Gbps = .666ns
Transmit Receive
1 Bit @ 3.0 Gbps = .333 ns
1 Bit40 Bits = Tword
Assume 40 Folds
Slide 21 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Change the tortoise and the hare fable
Throughput does not equal Bandwidth
Improve bit rate
Slide 22 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Parallel to Serial?
Back to that student metaphor
Shorter Cars! Both Directions!
Slide 23 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Serial - Higher bit rates & Even Shorter Cars!
Slide 24 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
• Multi-Lane, serial striping techniques goes “back to the future”!
Parallel the Serial – Aggregate for Higher bit rates
Slide 25 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Semantic “Anal”ism
• High Speed• High Bandwidth
Slide 26 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
iSCSI Overhead
iSCSIStandard Ethernet
Ethernet with Non-Standard Jumbo Frames
iSCSI/TCP/IP/ Standard Ethernet Ovehead
iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames Standard Ethernet
Ethernet with Non-Standard Jumbo Frames
iSCSI/TCP/IP/ Standard Ethernet Ovehead
iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames
iSCSI Data Out or Data in PDU Header Bytes 52 52 52 52
TCP Header Bytes 20 20 20 20
IPv4 Header Bytes 20 20 20 20
Inter Frame Gap Bytes 12 12 12 12 12 12 12 12
Preamble Bytes 7 7 7 7 7 7 7 7
SFD Bytes 1 1 1 1 1 1 1 1
Destination Address Bytes 6 6 6 6 6 6 6 6
Source Address Bytes 6 6 6 6 6 6 6 6
Length/Type Bytes 2 2 2 2 2 2 2 2
Maximum Data/Pad Bytes 1,500 9,000 1,500 9,000 1,500 9,000 1,500 9,000
FCS Bytes 4 4 4 4 4 4 4 4
Ethernet Frame overhead including Preamble and SFD fields Bytes 38 38 38 38 38 38 38 38
Total Overhead Bytes 76 76 168 168 76 76 168 168
Slide 27 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
iSCSI Performance
Payload Bytes 1,500 9,000 1,500 9,000 1,500 9,000 1,500 9,000
Efficiency % 97.53% 99.16% 89.93% 98.17% 97.53% 99.16% 89.93% 98.17%
Bit Rate gbps 1.25 1.25 1.25 1.25 12.5 12.5 12.5 12.5
Data Rate MBps 121.91 123.95 112.41 122.71 1219.12 1239.53 1124.10 1227.09
Encode/Decode Penalty 0.80 0.80 0.80 0.80 0.80 0.80 0.80 0.80
Effective Data Rate MBps 97.53 99.16 89.93 98.17 975.29 991.63 899.28 981.68
iSCSIStandard Ethernet
Ethernet with Non-Standard Jumbo Frames
iSCSI/TCP/IP/ Standard Ethernet Ovehead
iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames Standard Ethernet
Ethernet with Non-Standard Jumbo Frames
iSCSI/TCP/IP/ Standard Ethernet Ovehead
iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames
Slide 28 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel PerformanceBytes Bytes Bytes Bytes Bytes Bytes
Fibre Channel
Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 2
Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 2
SOF Bytes 4 4 4 4 4 4Frame Header Bytes 24 24 24 24 24 24Maximim Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048CRC Bytes 4 4 4 4 4 4EOF Bytes 4 4 4 4 4 4
InterFrame Word Minimum Bytes Bytes 24 24 24 24 24 24
Acknowledgement Frame Overhead 60 60Total Overhead Bytes 60 60 120 60 60 120
Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048
Efficiency % 97.24% 97.15% 91.92% 97.24% 97.15% 91.92%
Bit Rate gbps 1.0625 1.0625 1.0625 2.125 2.125 2.125
Data Rate MBps 103.31 103.23 97.67 206.63 206.45 195.33
Encode/Decode Penalty 0.8 0.8 0.8 0.8 0.8 0.8
Effective Data Rate MBps 82.65 82.58 78.13 165.30 165.16 156.27
Slide 29 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Performance
Fibre Channel
SOF BytesFrame Header BytesMaximim Payload BytesCRC BytesEOF BytesInterFrame Word Minimum Bytes BytesAcknowledgement Frame OverheadTotal Overhead Bytes
Payload Bytes
Efficiency %
Bit Rate gbps
Data Rate MBps
Encode/Decode Penalty
Effective Data Rate MBps
Bytes Bytes Bytes Bytes Bytes Bytes
Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 2
Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 3
SCSI FC DATA Out or IN / Fibre Channel Class 2
4 4 4 4 4 424 24 24 24 24 24
2,112 2,048 2,048 2,112 2,048 2,0484 4 4 4 4 44 4 4 4 4 4
24 24 24 24 24 24
60 6060 60 120 60 60 120
2,112 2,048 2,048 2,112 2,048 2,048
97.24% 97.15% 91.92% 97.24% 97.15% 91.92%
4.25 4.25 4.25 8.5 8.5 8.5
413.26 412.90 390.66 826.52 825.81 781.33
0.8 0.8 0.8 0.8 0.8 0.8
330.61 330.32 312.53 661.22 660.65 625.06
Slide 30 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Bandwidth does not equal bit rate
Bit Rate Efficiency Effective Data Rate
Gigabit Ethernet Standard MTU 1500
1.25 Gbps 97.5% 97.5 MBps
iSCSI / TCP / IP / Gigabit Ethernet Standard MTU
1500
1.25 Gbps 89.9% 89.9 MBps
iSCSI / TCP / IP / Gigabit Ethernet Jumbo Frames
MTU 9000
1.25 Gbps 98.2% 98.2 MBps
Standard SCSI FCP Fibre Channel Class 3
1.0625 Gbps 97.15% 83 MBps
Gigabit Ethernet Standard MTU 1500
12.5 Gbps 97.5% 975 MBps
iSCSI / TCP / IP / Gigabit Ethernet Standard MTU
1500
12.5 Gbps 89.9% 899 MBps
iSCSI / TCP / IP / Gigabit Ethernet Jumbo Frames
MTU 9000
12.5 Gbps 98.2% 982 MBps
Standard SCSI FCP Fibre Channel Class 3
8.5 Gbps 97.15% 661 MBps
SAN Performance Comparison – Effective Data Rate
Good
8% Better
18% Best
Good
36% Better
49% Best
Slide 31 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
FC vs. iSCSI Performance
• Payload Efficiency is not the whole story• Latency and Delay play a role too!• Offered Load• Off Load• Ethernet plays a big role for both
Slide 32 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Harry Newton’s Telecom Dictionary • Delay
– The wait time between two events, such as the time from when a signal is sent to the time it is received.
– There are all sorts of reasons for delays• Propagation Delay• Queuing Delay• Processing Delay• Rotational Delay• Satellite Delay
• Latency– A fancy term for waiting delay. The time it takes to get
information through a network.– Accumulated delay
Slide 33 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Circuit Switch – Packet Switch
• Circuit Switch - Fixed path whether moving or not• Packet Switch - Routing frames or packets on the
current best path as they arrive and leave
Slide 34 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Circuit Switching
Circuit Switch Construction Circuit Switch Setting
Slide 35 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Circuit Switch
SAS Expanders
SATA Port Multipliers
FC Class 1
POTS (Plain Old Telephone Network)
Slide 36 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Frame Switch
Frame Direction
Slide 37 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Packet (Frame) Switch
FC Class 2,3
PCI Express
IP Datagram (Router)
VoIP
Slide 38 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Packet (Frame) Switch
RC
PCI Express Transaction
Slide 39 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Parts of a Physical Transport Network?
1. 2. 3.Media or the “appearance of” media
Ports - 2 or more (transceivers)
Protocol
Slide 40 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Port vs. PHY = Ports vs. Docks
Fibre Channel, Ethernet
1 Port – 1 PHY
SAS, Infiniband, PCIe
1 Port – Many PHYs
Slide 41 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Spraying - Striping
1 Link - 4 Phy Lanes
Aggregate Bandwidth(PCIe, InfiniBand)
Slide 42 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Striping the Transmission Word
1 Link - 4 Phy Lanes
0 1 2 30 1 2 3
0 1 2 3
PCIe, InfiniBand does Striping (4x, 8x, 12x, etc.)
Slide 43 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Striping the Transmission Word
1 Link - 4 Phy Lanes
0 1 2 30 1 2 3
0 1 2 3
PCIe, InfiniBand does Striping (4x, 8x, 12x, etc.)SAS is not Striping – See connection path
Slide 44 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Individual Pathways1 per Open Connection
1 Wide Link – 2 Connections
No Aggregate Bandwidth
Slide 45 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Wide Ports – Wide Link
1 Link - 4 Phy Lanes
Slide 46 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Wide Ports – Wide Link
1 Link - 4 Phy Lanes
SAS is not Striping – See connection path
1 2 30
Slide 47 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS 2 Expander – Open, Arbitration, Open Accept
Node Node
Node NodeExpander
ID03
ID04
ID01
ID02
ARB 02, 03
OPEN 02, 03
AIP (Normal)
Frame Operation 1 Primitive
ExpanderARB 02, 03AIP (Normal)
OPEN 02, 03
AIP Status (WoD)
OPEN 02, 03
OPEN Accept
Frame Operation 2
Slide 48 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS Expander – Move Revisited –Before Mux
Node Disk Node
HBA Node Disk NodeExpander
ID03
ID04
ID01
ID02
6 gbps 3 gbps
3 gbps
6 gbps Frame
3 gbpsFrame
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
04-01
Slide 49 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS Expander – Move Revisited –Before Mux
Node Disk Node
HBA Node Disk NodeExpander
ID03
ID04
ID01
ID02
6 gbps 3 gbps
3 gbps
6 gbps Frame
3 gbpsFrame
04-0104-01
04-0104-01 04-01ALIGN ALIGN
04-01
Slide 50 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS Expander – Move Revisited –After Mux
Node
HBA Node Disk NodeExpander
ID03
ID01
ID02
6 gbps 3 gbps
3 gbps
6 gbps Frame
3 gbpsFrame
04-01
04-01 02-0102-01
04-01
04-01 02-0102-01
04-01
02-0104-0102-01
04-01
02-01
ID04
04-0102-01
04-01
02-0104-0102-01
04-01
02-0104-0102-01
04-01
02-0104-0102-01
04-01
02-01
Slide 51 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS Expander – Move Revisited –After Mux
Node
HBA Node Disk NodeExpander
ID03
ID01
ID02
6 gbps 3 gbps
3 gbps
6 gbps Frame
3 gbpsFrame
ID04
02-01
04-01
02-0104-01 02-01
04-01
02-01
04-01
04-01 02-01
Slide 52 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Inter-Switch Links - FSPF
Fabric
N-PortE-Ports
N-Port
Node
F-PortF-Port
Node Node
Slide 53 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Inter-Switch Links
N-PortE-Ports
N-Port
Node
F-PortF-Port
Node NodeStandard FSPF Routing
Slide 54 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Round Robin
Slide 55 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Routing – Round Robin Load Share• Fabric Routing involves Inter-Switch Links
(ISLs)1. Port-Based2. Device-Based3. Exchange-Based
Slide 56 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Load Sharing –Port Based
Fabric
N-Port E-Ports N-Port
Node
F-PortF-Port
Node Node
1
3
2
4
5
Domain 1
Domain 2
Slide 57 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Load Sharing – Device Based
Fabric
N-Port E-Ports N-Port
Node
F-PortF-Port
Node Node
1
3
2
4
5
Domain 1
Domain 2
Slide 58 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Load Sharing – Exchange Based
Fabric
N-Port E-Ports N-Port
Node
F-PortF-Port
Node Node
1
3
2
4
5
Domain 1
Domain 2
Exchanges
Slide 59 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Trunking
Fabric
N-PortE-Ports
N-Port
Node
F-PortF-Port
Node Node
Slide 60 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
SAS: The Evolutionary Tale….
Evolution:
- Replaces Ultra320 - Preserves Legacy
SCSI Software- Renews the SCSI
Roadmap- Continues the
Serial Trend
CPU
Storage
Interface
3G
6G
Slide 61 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Industry Association FCIA Roadmap
Base2*
Product NamingThroughput
(MBps)Line Rate
(Gbaud)
T11 Spec Completed
(Year)
Market Availability
(Year)
1GFC 200 1.065 1996 1997
2GFC 400 2.125 2000 2001
4GFC 800 4.25 2003 2005
8GFC 1,600 8.5 2006 2008
16GFC 3200 17 2009 2011
32GFC 6400 34 2012 Market Demand
64GFC 12800 68 2016 Market Demand
128GFC 25600 136 2020 Market Demand
Base10**
10GFC 2400 10.52 2003 2004
FCoE
Slide 62 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The Road to 100G Ethernet
Year Standard Description1999 802.3ab Gig Ethernet TP (Copper)June 2006
802.3an Gig Ethernet (10 Gbps) TP
July 2006 IEEE High Speed Study Group
Gig Ethernet (40 Gbps, 100 Gbps)
2007 IEEE will form 100G Ethernet Task Force
Gig Ethernet (100 Gbps)
2009 / 2010
IEEE 100G Standard Complete
Gig Ethernet (100 Gbps)
Slide 63 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Traditional FC SAN
Host Applications
File Manager
Device Driver
SCSI Protocol
Operating SystemI/O Interface
HBA
FC2FC1
FC3
FC4
FC0
Transport
ApplicationServices
Internet
Internet Protocol Suite
Host NIC
NetworkInterfaceSublayer
Ethernet
FC2
Slide 64 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Read 8K 6 FC frames (4 - 2K Frames)
Host
HBA
FC2FC1
FC3
FC4
FC0
FC2
Fibre Channel Multi-Frame Sequence Minimizes Server I/O interrupts - 2
Read CommandDataDataDataDataStatus
Slide 65 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Traditional NAS
Host Applications
File Manager
Device Driver
SCSI Protocol
Operating SystemI/O Interface
HBA
FC2
FC1
FC3
FC4
FC0
Transport
ApplicationServices
Internet
Internet Protocol Suite
Host NIC
NetworkInterfaceSublayer
Ethernet
NFS/CIFS
FC2
Slide 66 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
8K 7 Ethernet Frames (6 - 1.5K Frames)
Host
Host IP Fragmentation - 6 IP Packet Fragments, 6 – 1500 byte MTU Ethernet Frames, Maximizes Server I/O interrupts - 7
NIC
NetworkInterfaceSublayer
Ethernet
Transport
ApplicationServices
Internet
Get FileDataDataDataData
DataData
NFS / CIFS / iSCSI
Slide 67 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
iSCSI Full Offload
Host Applications
File Manager
Device Driver
SCSI Protocol
Operating SystemI/O Interface
Transport
ApplicationServices
Internet
Internet Protocol Suite
Host IntelligentNIC/HBA
NetworkInterfaceSublayer
GigabitEthernet
iSCSI
TCP
IP
iSCSI
Slide 68 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Read 8K 8 (6 – 1.5K Ethernet Frames)
Host Offload IP Fragmentation (Like FC Multi-Frame Sequence)Minimizes Server I/O interrupts - 2
Read CommandDataDataData
Status
IntelligentNIC/HBA
NetworkInterfaceSublayer
GigabitEthernet
TCP
IP
iSCSI
DataData
Data
Slide 69 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Not All TOEs Are Created Equal!
Even Between Different ASICs …
Full HW State-Based TOE
Firmware ASIC TOEFirmware TOELittle TOE
TCP/IP Offload Engines (TOE)
Slide 70 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The TOE Spectrum
Adapter Driver
TCP/IP
iSCSI
SCSI Port to OS
Software iSCSI“NIC + Driver”
Media Interface
Ethernet
Media Interface
Ethernet
Fast Path TCP/IP
Software iSCSIWith Partial TCP Off-load
TCP/IP
iSCSI
SCSI Port to OS
Media Interface
Ethernet
TCP/IP
iSCSI
Firmware TCPand iSCSI Off-load
SCSI Port to OS
= SW = Hardware
Hos
tA
dapt
er
Media Interface
Ethernet
TCP/IP
iSCSI
Hardware TCP and Firmware iSCSI
Off-load
SCSI Port to OS
= FW
Slide 71 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Game Changers
• PCIe no longer has “Hard I/O Interrupts”– Message Signaled Interrupts (MSI)
• Multi-core processers have cycles to spare– Message to Virtual Machines – Let us have a few cycles for
TCP/IP overhead!
• Parallel the Serial – Leverages physical barriers
Slide 72 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Summary
• Achieving high bit rates are important for the data center
• Understand the information flow path• Asset utilization is key• Hope this helps• Thanks for coming
Fibre Channel for the Data CenterFC Virtualization and FCoE
Presented and Developed by Howard Goldstein of Howard Goldstein Associates, Inc.
© Copyright 2009 Howard Goldstein
Slide 74 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
FC and the Data Center
• Virtualization• N-Port ID Virtualization• Distributed Services• FC Fabric Gateways & Routers• Virtual Fabrics• FCoE
© Copyright 2009 Howard Goldstein
Slide 75 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What is Virtualization?
• Virtualization is an Abstraction– Server– Network– Storage
• What is an Abstraction?
© Copyright 2009 Howard Goldstein
Slide 76 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtualization is Everywhere
• Benefits– Pooling of resources– Rapidly deploy new applications– Increase resource utilization– Over-subscribe resources– Lower acquisition cost and TCO
© Copyright 2009 Howard Goldstein
Slide 77 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Where is Virtualization?Servers
Storage
Network
Storage Storage
VirtualMachine
VM File System
Scale Servers & Storage
VirtualMachineVirtual
Machine
VirtualMachineVirtual
Machine VirtualMachine
VM File System VM File System
VirtualMachine
VirtualMachine
VirtualMachine
Storage
© Copyright 2009 Howard Goldstein
Slide 78 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What is Storage Virtualization?
• An abstraction of storage that separates – Host view – Storage system implementation
• Makes invisible to host:– physical pathing– device characteristics– physical data location
• Provides Location and Implementation Transparency
• Dynamic– Enables transparent “on the fly” reconfiguration– Allow data location to change transparently to host environment
• There are a lot of different types and approaches and degrees of storage virtualization
© Copyright 2009 Howard Goldstein
Slide 79 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtualization in the Network (In-Band)
Virtualization
Data center-wide
Provisioning
Host
RAID subsystem
SANHost accessDevice access
Network Device: Appliance
The appliance is in the data path; Plug and play from the host perspective
Slide 80 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtualization in the Network (Out-of-Band)
Virtualization
Data center-wide
Provisioning
Host
RAID subsystem
“SAN appliance”separate boxIn-host Network
The appliance is not in the data path; Requires agent software on each host; Separates the data from the control path
Slide 81 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtualization in the Network (In-Band with Switches)
Virtualization
Data center-wide
Provisioning
Host
RAID subsystem
NetworkDevice: Switch
SANHost accessDevice access
The FC switch is in the data path; Plug and play from the host perspective
Fibre Channel Names & Addresses NPIV
Slide 83 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric
World Wide (Port & Node) Names
FC-ALHub
Point-to-point
WWPN
WWPN
WWPNWWPN
FCIP gateway
WWPN
WWPN
WWPN
WWNN WWNN
WWNN WWNN
WWNN
WWNN
Slide 84 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Routing
010600 to 0302EF
Domain ID 01
Domain ID 03
Domain ID 02
Hub
010600
0302EFSwitch
Switch
Switch
Slide 85 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Standard Fabric Login
Switch Domain
ID 02
Switch
WWPN 1
HBA Name Server
WWNN WWPN Port_ID
Fabric Login Controller
Port_ID Allocate
Area_IDs00 01 02 03 04 05 06 07Fabric Login
WWPN1
Login AcceptWWPN1 020000
Port_ID is tightly couple with switch port
Area_ID often represents switch port
Device_ID represents:
• “00” FL_Port for FCAL
• ALPA attached device for FCAL,
• “00” in most cases for Point-to-Point
Slide 86 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
NPort_ID Virtualization
Switch Domain
ID 02
Switch
WWPN 1
NPIV Switch
Name Server
WWNN WWPN Port_ID
Fabric Login Controller
Port_ID Pool
Area_IDs00 01 02 03 04 05Fabric Login
WWPN1
Login AcceptWWPN1 020000
Virtual HBAs
FDISCWWPN2
FDISC AcceptWWPN2 020001
FDISCWWPN3
FDISC AcceptWWPN3 020002
FDISCWWPN4
FDISC AcceptWWPN4 020003
Port_ID TableWWPN1 020000WWPN2 020001WWPN3 020002WWPN4 020003
WWPN 2 WWPN 3 WWPN 4
Slide 87 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
N_Port ID Virtualization
• N_Port virtualization enables an N_Port to acquire multiple N_Port IDs (addresses)– Each address has the appearance of a separate N_Port to other
ports– Separate WWN is used for each address– Each address maintains a separate context
• The initial address is acquired using FLOGI– Additional addresses are acquired by FDISC
• An address can be relinquished using LOGO• All virtual N_Ports share the same physical link
– BB_Credit is shared by all virtual N_Ports– Link Reset, Link Initialization or Link Failure affects all virtual
N_Ports on that link
Fibre Channel Gateways, Routers & Virtual Fabrics
Slide 89 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Distributed Services
Slide 90 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Expansion – 2 Separate Fabrics
Slide 91 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Expansion – 2 Separate Fabrics
Fabric 1
Fabric 2
Slide 92 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Expansion – Inter Switch Link
ISL
Slide 93 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Merged Fabric
Fabric Expansion – Inter Switch Link
ISL
Slide 94 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fabric Initialization
• Fabric initialization includes a series of actions: – Negotiate the link speed, if supported – Determine the switch port operating mode – If an F_Port or FL_Port, wait for node port to initiate
login – If an E_Port, Exchange Link Parameters and Switch
Capabilities with neighbor – Select a principal switch – Request/Assign Domain_IDs – Build routing tables and select paths
Slide 95 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP FC Switch
Fibre Channel over IP (FCIP)
FC
FC
FC
FC
FC Switch
SCSI
SAN ASAN A
Virtual ISL
Merged Fabric
Slide 96 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP
Fibre Channel over IP (FCIP)
FC
FC
FC
FC Switch
FCIP Gateway
Blade
FCIP Gateway
Blade
SCSI
SAN ASAN AFC
Virtual ISLFC Switch
Merged Fabric
Slide 97 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP
Multipoint Fibre Channel over IP (FCIP)
FC
FC
FC
FC Switch
FCIP Gateway
Blade
FCIP Gateway
Blade
SAN ASAN AFC
Virtual ISL FC Switch
FC
FCIP Gateway
Blade
SAN A
FC
FC Sw
itch
Merged Fabric
Slide 98 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP
Multipoint FC-FC Router FCR with FCIP
FC
FC
FC
FC Switch
FCIP Gateway
Blade
FCIP Gateway
Blade
SAN BSAN AFC
Virtual IFL FC Switch
FC
FCIP Gateway
Blade
SAN C
FC
FC Sw
itch
FCR – Connectivity with Isolation
Not Merged Fabric
Slide 99 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Router
Slide 100 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Multiple Fabric Address Translation
Page 326: Figure 140
Slide 101 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Security, Zoning and Virtual Fabrics
• Zoning limits the ability to discover (soft zoning) or access (hard zoning) other ports– Does not limit access to well-known services– Does not provide authentication of entities
• Virtual fabrics provide isolation between instances of services– Separate Name Server, Fabric Controller, Zoning, FSPF
Database and routing tables• Security provides authorization, authentication and
encryption– Which devices can join a fabric and which ports can connect to
other ports– Means to authenticate the identity of entities– Encryption provides confidentiality
Slide 102 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtual Fabrics
• Virtual fabrics enable a single physical fabric to appear as multiple virtual fabrics (VSANs, LSANs)
• Each virtual fabric has all the characteristics of a normal fabric. Each has its own:– Address space– FSPF routing database– Zoning– Name Server– RSCN registration and distribution
• Virtual fabrics enable consolidating multiple separate fabrics into a single physical fabric
Slide 103 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Example Fabric Physical Configuration
Slide 104 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtual Fabric 1
Slide 105 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Virtual Fabric 2
Slide 106 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Shared ISLs and Frame Tagging
FCoE
Slide 108 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Applications and Networks
• Today, each application class has its own interface– Networking: Ethernet– Storage: Fibre Channel (or SAS or SATA)– Clustering: Infiniband
• This results in three different networks– Three different adapters for each system or server– Three different fabrics, cables and switches– Three different skill sets and tools– Three different management facilities
Slide 109 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
I/O Consolidation in the Data Center
FC HBA IB HCA Ethernet NIC
FC IB Ethernet
Processor Memory
SAN IPC LAN
HBA HCA NIC
Unified Data Center FabricSAN, IPC, LAN Information Flow Requirements
Slide 110 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
I/O Consolidation in the Data Center
FC HBA IB HCA Ethernet NIC
FC IB Ethernet
Processor Memory
Unified Data Center Fabric
Slide 111 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
I/O Consolidation Benefits
– Adaptor: NIC for Ethernet/IP, HCA for InfiniBand, Converged Network Adaptor (CNA) for FCoE
– Customer Benefit: Fewer NIC’s, HBA’s and cables, lower CapEx, OpEx
CNA
CNA
FC HBA
FC HBA
IB HCA
FC Traffic
FC Traffic
IB Traffic
iSCSIorInfiniBand orFCoEIB HCA IB Traffic
NIC Ethernet Traffic
NIC Ethernet Traffic
Slide 112 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The Drive for I/O Consolidation
• Multicore – Multisocket CPUs• Server Virtualization software (Hypervisor)• High demand for I/O bandwidth• Reductions in cables, power and cooling,
therefore reducing OpEx/CapEx• Limited number of interfaces for Blade
Servers• Consolidated Input into Unified Fabric
Slide 113 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
I/O Consolidation in the Data Center
FC HBA IB HCA Ethernet NIC
FC IB Ethernet
Processor Memory
Unified Data Center Fabric
Slide 114 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Ethernet FC IB
CNA
CNA
CNA
I/O Consolidation in the Data Center
FC HBA IB HCA Ethernet NIC
Unified Data Center Fabric
Converged Network Adapter (CNA)
Processor Memory
Slide 115 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Ethernet FC IB
CNA
CNA
CNA
I/O Consolidation in the Data Center
FC HBA IB HCA Ethernet NIC
Converged Enhanced Ethernet – 10 GE
Converged Network Adapter (CNA)
Processor Memory
Slide 116 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Separate Interfaces and Networks
Slide 117 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Ethernet as a “Fat Pipe”
• One barrier to using Ethernet as the basis for a converged network has been the limited bandwidth that Ethernet has historically provided
• As Ethernet bandwidth increases, more traffic can be carried via fewer physical links– With the advent of 10 gigabit Ethernet (10 GBE), the available
bandwidth now offers the potential to consolidate all of the traffic types over the same link
Slide 118 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Converged Enhanced Ethernet
• Why not use a single “converged” network?– Fewer adapters, cables and switches– Especially important in the data center or blade servers
Slide 119 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Benefits of a Converged Network Approach
• The benefits of a converged network are:– A server now has a single adapter replacing multiple
different types of adapters that were required when a different type of network was used for each class of application traffic.
– The datacenter has a single network Instead of two, or three, different networks. This means a single network to install, manage and maintain.
– The number of cables and connections is dramatically reduced.
Slide 120 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Levels
Slide 121 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP & Fibre Channel in SANs
IP Network FC NetworkIntegrated IP & FC Networks
Slide 122 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
IP Storage Protocol Options
Ethernet
IP
TCP
iSCSI
Ethernet
IP
TCP
FCP FC-4
Ethernet
IP
TCP
FC 0, 1, 2
FCP FC-4
Standard SCSI Command Set
Operating System
Applications
iSCSI iFCP FCIP
Slide 123 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
FC & IP Storage Protocol Options
Ethernet Phy
FC Base T
Ethernet Phy
Ethernet MAC
FCoE
FC 2
FCP FC-4
FC FC Base T FCoE
FC 2
FCP FC-4
FC 0FC 1
FC 2
FCP FC3
FCP FC-4
Standard SCSI
FCP FC3
FC 0FC 1
FCP FC3
Slide 124 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The Death of Fibre Channel or iSCSI?
iSCSI lives!FC lives!
Slide 125 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
The Resurrection of Fibre Channel?
Slide 126 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel DriversEthernet Drivers
Operating System
Fibre Channel DriversEthernet Drivers
Operating System
PCIe
Fibr
e C
hann
el
Ethe
rnet
10G
bEE
10G
bEE
Link
Fibr
e C
hann
el
PCIe
4GH
BA
4GH
BA
Link
PCIe
Ethe
rnet
10G
bE
10G
bE
Link
FCoE Enabler: Converged Network Adapter
LAN - NIC CNASAN - HBA
Slide 127 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel
Host
FC HBA
FC
SCSI
Storage
FC SA
SCSI
FC ServicesLogin, Zone, Name
Server, etc. FCFC
Moving
Routing - Switching
Slide 128 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Over Ethernet (FCoE)
Host
FC HBA
FC
SCSI
Storage
FC SA
SCSI
FC ServicesLogin, Zone, Name
Server, etc. FCCEERouting - Switching
Moving
MPS FCF
Slide 129 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Host
FC HBA
FC
SCSI
Storage
FC SA
SCSI
FC ServicesLogin, Zone, Name
Server, etc.Ethernet Ethernet
Moving
Routing - Switching
FCoE for Storage?
MPS FCF MPSFCF
Slide 130 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Host
FC HBA
FC
SCSI
Storage
FC SA
SCSI
FC ServicesLogin, Zone, Name
Server, etc.Ethernet Ethernet
Moving
Routing - Switching
FCoE Switches
FCoE
Slide 131 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Host
FC HBA
FC
SCSI
Storage
FC SA
SCSIFC Services
Ethernet Ethernet
Moving
Ethernet
Login, Zone, Name Server, etc.
How about a FC software alternative
Why FC services in an expensive FC switch?
Slide 132 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Host
iSCSI HBA
iSCSI
SCSI
Storage
iSCSI SA
SCSIiSNS ServicesLogin, Zone, Name
Server, etc.
Ethernet Ethernet
IP Routing (optional)
Moving
Ethernet
Why FC Switch? – Why not Server model
Slide 133 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What is FCoE?
• Fibre Channel over Ethernet (FCoE) is the transport of encapsulated Fibre Channel frames over “lossless” Ethernet– Ethernet provides the physical interface
• Lossless Ethernet NICs or Converged Network Adapters (CNAs)• Lossless Ethernet switches and/or Fibre Channel switches make up
the “Fabric”– Fibre Channel provides the transport protocol
• Fibre Channel frame content is delivered in the Ethernet frames– Most environments will be a mixture of FCoE and FC devices.
• FCoE end devices and switches and FC end devices and switches
Slide 134 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
FCoE Has “Virtual” Ports
• FCoE ports do not meet the strict definitions in the Fibre Channel standards– They do not have a Fibre Channel Physical interface– They emulate the behavior of a native Fibre Channel
port• Because of this, FCoE ports are referred to a
“virtual” ports– Virtual Node Port, or VN_Port– Virtual Fabric Port, or VF_Port– Virtual Expansion Port, or VE_Port
Slide 135 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fibre Channel Roadmap
Slide 136 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
General FCoE Objectives
• Seamlessly and transparently replace the Fibre Channel physical interface with Ethernet– FCoE devices and Fibre Channel devices can communicate with
one another without awareness of the underlying physical link
• No changes to protocol mappings, information units, initialization, Fibre Channel services, etc.
• Develop a mapping that can be implemented totally in software– It should be possible to implement FCoE functions totally within
a software driver
• Enable high-performance implementations through the use of hardware assists
Slide 137 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What’s Different About FCoE?
• There are existing storage mappings to Ethernet– Internet SCSI (iSCSI)– Internet FCP (iFCP)– Fibre Channel over IP (FCIP)
• All are based on the use of TCP/IP– TCP/IP adds complexity and overhead– Argument is that TCP/IP is not required in a local network
• TCP/IP is still required for the WAN or long-haul
• FCoE bypasses TCP/IP for efficiency and simplicity– Assumes a reliable, lossless Ethernet
Slide 138 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Protocol “Stack” Comparison
Slide 139 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Why Maintain Fibre Channel Content?
• Why not get rid of Fibre Channel altogether and use something else, such as iSCSI?– iSCSI has made inroads into storage– iSCSI is often used where Fibre Channel is not already in use
• Significant install base of Fibre Channel today– Fibre Channel is a proven technology– Customers don’t want to “rip and replace”– Fibre Channel supports protocols other than SCSI
• e.g., FICON, FC-SATA etc.
• Fibre Channel will likely continue to provide the highest performance for the data center
Slide 140 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Rationale for FCoE
• FCoE preserves existing FC infrastructure– FCoE can blend seamlessly into an existing FC environment
• No need to “rip and replace”– FCoE is an evolutionary step– New FCoE enabled servers can communicate with existing FC
storage
• FCoE leverages Ethernet technology– Ethernet is ubiquitous – its everywhere– Ethernet is mature and well understood– As Ethernet speeds increase FCoE can seamlessly take
advantage of the increases– Ethernet should provide cost-effective transport with
performance (almost?) equivalent to native Fibre Channel
Slide 141 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
FCoE Configurations
• Many different FCoE configurations are possible, but all probably boil down to a few basic variations– Point-to-Point Ethernet– Point-to-Point Ethernet to Fibre Channel– FCoE devices directly connected to FCoE Forwarder (FCF)– FCoE devices connected to FCoE Forwarder (FCF) via
intervening Lossless Ethernet switches– Hybrid configurations consisting of:
• FCoE devices and switches• Fibre Channel devices and switches• Standard Ethernet devices and switches
Slide 142 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Hybrid Configuration: Example 1
• Most FCoE will be deployed into existing environments– FC hosts, fabric and storage– Ethernet NICs, fabric and Network Attached Storage
• FCoE hosts and switches will co-exist with existing devices– New FCoE enabled devices can access legacy storage– Legacy FC hosts can access new FCoE enabled storage
Slide 143 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Today’s Main Issue
• Is Ethernet convergence the right answer?• What happened to the separate networks
argument?• Are we ready politically and operationally
– Who handles the network now?• Economics and existing vendors will help
drive this decision• YES to convergence (over time)
The New Ethernet
Slide 145 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Fundamental I/O Consolidation Technologies
• PCI-Express– I/O Bandwidth implements multi-lane bit rate
aggregation, • 2.5 & 5.0 Gbps• 1x, 2x,4x, 8x, 16x, 32x
• Gigabit Ethernet– I/O Bandwidth single-lane bit rate
• Evolved from a shared media (CSMA/CD) to a serial full duplex point-to-point network
• 10 Gbps & 100 Gbps• 1x• SFP, SFP+, SFP+ Cu
Slide 146 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
What does Ethernet need?
• Losslessness• Why are frames dropped?
– Frame Errors– Collisions (Old Ethernet)– Congestion
• PAUSE, FC Credit & Priority Flow Control (PFC)
Slide 147 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Ethernet Flow Control
• Ethernet is connectionless– No concept of a virtual circuit– On bit error Ethernet throws frame away– Errors are infrequent
• Ethernet frame loss can also occur due to buffer unavailability too!– Ethernet “brute force” technique of high bit rate and
high switch memory has limits– TCP will retry, with a Positive Acknowledgement and
Retransmission (PAR) mechanism– But how much TCP is too much TCP?
Slide 148 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Priority-Base Flow Control
Receivebuffers
Link-level flow control per priority• 8 priority levels • Traffic paused based on priority level• Protects from transient congestion• Enables Loss-Less Ethernet fabric for latency sensitive applications like FCoE and HPC
Only one priority queue is paused
Transmitqueues
Pause
8 VirtualLanes
Slide 149 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Enhanced Transmission Selection
t1 t2 t3 t40
1
2
3
4
5
6
7
8
9
10
EmailHTTPHPCFCoE
For effective fabric consolidation, important traffic like storage and HPC needto be assigned high priorities and guaranteed bandwidths.
• To improve network efficiency 802.1Qaz allows lower priority traffic to use unused bandwidth from high priority queues.
Assigned Bandw idth0
1
2
3
4
5
6
7
8
9
10
Example:Email = best effortHTTP = 1Gb minHPC = 3Gb minFCoE = 5Gb min
Slide 150 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Link Layer Congestion Management
Slide 151 © Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information
Shortest Path Bridging
• An incremental advance to MSTP• Uses link state protocol (IS-IS) to share learned topologies between switches• Enables learning of shortest paths as the fabric changes
Ensures forward and reverse paths are aligned
IEEE – TRILL Internet draft is similar(Transparent Interconnect over Lots of Links )
End