Hilda Training May 10, 2012 16 GbFC. Agenda Afternoon May 11, 2012 Fibre Channel Review Putting it...

Hilda TrainingMay 10, 2012

16 GbFC

Agenda

• Afternoon May 11, 2012 • Fibre Channel Review

• Putting it into perspective

• Morning May 12, 2012• Motivation for 16Gb FC• 16Gb FC Performance Advancement• Advancing the Technology• Efficiency• End to End Protection• QLE2600 Features• Competitors• QLogic FC/FCoE Target Reference Kit

• Lab

May 10 & 11, 2012QLogic Confidential! DO NOT Distribute!

Class Scope

• Fibre Channel supports several Upper Layer Protocols• Qlogic Supports

• FCP SCSI • FC-VI • SB-4 (FICON)

• SCSI is by far the predominate protocol supported by FC and the remainder of this presentation will focus on FCP SCSI


Computer System

OS

Fibre Channel Review


SCSI Initiator Driver

File System

SCSI Target HW

SCSI BU

SLUN

ApplicationOriginally SCSI was the technology that physically connected a hard drive in a computer system.

The computer accessed the hard drive through an interface known as the SCSI Initiator

The SCSI Initiator was connected to one or more SCSI devices know as SCSI Targets through a physical cable know as the SCSI Bus

SCSI Initiator HW

Computer System

OS



SCSI Initiator Driver

File System

SCSI Target

LUN

ApplicationFibre Channel allows the physical hard drive to be removed from the system and remotely accessed

The computer accesses the hard drive through the same SCSI Initiator interface as with original SCSI

The SCSI Initiator is connected to one or more SCSI Targets through the Fibre Channel

Fibre Channel

Fibre Channel

FC Fabric


• FC-0 Link Level Protocols• The physical interface (FC-0) consists of

transmission media, • transmitters, and • receivers and their interfaces.


FC-0 FC-0 Physical Interface


• FC-1 Link Level Protocols• The transmission protocol used by layer FC-0

• Serial encoding,decoding• 16Gb Fibre Channel transmits

information using a 64B/66B transmission code

• Error control


FC-1


FC-1 Link Level Protocols


• FC-2 Transport Layer: • Consists of three sub layers

• FC-2 Virtual Sub LayerProvides functions and facilities for use by

FC-4 level Classes of service• Frame content construction and

analysis • Sequence disassembly and

reassembly• Exchange management• Name Identifiers

• FC-2 Multiplexer Sub Layer• Specifies the routing of frames

between a VN_Port and a Specific Physical Layer’s Link Level

• FC-2 Physical Layer

Specifies functions for • Frame transmission and

reception • Buffer-to-buffer flow control• Clock synchronization by use of

Primitive SignalsMay 10 & 11, 2012QLogic Confidential! DO NOT Distribute!

FC-1



Physical Node (PN_PortFC-2P

FC-2V Virtual Node (VN_Port)

Multiplexer (VN_Port - to - PN_Port )FC-2M


• FC-3 Common Services• Provides a set of services that are common

across multiple Nx_Ports of a node. • Includes protocols for Basic Link Services and

Extended Link Services


FC-1


Common Services


FC-3





• FC-4 Fibre Channel Protocol • Maps the lower levels of the Fibre Channel to

an Upper Level Protocol (e.g. SCSI)


FC-1


Common Services


FC-3




FCPFC-4


• SCSI – • Standard for physically connecting and

transferring data between computers and peripheral devices


FC-1


Common Services


FC-3




FCPFC-4

Small Computer System Interface SCSI

HBA

Server



FC FabricN_Port

Storage Device

N_Port

FC supports a fabric topology

Node Node

End points that are the source or destination of data are called

Nodes

Nodes’ external access is through Node Ports (N_Ports)

Initiator

Target

HBA

F_Port

Server

F_Port



FC FabricN_Port

Storage Device

N_PortNode Node

N_Ports Login to the Fabric through the F_Ports with FLOGI

In a Fabric Topology Each N_Port is connected to a

Fabric Port (F_Port) through dual conductor cable

FLOGI establishes a transmit and receive connection between the N_Port and the F_Port

Initiator

Target

The Receive side of an FC Port provides memory space as Receive Buffers to catch incoming data The Receive side of an FC Port provides memory space as Receive Buffers to catch incoming data

N_Port

Receive Buffer

F_Port



Receive Buffer

Buffer Credit

Buffer Credit

Receive Buffer

Buffer Credit

R_RDY

Receive Buffer

Receive Buffer

Buffer CreditBuffer Credit

FC Flow Control is accomplished using Buffer Credits

When an N_Port FLOGIs, the N_Port and F_Port exchange Buffer Credits An FC Port will extend a Buffer Credit equal to the number of Receive Buffers it provides

Receive Buffer

Buffer Credit

R_RDY is an Ordered Set which does not require a buffer to be received

When a port transmits data the receiving port’s Receive Buffer fills and the transmitting port’s Buffer Credit decrements

If the receiving port provides multiple Receive Buffers, the transmitting port can continue to send data until it has exhausted its Buffer Credits

The receiving port handles the received buffer and sends an R_RDY so the sending port can increment its Buffer Credit

An FC Port will extend a Buffer Credit equal to the number of Receive Buffers it provides When an N_Port FLOGIs, the N_Port and F_Port exchange Buffer Credits When a port transmits data the receiving port’s Receive Buffer fills and the transmitting port’s Buffer Credit decrements The receiving port handles the received buffer and sends an R_RDY so the sending port can increment its Buffer Credit

Name ServerFabric Controller

HBA

F_Port

Server

F_Port

Fibre Channel Review mod


FC FabricN_Port

Storage Device

N_PortNode Node

In a Fabric Topology When an Initiator N_Port has logged

into the fabric it uses the Name Server to discover the visible Target N_Ports

Register w/Name Server RSCNGID_FF

The Initiator Registers with the fabric’s Name Server

CT_IU (N_Port List)

The Name Server then notifies the registered nodes of the new node with a Registered State Change Notification

(RSCN)

The Initiator then issues a Get Identification FC Features (GID_FF)

request to the Name Server to obtain a list of features for the visible N_Ports

The Name Server then sends the Initiator a Common Transport

Information Unit (CT_IU) with a list of visible N_Ports and their features,

including whether or not an N_Port is a Target

SCR Request

The Initiator issues a State Change Registration Request (SCR) to the

Fabric Controller to register for State Change Notification

Initiator

Target

HBA

F_Port

Server

F_Port



FC FabricN_Port

Storage Device

N_PortNode Node

In a Fabric Topology When an Initiator N_Port has obtained

a list of Target Nodes

For each Target Port ID returned in the CT_IU:1. The Initiator Logs into the Target N_Port and the FCP SCSI process to establish a IT_nexus2. The Initiator issues a REPORT LUNS command 3. Then the initiator issues for each LUN identified by the REPORT LUNS command an INQUIRY to identify the peripheral device

type of each LUN4. Then the Initiator issues another INQUIRY command to obtain the LUN’s VPD containing the logical unit’s Worldwide_Name 5. The Initiator has now established an ITL_Nexus with the Target and its LUNs

LUN 0 LUN 1

REPORT LUNSLUN ListINQUIRY INQUIRY (VPD)Peripheral Device TypeLUN Worldwide NamesPRLILS_ACCPLOGILS_ACC

Initiator

Target

16Gb FC Service Classes

• The QLE2600 16Gb FC Adapter supports Service Classes 2 and 3 • Service Class 2:

• Connectionless class of service • Confirmation of delivery• Notification of frame non-deliverability

• Service Class 3: (Datagram)• Connectionless class of service• No confirmation of delivery• No notification of frame non-deliverability• If a frame can’t be delivered or processed it is discarded without notification


IT_Nexus

PRLI

Buffer Credit Cnt 0Buffer Credit Cnt 0Buffer Credit Cnt 1

Buffer Credit 1

Fabric Login session

Port Login session

Fabric Login session

FLOGIFLOGI



N-Port

Exchange

F-Port F-Port

Sequencing

Encoding frame frame

PLOGI

Fabric R_RDYR_RDY

Flow control

Framing

FC-4

FC-2 P

FC-1

FC-0 FC-0

FC-4

FC-2 V

FC-2 P

FC-1

N-Port

Exchange

Sequencing

Decoding

Flow control

Framing

Exchange Information Unit

Information Unit Information Unit

MUXFC-2 M MUX FC-2 M

Initiator Target

Class 2 ACK

FLOGI ReplyLS_ACC

FLOGI ReplyLS_ACC

Buffer Credit Cnt 1

Buffer Credit 1

FCP SCSI

Command Block

FC-2 V

SCSISCSI

FCP SCSI

Command Block

SCSI SCSI

Hilda TrainingMay 11, 2012

16 GbFC

Agenda: Hilda Training May 11, 2012

• Motivation for 16GbFC• 16Gb FC Performance• Under Laying Technologies• QLE2600 Features• Comparing QLE2600 with Emulex LPe16002• Comparing QLE2600 with Brocade 1860• QLogic 16Gb FC Target• LAB


Hilda Training May 11, 2012


•Market Conditions for 16GbFC•Deficiencies of 8Gb FC in the

Eco System•Advantages of 16Gb FC in the

Eco SystemMotivation for

16Gb FC

Market Conditions for 16GbFC

Market Drivers

• Server virtualization• Increasing server

workloads• Applications growth• 12-Core processors• SSDs• PCIe 3.0

Applications

• High-end backup• Disaster recovery• High-end databases• Fabric tiers• Big Data• Cloud deployments

Benefits

• Higher performance• Conserves PCIe slots• Improved SAN I/O

density• Increased VM I/O

channels per port• Lower power

QLogic Confidential! DO NOT Distribute! May 10 & 11, 2012

8GFCStorage Adapter

Deficiencies of 8Gb FC in the Eco System


Hypervisor

8GFC Adapters

Port 0

FC Storage

8GFC Adapters

Storage I/O Path

Legend Port 1

VMware Virtual HDD

VM VM VM VM VM VM VM8Gb FC Adapters: Deficiencies

In this example the system is configured for two dual port 8Gb FC adapters

There are 16 VMs each requiring 200MBps

Four 8Gb FC ports are required for the cumulative bandwidth needs of the VMs.

The 8Gb FC adapters * Consume two PCIe slots! * Both adapters require 2 cables (4 cables total) * SAN I/O spread across 4 ports * Both adapters are drawing power

VM

8GFCStorage Adapter

8GFCPort

8GFCPort

8GFCPort

8GFCPort

VM VM VM VM VM VM VMVM

FC Switch

Advantages of 16Gb FC in the Eco System


Port 0

16GFC Adapters

Storage I/O Path

Legend

Port 1

VMware Virtual HDD

16Gb FC Adapters: Solutions

16 Virtual Machines each require 200MBps

Only two 16GFC ports are required for the cumulative bandwidth needs of the VMs.

In this example the system is configured for one16GFC adapter

The 16GFC adapter: * Increases bandwidth by 100% * Consumes only one PCIe slot! * Reduces cabling by half * Increases SAN I/O density by 100% * Increased VM I/O Channels * Only one adapter drawing power

FC Switch

VM VM VM VM VM VM VMVM VM VM VM VM VM VM VMVM

Hypervisor

16GFC Adapters

16GFCStorage Adapter

16GFCPort

16GFCPort

FC Storage



•16Gb FC vs 10Gb FCoE•16Gb FC vs 8Gb FC•16Gb FC IOPS16Gb FC

Performance Advancement

Performance: 16Gb FC to FCoE


16Gb FC

TheoreticalMAX throughput for a

10GbE FCoE port is

1.2GBps

33%10GbEFCoE


16Gb FC port is

1.6GBps

MAX throughput for 16Gb FC is 33% higher than

10GbE FCoE

Performance: 16Gb FC to 8Gb FC


16Gb FC

TheoreticalMAX throughput for an

8Gb FC port is

801MBps

2X8GFC


16Gb FC port is

1.6GBps

MAX throughput for 16Gb FC is 2X higher than 8GFC

QLogic’s8Gb FC’s Stellar

Performancesurpassed

QLogics’s 16Gb FCDoubles

Performance

500,000IOPS2500

Performance: 16Gb FC Doubles IOPS


8Gb FC 250K IOPS

8Gb FC 250K IOPS

2600

FC Port

FCPort

1,000,000IOPS

16Gb FC500K IOPS

Per Port

16Gb FC500K IOPS

Per Port



•Increased Line Rate•Increasing efficiency of

encoding•Increased Buffer to Buffer

Credits•NPIV QoS: QLogic Only!•Increased number of NPIV

vPorts supported •Increased Number of

Exchanges supported•Increased Number of Login

supported •PCIe Gen3

•Increased number of Request/Response Queue Pairs

•Increased Speed•Dynamic Power Management•Auto Configure Lane Configuration

16Gb FC Advancing the

Technology

Changed Under laying Technology between 8GbFC and 16GbFC

• FC – 1: Increased Line Rate• FC – 1: Increasing efficiency of encoding• FC – 2P: Increased Buffer to Buffer Credits• FC – 2P: NPIV QoS: QLogic Only!• FC – 2M: Increased number of NPIV vPorts supported • FC – 2V: Increased Number of Exchanges supported• FC – 3 Increased Number of Login supported • PCIe Gen3

o Increased number of Request/Response Queue Pairso Increased Speedo Dynamic Power Managemento Auto Configure Lane Configuration


16Gb FC 1: Increased Line Rate

• 16Gb FC is called 16Gb by virtue of the fact that it delivers twice the throughput of 8Gb FC

• The actual Line Rate of 16Gb FC is 14.025Gbps• The natural value for 16Gb FC was arrived at by the following formula

• 16Gb FC line rate = 2 * 8.5Gbps * 8b/10b * 66b/64b = 14.025Gbps• 8Gb FC line rate = 8.5Gbps• 8Gb FC encoding overhead 8b/10b• 16Gbps encoding overhead 64b/66b

o (Inverted for this calculation - 66b/64b or 1.03%) • Natural line rate is 14.025Gbps


16Gb FC 1: Increased Line Rate – Throughput Doubles

• 16Gb FC Throughput = (line rate * encoding) • 8Gb FC Throughput = 8.5Gbps * (8b/10b) = 6.8Gbps = 850MBps• 16Gb FC Throughput = 14.025Gbps * (64b/66b) = 13.6Gbps = 1.7GBps

• Therefore: • 16Gb FC Throughput = 2 X 8Gb FC Throughput

• In bits per second: 2 * 6.8Gbps = 13.6Gbps • In Bytes per second: 2 * 850MBps = 1.7GBps


Why Encode data?

• Encoding data on a serial bus allows for the clock to be embedded into the serial data stream • This eliminates the necessity of a separate clock line and is the technique which

allows the data to flow through a single conductor


Doesn’t the data transition? Can’t it carry the clock?

• True data on any serial bus can have long periods in which the majority of the bits can be one polarity or the other • This means that the natural bit transitions on the serial bus can’t be relied on to carry

the clock.


Why would I care about data encoding?

• The standard objective for a new generation of a bus technology is to double the previous generation’s performance • This sounds simple

• Just double the clock frequency

• But at some point physics gets in the way • Some existing technology can’t handle the higher frequencies • These technologies are often required for backwards compatibility

• This time around there was an alternative to doubling the clock• 8Gb FC used 8b/10b encoding

• This is a 20% overhead

• 16Gb FC uses 64b/66b encoding• This has a 3% overhead


What is Encoded data?


Embedded Clock

Data Timing

Encoded DataGuarantees Transitions

DataData DataData DataDataData

Actual DataInsufficientTransitions

DataData DataData DataDataData Data

Receiver ClockSynchronized

with embedded clock

T0 T1 T2 T3 T4 T5 T6 T7 T8

Buffer Credits

• The number of Buffer Credits a receiving port should provide depends on the Link Speed and the Distance from the sending port

• The Maximum Buffer Credits allocated for FC ports are based on the physical capacity of 20Km of cable to hold the frames.• This is to account for the transit of an FC frame across 10Km of cable and its

R_RDY return• In this case the transmitting port should be allowed to burst an amount of frames

equal to the cables carrying capacity, plus one buffer at the receiving port


Why do Buffer Credits matter?

• An Exchange can only transfer an Information Unit if it has a buffer credit with the receiving port.• A frame can not be sent between FC two ports unless the sending port has at least

one buffer credit with the receiving port• If a sending port has more than one buffer credit with the receiving port then it can

continue sending frames until it runs out of buffer credits• Buffer Credits are allocated when an N_Port logins on to a fabric port with FLOGI

• With NPIV the amount of simultaneous Exchange activity could be quite large

• The receive buffers traded in FC flow control • Are implemented in the ASIC• Are of a finite number• Are shared by all of the vPorts on an N_Port


If FC had to wait for every R_RDY before sending the next frame, performance across long cables would be very poor

N_Port F_Port

Buffer Credits


Receive Buffer

Max Receive Buffers Credit is based on the number of frames that physically

occupy the cable

This means that the receive port can extend Buffer Credits equivalent to the cables capacity To improve performance across long cables,(such as 10Km) the storage capacity of the cable itself is utilized

frame frame frame frame…

frame frame frame…R_RDYframe

Buffer Credits

86

Buffer Credits

86

If FC had to wait for every R_RDY before sending the next frame, performance across long cables would be very poor To improve performance across long cables,(such as 10Km) the storage capacity of the cable itself is utilized

Buffer Credits

• Credit = (Round Trip Time + Receive Port Processing)/Frame Transmission Time = wave length * bits in frame• The Round Trip Time of light through an 10Km Optical Cable is 100usec• Receive Port Processing time is 8.92usec (Assumes 7 times receive time)• Frame Transmission time is 1.27

• Buffer Credits for 10 Km @ 14.25 Gbps = (100usec + 8.92usec)/1.27usec = 85.5


Buffer Credits

• QLogics’ QLE2600’s default Buffer to Buffer Credit allocation is 10 receive buffers

• Even though the commonly accepted specification for receive buffer quantities is based on 10Km of cable,

• In practical terms this only applies to a very small set of actual cables. • In a real data center the length of cable between FC_Ports is probably less than the length of a single frame o • This means that a receiver will receive the beginning of a frame before the transmitter has sent the end.

• This renders the storage capacity of the cable as zero.

• The consideration of the default number of allocated receive buffers isn’t based on 10Km. • The factors to consider are the processing time of the receiver and the buffering depth necessary for it to service

incoming traffic. • If the receive processing time to service a single receive buffer is assumed to be 7 times the frame receive time

then for a single • Then a port can be serviced by rotating 8 receive buffers on line

• However, allocating only eight receive buffers would impact performance• This is because the propagation time of the R_RDY returning the Buffer Credits prolongs the turn around for

buffer reuse• To accommodate for R_RDY it is necessary to add additional buffers to compensate for the transit time• QLogic’s default value of 10 gives a 12.5 % pad for Receive Buffer allocation



Time to fill

Buffer 1Time to process Buffer 1

Sending System

Receiving System

Buffer 1Again

available for use

R_RDYFor Buffer 1

Issued

B1 B2 B3 B4 B5 B6 B7 B8

R_RDY received

Transmit delayed

waiting for R_RDY

R_RDY propagation

Frame transmission

begins

B1 Transmit skewed

Waiting for R_RDY


Time to fill

Buffer 1Time to process Buffer 1

Sending System

Receiving System

Buffer 1Again

available for use

R_RDYFor Buffer 1

Issued

B1 B2 B3 B4 B5 B6 B7 B8

R_RDY received

R_RDY propagation

Frame transmission

begins

Transmit Unaffected

byR_RDY

Transmit timing

UnaffectedWaiting for

R_RDY

B9 B10

HBA

Server

F_Port

N_Port Interface Virtualization (NPIV)


N_Port

Node vPorts Login to the Fabric through the F_Ports with FDISC

N_Port ID VirtualizationAllows a single physical N_Port

to appear as if it were multiple N_Ports

FDISC establishes a transmit and receive connection between the

vPort and the F_Port

vPort

FDISCvPort

vPort

FDIS ReplyLS_ACC FC Fabric

Once an NPIV vPort is established with FDISC. Its behavior is the same as an N_Port

F_Port

F_Port

F_Port

VM 3

VM 2

VM 1HBA

Server

F_Port

N_Port Interface Virtualization (NPIV)


N_Port

Node vPorts can be assigined to VMs

vPorts connect to Storage Devices

Using PLOGI & PRLI

The vPorts appear to the VM as SCSI devices

SCSISCSI

SCSI

N_Port

Storage Device

Storage Device

N_Port

N_Port

PLOGI

PLOGI

PLOGI

PLOGI Storage Device

vPort

vPort

vPort

PRLI

LUN

PRLI

PRLILUN

PRLI

FC Fabric

LUN

Fibre Channel Overview


FC-1


Common Services


FC-3




FCP SCSIFC-4

Small Computer System Interface SCSI

Common Services

Virtual Node (VN_Port)

FCP SCSI

Small Computer System Interface

Common Services

Virtual Node (VN_Port)

FCP SCSI

Small Computer System Interface

vPort vPort vPort

NPIV Expansion


8GFCPort

QLogic’s 8Gb FCNPIV

Supports 64

vPorts

NPIV vPorts

WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN




NPIV Expansion


QLogic’s 16Gb FC

NPIVSupports

255 vPorts

16GFCPort

NPIV vPorts

WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN WWN








16Gb FC Parameters: NPIV V_Ports

• The total number of NPIV vPorts supported by an adapter is a fixed number • This is because the memory, where the necessary tables are implemented, is

onboard the adapter card and is of a limited amount

• This value determines the number of targets an initiator HBA port can present to a hypervisor as storage for its VMs

• The Total number of vPorts supported by QLogic is 255• The Total number of vPorts supported by Emulex is 255• The Total number of vPorts supported by Brocade is 255


16Gb FC Features: NPIV QoS


• The QLogic 2600 16Gb FC adapter provides NPIV QoS• NPIV QoS can be configured based on

• Priority• Bandwidth Percentage • Bandwidth Speed

• This feature ensures the user ensure high-quality performance for critical applications that require preferential delivery

• QLogic is the only vendor with this feature• Neither Emulex nor Brocade have support for this feature

Exchange

• An Exchange transfers IUs as one or more segments• In FC the segments of an IU are called a Sequence

• An Exchange can have multiple Sequences open at a time• An Exchange can be unidirectional or bidirectional

• Bidirectional Exchanges CAN ONLY BE HALF DUPLEX • Only one direction at a time

• An Exchange changes direction by passing the Initiative• The owner of the Initiative can transmit• The Initiative of different Exchanges are independent of each other• Different Exchanges can be sending and receiving at the same time.

• An Exchange can only be sending or receiving a single sequence at a time• Different Exchanges can be multiplexed• The bandwidth on the link is the agregation of the


16Gb FC Parameters: Exchanges

• The total number of Exchanges supported by an adapter is a fixed number • This is because the necessary information to maintain the exchange is stored in a

table entry in memory• The memory allocated to this table determines the Total number of Logins an

adapter can support

• If the Total number of Exchanges were currently being used• A new SCSI command sequence would block until an entry in the Exchange table

were freed and a new Exchange could be established

• To accommodate the increase number of vPorts supported by 16GbFC the number of Exchanges supported by 16GbFC HBAs has been increased


Information Units

• If a LUN determines an Initiator wants to initiate a write operation • It calls the Receive Data-Out • The target FCP_Port transmits a data descriptor IU containing the FCP_XFER_RDY IU

indicating the portion of the data is to be transferred

• The initiator FCP_Port then • Transmits a solicited data IU to the target FCP_Port containing the FCP_DATA IU

payload requested by the FCP_XFER_RDY IU.

• Data delivery requests • Containing FCP_XFER_RDY IU and returning FCP_DATA IU payloads • Continue until the data transfer requested by the command is complete. • One FCP_DATA IU shall follow each FCP_XFER_RDY IU.


Mapping SCSI to FC: Exchanges

• The SCSI Protocol transfers data between Initiators and Targets through sequences of Command Data Blocks (CDB)

• A transfer of CDB sequences in the context of an Exchange• An Exchange transfers Information Units (IU) between an Initiator and Target

• The SCSI CDBs between an Initiator and a Target through an Exchange as IUs• The first SCSI Command Block in a command sequence between an Initiator and

Target causes the Initiator’s FCP to establish an Exchange• Subsequent Command Blocks in the SCSI command sequence continue to utilize

the established Exchange until SCSI command sequence is complete

• There can be multiple Exchanges established between an Initiator and a Target


Mapping SCSI to FC: Exchanges

• A SCSI Application Client uses Send SCSI Command • To request a SCSI initiator port to send a command

Send SCSI Command (IN ( I_T_L_Q Nexus, CDB, Task Attribute,

[Data-In Buffer Size], [Data-Out Buffer],

[Data-Out Buffer Size], [CRN],

[Command Priority], [First Burst Enabled] ))

• A LUN’s device server uses the Receive Data-Out• To request that a SCSI target port receive data

• Receive Data-Out (IN ( I_T_L_Q Nexus, Application Client Buffer Offset,

Request Byte Count, Device Server Buffer ))


The Life Cycle of an Exchange


SCSI Application Client Send SCSI Command <Write Data Buffer>

Initiator FCP_Port function Target FCP_Port functionIU IUExchange

Command requestFCP_CMND

The Exchange is created with receipt of FCP_CMD IU

The Initiator passes Initiative to the Exchange Responder

FCP_CMND Prepare Data-Out transfer buffer]

First data delivery requestFCP_XFER_RDYThe Target LUN responds with a receive buffer location and size & passes it back to Initiator

FCP_XFER_RDYFirst Data-Out Action

FCP_DATA The Initiator sends SCSI data buffer as FCP_DATA IU which probably exceeding frame size and are segmented into multi frame sequences

FCP_DATA

Data delivery request

Receive Data-Out transfer buffer]

FCP_XFER_RDYThe Initiator continues to exchange FCP_XFER_RDY and FCP_DATA IUs until the data transfer is complete

FCP_XFER_RDYData-Out Action

FCP_DATA When the last IU of the Write Transfer completes the Exchange terminates

FCP_DATA Receive Data-Out transfer buffer]

Duration of Exchange

FC-2V

LUN

Exchange


SCSI FCP-4 SCSI FC-3

SCSI Application

Client

FC-2V FC-2M

FC-2P FC-1

SCSI DataOut

IUFCP

DATA

Exchange

Sequence

Segment 1 Segment 2Segment 3Segment 4Segment 5

Encoded

Check Buffer Credit

Frame

Segment 4Segment 5Segment 3Segment 4Segment 5

Segment 2Segment 3Segment 4Segment 5

Frame

SCSIFCP-4 SCSIFC-3FC-2M

FC-2PFC-1

Decoded

Frame

ReturnBufferCredit

Frame

Exchange

SequenceIU

FCP DATA

SCSI DataOut

FC Fabric Ports

R_RDY R_RDY

Application Data

Segment 1

Frame

Segment 2Segment 1Segment 1 Segment 2Segment 3

Segment 1 Segment 2Segment 3Segment 4Segment 5

Segment 1 Segment 2Segment 3Segment 4

Segment 5

Application

Databuffer Initiator Target

Login

• When Fabric Login has ended successfully, • The buffer-to-buffer Credit values are initialized

• The Login procedure shall• Follow the Exchange and Sequence management rules• The buffer-to-buffer flow control rules, • The end-to-end Flow control rules• The Nx_Port shall transmit the FLOGI in a new Exchange

• The Payload of FLOGI contains • The Service Parameters of the Nx_Port• A 64-bit N_Port_Name of the Nx_Port• A 64-bit Node_Name


Login

• Possible responses to an Nx_Port’s attempt to FLOGI to a Fabric:• LS_ACC reply Sequence

• This is the normal response to a Fabric Login request

• F_BSY • The Fabric is busy • The Nx_Port may retry the FLOGI again later

1. F_RJT Sequence1. The Fabric has rejected the FLOGI request

2. The reason code contained in the Payload determines the Nx_Port’s action

3. If the reason code is "Class not supported"

1. the Nx_Port may originate a FLOGI in a different class.

4. If the reason code is "Invalid S_ID"

1. the Nx_Port may originate a FLOGI with a different S_ID:


Login

• Possible responses to an Nx_Port’s attempt to FLOGI to a Fabric: (continued)

1. LS_RJT Sequence 1. The reason code contained in the Payload determines the Nx_Port’s action

2. The Nx_Port may alter the Service Parameters based on the reason code and originate a new FLOGI

• No Response • May indicate a delivery error, e.g., error on the physical transport• The Nx_Port may originate a new FLOGI after recovery

• If the received N_Port_Name is equal to its N_Port_Name• Then the Nx_Port is connected to itself

o This case is undefined and the FLOGI is discarded


Mapping SCSI to FC: Logins

• Before a SCSI Initiator can communicate with a SCSI Target it must first establish a logical connection between Initiator and Target• In SCSI parlance this Logical Connection is called Initiator-Target Nexus (IT_Nexus)

• FC accomplishes the establishment of a SCSI IT_Nexus using the Port Login (PLOGI) and the Process Login (PRLI)


Login

• N_Port Login accomplishes the following functions:a) It provides each Nx_Port with the other Nx_Port's operatin characteristics

a) N_Port_Name and Node_Name.b) If a Fabric is not present, it assigns the native N_Port_ID for both Nx_Ports

c) If initializes the Nx_Port end-to-end Credit.

• N_Port Login between two Nx_Ports is complete • When each Nx_Port has received the Service Parameters of the other Nx_Port

• An Nx_Port must Login with each Nx_Port with which it intends to communicate • This includes reserved and well-known address identifiers since they are considered

to be N_Ports (see FC-FS-2)


Login

• The N_Port Common Service Parameters during N_Port Login are• Buffer-to-buffer Credit• Common Features

• Continuously increasing relative offset• Multiple N_Port_ID Support (N_Port supports NPIV) • Random relative offset• Virtual Fabrics bit• Valid Vendor Version Level• Multiple N_Port_ID Assignment (F_Port supports NPIV)• N_Port/F_Port • Buffer-to-Buffer Receive Data_Field Size• Nx_Port Total Concurrent Sequences• Relative offset by Info Category

• R_A_TOV - indicates support for the short value of R_T_TOV

• E_D_TOV Value (1 millisecond increments/ 1 nanosecond increments)

• Explanation of the common features are beyond the time constraints of this lecture.May 10 & 11, 2012QLogic Confidential! DO NOT Distribute!

16Gb FC Parameters: Total Logins

• The total number of Logins supported by an FC physical port • This includes all PLOGIs an adapter’s physical port can simultaneously be logged

into• This total is divided between the physical N_Port and the NPIV vPorts

• The reason for the limitation is that • There needs to be control structure for each Login

• For optimal performance these control structures should be located in fast access ram

• Each Login control structure is stored in a table located in fast ram on the ASIC• The memory allocated to this table determines the Total number of Logins an

adapter supports with the high performance memory• If overflow were to happen, (Which is actually an unlikely case)

• Memory from the host memory can also be used


PCIe Gen 3Configured for 4 lanes

PCIe Gen 2Configured for 4 Lanes

PCIe Support


PCIe Slot

QLogic’s 16Gb FC Adapter

IsDesigned for

PCIe Gen3It Also Supports

PCIe Gen 2 And

PCIe Gen 3

The adapter automatically is configured for the correct lane configuration for the PCIe generation

PCIe Gen 1Configured for 8 Lanes

PCIe Gen 3 Dynamic Power Management

Extend existing PCIe device PM to provide active device

power management

sub-states

• Support for up to 256 sub-states per function

• Maximum power allocation advertised per sub-state in Watts

• Contiguously numbered sub-states 0 to N sub-state w/o

• Gaps• Each successive sub-state

reports maximum power• allocation less than or equal to

prior sub-state

Software permitted to dynamically configure a function

to

operate at any sub-state in any sequence

• Function must operate at or below the maximum power allocation corresponding to the selected sub-state

New DPA capability only for endpoints

• DPA capability to enable software to discover and actively manage endpoint function power usage

QLogic Confidential! DO NOT Distribute! May 10 & 11, 2012

PCIe x8 G2 vs PCIe x4 G3

• PCIe Gen 2 provides 4.0 Gigabits/second/Lane/direction bandwidth.• Bandwidth for 8 lanes = 4Gb * 8 lanes * 2 (BiDi) = 64Gb • PCIe Protocol Overhead = ~15%

• Usable PCIe Gen2 Payload Bandwidth with 8 lanes = 54Gb

• PCIe Gen 3 provides 8.0 Gigabits/second/Lane/direction bandwidth.• Bandwidth for 4 lanes = 8Gb * 8 lanes * 2 (BiDi) = 64Gb • PCIe Protocol Overhead = ~15%

• Usable PCIe Gen3 Payload Bandwidth with 4 lanes = 54Gb




•Theoretical Calculation Caveat•Defining Overhead•Encoding •Framing•Fibre Channel Efficiency

Calculations•Data Rate Calculation Theory•Data Rate Calculations

16Gb FC Efficiency

Theoretical Calculation Caveat

Caveats: • All calculations are theoretical• In a real environment the traffic will probably be a mix of two or more protocol types.

For example: FC - class 2 and class 3• It is not within the scope of this document to predict the ratio of one protocol type to

another • Therefore all calculations are based on a single protocol type


Defining Overhead

• Payload = Line Rate - Overhead• Overhead Calculation

• The overhead for a protocol frame consists of Two Components

oEncoding OverheadoFrame Header Overhead


Encoding

• Encoding Overhead:• The quantity of transmitted data bits used to encode the data stream

• Examples: • 8b/10b Encoding – Each eight bits of data requires ten bits of encoding to be

transmitted

o This results in 20% Encoding Overhead • 64b/66b Encoding – Each 64 bits of data requires 66 bits of encoding to be transmitted

o This results in 3% Encoding Overhead • Line Rate:

• The rate at which encoded data bits are transmitted

• Throughput: • The rate at which un-encoded data is transmitted• Data Rate = Line Rate – Encoding Overhead


Framing

• Frame Header Overhead• The cumulative amount of header information needed to define protocol• Example: Fibre Channel - Class 2 Frame

• FC SOF + FC Header + Payload + FC CRC + FC EOF + RTW + ACK• FC Frame size = 2168 bytes

• Efficiency• The ratio of Payload Data to Frame Size• Example: Fibre Channel

• Efficiency = 2048 / 2168 = 0.944649


FC SOF 4 bytes

FC Header 24 bytes

FC CRC 4 bytes

RTW24 bytes

Payload Data 2048 bytes

FC EOF 4 bytes

ACK60 bytes

2168

Fibre Channel Efficiency Calculations


Fibre Channel Class 2 Fibre Channel Start Of Frame = 4 bytes Fibre Channel Header = 24 bytes Payload data = 2048 bytes Cyclic Redundancy Check = 4 bytes End Of Frame = 4 bytes Required Transmit Words = 24 bytes ACK = 60 bytesTotal Frame Size = 2168 bytes

2048 2168 0.944649 5.54%Protocol Frame components

Data Payload in Bytes Frame Size

Overhead in Bytes Efficiency

Overhead Percentage

120

Fibre Channel Class 3 Fibre Channel Start Of Frame = 4 bytes Fibre Channel Header = 24 bytes Payload data = 2048 bytes Cyclic Redundancy Check = 4 bytes End Of Frame = 4 bytes Required Transmit Words = 24 bytes Total Frame Size = 2108 bytes

2048 2108 60 0.971537 2.85%

Data Rate Calculation

• Data Rate• The amount of Payload Data a protocol actually delivers

• Data Rate Calculation• Data Rate = Line Rate – Encoding Overhead – Header Overhead


Data Rate Calculations


Line Rate Throughput EfficiencyData Rate in Bytes

Data Rate in bits

Normalized Data RateProtocol

8Gb FC Throughput Class 2 w/ 2048 Payload 8,500,000,000 6,800,000,000 0.944649 6,423,613,200 802,951,650 803MBps8Gb FC Throughput Class 3 w/2048 Payload 8,500,000,000 6,800,000,000 0.971537 6,606,451,600 825,806,450 826MBps8Gb FC Throughput Class 2 w/2112 Payload 8,500,000,000 6,800,000,000 0.945237 6,427,611,600 803,451,450 803MBps

8Gb FC Throughput Class 3 w/2112 Payload 8,500,000,000 6,800,000,000 0.972376 6,612,156,800 826,519,600 827MBps16Gb FC Throughput Class 2 w/2048 Payload 14,025,000,000 13,600,000,000 0.944649 12,847,226,400 1,605,903,300 1.6GBps16Gb FC Throughput Class 3 w/2048 Payload 14,025,000,000 13,600,000,000 0.971537 13,212,903,200 1,651,612,900 1.65GBps16Gb FC Throughput Class 2 w/2112 Payload 14,025,000,000 13,600,000,000 0.945237 12,855,223,200 1,606,902,900 1.6GBps

16Gb FC Throughput Class 3 w/2112 Payload 14,025,000,000 13,600,000,000 0.972376 13,224,313,600 1,653,039,200 1.65GBps



•End to End Data Protection•T10 – PI •Over Lapping Protection

Domains16Gb FC

End to End Protection

End to end T10 data protection

• The ANSI T10 standard defines the Protection Information (PI) • Which provides data integrity checking capability for data written and read across

the SAN fabric to a storage device and back

• End to end protection protects against transmission data corruption and misdirected and out of order writes

• The storage devices used with End to End T10 data protection must• Support T10–PI• Be formatted for 520 byte sectors


T10-PI

• Also Know As T10-DIF• The Data Protection Information is 8 bytes long and is defined as:

• 2 Byte Cyclic Redundancy Check (CRC) generated by the HBA • Protects against transmission data corruption

• 2 Byte Application Tag • Application Defined

• 4 Byte Reference Tag • Equals the lower 32 bits of Logical Block Address from the SCSI command• Protects against out of order and misdirected writes


CRC Application Tag Reference Tag

Overlapping Protection Domains


OPD on the data path was firstintroduced with QLogic’s 4Gb ASICs and HBAs

Data received from the host bus is protected by PCIe CRC.

Before data is put into the frame buffer, frame buffer byte parity iscalculated on the data

Then PCI parity/ECC or PCIe CRC is checked; if itsintegrity is verified, it is stripped

Similarly, when data is put on the FCinterface, FC CRC is calculated on the data

The data is checked againstthe frame buffer parity; if its integrity is verified, it is stripped

OPD is a separate technology from End to end T10 data protection



•Backwards Compatabile•Hilda Features: PCIe•Hilda Features: FC/FCoE•Hilda Firmware Features:

FC/FCoE16Gb FC QLE2600Features

16Gb FC8Gb FC16/8/4 Gb Fibre Channel

16Gb FC Backwards Compatable


PCIe Slot

QLogic’s 16Gb FC Adapter

isBackward

Compatible To

QLogic’s 8Gb FC And

QLogic’s 4Gb FC

The Power Management automatically reduces the adapter’s power consumption in the 8Gb and 4Gb configurations

Hilda Features – PCIe

Category Feature Description ValueComments / Additional

Information

PCIe

Width

PCIe Gen 1 x8

PCIe Gen 2 x8

PCIe Gen 3 x4

Physical Functions

Ports configured as Ethernet

Max Number of PFs per Port = 8 Max one PF for FCoE functionMax one PF for iSCSI function

Max up to 8 NIC functions

Ports configured as 16Gb FC Max one PF for FC function

IO BAR No


Hilda Features – PCIe (Cont.)


Information

Host Bus Virtualization

MSI No

MSI-X Yes 2K vectors per PFs and associated VFs

Request/Response Queue

Pairs

Ports configured as Ethernet 4K per adapter

Ports configured as 16Gb FC 256 per port


Hilda Features – FC/FCoE


Information

FCoE / 16Gb FC Features

Number of ports Initiator/Target 2 Supports simultaneous initiator and target mode functionality

Speed16Gb Xcvr 16Gb/8Gb/4Gb

8Gb Xcvr 8Gb/4Gb

Protocol Support

FCP-3 SCSI Yes

FC-Tape Yes

SB-4 (FICON) Yes

FC Class SupportFCoE Class 3

16Gb FC Class 2 and Class 3

FC TopologyFCoE Switching Fabric

16Gb FC Switching Fabric, FC-AL(8/4), Point-to-Point


Hilda Features – FC/FCoE (Cont.)


Information

FCoE / 16Gb FC Features

B2B Credits 86 (to support 10Km); Default - 10

iiDMA Yes

OoOFR Yes

OPD Yes

FC-SP Authentication Only

T10 DIF (A.K.A. T10-PI) Yes


Hilda Virtualization Features – FC/FCoE (Cont.)


Information

FCoE / 16Gb FC

Virtualization Features

NPIVInitiator 256 per port

Target 512 per port

Virtual FabricsFCoE Yes Implementation depends on the

standards definition

16Gb FC Yes

MQ-IO Yes

• Bandwidth allocation across multiple queues in FC/FCoE• Priority based IO processing across multiple queues in FC/FCoE


Hilda Firmware Features – FC/FCoE (Cont.)


Information

FC Firmware Features

Enhanced Hardware Assisted firmware tracing Yes

DMI Yes Diagnostics

API compatibility 8Gb FC and Schultz

FC-IP No

FCoE / 16Gb FCLogin/Exchange

Features

Number of logins per port Initiator/Target 2K – 16K Beyond 2K uses Host

Memory

Number of exchanges per

portInitiator/Target 2K – 32K

Beyond 2K uses Host Memory


Out of Order Frame Reassembly

• Method and system for processing out of orders frames • United States Patent 7676611 (QLogic is Patent holder)

• A method and system for processing out of order frames received by a host bus adapter is provided. • The method includes,

• determining if a current frame is out of order;• determining if a frame is within a range of transfer for an Exchange; • and creating (or appending if not the first out-of-order frame) an out of order list if the current

frame is a first out of order frame.

• The method also includes, • determining if an entry in an out of order list has a relative offset value of zero; • determining if at least one entry has a relative offset value equal to a total transfer length of an

Exchange; • and determining if every non-zero starting relative offset has a matching entry.

• The method also scans an out of order list and combines a last entry with an entry whose starting point matches the end point of the last entry.




•Emulex LPe16000•Brocade 186016Gb FC

Competitors

QLogic 2600 vs Emulex LPe16002

Point of Comparison Emulex QLogic

Logins: 8192 2048 on chip 16K in Host memory

Exchanges: 8192 2048 on chip 32K in Host memory

Buffer to Buffer Credits: See Note: 86

NPIV IDs ( vPorts) 255 255

NPIV QoS No Yes

Classes of Service 2 & 3 2 & 3

Payload Size 2048 2048

Note: Emulex claims to provide sufficient Buffer to Buffer credits support a 10Km cable


QLogic 2600 vs Brocade 1860

Point of Comparison Emulex QLogic

Logins: 2048 2048 on chip 16K in Host memory

Exchanges: 4096 2048 on chip 32K in Host memory

Buffer to Buffer Credits: 80 86

NPIV IDs ( vPorts) 255 255

NPIV QoS No Yes

Classes of Service 2 & 3 2 & 3

Payload Size 2048 2048 & 2112




•QLogic Free BSD Target Reference Kit

16Gb FC QLogic

FC/FCoE Target Reference Kit

QLogic Free BSD Target Reference Kit

• FC/FCoE Target Reference Driver Kit• Provides a reference driver and target application for Qlogic’s FC HBAs and CNAs

to provide native FC and FCoE target solutions• Is supported in PC-BSD 7.1.1• The Driver Reference Kit emulates One Native FC Target with One LUN of

256Mbytes• The initiator may be either Linux or Windows

• This is a preliminary release and currently supports only QLE83xx FC (QLE26xx) interface


QLogic Free BSD Target Reference Kit


InitiatorQLE2600

TargetQLE2600

FCSwitch



•LAB

16Gb FC LAB

Students will use the QLogic QCC GUI to create an NPIV vPort through a QLogic 2600 HBAStudents will use the QLogic QCC GUI to create an NPIV vPort through a QLogic 2600 HBAStudents will use the QLogic QCC GUI , QCC CLI , & Windows Device Manager to determine the state of the vPort createdStudents will use the QLogic QCC GUI , QCC CLI , & Windows Device Manager to determine the state of the vPort created

Students will Login into the VM utilizing the created vPort and establish and validate the FC connected storage device Students will configure a Virtual Machine to use the created vPort

Students will access the QLogic switch through a user access login and determine the Ports’ visibility to the FC Switch Students will access the QLogic switch through a user access login and determine the Port’s visibility to the FC Switch Students will login to the QLogic Free BSD FC/FCoE Target which is executing the QLogic Target Reference Driver & determine the state of the Targets LUNs and vPorts

QLogic End to End FC LAB


16Gb FC Initiator

N_Port

vPort

vPort

QLogic 8 Gb FC Switch

F_PortF_Port

16Gb FC Target

LUN

N_Port

vPort

vPort LUN

This LAB employees QLogic equipment from End to End

Students will login to the QLogic Free BSD FC/FCoE Target which is executing the QLogic Target Reference Driver & determine the state of the Targets LUNs and vPorts Students will configure a Virtual Machine to use the created vPort Students will Login into the VM utilizing the created vPort and establish and validate the FC connected storage device

Hilda Training May 10, 2012 16 GbFC. Agenda Afternoon May 11, 2012 Fibre Channel Review Putting it...

Documents

Transcript of Hilda Training May 10, 2012 16 GbFC. Agenda Afternoon May 11, 2012 Fibre Channel Review Putting it...