Understanding NPIV and the Performance of Channels With ZLinux

8/11/2019 Understanding NPIV and the Performance of Channels With ZLinux

1/51

Understanding NPIV and the Performanceof Channels with zLinux

Dr. Steve Guendert, Brocade Communications


2/51

Abstract

This session will be two parts. The first part will discuss

how mainframe I/O channels perform using QDIO in a

zLinux environment. The functionality of QDIO will be

explained and how the zLinux I/O works on the modern

day System z. The second part of the presentation willdiscuss Node Port ID Virtualization (NPIV) and how a

System z10 can make use of NPIV technology to virtualize

its connectivity to storage, and realize TCO savings in the

process.


3/51

Agenda

Background/introduction

FCP Channels on the mainframe

FCP channel performance

Challenges of access to distributed storage NPIV


4/51

Key References

S. Guendert. Understanding the Performance and Management Implications of FICON/FCPProtocol Intermix Mode (PIM). Proceedings of the 2008 CMG. Dec 2008.

I. Adlung, G. Banzhaf et al. FCP For the IBM eServer zSeries Systems: Access To DistributedStorage. IBM Journal of Research and Development. 46 No.4/5, 487-502 (2002).

J. Srikrishnan, S. Amann, et al. Sharing FCP Adapters Through Virtualization. IBM Journal ofResearch and Development. 51 No. , 103-117 (2007).

J. Entwistle. IBM System z10 FICON Express8 FCP Channel Performance Report. IBM 2009.

American National Standards Institute. Information Technology-Fibre Channel Framing andSignaling (FC-FS). ANSI INCITS 373-2003.

G. Bahnzhaf, R. Friedrich, et al. Host Based Access Control for zSeries FCP Channels,z/Journal 3 No.4, 99-103 (2005)

S. Guendert. The IBM System z9, FICON/FCP Intermix, and Node Port ID Virtualization (NPIV).NASPA Technical Support. July 2006, 13-16.

G. Schulz. Resilient Storage Networks. pp78-83. Elsevier Digital Press. Burlington, MA 2004.

S. Kipp, H. Johnson, and S. Guendert. Consolidation Drives Virtualization in Storage Networks.z/Journal. December 2006, 40-44.

S. Kipp. H. Johnson, and S. Guendert. New Virtualization Techniques in Storage Networking:Fibre Channel Improves Utilization and Scalability. z/Journal, February 2007, 40-46


5/51

Background/Intro: Linux on System z

Linux on System z was ten years

old in 2009

Virtualization is a key component to

address ITs requirement to control

costs yet meet business needs with

flexible systems System z Integrated Facility for

Linux (IFL) leverages existing

assets and is dedicated to runningLinux workloads while containingsoftware costs

Linux on System z allows you toleverage your highly available,reliable and scalable infrastructurealong with all of the powerfulmainframe capabilities

Your Linux administrators nowsimply administer Linux on a BigServer


6/51

Background-Intro: Protocol Intermix

FCP channel support introduced in May 2002

What is PIM?

Why PIM?


7/51

FCP channels on the mainframe


8/51

FICON and FCP Mode

A FICON channel in Fibre Channel Protocol mode (CHPID type FCP)

can access FCP devices through a single Fibre Channel switch or

multiple switches to a SCSI device

The FCP support enables z/VM, z/VSE, and Linux on System z toaccess industry-standard SCSI devices. For disk applications, these

FCP storage devices use Fixed Block (512-byte) sectors instead of

Extended Count Key Data (ECKD) format.

FICON Express8, FICON Express4, FICON Express2, and FICONExpress channels in FCP mode provide full fabric attachment of SCSI

devices to the operating system images, using the Fibre Channel

Protocol, and provide point-to-point attachment of SCSI devices.


9/51

FICON and FCP Mode (Continued)

The FCP channel full fabric support enables switches and

directors to be supported between the System z server

and SCSI device, which means many hops through a

storage area network (SAN).

FICON channels in FCP mode use the Queued Direct

Input/Output (QDIO) architecture for communication with

the operating system.


10/51

FCP channels

Classic zSeries operating systems such as z/OS and

z/VM were designed for use only with storage controllers

that support the I/O protocols defined by the

z/Architecture.

This changed with the advent of Linux for zSeries since its

storage I/O component is oriented toward SCSI protocols.

This necessitated adding specific support to Linux for

zSeries to enable it to function in a CCW based zSeries

I/O environment.


11/51

FCP channels and QDIO

FICON channels in FCP mode use the Queued Direct

Input/Output (QDIO) architecture for communication with

the operating system.

This is derived from the QDIO architecture initially defined

for OSA Express and for Hipersockets communications.

FCP channels do not use control devices.

Instead, data devices representing QDIO queue pairs are

defined:

Request queue

Response queue


12/51

QDIO queue pair basics

Each of these QDIO queue pairs represent a

communication path between an operating system and the

FCP channel.

FCP requests are sent by the operating system to the FCP

channel via the request queue.

Response queue is used to pass complete indications and

unsolicited status indications to the operating system.

Use of Sense CCWs for channel programs


13/51

QDIO and hardware definitions

HCD/IOCP is used to define the FCP channel type and QDIO

data devices.

There is no definition requirement for fibre channel storage

controllers and devices, nor switches and directors.

These devices are addressed using World Wide Names(WWNs), Fibre Channel Identifiers (IDs) and Logical Unit

Numbers (LUNs).

These addresses are configured on an operating system level

They are passed to the FCP channel together with thecorresponding Fibre Channel I/O or service request via a logical

QDIO device (queue).


14/51

I/O definit ion comparisonClassical System z I/O definitions System z FCP (SCSI) I/O definitions

Device Number

Host Program

IOCP

FICON/ESCON Channel

Director

Director

Control UnitDevice

LINK CUADD

CHPID Switch

Device Number

WWPN

LUN

FCP

SAN

FCP Channel

Host Program

IOCP

Device

Controller

CHPID

IO Device

Add ress

QDIO Device

queues

Device number definitions identify

the communication path to the FCP

channel and QDIO device queue.

WWPN (8 byte)-

World Wide Port Name is used to

query the controller port connectionin the fabric.

LUN (8 Byte)

Logical Unit Number

Identifies the device


15/51

FCP channel overview

Reference: J. Srikrishnan, S. Amann, et al. Sharing FCP Adapters Through Virtualization. IBM Journal of Research and Development. 51 No. , 103-117 (2007).


16/51

FCP CHANNELPERFORMANCE

Graphics in this section from IBM performance documentation


17/51

FICON Express 8 Performance Start I/Os


18/51


19/51


20/51


21/51


22/51


23/51

Channel Path Activity

All FICON Express8 channels provide Channel Path Activity information,which includes FICON processor and bus utilization, and MBytes read and

written. If z/OS is running in a separate LPAR from Linux on System z LPAR,

z/OS can report FCP channel usage in the Linux LPAR


24/51

Performance notes

FICON Express8 channels operating as FCP channels

yield significant performance benefits

Little channel I/O performance difference between Linux

native vs VM guest

Little difference whether QIO assist is on or off

Available performance metrics tools:

IOSTAT

zFCP Device driver Additional detailed info:

Running Linux on IBM System z9and IBM eServerzSeries

under z/VM (SG24-6311).


25/51

CHANNEL SHARING ANDMANAGEMENT CHALLENGES

Part 2


26/51

FCP channel and device sharing

An FCP channel can be shared between multiple Linuxoperating systems, each running in a logical partition or as aguest operating system under z/VM.

To access the FCP channel, each operating system needs itsown QDIO queue pair defined as a data device on an FCPchannel in the HCD/IOCP.

These devices are internal software constructs and have norelation to physical devices outside of the adapter.

These QDIO devices are also referred to as subchannels.


27/51

FCP channel and device sharing

An FCP channel can be shared between multiple Linux operating systems, each running in a logical partition or as aguest operating system under z/VM.

To access the FCP channel, each operating system needs its own QDIO queue pair defined as a data device on anFCP channel in the HCD/IOCP.

These devices are internal software constructs and have no relation to physical devices outside of the adapter.

These QDIO devices are also referred to as subchannels.

The host operating system uses these subchannels as vehicles

to establish conduits to the FCP environment. Each subchannel represents a virtual FCP adapter that can be

assigned to an operating system running either natively in anLPAR or as a guest OS under z/VM.


28/51

FCP channel and device sharing Initially, support was for up to 240 z/Architecture defined

subchannels. Currently each FCP channel can support up to 480

subchannels/QDIO queue pairs.

Each FCP channel can be shared among 480 operating systeminstances, with the caveat of a maximum of 252 guests perLPAR).

Host operating systems sharing access to an FCP channel canestablish a total of up to 2048 concurrent connections to up to 512different remote fibre channel ports associated with fibre channelcontrollers.

Total number of concurrent connections to end devices, identifiedby logical unit numbers (LUNs) must not exceed 4096.

WAT keeps track of it all


29/51

FCP channel sharing

When NPIV is not implemented, multiple Linux images

share an FCP channel and all of the Linux images have

connectivity to all of the devices connected to the FCP

fabric.

Since the Linux images all share the same FCP channel,

they all use the same WWPN to enter the fabric.

Makes them indistinguishable from each other within the

fabric.

Hence the use of zoning in switches and LUN-masking incontrollers cannot be effective.

Creates a management challenge


30/51

Management challenge

When sharing an FCP adapter, System z must ensure the

OS images sharing System z resources have the same

level of protection and isolation as if each OS was running

on its own dedicated server.

For accessing storage devices via a shared host adapter,

this means that the same level of access protection must

be achieved as if each OS was using its own dedicated IO

adapter.

FCP LUN Access Control (pre System z9)

NPIV


31/51

Integrating System z using z/OS, zLinuxand Node Port ID Virtualization (NPIV)


32/51

zSeries/System z server virtualization

zSeries/System z support of zLinux

Mainframe expanded to address open system applications

Linux promoted as alternative to Unix

Mainframe operating system virtualization benefits

Availability, serviceability, scalability, flexibility

Initial zSeries limits

FCP requests are serialized by the operating system

FCP header does not provide image address

FICON SB2 header provides additional addressing

Channel ports are underutilized

Resulting cost/performance benefit is not competitive


33/51

FCP Device sharing

Without using NPIV, an FCP channel prevents logical units

from being opened by more than one Linux image at a

time.

Access is on a first come, first served basis.

This prevents problems with concurrent access from Linux

images that are sharing the same FCP channel (same

WWPN).

This usage serialization means that one Linux image can

block other Linux images from accessing the data on one ormore logical units, unless the sharing images (z/VM guests)

are well behaved and not in contention.


34/51

FCP channel/device sharing-summary

Different host operating systems sharing access to a single FCP

channel may access the same fibre channel port via this channel.

While multiple operating systems can concurrently access the same

remote fibre channel port via a single FCP channel, fibre channel

devices (identified by their LUNs) can only be serially re-used.

In order for two or more unique operating system instances to share

concurrent access to a single fibre channel or SCSI device (LUN),

each of these operating systems must access this device through a

different FCP channel.

Should two or more unique operating system instances attempt toshare concurrent access to a single fibre channel or SCSI device

(LUN) over the same FCP channel, a LUN sharing conflict will occur,

resulting in errors.


35/51

With and without NPIV

Without NPIV With NPIV

Reference: J. Srikrishnan, S. Amann, et al. Sharing FCP Adapters Through Virtualization. IBM Journal of Research and Development. 51 No. , 103-117 (2007).


36/51

NPIV

NPIV is designed to allow the sharing of a single physical

FCP channel among operating system images, whether it

be in logical partitions (LPARs) or as z/VM guests in virtual

machines.

This is accomplished by assigning a unique World Wide

Port Name (WWPN) to each operating system connected

to the FCP channel.

Each OS then appears to have its own distinct WWPN in a

SAN environment, enabling traffic separation via zoningand LUN masking.


37/51

The road to NPIV-

Alternatives looked at by IBM included:

FC process associators

Hunt groups/multicasting

Identifying the OS image at the upper level protocol layer

Emulating sub fabrics Finally settled on NPIV


38/51

Server Consolidation-NPIV

N_Port Identifier Virtualization (NPIV)

Mainframe world: unique to System z9 and System z10

zLinux on System z9/10 in an LPAR

Guest of z/VM v 4.4, 5.1 and later

N_Port becomes virtualized Supports multiple images behind a single N_Port

N_Port requests more than one FCID FLOGI provides first address

FDISC provides additional addresses

All FCIDs associated with one physical port

OS

Applications

Fabric Login

Fabric Discover

Fibre Channel Address

Fibre Channel Address


39/51

Node Port ID Virtualization (NPIV)

Allows each operating system sharing an FCP channel to beassigned a unique virtual world wide port name (WWPN). Used for both device level access control in a storage controller

(LUN masking) and for switch level access control on a fibrechannel director/switch (zoning).

A single, physical FCP channel can be assigned to multipleWWPNs and appear as multiple channels to the externalstorage network.

The virtualized FC Node Port IDs allow a physical fibre channelport to appear as multiple, distinct ports.

IO transactions are separately identified, managed, transmitted,and processed just as if each OS image had its own uniquephysical N port.


40/51

Using ELS to assign N_Port IDs

FC standard defines a set of services used to establish

communications parameters, each of which is called an

extended link service (ELS).

An ELS consists of a request sent by an N_Port and a response

returned by a recipient port. One ELS, called a fabric login (FLOGI), is sent from an N_Port

to its attached fabric port (F_Port) in the adjacent switch to

request the assignment of an N_Port ID.


41/51

Using ELS to assign N_Port IDs: FLOGI

The FLOGI request is the first frame sent from an N_Port to its

adjacent switch.

The purpose of the FLOGI ELS is to enable the switch and the N_Port

to exchange initialization parameters

Includes unique identifiers known as worldwide port names (WWPNs)

Allows the fabric to assign an N_Port ID to the N_Port.

The switch responds with the N_Port assigned to the requesting

N_Port.

Because the N_Port that sends the FLOGI request does not yet have

an N_Port ID, it sets the S_ID in the FLOGI request to zero. The switch responds with a FLOGI-accept response that contains the

assigned N_Port ID.

The HBA uses this assigned N_Port ID as the S_ID when sending

subsequent frames.


42/51

FLOGI (Contd)

The N_Port ID assigned to a given N_Port may change each

time the N_Port is reinitialized and performs the FLOGI ELS,

but the WWPN of the N_Port does not change.

This allows the fabric to more effectively manage N_Port ID

assignments. Provides for persistent and repeatable recognition of the identity

of an N_Port (WWPN) regardless of the physical fabric port it is

attached to.

N_Ports become associated with a specific OS image

The WWPN can be used to identify the owning OS and the access

privileges it requires.


43/51

Need to request multiple N_Port IDs:FDISC

Fabric Discovery (FDISC) is another ELS

Original purpose is to verify an existing login with the fabric is

still valid.

The FDISC was always sent with a non-zero S_ID (the

presumed S_ID of the sender). This made it possible to obtain additional N_Port IDs by an

extension of the FDISC ELS.

An unlimited number of additional N_Port IDs could be obtained

by using the following protocol:


44/51

The FLOGI request is sent (with the S_ID set to zero and the WWPN

set to that of the first OS image requiring an N_Port ID) to request the

first N_Port ID.

The fabric assigns the first N_Port ID to the N_Port, and that N_Port ID

is used by the first OS image.

The FDISC request is sent (with the S_ID set to zero and the WWPN

set to that of the next OS image requiring an N_Port ID) to request the

next N_Port ID.

The fabric assignes the next N_Port ID to the N_Port, and that N_Port

ID is used by the next OS image.

Need to request multiple N_Port IDs:FDISC(2)


45/51

FDISC (3)

The FDISC request can be repeated until all N_Port IDs have

been assigned to the N_Port.

It may also be executed in parallel in order to decrease the time

taken to obtain all N_Port IDs.

This protocol for obtaining the additional N_Port IDs has beenincorporated into the FC standards and is referred to as NPIV.


46/51

Domain Area AL (Port)

1 byte 1 byte 1 byte

FC-FS 24 bit fabric addressing Destination ID (D_ID)

Domain Port @ Virtual Addr.

1 byte 1 byte 1 byte

FICON Express2, Express4 and Express 8 adapters now support NPIV

CU Link @

Identifies the Switch Number

Up to 239 Switch Numbers

Identifies the Switch Port

Up to 240 ports per domain

AL_PA, assigned during LIP,

Low AL_PA, high Priority

Switch 00 - FF

System z N-port ID Virtualization


47/51

Linux Partition

System z10

Line CardA Simplified SchematicLinux using FCP on a System z10 with NPIV

Linux A

Linux B

Linux C

Linux D

One

FCP

Channel

for

many

Linux

images

A

B

C

D

D C B A

Lots of

Parallelism

Much better I/O

bandwidth

utilization

per path

200 - 800 MBpsper port


48/51

NPIV on the System z

FCP Driver for System z

Same CHPIDs as used for FICON

VM

Linux

VM

Linux

VM

Linux

All to the same

Director Port

Domain 7A

FLOGI

FCID (24 bit port ID)7A 63 00

FDISC

FCID + 1

FDISC

FCID + 2

Etc. for up to 256 FCIDs per channel path

7A 63 01

7A 63 02

One System z server port can have up to 255 NP-IDs

IBM has told us it wants this expandable to thousands

System z N-port ID Virtualization-summary

System z

After the first N_Port FLOGIs,

up to 255 FDISCs will acquire

the other N_Port_IDs


49/51

NPIV summary

NPIV allows multiple zLinux servers to share a single

fibre channel port

Maximizes asset utilization

Open systems server ROT is 10 MB/second

4 Gbps link should support 40 zLinux servers from a bandwidthperspective

NPIV is an industry standard


50/51

Standards and NPIV

FC-FS

Describes FDISC use to allocate additional N_Port_IDs Section 12.3.2.41

NV_Ports are treated like any other port Exception is they use FDISC instead of FLOGI

FC-GS-4

Describes Permanent Port Name and Get Permanent Port Name command

Based on the N_Port ID (G_PPN_ID)

The PPN may be the F_Port Name

FC-LS Documents the responses to NV_Port related ELSs

FDISC, FLOGI and FLOGO

Reference 03-338v1


51/51

More Standards on NPIV

FC-DA

Profiles the process of acquiring additional N_Port_IDs

Clause 4.9

FC-MI-2

Profiles how the fabric handles NPIV requests

New Service Parameters are defined in 03-323v1

Name Server Objects in 7.3.2.2 and 7.3.2.3

Understanding NPIV and the Performance of Channels With ZLinux

Documents

Transcript of Understanding NPIV and the Performance of Channels With ZLinux