FICON for CCIEstudy May

© 2010 Cisco Systems, Inc. All rights reserved. FICON 1

FICON 101

CCIE SAN Study Group

By Ozden Karakok – May 2011


Objective & Agenda At the end of this session, attendees will be able to answer what is FICON is all about!   FICON Concepts

•  Terminology •  Topologies •  Addressing •  Frames

  FICON Applications

  MDS Requirements •  FICON Manager •  FICON Control Unit Process (FCD)

  Configuring FICON •  SAN-OS •  Mainframe

  SAN-OS 3.0 Update

  Troubleshooting FICON

  Summary


FICON Terminology


IBM Mainframe Systems

 Current IBM mainframe systems: S/390 G5 S/390 G6 zSeries 800 zSeries 890 zSeries 900 zSeries 990 z9 Ent. Class z9 Business C. z10 Ent. Class z10 Business C.

OS/390 VM VSE/ESA MVS

z/OS z/VM Linux


Introduction to FICON FICON Protocol Layer

ULP (Upper Level Protocol)

FC - 4

FC - 3

FC - 2

FC - 1

FC - 0

SCSI – 3/4

Common Services

SCSI— Command Set Mapping (FCP)

Framing Protocol/Flow control

Encode/Decode (8/10)

Physical Variant

FC-SB2/FC-SB3 (FICON)

Single Byte Command Code

Sets

•  FICON is Fibre Channel FC-4 layer protocol

•  It is at the same layer as FCP protocol

•  Fibre Channel –”Single Byte Command Code” mapping protocol is described in FC-SB3, FC-SB4

•  FC-SB2 is an older version. FC-SB3-4 is superset.

Ref: www.t11.org


FICON Terminology

  A Channel is a physical entity/element of the processor, acting as the interface between the CPU and the CU

–  A Channel Image instantiates multiple, logical channels behind a physical channel

  A Channel Path includes the interface adapter, physical fiber pair, and up to two intervening FICON switches/directors between the channel and the CU

  A Channel Path Identifier (CHPID) is a single hexadecimal byte value identifying each channel path to the system


FICON Terminology

 Outboard I/O processors called Control Units (CUs) –  CUs can be stand-alone or integrated in peripheral devices –  CUs are identified by a two-byte hexadecimal CUNUMBER –  A Control Unit Image instantiates multiple, logical CUs behind a physical CU

 A Device Address (aka Unit Address) is a decimal value that selects a specific device on a given CU image


FICON Terminology

Legend

M A I N F R A M E

IBM

N Port

N Port

  A “Channel” is an entity typically mapped to an N-port in a host computer which performs access to I/O devices

Channel

Control Unit (CU)

  A “Control Unit” is the interface to I/O devices such as printer, disks etc.

Control Unit Image

  There can be multiple “Control Unit images” behind each “Control Unit”

Channel Images

  There can be multiple “Channel Images” behind one N port, each of which represent a separate logical channel.

Physical

Logical


FICON Terminology

 A Logical Partition (LPAR) is a subset of mainframe resources, dedicated to implementing a particular programming environment

–  Each LPAR functions independently of other LPARs and is protected against the actions of poorly-behaved LPARs –  Resources such as CHPIDs (Channel Path Identifier) may be shared between LPARs


LPARs can share channels Multiple Image Facility (MIF)

Channel

MIF

CU

CU

N_Port

N_Port

N_Port

MIF

LPAR 1

LPAR 2

LPAR n


FICON Terminology

  An Output Port is the FICON director or switch port that connects to the CU

–  In a switched point-point environment, it can be a single byte hexadecimal value identifying the director/switch port number connected to the CU –  FICON port numbering schemes are proprietary to the switch vendor/model; typically different than what’s used for FC config –  The port number may also be added with the switch domain ID to form a two-byte Output Port value if supported by the CU – In a cascaded environment, Output Port must be a two-byte value derived from the non-Entry Switch


FICON Cascaded Topology Output Ports

Non-entry switch Entry Switch

CH = Initiator

CU = Target

Switch ID = Domain ID Portaddress = int fc X/Y


FICON Terminology

 The Input Output Control Program (IOCP) is a mainframe application that defines the Input Output Control Data Set (IOCDS), which the mainframe uses to define & identify control units, devices, channel paths, channels and their relationships to each other.

  I/O definition file (IODF)

 All of the terms we’ve just covered are required IOCP/IOCDS parameters


Objective & Agenda   FICON Concepts







  Summary


FICON Topologies

  A Point-to-Point (direct connection)   A Switched Point-Point Environment results when there is one

FICON director/switch in the channel path between the channel and CU

  A Cascaded Environment results when two FICON directors/switches are connected in series along the channel path between the channel and CU

  An Entry Switch is the FICON director or switch connected to the channel via the channel path –  In a cascaded environment, the Entry Switch is first switch

encountered in the channel path from the channel to the CU –  The Entry Switch is identified by a single hexadecimal byte equal

to it’s FC domain ID


FICON Topologies (Contd)

  In Point to Point topology, channel and control unit are directly connected. There is no switch between them.

Point to Point

FICON Channel

FICON Control

Unit



  In Single Switch Fabric path configuration, all channels and control units are attached to a single switch

  This is called ‘switched point-to-point’ configuration.

Single Switch Fabric Path Configuration FICON

Channel

FICON Control

Unit

FICON Control

Unit

FICON Control

Unit

FICON Channel



  In Two Switch Fabric path configuration there may be more than 2 switches in the fabric, but only one hop between any pair of channel and control units.

  Multiple ISLs between connected switches are allowed. However, the fabric should perform load balancing on these ISLs.

Two Switch Fabric Path Configuration

FICON Control

Unit

FICON Channel

FICON Control

Unit

FICON Channel

FICON Control

Unit


FICON Addressing

  FICON defines concept of link address.

  Each channel and control unit have one unique link address.

  In older versions, link address used to be single byte and could only be used in single switch environment

  FC-SB3 defines two byte link addresses which allow more than one switch in FICON path.

  Link addresses are mapped to FC-ID as depicted in figure.

Link Address

Port Area

16

Link address

Domain

Static in FICON fabric (usually 0)

Two Byte Link address to FCID mapping (for Cascaded environments)

Single Byte Link address to FCID mapping

Port Area

8

Link address

Domain

Static in FICON fabric (usually 0)


HCD – Point to Point


HCD – Cascaded


Using DM to identify Port Addresses

Notice FICON port numbers are displayed

9513 - 16 ports per slot 9509 - 32 ports per slot









  Summary


FICON Channel Initialization

CH to Switch

CH to CU

Same Well Known Fibre Channel Addresses


FICON Control Unit Initialization

Defined In FC-FS

Defined In FC-GS

Can register both SCSI and FC-SB FC-4 types

port-wwn (vendor) :50:05:07:63:00:c2:04:2b (IBM) node-wwn :50:05:07:63:00:c0:04:2b class :2,3 node-ip-addr :0.0.0.0 ipa :ff ff ff ff ff ff ff ff fc4-types:fc4_features:scsi-fcp fcsb2-ch-cu fcsb2-cu-ch

[snip]


QSA – Query Security Attributes

 Used by Channel to verify Fabric Binding and Insistent Domain IDs is configured on MDS

 Only used on Cascade environment

•  QSA = 0x7E 00 00 00 •  Revision field = 0x00 00 00 01


QSA Accept  QSA Accept sent if Security Features enabled

Bit Meaning 0 Fabric Binding 1   Static DomainID 2   Soft Zoning 3   Hard Zoning 4   SLAP 5   SRP 6   MS CT

Authentication 7-31 Reserved

0000 0011 Bit 0 & 1 are on

7654 3210 BIT numbers


ESA – Exchange Security Attributes

  Used between Switches (ISL) to validate QSA information   Passed during ISL link initialization   Address by and to MDS Fabric Controller 0xFF FF FD   ESA frame command code is 0x32 00 00 00

272.210534 ff.ff.fd -> ff.ff.fd 0xa 0xffff SW_ILS 0x32000000!

272.210594 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!

272.241828 ff.ff.fd -> ff.ff.fd 0xa 0xa SW_ILS SW_ACC (0x32000000)!

272.242168 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!

ESA

Class F Ack

ESA Accept

Class F Ack


RNID – Request Node Id Data

•  Used to determine characteristics of node at other end


RNID Accept

 Accept is broken into two sections:

Common ID Field - shows pWWN and nWWN. Ignored by FICON

Specific ID Data – shows SB-2, FCID, Name, Manufacturer, Model…..


LIRR, RLIR, DRLIR   LIRR (Link Incident Record Registration) used to

register for RLIR (Registered Link Incident Record) - similar to SCR

 RLIR used by N-Ports to send link incidents to registered ports - similar to RSCN

 DRLIR (Distribute Registered Link Incident Records) used to send incidents to entire fabric

 Switch receives DRLIR and send RLIR to registered N-Ports

  LIRR sent to all Switches in path and Control Unit

 Sent to 0xFF FF FA (MDS Management Server)


ELP – Establish Logical Path LPE – Logical Path Established

•  ELP - Sent from Channel to Control Unit to verify optional parameters and verify logical path • LPE – Sent from Control Unit to Channel as verification of logical path and pass optional parameters • LACK – Close exchange


Idles FC Header

4 24 0 - 8160 4 4 Bytes:

Frame Format

FICON Header

32

Data Payload CRC

FC-SB-3 Frame Format

Reserved CH_ID Reserved CU_ID Device Address Reserved

IUI DH_FLGS CCW Number Reserved Token

Command / Data / Status / Control / Link Header

Reserved IU Count DIB Data Count LRC

SB-3 Header

IU Header

DIB Header

Bit 0 7 8 15 16 31 23 24

S O F

E O F

Idles


FICON Read Operation

Command DIB (READ)

Command DIB (RSP)

Data DIB

Data DIB

Status DIB

FICON Read Operation

FCP_READ

FCP_DATA

FCP_DATA

FCP_RSP

FCP Read Operation

FICON Channel

FICON CU

FCP Initiator

FCP Target


FICON Write Operation

Command DIB (WRITE)

Command DIB (RSP)

Data DIB

Data DIB

Status DIB

FICON Write Operation

FCP_WRITE

FCP_DATA

FCP_DATA

FCP_RSP

FCP Write Operation

FICON Channel

FICON CU

FCP Initiator

FCP Target

FCP_XFER_RDY


Flow Control

FICON uses standard FC-2 BB_Credit flow control FICON also uses IU Pacing:

Similar to Fibre Channel EE_Credits IU Pacing credit is sent to the channel by the CU Indicates the maximum number of IUs a channel can send before a command response IU is expected

CU

N_Port

Channel

N_Port F_Port E_Port E_Port F_Port Frames

R_RDYs

Frames

R_RDYs

FC-2 BB_Credit FC-2 BB_Credit

CMR CRR

IU Pacing









  Summary


Linux Support

IBM zSeries supports Linux LPARs: Allows Linux to access ESCON/FICON storage Linux also supports open systems FCP:

Same adapters used for FICON and FCP Different firmware versions

Applications: CPU-intensive workloads Server/storage consolidation NPIV (like MIF for FCP channels)

zSeries server

FC FICON

LINUX FCP

FICON

z/OS FICON FICON

FC

FC FC FC

FC

FC FC


Disaster Recovery Applications

Data replication over FICON: Backup, snapshot, and disaster tolerance Local and remote replication Globally Dispersed Parallel Sysplex (GDPS) w/ IBM XRC IBM PtP VTS GDPS w/ IBM PPRC (runs over FCP) HDS HXRC


Disaster Recovery Applications XRC - eXtended Remote Copy


Channel Extension

 Channel extension is different than distance extension: Distance extension simply means extending FICON over a WAN Channel extension involves protocol translation Channel extenders proxy some commands—e.g. writes—to increase performance over distance MDS does not do channel extension in SAN-OS 3.X









  Summary


MDS FICON Requirements

Feature Explanation Predictable and persistent FCID allocation

Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be predictable and persistent.

FICON also requires uniform “port” byte (last byte in FC-ID) in FICON fabric.

Support for new LIRR, RLIR, and DRLIR ELS

Link Incident Record (LIR) ELS reports link level events in the fabric to the channel N-port.

Examples of link incidents are loss of signal or synchronization, primitive sequence timeout, Invalid primitive sequence, NOS received

Concept similar to RSCN

Report Node Identification (RNID) ELS

RNID is an ELS command, which can have common and specific Node data.

FICON defines a specific node data field format.

Large Buffer to Buffer Credit Allocation

FICON requires 60 B2B credit for 100 Km and 6 B2B credit for 10 Km.

MDS 9000 FICON supports up to 255 B2B Credits per Port + extended buffer credits


MDS FICON Requirements

Feature Explanation Insistent Domain ID Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be

predictable and hence Domain ID to be static in cascaded FICON switches.

In-order Delivery FICON requires in-order delivery for class 2 and class 3 frames.

Fabric Binding and EFMD ELS Fabric Binding feature prevents unauthorized switches joining the fabric based on switch WWNs.

Exchange Fabric Membership Data (EFMD) is used to distribute this information fabric wide.

Security Attributes (QSA and ESA)

FICON requires Fabric Binding and Insistent Domain ids to be enabled in the entire fabric.

Query Security Attribute (QSA) is used by Channels to ensure these features are turned on.

Exchange Security Attribute (ESA) is used by switches to validate these attributes during E port bringup.









  Summary


What is FICON manager?

FICON Manager

Running process that manages functionality required to support FICON in a VSAN, doing the necessary checks for enabling FICON and holding all FICON specific configuration information.


FICON Manager

  FICON Manager is responsible for: -  Providing CMI interface for configuration -  Serialization of FICON config -  Maintain and apply FICON configuration files -  Port Number Allocation and assignment -  Port Address Allocation and assignment -  Consistency Checks during enabling of FICON -  Maintaining port numbers in FICON VSANs -  Interact with ACL Manager for necessary ACLs during port bringup -  Exchanging new ELS During E-port bringup


FICON Manager Role

FCD (CUP)

FC2

Vsan Manager

Port Manager

Others

SNMP

CLI

FICON Manager

Mainframe GUI


FICON Configuration Files

 Up to 16 configuration files per FICON VSAN - Restricted to 8 alphanumeric characters

 Shared lock for file access

 Proprietary IBM format IPL contains the following attributes for each implemented port:

-  Port Address Name -  Host Control -  Block -  Prohibit Dynamic Connectivity Mask

-  ‘show ficon vsan <x>’ will show saved IPL files


Port Number and Port Address   Implemented/Unimplemented Port Number

Refers to any available port in the chassis Reflected in the table to the right

  Installed/Uninstalled Port Only ports participating in FICON VSANs are

considered ‘installed’ i.e. If port 4 is in a FICON VSAN on a 16 port line

card in a chassis, then ports 0-3, 5-31 are considered ‘uninstalled’

  Port number is the physical location on the switch   Port address is:

-  Port name -  Block/Unblock state -  Connectivity attributes -  Same as port number unless port “swapped”

  256 possible port number/addresses for 9509 (8 bits, some unimplemented)

  0xFE and 0xFF reserved   An FCIP tunnel must be explicitly bound to a port

number for FICON - Available port numbers depends on the slot the

IPS blade is located in   A Port Channel must be explicitly bound to a port

number for FICON -  Available port numbers depends on chassis ‘show ficon first-available port-number’ reveals next port-number free for allocation

Slot 91xx 92xx 9506 9509 9513 1 0 - 39 0 - 31 0 - 31 0 - 31 0 - 15 2 32 - 63 32 - 63 32 - 63 16 - 31 3 64 - 95 64 - 95 32 - 47 4 96 - 127 96 - 127 48 - 63 5 -- -- 64 - 79

6 -- -- 80 - 95 7 128 - 159 -- 8 160 - 191 -- 9 192 - 223 96 - 111 10 112 - 127 11 128 - 143 12 144 - 159 13 160 - 175

Max Physical* 40 64 128 224 528

Port Channels 40-55 64-79 128-143 224-249 224 - 249


Using DM to identify Port Addresses

Notice FICON port numbers are displayed

9513 - 16 ports per slot 9509 - 32 ports per slot


FICON States

  Various states for FICON

STATE Meaning

Enabled User has configured FICON on a given VSAN.

Disabled User configures a non-FICON vsan.

Up FICON enabled and all prerequisites met

Down FICON enabled, but one or more prerequisites not satisfied


FICON Prerequisites

 Before FICON is ‘up’, FICON manager must check prerequisite conditions:

  In-order delivery enabled   Fabric Binding   Insistent Domain ID   No conflicting persistent FCIDs

-  Does not allow user configuration for reserved range: D.0.0 thru D.MaxPa.0 where:

D is domain MaxPa is max Port Addr implemented in FICON


FICON Port Initialization

  Blocked/Unblocked State -  A blocked port is configured ‘down’ by Port manager, at request of FICON

manager -  Blocked ports will continue to transmit an Off-Line State (OLS) primitive

sequence. -  Similar to shut/no shut, as blocked port must be reinitialized

- Exception is TE port: - Removal of the VSAN from the VSAN allowed list - Port Channels - Members get port channel configuration (block/unblock)

  Prohibit State -  FICON manager programs ACLs to allow communication between certain

ports and prohibit others -  Similar concept to zoning -  Port remains ‘up’


FICON Port Initialization ELS

 Replies to Query Security Attributes (QSA) -  For Fx, Responsible for sending:

-  Fabric Binding -  Insistent Domain ID -  In-order delivery

 Sends Exchange Security Attributes (ESA) -  Upon bring up of (T)E Ports

 Sends Request Node Identification (RNID) -  Sent after QSA reply


FICON Port Swap Support

 Port Swap feature -  Must be supported for FICON directors -  Enables swapping configuration on two physical ports

-  FICON manager swaps all config except Port # -  Ports left in “shut” state following…’no shut’ required

FICON Configuration

Including Port Address,

Block State, etc.

FICON Configuration

Including Port Address,

Block State, etc.

Physical Port A Physical Port B









  Summary


FICON Control Unit Port (CUP)   CUP protocol provides inband

management of switch from mainframe

  In order to be classified as “FICON director”, CUP protocol needs to supported.

  CUP specification is proprietary to IBM and is available only through special license agreement with them.

  CUP can configure a port to be prohibited from another port, blocked, etc.

  Complimentary to SNMP, CLI, and is serialized through CMI

  FICON Manager holds all configuration and FICON specific information; FCD acts as a client

SANOS 3.0(2) View of FICON IPL File


FICON Control Device (FCD)

 FCD process is responsible for: -  Acquiring persistent pWWN for Control Device

-  Uses Key-Based Chassis Specific (KCS) WWNs -  Uses VSAN based nWWN

-  Acquiring FCID from FPort server -  0xFE is well defined address required -  FCID will be <domain, 0xFE, 00> for Control Device

-  Device allegiance based Serialization of FICON config -  Support for both SB-3 configuration and query commands


FICON Port Configuration

Block Block all communication from a port. In this state port is down and will not send any information.

Prohibit Each port (identified by port address) can block any other port. This is done via ACLs.

Port Address Name

Each port (address) can have a 24 character port address name.

Port Swap Reassigns port address, connectivity attributes and port address name of ports by swapping.

Service Required

This is set when FICON Director determines that some service is required on the port. Port is kept offline. Cleared when required service performed.

Port Statistics The MDS needs to maintain detailed statistics on each port. The statistics are reported based on port numbers and not port addresses


HCD - CUP


FCD Control Device

An example of an FCD Control Device on a FICON vsan:

switch2# show ficon control-device sb3 vsan 1977

Control Unit Image:0x80b9df4 VSAN:1977 CU:0x4dfe00 CUI:0 CUD:0 CURLP:(nil) ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0

Control Unit Image:0x80bcbec VSAN:1978 CU:0x4efe00 CUI:0 CUD:0 CURLP:(nil) ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0

switch2# show flogi database vsan 1977 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- sup-fc0 1977 0x4dfe00 22:00:00:05:30:00:46:e0 27:b9:00:05:30:00:46:df

Total number of flogi = 1.

Domain ID = 77 (0x4d) Well known port = 0xFE

pWWN assigned









  Summary


CLI Configuration of FICON An example of using the ‘setup ficon’ script: switch1# setup ficon --- Ficon Configuration Dialog --- This setup utility will guide you through basic Ficon Configuraton on the system. Press Enter if you want to skip any dialog. Use ctrl-c at anytime to skip all remaining dialogs. Would you like to enter the basic configuration dialog (yes/no)[yes]:yes Enter vsan [1-4093] :3 vsan 3 does not exist, create it? (yes/no)[yes]:yes Enable ficon on this vsan? (yes/no)[yes]:yes The following options required for ficon will be enabled: In-order-delivery Fabric binding Configure domain-id for this ficon vsan (1-239) :3 Would you like to configure ficon in cascaded mode: (yes/no)[no]:yes Configure peer wwn (hh:hh:hh:hh:hh:hh:hh:hh): 20:00:00:0d:ec:0e:5b:80 Configure peer domain (1-239) :4 Would you like to configure additional peers: (yes/no)[no]: Enable SNMP to modify port connectivity parameters? (yes/no)[yes]: Disable Host from modifying port connectivity parameters? (yes/no)[no]: Enable active=saved? (yes/no)[yes]:no

Would you like to configure additional ficon vsans (yes/no)[yes]:no The following configuration will be applied: vsan database vsan 3 in-order-guarantee vsan 3 fcdomain domain 3 static vsan 3 fabric-binding enable fabric-binding database vsan 3 swwn 20:00:00:0d:ec:0e:5b:80 domain 4 fabric-binding activate vsan 3 zone default-zone permit vsan 3 ficon vsan 3 no active equals saved

swwn from remote switch

# show wwn switch


CLI Configuration of FICON A FICON port-channel configuration example: switch1# conf t Enter configuration commands, one per line. End with CNTL/Z. MDS9506(config)# int port-channel 1

Remember that PortChannels need to be assigned FICON port numbers, so in the next command we are determining what the next available FICON port number is.

switch1(config-if)# do show ficon first-available port Port number 128(0x80) is available

Next step is to bind the FICON port number, set the port type to an ISL (E_Port), and set it up so that only VSANs 1 and 10 can trunk across the PortChannel.

switch1(config-if)# ficon portnumber 0x80 switch1(config-if)# switchport mode e switch1(config-if)# switchport trunk allowed vsan 1 switch1(config-if)# switchport trunk allowed vsan add 10

Next we move interfaces fc1/9 and fc2/1 into PortChannel 1

switch1(config-if)# int fc1/9 switch1(config-if)# channel-group 1 force fc1/9 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel,then do "no shutdown" at both ends to bring them up

switch1(config-if)# int fc2/1 switch1(config-if)# channel-group 1 force fc2/1 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel,then do "no shutdown" at both ends to bring them up


CLI Configuration of FICON A FICON port-channel configuration example continued:

Next we no shut interfaces fc1/9, fc2/1, and PortChannel 1.

switch1(config-if)# int fc1/9, fc2/1 switch1(config-if)# no shut switch1(config)# int port-channel 1 switch1(config-if)# no shut

Next we repeat the configuration steps on the switch2.

switch2(config)# int port-channel 1 switch2(config-if)# do show ficon first-available port-number Port number 64(0x40) is available swich2(config-if)# ficon portnumber 0x40 switch2(config-if)# switchport trunk allowed vsan 1 switch2(config-if)# switchport trunk allowed vsan add 10 swich2(config-if)# int fc1/1 switch2(config-if)# channel-group 1 force fc1/1 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring them up

switch2(config-if)# int fc1/9 switch2(config-if)# channel-group 1 force fc1/9 added to port-channel 1 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring them up

switch2(config)# int fc1/1, fc1/9 switch2(config-if)# no shut switch2(config)# interface port-channel 1 switch2(config-if)# no shut switch2(config-if)# end


CLI RLIR MDS-1# show rlir ? erl Show Established Registration List history Show link incident history internal Show RLIR internal recent Show recent link incident statistics Show RLIR Statistics

MDS-1# show rlir internal vsan 3

Internals for VSAN: 3 ------------------------

VSAN state : Active Ficon State : Up Fabric Name : 20:03:00:05:30:00:51:1f R_A_TOV : 10000 ms Local Domain : 0x3(3) Remote Domains : None

MDS-1# show rlir statistics vsan 3

Statistics for VSAN: 3 -------------------------

Number of LIRR received = 0 Number of LIRR ACC sent = 0 Number of LIRR RJT sent = 0 Number of RLIR sent = 0 Number of RLIR ACC received = 0 Number of RLIR RJT received = 0 Number of DRLIR received = 0 Number of DRLIR ACC sent = 0 Number of DRLIR RJT sent = 0 Number of DRLIR sent = 0 Number of DRLIR ACC received = 0 Number of DRLIR RJT received = 0


CLI RNID and Port attributes

MDS-1# show ficon vsan 5 portaddress 0x21 Port Address 33(0x21) is up in vsan 5 Port number is 33(0x21), Interface is fc1/3 Port name is Port is not admin blocked Prohibited port addresses are 250-253,255(0xfa-0xfd,0xff) Admin port mode is FX Port mode is F Peer is type 002064 model 104 manufactured by IBM

MDS-1# show ficon vsan 5 portaddress brief installed ----------------------------------------------------------------------------- Port Port Interface Admin Status Oper FCID Address Number Blocked Mode ----------------------------------------------------------------------------- 0x40 0x40 port-channel 5 off up TE -- 0xfe 0xfe -- off up -- --


CLI FICON File Configuration switch1# show run

ficon vsan 2 file IPL

switch1(config)# ficon vsan 2 switch1(config-ficon)# ? FICON Configuration: active Configure automatic saving of ficon configuration code-page Ficon configuration for codepage on a VSAN do EXEC command end Exit from configure mode exit Exit from this submode file Ficon file configuration on a VSAN host Enable host control of ficon no Negate a command or set its defaults portaddress Ficon configuration for portaddress on a VSAN snmp Enable snmp control of ficon

switch1(config-ficon)# portaddress 1 switch1(config-ficon-portaddr)# ? FICON config for portaddress: block Block a range of portaddress do EXEC command end Exit from configure mode exit Exit from this submode name Give name for a portaddress no Negate a command or set its defaults prohibit Prohibit communication with a range of portaddress


Creating FICON VSAN from DM


Creating FICON VSAN from FM


Creating/Modifying FICON Files from DM


FICON Configuration File (IPL)

Edit IPL file, Block – Prohibit – Name Ports


Creating Port Channel through FM

Determine the ISLs to add to the Port Channel Now select the Port Channel Wizard


Creating Port Channel through FM (cont’d)

Fill in Port Channel Information

Select a Port number for the logical PC

Port Channel is now created


Showing Port Channel through DM

Select Interface->Port Channels

All Port channel information is displayed


Swapping FICON Ports through DM

Select Ports to swap Select FICON->Swap Selected Ports


Swapping FICON Ports through DM (contd)

Notice the swapped ports


FICON VSAN Options

•  Offline SW = Enables ability for Host to disable FICON or offline MDS •  Sync Time = Enables ability for Host to sync MDS time with Host

•  Port Control – Host = Enables ability for Host to Name, Prohibit, Block MDS ports •  Port Control – SNMP = Enables ability for Device Manager to Name, Prohibit, Block MDS ports •  Active=Saved = Enables ability for IPL (ficon running configuration) to be saved automatically and immediately


FICON Load Balancing

 Packets of a flow (SID,DID) or an exchange (SID, DID, OXID) are received in the order they were sent. Some applications can not handle Out Of Order exchanges

 FICON is one such application. In fact FICON uses two unique exchanges, one for the entire outbound exchange (For example a write chain) with an OXID allocated by the CH. The CU sends back a command response with a different OXID representing the inbound exchange


FICON Load Balancing

 The FICON load balancing tool has been enhanced in the Fabric Manager 3.0 release.









  Summary


Configuring Devices on Mainframe Consider the following example and IOCP definition


Configuring Devices on Mainframe

Configuring CHPID 89 online HCD

SDSF TSO Command Input


Configuring Devices on Mainframe Varying devices online

HCD

SDSF TSO Command Input

Vary devices D40 through D74 online. I/O errors occur because the volumes at these addresses are not initialized.

V D40-D74,ONLINE IEF503I UNIT 0D50 I/O ERROR IEF503I UNIT 0D51 I/O ERROR ...

Following command automatically executed based on the range that was varied on in previous command.

D U,,,0D40,00052,L=OZI-Z VARY RANGE DISPLAY IEE457I 10.55.06 UNIT STATUS 198 UNIT TYPE STATUS VOLSER VOLSTATE 0D40 3390 O PPRD40 PRIV/RSDNT 0D41 3390 O PPRD41 PRIV/RSDNT … 0D50 3390 OFFLINE /RSDNT


Displaying Devices on Mainframe

Notice the online state for configured CHPID 86

Notice the availability of devices configured in HCD

HCD

TSO

HCD

HCD

TSO


Displaying Devices on Mainframe Displaying CHPID 86 and 89 Matrix

52 Devices Unit = 3390

Begins at D40

CUP Device Unit = 2032

at well-known 0xFE


Displaying Devices on Mainframe Displaying Path Matrix for Device D40

CHPID with corresponding Destination Link Address Entry Link Address









  Summary


New FICON Port Numbering scheme  Port-numbers are user-configurable

 Duplicates allowed -  Port-number 1 can be assigned to fc1/1 and fc2/1

 Assigned on per-slot or logical basis ficon slot <x> assign port-numbers <range> ficon logical-port assign port-numbers <range>

 Default configuration -  No change for 1, 2, 6 and 9 slot chassis -  13 slot chassis will have 11 slots w/ 16 port-numbers each

 HA compatibility


Configuring Port-Numbers

switch1# sh module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 2 24 1/2/4 Gbps FC Module DS-X9124 ok 3 8 IP Storage Services Module DS-X9308-SMIP ok 4 16 1/2 Gbps FC Module DS-X9016 ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 8 4 10 Gbps FC Module DS-X9704 ok

Consider the following 9509 with LC modules in slots 2, 3, 4, and 8

Finding out what is assigned – this example show the defaults for a 9509

switch1# show ficon port-numbers assign ficon slot 1 assign port-numbers 0-31 ficon slot 2 assign port-numbers 32-63 ficon slot 3 assign port-numbers 64-95 ficon slot 4 assign port-numbers 96-127 ficon slot 7 assign port-numbers 128-159 ficon slot 8 assign port-numbers 160-191 ficon slot 9 assign port-numbers 192-223 ficon logical-port assign port-numbers 224-249



switch1# show ficon port-numbers assign ficon slot 1 assign port-numbers 0-31 ficon slot 2 assign port-numbers 32-63 ficon slot 3 assign port-numbers 64-95 ficon slot 4 assign port-numbers 96-127 ficon slot 7 assign port-numbers 128-159 ficon slot 8 assign port-numbers 160-191 ficon slot 9 assign port-numbers 192-223 ficon logical-port assign port-numbers 224-249

switch(config)# no ficon slot 1 assign port-numbers 0-31 switch(config)# no ficon slot 7 assign port-numbers 128-159 switch(config)# no ficon slot 9 assign port-numbers 192-223

Unreserve FICON port-numbers from an unused slot, slots 1, 7, & 9 in this case

Reserve port-numbers for slot 2, a 24 port card

switch(config)# ficon slot 2 assign port-numbers 128-151



switch(config)# do show ficon vsan 2 portaddress brief | inc fc2

0x80 0x80 fc2/1 off not installed -- -- 0x81 0x81 fc2/2 off not installed -- -- 0x82 0x82 fc2/3 off not installed -- -- 0x83 0x83 fc2/4 off not installed -- -- 0x84 0x84 fc2/5 off not installed -- -- 0x85 0x85 fc2/6 off not installed -- -- 0x86 0x86 fc2/7 off not installed -- -- 0x87 0x87 fc2/8 off not installed -- -- 0x88 0x88 fc2/9 off not installed -- -- 0x89 0x89 fc2/10 off not installed -- -- 0x8a 0x8a fc2/11 off not installed -- -- 0x8b 0x8b fc2/12 off not installed -- -- 0x8c 0x8c fc2/13 off not installed -- -- 0x8d 0x8d fc2/14 off not installed -- -- 0x8e 0x8e fc2/15 off not installed -- -- 0x8f 0x8f fc2/16 off not installed -- -- 0x90 0x90 fc2/17 off not installed -- -- 0x91 0x91 fc2/18 off not installed -- -- 0x92 0x92 fc2/19 off not installed -- -- 0x93 0x93 fc2/20 off not installed -- -- 0x94 0x94 fc2/21 off not installed -- -- 0x95 0x95 fc2/22 off not installed -- -- 0x96 0x96 fc2/23 off not installed -- -- 0x97 0x97 fc2/24 off not installed -- --

The output should reflect the current status for all the ports



Switch-9509(config)# do sh run | incl assign no ficon slot 1 assign port-numbers ficon slot 2 assign port-numbers 128-151 force no ficon slot 7 assign port-numbers no ficon slot 9 assign port-numbers

Also the running config will show the newly reserved ports



switch(config)# ficon logical-port assign port-numbers 16-31,224-249

Assign logical port-numbers for FCIP interfaces and Port-channels

Logical ports are shown in a show running config

Switch-9509(config)# do sh run | incl assign no ficon logical-port assign port-numbers no ficon slot 1 assign port-numbers ficon slot 2 assign port-numbers 128-151 force no ficon slot 7 assign port-numbers no ficon slot 9 assign port-numbers ficon logical-port assign port-numbers 16-31, 224-249 force









  Summary


Fabric Binding Troubleshooting   First, verify that identical Fabric Binding Databases are active on both

switches   Debug information:

debug fabric-binding all (collect from both switches) Activate the fabric binding database on both switchs while the debug is running. fabric-binding activate vsan X Flap the port-channel or ISL interface (Only if customer can take the outage)

  Collect the following show fabric-binding statis vsan X (x=vsan 500) show fabric-binding efmd statistics vsan X show fabric-binding database vsan X show fabric-binding database act vsan X show fabric-binding internal global show fabric-binding internal info vsan X show fabric-binding internal event show fabric-binding internal efmd event interface fcx/x or port-channel x


CUP Troubleshooting Collect the following:

show tech-support details show ficon control-device sb3 show ficon control-device internal info show ficon control-device internal event-history vsan x (x=ficon vsan) show ficon control-device internal event-history errors show ficon control-device internal event-history msgs show ficon control-device internal event-history vsan <x> sh ficon internal event-history errors sh ficon internal event-history msgs sh ficon internal event-history vsan <x>

From the Mainframe: MVS SYSLOG before, during, and after the failure. Output from D IOS,FICON


Basic Troubleshooting

  Troubleshoot port problems (performance, failing) like any other case. Assume the switch is the problem and rule it out!

attach mod x term len 0 show hardware internal debug-info interface fc x/x clear asic-cnt list-all-devices clear asic-cnt all clear asic-cnt device-id 3

  From Supervisor show flogi internal event interface fcx/y (for both CH and CU interfaces) show port internal event-history interface fcx/y show ficon internal event-history vsan x interface fcx/y. show ficon internal event-history vsan 1 show ficon internal event-history msgs show ficon internal event-history errors show ficon internal info


Basic Troubleshooting

The list of commands that are useful from SAN-OS(later NX-OS) CLI: Setup ficon ficon slot X assign port-numbers 48-63 show ficon port-numbers assign slot 3 show ficon port-numbers assign logical-port show ficon vsan X portaddress brief show ficon vsan vsan-id file name filename portaddress show ficon vsan 20 portaddress 8 counters

ficon swap portnumber old-port-number new-port-number ficon swap interface old-interface new-interface

clear ficon vsan 1 allegiance

ficon vsan 2 apply file SampleFile ficon vsan vsan-id offline ficon vsan vsan-id online show ficon control-device sb3 vsan X show flogi database vsan X


Basic Troubleshooting continued

 Gather the following Mainframe displays before and after any failures or recreates

D M=DEV(nnnn) D M=CHP(mmm) Collect the MVS SYSLOG before, during, and after the failure. Collect any messages that may appear on Hardware Console, for example, "LINK LEVEL ERROR - FICON PLOGI ELS ERROR",


Summary / Review

  FICON uses the Single-Byte Command Code Sets-3 (SB-3) and

Single-Byte Command Code Sets-4 (SB-4)

  Three types of FICON topologies: Point-to-point, Switched point-to-

point (via a FICON Director) ,Cascaded FICON Directors (through two FICON Directors)

  For FICON, there is a maximum of 1 hop from CH to CU

  Allow for fabric-binding between FICON fabrics

  Utilize ‘setup ficon’ command

  Utilize GUI (FM - DM) for basic troubleshooting / configuration


References   FICON Planning and Implementation Guide:

http://www.redbooks.ibm.com/abstracts/sg246497.html?Open   MDS FICON Configuration Guide:

http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/4_1/configuration/guides/cli_4_1/ficon.html

  MDS Fabric Manager Configuration Guide http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/4_1/configuration/guides/fm_4_1/ficon.html


Question and Answer…

Thanks for attending this session and Good Luck in your CCIE Journey

FICON for CCIEstudy May

Documents

Transcript of FICON for CCIEstudy May