HBA Troubleshooting Guide

53-1000885-02January 30, 2009

Fibre Channel HBA Troubleshooting Guide

Supporting Models 415, 425, 815, 825

Copyright 2006-2009 Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol, DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity with respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that accompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements. To find-out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

Brocade Communications Systems, Incorporated

Document History

Corporate and Latin American HeadquartersBrocade Communications Systems, Inc.1745 Technology Drive San Jose, CA 95110 Tel: 1-408-333-8000 Fax: 1-408-333-8101 E-mail: [email protected]

Asia-Pacific HeadquartersBrocade Communications Singapore Pte. Ltd.30 Cecil Street#19-01 Prudential TowerSingapore 049712SingaporeTel: +65-6538-4700Fax: +65-6538-0302E-mail: [email protected]

European HeadquartersBrocade Communications Switzerland SrlCentre SwissairTour B - 4me tage29, Route de l'AroportCase Postale 105CH-1215 Genve 15Switzerland Tel: +41 22 799 5640Fax: +41 22 799 5641E-mail: [email protected]

Title Publication number Summary of changes Date


53-1000885-01 New document December 2008


53-1000885-02 Revised with corrections. January 2009

Host system freezes or crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

HCM GUI fails to connect with HCM agent . . . . . . . . . . . . . . . . . . . . 17

Verifying Fibre Channel links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

Chapter 3 Tools for Collecting Data

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23Contents

About this Document

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Whats new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Document conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Notice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Getting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii

Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1 Introduction to troubleshooting

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

How to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . 1

Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Chapter 2 Isolating Problems

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

General problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Resolving installation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12Verifying installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12Errors when installing driver . . . . . . . . . . . . . . . . . . . . . . . . . . . .13Installer program does not autorun from CD (Windows only) . .13Files needed for bfad.sys message appears when removing driver13Cannot roll back driver on all HBA instances using Device Manager14Host not booting from remote LUN . . . . . . . . . . . . . . . . . . . . . . . 14Confirming driver package installation. . . . . . . . . . . . . . . . . . . . 14Brocade Fibre Channel HBA Troubleshooting Guide iii53-1000885-02

For detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Data to provide support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Collecting data using host system commands . . . . . . . . . . . . . . . . . 24

Collecting data using BCU commands and HCM . . . . . . . . . . . . . . .25Using Support Save . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25HBA data collection using HCM . . . . . . . . . . . . . . . . . . . . . . . . .28Collecting data using BCU commands . . . . . . . . . . . . . . . . . . . .28

Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . .28

Event logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30Host system logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30HCM logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32Syslog support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34Windows Event Log support . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34Port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34IOC statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35Fabric statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36Remote port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36FCIP initiator mode statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Logical port statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Virtual port statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38Quality of service (QoS) statistics . . . . . . . . . . . . . . . . . . . . . . . .39

Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40Beaconing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40Loopback tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41PCI loopback test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42Memory test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42HBA temperature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43Ping end points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43Trace route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44Echo test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44SCSI test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

Collecting SFP data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45SFP diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45Port power on management (POM) . . . . . . . . . . . . . . . . . . . . . .46

Collecting port data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46Base port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46Remote port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46Logical port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Virtual port properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Port speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48

Authentication settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48Displaying settings through HCM . . . . . . . . . . . . . . . . . . . . . . . .48Displaying settings through BCU. . . . . . . . . . . . . . . . . . . . . . . . .48iv Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

QoS and target rate limiting settings . . . . . . . . . . . . . . . . . . . . . . . . .48BCU commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49HCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49

Persistent binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50

Chapter 4 Performance optimization

In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Linux tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Solaris tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52

Windows tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52Driver tunable parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52OS tunable parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

VMware tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

IndexBrocade Fibre Channel HBA Troubleshooting Guide v53-1000885-02

vi Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Administrators Guide. That publication provides detailed information on HBA monitoring and diagnostic tools in Host Connectivity Manager (HCM) and the BCU. About this Document

In this chapterHow this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiSupported hardware and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiWhats new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixDocument conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixNotice to the reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAdditional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiGetting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiDocument feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

How this document is organizedThis document is organized to help you find the information that you want as quickly and easily as possible.

The document contains the following components:

Chapter 1, Introduction to troubleshooting provides an introduction and approach to troubleshooting HBA problems, as well as tips for gathering problem information. A checklist is also provided to verify that required procedures have been followed during installation.

Chapter 2, Isolating Problems provides information on common HBA problems and procedures to diagnose and recover from these problems

Chapter 3, Tools for Collecting Data provides a summary of diagnostic and monitoring tools available through the HCM, Brocade Command Line Utility (BCU), fabric switch operating system and host system to help you isolate and resolve HBA-related problems.

Chapter 4, Performance optimization contains guidelines for optimizing HBA performance on your host system.

NOTEThis publication is a companion guide to be used with the Brocade Fibre Channel HBA Brocade Fibre Channel HBA Troubleshooting Guide vii53-1000885-02

Supported hardware and softwareThis section describes HBA hardware and software support.

HBA supportThe following Fibre Channel host bus adapters (HBAs) are supported in this release.

Brocade 815. Single-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+. Brocade 825. Dual-port HBA with a per-port maximum of 8 Gbps using an 8 Gbps SFP+. Brocade 415. Single-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP. Brocade 425 Dual-port HBA with a per-port maximum of 4 Gbps using a 4 Gbps SFP.

Notes: This publication only supports the HBA models listed above and does not provide information

about the Brocade 410 and 420 Fibre Channel HBAs, also known as the Brocade 400 Fibre Channel HBAs.

Although you can install an 8 Gbps SFP+ into a Brocade 415 or 425 HBA, only 4 Gbps maximum port speed is possible.

Install only Brocade-branded SFPs in these HBAs.

Fabric OS and switch supportFor a current list of compatible servers, switches, storage, and applications, refer to the Brocade HBA web site at www.brocade.com/hba.

Host operating system supportThe following operating systems support Brocade Host Connectivity Manager (HCM), Brocade Command Line Utility (BCU), and HBA drivers:

Windows Server 2003, version R2 with SP2 Windows Server 2008 Windows NT (HCM support only) Windows 2000 (HCM support only) Linux RHEL4, RHEL5, SLES9, and SLES10 Solaris 10 (x86 and SPARC) VMware ESX Server 3.5

NOTEDrivers, BCU, and HCM Agent are supported only on the VMware console Operating System. HCM is supported only on the guest operating system on VMware.

Windows Vista (HCM only) Windows XP (HCM only)viii Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

NOTESpecific operating system service pack levels and other patch requirements are detailed in the current HBA release notes.

Whats new in this documentThis is a new document. For further information about new features not covered in this document and documentation updates for this release, refer to the HBA release notes.

Document conventionsThis section describes text formatting conventions and important notice formats used in this document.

Text formattingThe narrative-text formatting conventions that are used are as follows:

bold text Identifies command namesIdentifies the names of user-manipulated GUI elementsIdentifies keywords and operandsIdentifies text to enter at the GUI or CLI

italic text Provides emphasisIdentifies variablesIdentifies paths and Internet addressesIdentifies document titles

code text Identifies CLI outputIdentifies command syntax examples

For readability, command names in the narrative portions of this guide are presented in mixed lettercase: for example, switchShow. In actual examples, command lettercase is often all lowercase. Otherwise, this manual specifically notes those cases in which a command is case sensitive. Brocade Fibre Channel HBA Troubleshooting Guide ix53-1000885-02

Command syntax conventionsCommand syntax in this manual follows these conventions:

Notes, cautions, and warningsThe following notices and statements are used in this manual. They are listed below in order of increasing severity of potential hazards.

NOTEA note provides a tip, guidance or advice, emphasizes important information, or provides a reference to related information.

ATTENTIONAn Attention statement indicates potential damage to hardware or data.

CAUTIONA Caution statement alerts you to situations that can be potentially hazardous to you or cause damage to hardware, firmware, software, or data.

DANGERA Danger statement indicates conditions or situations that can be potentially lethal or extremely hazardous to you. Safety labels are also attached directly to products to warn of these conditions or situations.

Key termsFor definitions specific to Brocade and Fibre Channel, see the technical glossaries on Brocade Connect. See Brocade resources on page xi for instructions on accessing Brocade Connect.

For definitions specific to this document, see the Brocade Fibre Channel HBA Installation and

command Commands are printed in bold.

--option, option Command options are printed in bold.

-argument, arg Arguments.

[ ] Optional element.

variable Variables are printed in italics. In the help pages, values are underlined or enclosed in angled brackets < >.

... Repeat the previous element, for example member[;member...]

value Fixed values following arguments are printed in plain font. For example, --show WWN

| Boolean. Elements are exclusive. Example: --show -mode egress | ingressx Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Reference Manual.

For definitions of SAN-specific terms, visit the Storage Networking Industry Association online dictionary at:

http://www.snia.org/education/dictionary

Notice to the readerThis document may contain references to the trademarks of the following corporations. These trademarks are the properties of their respective companies and corporations.

These references are made for informational purposes only.

Additional informationThis section lists additional Brocade and industry-specific documentation that you might find helpful.

Brocade resourcesFor HBA resources, such as product information, software, firmware, and documentation, visit the Brocade HBA web site at www.brocade.com/hba.

To get up-to-the-minute product information, join Brocade Connect. Go to http://www.brocadeconnect.com to register at no cost for a user ID and password.

For practical discussions about SAN design, implementation, and maintenance, you can obtain Building SANs with Brocade Fabric Switches through:

http://www.amazon.com

For additional Brocade documentation, visit the Brocade Web site:

http://www.brocade.com

Corporation Referenced Trademarks and Products

Microsoft Corporation Windows, Windows Server 2003, Windows Server 2008, Vista, XP, PE for Windows, Hyper V for Windows, Windows Automated Installation Kit (WAIK)

Sun Microsystems, Inc. Solaris

Red Hat Inc. Red Hat Enterprise Linux (RHEL)

Novell, Inc SUSE Linux Enterprise Server (SLES)

VMware Inc. ESX Server

SPARC International, Inc SPARCBrocade Fibre Channel HBA Troubleshooting Guide xi53-1000885-02

Other industry resourcesFor additional resource information, visit the Technical Committee T11 Web site. This Web site provides interface standards for high-performance and mass storage applications for Fibre Channel, storage management, and other applications:

http://www.t11.org

For information about the Fibre Channel industry, visit the Fibre Channel Industry Association Web site:

http://www.fibrechannel.org

Getting technical helpContact your Brocade HBA support supplier for hardware, firmware, and software support, including product repairs and part ordering. Provide the following information:

1. General information:

Brocade HBA model number Host operating system version Software name and software version, if applicable syslog message logs bfa_supportsave output.

To expedite your support call, use the bfa_supportsave feature to collect debug information from the driver, internal libraries, and firmware. You can save valuable information to your local file system and send it to support personnel for further investigation. For details on using this feature, refer to Using Support Save on page 25.

Detailed description of the problem, including the switch or fabric behavior immediately following the problem, and specific questions.

Description of any troubleshooting steps already performed and the results.2. HBA serial number:

The HBA serial number and corresponding bar code are provided on the serial number label illustrated below. The serial number label is located on the end of the HBA opposite from the SFP receivers.

You can also display the serial number through the following HCM dialog boxes and BCU commands:

HBA Properties tab in HCM.Select an HBA in the device tree, then click the Properties tab in the right pane.

BCU adapter --list command.

*FT00X0054E9*

FT00X0054E9xii Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

NOTEFor details on using HCM and BCU commands, refer to the Brocade Fibre Channel HBA Administrators Guide.

3. Port World-Wide Port Name (PWWN).

Determine this through the following resources:

Label located on the end of the HBA opposite the SFP receiver slots. This label provides the WWPN for each port.

Brocade BIOS Configuration Utility. Select the appropriate HBA port from the initial configuration utility screen, then select Adapter Settings to display the WWNN and PWWN for the port. For details, refer to the Brocade Fibre Channel HBA Installation and Reference Manual.

HCM Port Properties dialog box.This dialog box displays the PWWN for each HBA port. To display this dialog box, click a port in the device tree, then click the Properties tab in the right pane.

The following BCU commands:adapter --query

This command displays HBA information. The parameter is the HBAs serial number.

port --list

This command lists all the physical ports on the HBA along with their basic attributes. The parameter is the HBAs serial number.

Document feedbackQuality is our first concern at Brocade and we have made every effort to ensure the accuracy and completeness of this document. However, if you find an error or an omission, or you think that a topic needs further development, we want to hear from you. Forward your feedback to:

[email protected]

Provide the title and version number of the document and as much detail as possible about your comment, including the topic heading and page number and your suggestions for improvement.Brocade Fibre Channel HBA Troubleshooting Guide xiii53-1000885-02

xiv Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02system BIOS, helpful data, such as event logs, operating statistics, and diagnostics. Note that Table 5 on page 24 in Chapter 3 provides a list of useful host system commands for each supported operating system that you can use to gather data.

4. Consider these factors when isolating and resolving the problem:

Can the issue be resolved by using the latest supported combination of host operating system, operating system updates, or HBA drivers?Chapter

1Introduction to troubleshooting

In this chapterHow to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . 1Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

How to use this manual for troubleshootingA Fibre Channel HBA is one component in a SAN consisting of Fibre Channel switches, storage devices, and the fiber optic cabling and connections for these components. Although there may be a problem in the HBA or an HBA component, such as a small form factor pluggable (SFP) optical transceiver, the problem could also originate in another SAN component or connections between these components. Before removing and replacing HBAs, launching HBA diagnostics, or even gathering statistics on HBA operation, it is important that you perform the following tasks:

1. Fully describe the problem and gather complete information about the symptoms that suggest a problem exists. Refer to Gathering problem information on page 2.

2. Isolate or resolve the problem by first using information in Chapter 2, Isolating Problems.

First study information in the general HBA problem symptoms, causes, and fixes or actions in Table 2 on page 6. This table should provide help on many general problems that you may encounter with HBA operation. Fixes and actions often reference the BCU commands, HCM features, and host operating system commands described in Chapter 3, Tools for Collecting Data to gather data for problem isolation or resolution.

If you still require more information to isolate problems, use the following sections in Chapter 3. Note that these sections are referenced from Table 2 (Troubleshooting General Problems) when appropriate to further isolate problems. Resolving installation problems on page 12. Host system freezes or crashes on page 16. Verifying Fibre Channel links on page 20.

3. Use the BCU commands, HCM features, and host operating system commands described in Chapter 2, Isolating Problems to gather data to help you isolate problems. Although many of these tools are specifically referenced as actions for problems described in Table 2 (Troubleshooting General Problems) in Chapter 2, many more are included that can provide 1

Gathering problem information1 Does the issue persist when the HBA is installed in a different platform or is connected using a different switch port, SFP, and cable?

Can this problem be reproduced on one or more HBAs, port, or host system? Can you identify specific steps that consistently reproduce this problem on one or more hosts?

Is the problem documented in release notes for the HBA, operating system, or host system BIOS?

Is the problem documented in release notes for the switch and target storage system? Is unexpected behavior intermittent or always present?If the problem is in a Fibre Channel switch, cabling, storage device, or in connectivity between these components, refer to documentation, help systems, or service providers of that equipment.

5. If you cannot resolve the problem, gather and provide problem information to your HBA support provider for resolution.

Gathering problem informationPerform the following tasks to obtain as much information as possible before contacting technical support. Be sure to take careful notes for use as a record and reference.

Describe the symptoms that you are observing. Be specific. Here are some examples:- User experiences, such as slow performance or file access.- LEDs not functioning on an HBA port that is connected to the fabric.- All LEDs on HBA port flashing amber.- Expected storage devices not visible from the HCM or host systems storage management

application.

- HBA not recognized by host system BIOS.- HBA not recognized as PCI device by host system operating system.

What happened prior to the observed symptoms? Describe all observed behavior that is unexpected and compare against expected behavior. Gather information for support.

- Use appropriate tools on storage targets to gather information such as disk, tape, and controller model and firmware levels.

- Run the bfa_supportsave BCU command on the host system and save output to a file on your system. This command captures all driver, internal libraries, firmware, and other information needed to diagnose suspected system issues.You can save captured information to the local file system and send it to support personnel for further investigation.

- Run the Fabric OS supportSave command on any Brocade switch and save output. This command collects RASLOG, TRACE, supportShow, core file, FFDC data and other support information.

For details on using the Support Save feature, refer to Using Support Save on page 25.

Draw a topology map of the SAN from the HBAs to the storage targets. Include the following:2 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Gathering problem information 1l

The bfa_supportSave and FOS supportsave commands can provide current information for the topology map. Also, consider using the Brocade SAN Health products to provide information on your SAN environment, including an inventory of devices, switches, firmware versions, and SAN fabrics, historical performance data, zoning and switch configurations, and other data. Click the Support tab on www.brocade.com for more information on these products.

Run appropriate diagnostic tools for storage targets. Use additional HCM, BCU, host system, and Fabric OS commands summarized in Chapter 3,

Tools for Collecting Data to gather statistics and problem data on the HBA, host, Fibre Channel links, and connected devices.

Determine what has changed in the SAN. For example, if the SAN functioned without problems before installing the HBA, then the problem is most likely in the HBA installation or configuration, HBA hardware, or HBA driver package. Other examples to investigate could be changes in the switch or storage system firmware, an offline switch, or a disconnected or faulty cable between the HBA, switch, or storage controller fiber optic ports.

Record the time and frequency of symptoms and the period of time symptoms have been observed.

Determine if unexpected behavior is intermittent or always present. List steps that have been taken to troubleshoot the problem, including changes attempted to

isolate the problem.

TABLE 1 Topology map details

Component How to identify

HBA Model, World-Wide Name (WWN), and driver release level.

Fibre Channel switches. Model, WWN, and Fabric OS version.

Fiber optic links between HBA, switches, and storage ports.

Port WWNs connected to all links.

Host hardware Model and hardware revision.Brocade Fibre Channel HBA Troubleshooting Guide 353-1000885-02

Gathering problem information14 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02Chapter

2Isolating Problems

In this chapterGeneral problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Resolving installation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Host system freezes or crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16HCM GUI fails to connect with HCM agent. . . . . . . . . . . . . . . . . . . . . . . . . . . 17Verifying Fibre Channel links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

General problemsTable 2 on page 6 describes general problems related to HBA operation, possible causes, and recommended actions that may fix the problem. Recommended actions may refer you to information in the following locations as appropriate to gather information to further isolate and resolve the problem.

Chapter 3, Tools for Collecting Data in this manual.Contains procedures to perform HBA diagnostics, display HBA statistics and event logs, and collect data for troubleshooting using BCU commands, HCM options, Fabric OS commands, and your host system commands.

Fabric OS Administrators GuideProvides detailed information on features available on Brocade storage area network (SAN) products, and how to configure and administer these products

Fabric OS Command Reference Manual.Provides detailed descriptions of command line interface commands to help system administrators and technicians operate, maintain, and troubleshoot Brocade SAN products.

Fabric OS Troubleshooting and Diagnostic Guide. Provides help with isolating problems in other Brocade SAN components.

Your hosts operating system documentation and help system.Provides details on commands for gathering information and isolating problems.5

General problems2TABLE 2 Troubleshooting general problemsSymptom Possible Cause Fix or Action

HBA not reported under servers PCI subsystem. 1 Card not seated properly.2 Server slot issues.3 HBA not compatible with host

operating system or connected storage systems.

1 Execute host operating system command to list PCI devices. Refer to the List PCI Devices row in Table 5 on page 24. If the HBA is not listed, perform the following steps.

2 Reseat the HBA.3 Replace the HBA with an HBA in

known working condition to determine whether there is a slot malfunction.

4 Verify compatibility by reviewing the Brocade Server Connectivity Compatibility Matrix. To find this document, log into Brocade Connect on www.brocade.com, then select the Compatibility Information quick link under Documentation Library.

No HBAs reported when using the adapter --list BCU command:

1 HBA not reported under servers PCI sub-system.

2 HBA driver not loaded.

1 Execute your hosts operating system command to list PCI devices. Refer to the List PCI Devices row in Table 5 on page 24. If the HBA is not listed in the output from this command, go on to the next step.

2 Refer to HBA not reported under servers PCI sub-system under the Symptom column in this table.

3 The HBA driver may not be loaded. Refer to Confirming driver package installation on page 14 for methods to verify driver installation.6 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

General problems 2Port link does not come up. 1 SFP or cable problems.2 Link between HBA and switch is

down.3 Switch port is disabled or switch is

disabled.4 HBA port speed or topology

mis-match with the switch port.5 Non-Brocade branded SFP

installed.

1 Ensure that the SFPs and cables are connected properly on both HBA and switch side. Check for any cable damage.

2 Verify HBA side link status by executing the BCU port --list command. Check the FC Addr field for an address and the State field for Linkup. For details on using this command, refer to Port list on page 47.

3 Execute either the Fabric OS switchShow or portShow commands on the attached switch to ensure that the switch or individual port is not disabled or offline.

4 Check the port topology setting on the switch using the Fabric OS portCfgShow command to ensure that Locked L_Port is OFF. Use the portCfgLport command to change the setting to OFF if required.

Check the switch port speed using the Fabric OS portCfgShow command to verify that Speed is either AUTO or matches the speed of the attached HBA port (for example, the speed setting for both ports is 4 Gbps).

Check port speed on the HBA with the BCU port --list or port --query commands to display the current and configured speed. Refer to Port speed on page 48 and Port query on page 47 for details on using these commands.

5 If non-brocade branded SFPs are inserted on the HBA side or 8 Gbps switch or director, the port link will not come up.

On the switch, execute the Fabric OS switchShow command to verify that Mod_Inv (invalid module) does not display for the port state.

TABLE 2 Troubleshooting general problemsSymptom Possible Cause Fix or ActionBrocade Fibre Channel HBA Troubleshooting Guide 753-1000885-02

General problems2Port link does not come up (continued) On the HBA, execute the port --list and port --query BCU commands. Refer to Port list on page 47 and Port query on page 47. If an unsupported SFP is detected, the Sfp field displays us (unsupported SFP) for port --list and the Media field displays Unsupported SFP for port --query.

For additional actions and fixes for the port link not coming up, refer to Verifying Fibre Channel links on page 20.

Loss of sync and loss of signal errors in port statistics (refer to Port statistics on page 34).

Possible physical link problem. Ensure that the SFPs and cables are connected properly on both HBA and switch side. Check for any cable damage.

Refer to Verifying Fibre Channel links on page 20

Fabric authentication failures Authenticating configuration is incorrect. Check authentication settings on the switch and HBA. For the switch, execute the authutil --show Fabric OS command. For the HBA, execute the BCU auth --show command (refer to Authentication settings on page 48).

Use the BCU auth --show command on the HBA and Fabric OS authutil --show command on switch.

Check the shared secret configuration on the attached switch and HBA. For the switch, execute the secAuthSecret Fabric OS command. For the HBA, execute the auth -secret BCU command (refer to Authentication settings on page 48) for details on using the auth-secret command.

TABLE 2 Troubleshooting general problemsSymptom Possible Cause Fix or Action8 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

General problems 2Target not visible. 1 No fabric connectivity between HBA and target or target is not online.

2 The target and the HBA are not on the same zone.

3 The HBA driver is not loaded.4 There is a problem with remote

port.

1 Execute the Fabric OS nsAllShow command on the attached switch to verify that the target and the host are online in the fabric and registered in the name server.

2 Execute the Fabric OS cfgActvShow command on the attached switch and verify that the host and target are in the same zone (either using domain area members, port area members, or port or node WWNs).

3 The HBA driver may not be loaded. Refer to Confirming driver package installation on page 14 for methods to verify driver installation.

4 Verify that the remote target port (rport) is reporting itself online by comparing rport online and rport offline statistics (refer to Remote port statistics on page 36). The rport online counter should be one greater than the rport offline counter. If not, clear the counters and try connecting to the remote port again. Verify the rport online and rport offline statistics again.

LUN not visible 1 Missing or improper storage array LUN masking setting.

2 HBA driver not loaded.

1 Check LUN mapping and masking using storage array configuration tools.

2 The HBA driver may not be loaded. Refer to Confirming driver package installation on page 14 for methods to verify driver installation.

I/Os are not failing over immediately on a path failure in MPIO setup.

Improper driver mpiomode setting. Execute the port --query BCU command and ensure fcpim MPIO mode is enabled (which implies zero Path TOV values) or that fcpim MPIO mode is disabled with the expected Path TOV settings (default is 10 seconds).

Unable to create more than 126 Virtual (NPIV) ports.

The switchs maximum NPIV limit has been reached.

Execute the Fabric OS configure command on the attached switch and change the Maximum logins per port parameter under the F_Port login parameters menu to increase the maximum NPIV I/Ds allowed per port.

On Linux, the maximum IOPS numbers are very low.

The amount of disk I/O requests are causing low throughput and high latency.

Refer to Linux tuning on page 51 for suggestions to optimize HBA performance in Linux systems.

On VMware, the maximum IOPS numbers are very low.

The amount of disk I/O requests are causing low throughput and high latency.

Refer to VMware tuning on page 53 for suggestions to optimize HBA performance in Vmware systems.


General problems2QoS performance issues 1 QoS not enabled on both switch and HBA port.

2 QoS zones not created properly on switch for high, medium, and low priority targets.

1 Verify if QoS is enabled for an HBA port using the qos -query BCU command. Verify if it is enabled on the switch using the islShow command.

2 Verify zones on the switch using cfgActvShow command.

3 Verify that QoS is configured on switch using instructions in the Brocade Fabric OS Administrators Guide.

The HBA is not showing in the fabric. There is a problem in the fabric or a protocol issue between the HBA and fabric.

Check fabric statistics. Refer to Fabric statistics on page 36 for methods to display fabric statistics for the HBA. If counts for FLOGI sent and FLOLGI

accept fabric statistics do not match, suspect fabric problem or protocol issue between HBA and fabric.

If fabric offline counts increase and fabric maintenance is not occurring, this may indicate a serious fabric problem. Refer to your switch troubleshooting guide.

Errors when installing bfa_driver_linux--.noarch.rpm driver package.

Appropriate distribution kernel development packages are not installed on your host system for the currently running kernel.

Refer to Errors when installing driver on page 13 for more information to isolate this problem.

Installer program does not autorun (Windows only).

Autorun is not enabled on your system. Refer to Installer program does not autorun from CD (Windows only) on page 13 for more information to isolate this problem.

Host not booting from remote LUN. Boot from LUN not configured correctly. Refer to Host not booting from remote LUN on page 14 for more information to isolate this problem.

Host system freezes or crashes. 1 Fault fiber optic cabling and connections.

2 Faulty or unseated SFPs or unsupported SFPs.

3 Conflicts with port operating speed or topology of attached devices.

4 HBA not compatible with host system.

Refer to Verifying Fibre Channel links on page 20 for more information to isolate this problem.

Virtual devices not listed in name server. Problem in the fabric or a protocol issue between the HBA and fabric.

NPIV is not supported or is disabled on the switch

Check virtual port statistics, such as FDISK sent, FDISK accept, and No NPIV support statistics. Refer to Virtual port statistics on page 38 for methods to display virtual port statistics.

Operating system errors (blue screen). HBA driver not loaded. Refer to Confirming driver package installation on page 14 for methods to verify driver installation.


General problems 2Failed to connect to agent on host... error when using HCM.

The client application cannot connect to the HCM Agent listening on the configured port for the following reasons: The agent is not running. The agent not accepting

connections on the expected port. The agent is not listening on the

expected port. Communication between the client

and agent is blocked by a firewall preventing access to the port (usually only a consideration for remote HCM management).

Refer to HCM GUI fails to connect with HCM agent on page 17 to isolate and resolve the problem.

The HBA not registering with the name server or cannot access storage.

1 HBA cannot log in to the name server.

2 HBA has a problem registering with the name server.

3 HBA has a problem querying the name server for available storage.

Display logical port statistics (refer to Logical port statistics on page 37. Check for the following errors:1 Increasing name server port login

(NS PLOGI) error rejects and unknown name server port login response (NS login unknown rsp) errors means that HBA most likely cannot log in to the name server.

2 Increasing errors of the following types indicates that the HBA has a problem registering with the name server. Name server register symbolic

port name identifier (NS RSPN_ID) errors.

Name server register symbolic port name identifier response (NS RFT_ID rsp) errors

Name server register symbolic port name identifier response rejects (NS RFT_ID rejects).

3 Increasing name server get all port ID response (NS GID_FT rsp), rejects (NS_GID FT rejects), or unknown responses (NS_GID FT unknown rsp) means that the HBA has a problem querying the name server for available storage.

Driver event messages appearing in host system log files.

Various causes and severity levels. Follow recommended action in message.

Resolve critical-level messages and multiple major or minor-level messages relating to the same issue as soon as possible.

For details on event messages, refer to Event logs on page 30.


Resolving installation problems2NOTEIf troubleshooting actions in Table 2 do not resolve problems, check the installed version of the HBA (chip revision) and driver (fw version) using the adapter --query BCU command. To use this command, refer to Collecting data using BCU commands on page 28. Refer to release notes posted on the Brocade HBA web site (www.brocade.com/hba) for known problems relating to the HBA and driver versions.

Resolving installation problemsThis section provides a summary of common HBA problems that may occur during or immediately after installing HBAs and describes methods to resolve these problems.

Verifying installationProblems with HBA operation may be due to improper hardware or software installation, incompatibility between the HBA and your host system, unsupported SFPs installed on the HBA, improper fiber optic cable connected to the fabric, or the HBA not operating within specifications. Determine if problems may exist because of these factors by reviewing your installation with information in the Brocade Fibre Channel HBA Installation and Reference Manual listed in Table 3.

Files needed for bfad.sys message appears when removing driver

Installing the driver using the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI or command-based application), then subsequently removing the driver using the Device Manager.

Do not uninstall the driver using the Device Manager if you have used the Brocade installer programs to install driver instances. Always use the Brocade installer programs to remove the driver.Refer to Files needed for bfad.sys message appears when removing driver on page 13 for more information.

Cannot roll back driver on all HBA instances using Device Manager

Installing the driver using the Brocade driver installer program (bfa_installer.exe) or Software Installer (GUI or command-based application), then rolling back driver HBA instances using the Device Manager.

Install the driver for each HBA instances using the Device Manager, then roll back the driver using Device Manager.

Use the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI or command-based application) to install or upgrade the driver, then use the Brocade Software Uninstaller to roll back drivers on all HBA instances in one-step.

Refer to Cannot roll back driver on all HBA instances using Device Manager on page 14 for more information.


Resolving installation problems 2Errors when installing driver If you are installing the bfa_driver_linux--.noarch.rpm package, the driver module compiles on the system during installation. If driver build errors result when you install the package, verify that the appropriate distribution kernel development packages are installed on your host system for the currently running kernel. These should include the gcc compiler and the kernel sources. If these are not installed, you may need to reinstall the operating system before continuing installation. Be sure to install everything including the developer library options.

Installer program does not autorun from CD (Windows only)When installing software from a CD created from the ISO image downloaded from www.brocade.com/hba, and installer program does not autorun, make sure that autorun is enabled on your system. Enable autorun if possible and reinsert the CD. Try manually executing the LaunchWinSetup.cmd located on the installation the CD.

Files needed for bfad.sys message appears when removing driverWhen removing the driver with Device Manager or using the Brocade Software Uninstaller, program a message appears asking for Files needed for bfad.sys to complete the installation.

This occurs if you perform the following sequence of steps:

1. Install the driver using the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI or command-based application).

2. Uninstall the Brocade HBA using Device Manager.

3. Re-install the driver using the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI or command-based application).

4. Uninstall the driver using the driver installer program (bfa_installer.exe) program.

To avoid this problem, do not uninstall the driver using the Device Manager if you have used the Brocade installer programs to install driver instances. Always use the Brocade installer programs. If only one driver is present in the system, then the Brocade programs also remove the Fibre Channel devices from the Device Manager.

TABLE 3 Installation and Reference Manual referencesInformation Chapter

Hardware and software compatibility information. Product Overview

Software installation packages supported by host operating system and platforms.

Product Overview

Hardware and software installation instructions. Installation

Product specifications. SpecificationsBrocade Fibre Channel HBA Troubleshooting Guide 1353-1000885-02

Resolving installation problems2Cannot roll back driver on all HBA instances using Device Manager When using the Windows Device Manager, you can only roll back the driver for the first HBA instance.

This occurs if you perform the following sequence of steps:

1. Install the driver using the Brocade driver installer program (bfa_installer.exe) or Software Installer (GUI or command-based application).

2. Roll back driver HBA instances using Device Manager.

To avoid this problem, use one of the following methods:

Install the driver for each HBA instances using the Device Manager, then roll back the driver using Device Manager.

Use the driver installer program (bfa_installer.exe) or Brocade Software Installer (GUI or command-based application) to install or upgrade the driver, then use the Brocade Software Uninstaller to roll back drivers on all HBA instances in one-step.

Host not booting from remote LUNIf booting the host from a remote boot device, verify whether boot over SAN configuration is complete and correct. For example, verify the following:

A zone is created on the attached switch that contains only the PWWN of the storage system port where the boot LUN is located and the PWWN of the HBA port.

BIOS or EFI is enabled to support boot over SAN from a specific HBA port. BIOS or EFI is configured to boot from a specific LUN. The hosts operating system, HBA driver, and other necessary files are installed on the boot

LUN.

Confirming driver package installation HBA driver packages from Brocade contain the current driver, firmware, and HCM agent for specific operating systems. Make sure that the correct package is installed for your operating system. Refer to the installation chapter in the Brocade HBA Installation and Reference Manual.

An out-of-date driver may cause the following problems:

Storage devices and targets not being discovered by the device manager or appearing incorrectly in the hosts device manager.

Improper or erratic behavior of HCM (installed driver package may not support HCM version). Host operating system not recognizing HBA installation. Operating system errors (blue screen).

NOTEIf driver is not installed, try re-installing the driver or re-installing the HBA hardware and then the driver.

You can use HCM and tools available through your hosts operating system to obtain information such as driver name, driver version, and HBA port WWNs.14 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Resolving installation problems 2Host Connectivity Manager (HCM)Use the following steps to display the HBA PWWN, driver name and version, firmware name and version, and the BIOS version currently in operation

1. Launch HCM.

2. Select the HBA in the device tree.

3. Select the Properties tab in the right pane to display the Properties dialog box.

The dialog box displays HBA properties.

WindowsUse the Device Manager to determine driver installation. Verify if the driver is installed and Windows is recognizing the HBA using the following steps.

1. Open the Device Manager.

2. Expand the list of SCSI and RAID controllers.

3. Right-click the Brocade FC HBA model where you are installing the driver.

If you do not see this entry or Fibre Channel Controller displays with a yellow question mark under Other Devices, the driver is not installed.

4. Select Properties to display the Properties dialog box.

5. Click the Driver tab to display the driver date and version. Click Driver Details for more information.

NOTEIf driver is not installed, try re-installing the driver or re-installing the HBA hardware and then the driver.

LinuxVerify if the HBA driver installed successfully using the following commands:

# rpm -qa|grep -i bfaThis command prints the names of the Brocade HBA driver package (bfa) if installed.

# lspciThis is a utility that displays information about all PCI buses in the system and all devices connected to them.

# lsmodThis command displays information about all loaded modules. If bfa appears in the list, the HBA driver is loaded to the system.

# dmesgThis command prints kernal boot messages. For the bfa entry, HBA model and driver version should display if the hardware and driver are installed successfully.Brocade Fibre Channel HBA Troubleshooting Guide 1553-1000885-02

Host system freezes or crashes2 # modprobe -l bfaThis verifies that the module has loaded. If bfa displays, the module has been loaded to the system.

SolarisVerify if the HBA driver installed successfully using the following commands.

pkgchk -nv bfaThis checks for and lists the installed HBA driver package files.

pkginfo -l bfaThis displays details about installed Brocade HBA (bfa) drivers. Look for information as in the following example. Note that the VERSION may be different, depending on the driver version you installed. The ARCH and DESC information may also be different, depending on your host system platform. If the HBA driver package is installed, bfa_pkg should display with a completely installed. status.

PKGINST: bfa NAME: Brocade Fibre Channel Adapter Driver CATEGORY: system ARCH: sparc&i386 VERSION: alpha_bld31_20080502_1205 BASEDIR: / VENDOR: Brocade DESC: 32 bit & 64 bit Device driver for Brocade Fibre Channel adapters PSTAMP: 20080115150824 INSTDATE: May 02 2008 18:22 HOTLINE: Please contact your local service provider STATUS: completely installed

VMwareVerify if the HBA driver installed successfully using the following commands:

vmkload_mod -lThis lists installed driver names, R/O and R/W addresses, and whether the ID is loaded. Verify that an entry for bfa exists and that the ID loaded.

cat /proc/vmware/versionThis displays the latest versions of installed drivers. Look for a bfa entry and related build number.

Host system freezes or crashesIf the BIOS and the operating system recognize the HBA, but the host system freezes or crashes during startup and does not boot, disconnect all devices from the HBA, then reboot the host system.

If the system does not freeze when rebooted and operates correctly, use the following information to resolve the problem:

- Check for faulty fiber optic cable and cable connections. 16 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

HCM GUI fails to connect with HCM agent 2- Try rebooting the system without any connectivity to the switch. This will help isolate any hang caused by switch and device interactions.

- Reseat SFPs in the HBA. Determine whether the installed SFPs are faulty by observing LED operation by HBA ports. If all LEDs are flashing amber, the SFP is invalid and may not be a required Brocade model. You can also verify SFP operation by replacing them with SFPs in known operating condition. If the problem is resolved after replacement, original SFP is faulty.

- Check for conflicts with attached devices. Verify that data speed (1-8 Gbps) and connection topology (for example, point-to-point) for devices attached to the HBA are compatible with settings on the HBA port. Although auto may be set, configuring settings manually on the HBA port and devices may allow connection. Also, note that the HBA only supports point-to-point connection topology. Refer to the Brocade Fibre Channel HBA Administrators Guide for procedures to configure HBA ports.

NOTEObserve the LEDs by HBA ports. Illuminated LEDs indicate connection, link activity, and connection speed negotiated with the attached device. Refer to LED Operation in the Specifications chapter of the Brocade Fibre Channel HBA Installation and Reference Manual.

If the system freezes perform the following tasks:- Verify if the host system firmware supports PCIe specifications listed in the Hardware and

Software Compatibility section, Introduction chapter, of the Brocade Fibre Channel HBA Installation and Reference Manual. If not, download a firmware update to support the HBA.

- On Windows systems, determine when the system freezes during the boot process. If it freezes as the driver loads, uninstall and reinstall the driver. If it freezes during hardware recognition, uninstall both the driver and HBA, then reinstall both.

- Remove the HBA and reboot the system. If the system boots, reinstall the HBA. - Reseat the HBA.- Uninstall and reinstall the driver.- Try installing the HBA into another host system. If the problem does not occur, the HBA

may not be compatible with the original host system. If the problem occurs in the new system, replace the HBA.

HCM GUI fails to connect with HCM agentThe HCM Agent installs with the HBA driver package on the host system. It is independent of the HCM user interface (client application), which is installed on the local or remote host.

If you receive an error such as Failed to connect to agent on host... when using HCM, this indicates that the client application cannot connect to the HCM Agent listening on the configured port - normally TCP port 34568. This occurs for one of the following reasons:

The agent is not running. The agent not accepting connections on the expected port. The agent is not listening on the expected port.Brocade Fibre Channel HBA Troubleshooting Guide 1753-1000885-02

HCM GUI fails to connect with HCM agent2 Communications between the client and agent is blocked by a firewall preventing access to the port (usually only a consideration for remote HCM management).

Linux, Solaris, and VMware systemsPerform the following tasks to isolate and resolve the problem.

1. Verify that the agent is running by executing the appropriate status command for your operating system as described in the Installation chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. Refer to the section on modifying HCM agent operation.

2. If you receive a message that the hcmagent is stopped, restarting the agent should resolve the problem. To restart, use the appropriate start command for your operating system which is also described in the Brocade Fibre Channel HBA Installation and Reference Manual.

Note that one command described in the manual restarts the agent, but the agent will not restart if the system reboots or the agent stops unexpectedly. Another command restarts the agent, but the agent will restart if the system reboots.

3. Confirm the HCM agent is responding to requests using the expected user password. Execute the following command to connect to the HCM agent and force it to collect the HBA driver supportsave data.

NOTEThis command is a single line. The localhost can be replaced with a different IP address.

wget --no-check-certificate https://admin:password@localhost:34568/JSONRPCServiceApp/SupportSaveController.do

If successful, the file SupportSaveController.do (actually a zip format file) will contain the data from the HCM agent.

4. If you are managing a VMware host system through HCM from a remote system, the hosts firewall may be blocking TCP/IP port 34568, which allows agent communication with HCM.

Use the following command to open port 34568:

/usr/sbin/esxcfg-firewall-o 34568,tcp,out,https

Use Windows Firewall and Advanced Service (WFAS) to open port 34568.

NOTEYou can change the default communication port (34568) for the agent using procedures in the Installation chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. Refer to the section on modifying HCM agent operation.

5. If HCM is still unable to connect to the HCM agent after using the preceding steps, collect the following data and send to your Support representative for analysis:

Data collected from the previous step in SupportSaveController.do. Data from the HCM application SupportSave feature. Select Tools > SupportSave to

generate a supportsave file. The data file name and location displays when the SupportSave feature runs. 18 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

HCM GUI fails to connect with HCM agent 2 HBA agent files on the HBA host (where the HCM agent is installed). Collect these files using the following command:tar cvfz hbaagentfiles.tgz /opt/hbaagent

Output collects to hbaagentfiles.tgz.

Data collected on the HBA host from the bfa_supportsave feature using the following command:bfa_supportsave

Output collects to a file and location specified when the SupportSave feature runs.

Windows systemsPerform the following tasks to isolate and resolve the problem.

1. Verify that the agent is running by executing the appropriate status command for your operating system described in the Installation chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. Refer to the section on modifying HCM agent operation.

2. If you receive a message that the hcmagent is stopped, restarting the agent should resolve the problem. To restart, use the appropriate start command for your operating system which is also described in the Brocade Fibre Channel HBA Installation and Reference Manual.

Note that one command described in the manual restarts the agent, but the agent will not restart if the system reboots or the agent stops unexpectedly. Another command restarts the agent, but the agent will restart if the system reboots.

3. If the HCM agent starts, verify which TCP port the agent is listening on by executing the following command at the Windows command prompt:

netstat -nao | findstr 34568

Output similar to the following should display.

TCP 0.0.0.0:34568 0.0.0.0:0 LISTENING 1960

1960 in the last column is the process identifier for the Windows process listening on the TCP port. Note that this identifier may be different on your system.

4. Enter the following command to confirm that the process identifier bound to TCP port 34568 is for the hcmagent.exe process:

tasklist /svc | findstr 1960

The following should display if the identifier from step 3 is bound to TCP port 34568:

hcmagent.exe 1960 hcmagent

5. If you are managing a Windows 2008 host system through HCM from a remote system, the hosts firewall may be blocking TCP/IP port 34568.

Use Windows Firewall and Advanced Service (WFAS) to open port 34568.

NOTEYou can change the default communication port (34568) for the agent using procedures in the Installation chapter of the Brocade Fibre Channel HBA Installation and Reference Manual. Refer to the section on modifying HCM agent operation.Brocade Fibre Channel HBA Troubleshooting Guide 1953-1000885-02

Verifying Fibre Channel links26. If the hcmagent is running and listening on port 34568 and there are no firewall issues (as explained in step 5), but you get the same Failed to connect to agent on host..." error when using HCM, collect the following data. Send this data to your Support representative for analysis:

Copies of output from the commands in step 3 and step 4. Files from the output directory created after you execute the bfa_supportsave feature.

To collect these files, execute the following command:

bfa_supportsave

Data collected by this command saves to subdirectory named bfa_ss_out.

In Windows explorer, right-click this directory and select Send To > Compressed (zipped).

This creates a zip file that you can send to your Support representative.

Build information for the HCM application. Select Help > About in HCM to display the version, build identification, and build date.

Support data from the HCM application SupportSave feature. Select Tools > SupportSave to generate a supportsave file.

If HCM cannot connect to the agent, a message displays an error (Agent Support Save could not be collected) and explains that only a basic collection is possible. Messages also display that provide the location of the zip file created.

By default, a zip file is created in the following location if installed the application in C:\Program Files\BROCADE\FCHBA:

C:\Program Files\BROCADE\FCHBA\client\data\localhost\supportsave

The zip file will have a name similar to the following:

SupportSave_Basic_2008723_0_50_57.zip

Verifying Fibre Channel linksCheck for link problems by observing LED operation for HBA ports. LED operation other than expected or LEDs may indicate link problems. For example, all LEDS flashing amber for a port indicates that an invalid non-Brocade SFP may be installed. For details on HBA LED operation, refer to the specifications chapter in the Brocade HBA Installation and Reference Manual. If LEDs do not illuminate to indicate an active link, use appropriate Fabric OS and HBA diagnostic commands and HCM options in Table 4 on page 21. For additional diagnostics commands, refer to Chapter 3, Tools for Collecting Data for HCM and BCU commands and the Fabric OS Administrators Guide for Fabric OS commands.

NOTEAlso verify LED operation on switch port that is connected to an HBA port. Refer to the switch Hardware Reference Manual to analyze LED meaning.

Common link problems can be caused by the following:

Damaged fiber optic cables. (Note that damaged cables can also cause errors and invalid data on links.)20 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Verifying Fibre Channel links 2 Fiber optic cables may not be rated or compatible with HBA port speeds. Refer to Fibre Optic Cable specifications in the Brocade Fibre Channel HBA Installation and Reference Manual.

Faulty switch or HBA SFPs. Verify if an SFP is the problem by connecting a different link to the HBA port or, if convenient, replace the cable with a cable of known quality. If the errors or invalid data on the link still indicate a cable problem, the SFP may be faulty. Try replacing the SFP.

SCSI retries and timeouts determine communication between HBA and storage. Dropped packets cause timeouts. Packets can drop because of SFP issues on HBA or switch - possibly the SFP is not compatible with HBA, but is compatible with switch or vice versa. You can run the BCU port --stats command to display port statistics, such as error and dropped frames.

Table 4 lists HCM options and BCU commands, as well as Fabric OS commands that you can use to determine link status.

TABLE 4 Tools to determine link statusApplication Tool References

HCM Port Statistics Loopback and PCI loopback test Fibre Channel ping, echo, and trace route

tests Link Beaconing Port Properties SFP information

Chapter 3, Tools for Collecting Data

BCU BCU fcdiag and diag commands. Port commands, such as port --stats.


Switch Fabric OS switchShow portShow portStatsShow portErrShow fcpProbeShow fPortTest


Fabric OS Administrators Guide

Fabric OS Troubleshooting and Diagnostics GuideBrocade Fibre Channel HBA Troubleshooting Guide 2153-1000885-02

Verifying Fibre Channel links222 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02 Fabric OS Command Reference ManualFabric OS diagnostic and monitoring commands.

Your host systems operating system user and administrators guides.Host system diagnostics, logs, and system monitoring tools.Chapter

3Tools for Collecting Data

In this chapterFor detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Data to provide support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Collecting data using host system commands . . . . . . . . . . . . . . . . . . . . . . . 24Collecting data using BCU commands and HCM . . . . . . . . . . . . . . . . . . . . . 25Collecting data using Fabric OS commands . . . . . . . . . . . . . . . . . . . . . . . . . 28Event logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Collecting SFP data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Collecting port data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Authentication settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48QoS and target rate limiting settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Persistent binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

For detailed informationThis chapter provides basic instruction on tools useful for gathering information to isolate HBA problems. For more detailed information on using these tools, refer to the following publications:

Brocade Fibre Channel HBA Administrators GuideThe following chapters in this guide cover HBA HCM and BCU monitoring and diagnostics tools:

- Monitoring- Diagnostics and Troubleshooting- Command Line Utility (BCU)

Fabric OS Troubleshooting and Diagnostics GuideThis guide provides detailed information on collecting troubleshooting information and isolating general SAN problems between the switch, host systems, and storage systems.23

Data to provide support3Data to provide supportWhen problems occur requiring support assistance, provide a detailed description of the problem, as well as output collected from the following HCM and BCU tools:

Support Save Diagnostics Port logs Port statistics and properties HBA properties Host operating system commands

Collecting data using host system commandsThe following table describes commands common to each supported operating system that you can use to gather information for troubleshooting problems. For details on these commands, refer to your systems online help and documentation.

NOTEOutput from all of these commands is captured using the Support Save feature.

TABLE 5 Host system data collection commandsTask Linux Windows VMware Solaris

List PCI devices lspci -vv In Windows registry location HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI devcon find pci\*

lspci -vv, esxcfg-info -w prtdiag -v, prtconf -pv

List installed HW details lsdev msinfo32.exe Click the plus sign(+) next to Components to view hardware details.

esxcfg-info -a prtdiag -v, prtconf -pv

Process information ps -efl, top Windows Task Manager, tasklist.exe

ps -efl, top ps -efl, top

Memory usage top, vmstat -m Windows Task Manager, tasklist.exe

top, vmstat -m vmstat -s

Performance monitoring iostat, vmstat, sar Windows Task Manager, perfmon.exe

vmstat, VM Performance: esxtop [first type 'v', 'e' then enter vm# in the list down], Disk Performance: esxtop [type 'v' then 'd'].

iostat -nx 1 5, vmstat, mpstat, sar

List for driver modules lsmod driverquery vmkload_mod -l modinfo

To check for Brocade Fibre Channel adapter (BFA) driver module

lsmod | grep bfa driverquery /v | findstr bfad

vmkload_mod -l modinfo | grep bfa24 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Collecting data using BCU commands and HCM 3Collecting data using BCU commands and HCMYou can collect a variety of information on installed Brocade HBAs, such as firmware version installed, operational status, port speed, WWN, PCI data, configuration data, flash status, and other details for troubleshooting using BCU commands, HCM menu options, and host operating system commands.

Using Support SaveThe Support Save feature is an important tool for collecting debug information from the driver, internal libraries, and firmware. You can save this information to the local file system and send to support personnel for further investigation. Use one of the following options to launch this feature:

In HCM, launch Support Save through the Tools menu. Through BCU, enter the bfa_supportsave command. Through your internet browser (Internet Explorer 6 or later or Firefox 2.0 or later), you can

collect bfa_supportsave output if you do not have root access, do not have access to file transfer methods such as FTP and SCP, or do not have access to the Host Configuration Manager (HCM).

NOTEA bfa_supportsave collection can also occur automatically for a port crash event.

The Support Save feature saves the following information:

System log message locationNOTE: For more information,

refer to Host system logs on page 30.

dmesg, /var/log/message*

System Category in Windows Event Viewer (eventvwr.exe)

/var/log/message* , /var/log/vmkernel*, /var/log/vmkwarning*,/proc/vmware/log

dmesg, /var/adm/message*

Show OS distribution info (SuSE) cat /etc/SuSE-release,(RedHat) cat /etc/redhat-release

systeminfo.exe cat /etc/vmware-release uname -a, cat /etc/release

BFA configuration file location /etc/bfa.conf Windows Registry (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bfad\Parameters\Device), HBA Flash

/etc/bfa.conf /kernel/drv/bfa.conf

BFA device file location /dev/bfa* Windows Registry (HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi Port x)

/dev/bfa* (Release 1.0) - /devices/pci*/pci*/fibre-channel@0:devctl,

(Release 1.1 and later) - /devices/pci*/pci*/bfa@0:devctl

TABLE 5 Host system data collection commandsTask Linux Windows VMware SolarisBrocade Fibre Channel HBA Troubleshooting Guide 2553-1000885-02

Collecting data using BCU commands and HCM3 HBA model and serial number. HBA firmware version. Host model and hardware revision. All support information. HBA configuration data. All operating system and HBA information needed to diagnose field issues Information about all HBAs in the system. Firmware and driver traces. Syslog message logs. Windows System Event log .evt file. HCM GUI-related engineering logs Events HBA configuration data Environment information Data.xml file Vital CPU, memory, network resources HCM Agent (logs, configuration) Driver logs (bfa_supportsave output) Core files

NOTEMaster and Application logs are saved when Support Save is initiated through HCM, but not through BCU.

Initiating Support Save through HCMInitiate Support Save by selecting Tools > Support Save.

Messages display during the Support Save operation that provide the location of the directory where data is saved. If you are initiating Support Save from a remote management station and receive a warning message that support files and Agent logs could not be collected, the HCM Agent is unavailable on the remote host. Select Tools > Backup to backup data and configuration files manually.

For more information and additional options for using this feature, refer to the Brocade Fibre Channel HBA Administrators Guide.

Initiating Support Save through BCU commandsUse the bfa_supportsave command to Initiate Support Save through the BCU:

bfa_supportsave {output_dir]

where:

[output_dir] An optional parameter that specifies the directory where you want output saved. If not specified, output is saved as a directory in the current working directory as bfa_ss_out.26 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Collecting data using BCU commands and HCM 3Messages display as the system gathers information. When complete, an output file and directory display. The directory name specifies the date when the file was saved.

For more information on the bfa_supportsave feature, refer to the Host Connectivity Manager (HCM) Administrators Guide.

Initiating Support Save through the internet browserInitiate bfa_supportsave through an internet browser.

1. Open an Internet browser and type the following URL:

https://localhost:34568/JSONRPCServiceApp/SupportSaveController.do

where localhost is the IP address of the server from which you want to collect the bfa_supportsave information.

2. Log in using the factory default user name (admin) and password (password). Use the current user name and password if they have changed from the default,

The File Download dialog box displays, prompting you to save the supportSaveController.do file.

3. Click Save and navigate to the location where you want to save the bfa_supportsave file.

Initiating Support Save through a port crash eventIf the port crashes and triggers a port crash event, support save data is collected at a system-widelevel. An Application Log message is generated with the following message:

Port Crash Support Save Completed

Port crash events have a CRITICAL severity and you can view the details in the Master Log andApplication Log tables in HCM. For more information on these logs, refer to HCM logs on page 32.

Support Save differencesFollowing are differences in data collection for the HCM, BCU, and browser applications of bfa_supportsave:

BCU Collects only driver-related logs and configuration files.

BrowserCollects driver-related and HCM Agent logs and configuration files.

HCM Collects, HCM, driver-related, and HCM Agent logs and configuration files.

bfa_supportsave collection on a port crash eventIf the port crashes and triggers a port crash event, support save data is collected at a system-wide level. An Application Log message is generated with the following message:

Port Crash Support Save CompletedBrocade Fibre Channel HBA Troubleshooting Guide 2753-1000885-02

Collecting data using Fabric OS commands3Port crash events have a CRITICAL severity and you can view the details in the Master Log and Application Log tables in HCM.

HBA data collection using HCMUse the HBA Properties panel in HCM to display information about the HBA, such as its WWN, disabled or enabled status, temperature, installed driver name, driver version, firmware version, and BIOS version. This panel also displays PCI information for the HBA, such as vendor ID, device ID, subsystem ID, current number of lanes, and PCI generation.

1. Launch HCM.

2. Select an HBA in the device tree

3. Click the Properties tab in the right pane.

Collecting data using BCU commandsUse the BCU adapter command to list and query available HBAs seen by the driver.

The bcu adapter --list command lists all HBAs on the system with a brief summary of information such as model number, serial number, and HBA number. Enter the following command:

adapter --list

where:

list Lists all adapters in the system. For each adapter in the system, a brief summary line is displays.

The adapter --query command displays adapter information, such as the current version of the HBA (chip revision) and driver (fw version), maximum port speed, model information, serial number, number of ports, PCI information, pwwn, nwwn, hardware path, and flash information (such as firmware version).

adapter -query

where:

ad_id ID of the adapter (HBA) for which you want to query.

Collecting data using Fabric OS commandsUse the following Fabric OS commands on attached Brocade switches to gather information and help isolate connectivity and other problems between the HBA, switch, and storage ports. For details on using these commands, refer to the Fabric OS Command Reference Manual.

authUtilUse this command to display and set local switch authentication parameters.

cfgShowDisplays zone configuration information for the switch. You can use command output to verify target ports (by port WWN) and LUNs that are intended to be accessible from the HBA.28 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Collecting data using Fabric OS commands 3 fcpProbeShowUse this command to display the Fibre Channel Protocol daemon (FCPd) device probing information for the devices attached to a specified F_Port or FL_Port. This information includes the number of successful logins and SCSI INQUIRY commands sent over this port and a list of the attached devices.

nsShowUse this command to display local NS information about all devices connected to a specific switch. This includes information such as the device PID, device type, and the port and node WWN.

zoneshowUse this command without parameters to display all zone configuration information (both defined and enabled).

portErrShowUse this command to display an error summary for all switch ports.

portLogShowUse this command to display the port log for ports on a switch.

portLogShowPortUse this command to display the port log for a specified switch port.

portPerfShowUse this command to display throughput information for all ports on the switch.

portStatsShowUse this command to display hardware statistics counters for a specific switch port.

portShowUse this command to display information and status of a specified switch port, including the speed, ID, operating state, type, and WWN.

SecAuthSecret Use this command to manage the DH-CHAP shared secret key database used for authentication.This command displays, sets, and removes shared secret key information from the databases

sfpShowUse this command to display detailed information about specific SFPs installed in a switch.

switchShowUse this command to display switch and port information. Output may vary depending on the switch model. Use this information to determine the fabric port WWN and PID connected to an HBA port. Also display topology, speed, and state of each port on the switch.Brocade Fibre Channel HBA Troubleshooting Guide 2953-1000885-02

Event logs3Event logsEvent messages that occur during HBA and driver operation are important tools for isolating and resolving problems. Messages provide descriptions of the event, severity, time and date of the event, and in some cases, cause and recommended actions. These messages are captured in logs. Monitoring events in these logs allows early fault detection and isolation on a specific HBA.

The following types of logs are available:

Host operating system logs HCM logs

Host system logsBrocade HBA event messages are captured in host system log files. All messages related to the Brocade HBA are identified in these logs by BFAD or bfa (Brocade Fabric Adapter). These messages contain the following information:

Message ID Message text Message arguments Severity level Cause Recommended actionBrocade HBA event message files are installed in the HBA driver installation directory for each supported operating system. Table 8 provides the location of the message files for each system.

Table 7 describes the logs for each supported operating system, where the logs are stored, and how to view them.

TABLE 6 Message catalog locationOperating System Catalog Location

Linux /opt/bfa

VMware /opt/bfa

Solaris /opt/bfa

Windows aen.zip loaded to your driver installation directory. Unzip this file to obtain all message catalog files.

TABLE 7 System Event LogsOperating System Log Name Location Viewing Message Log

Solaris Syslog /var/adm/messages dmesg command

Windows Event Log Not applicable System category in Event Viewer (eventvwr.exe)30 Brocade Fibre Channel HBA Troubleshooting Guide53-1000885-02

Event logs 3You can view all event messages that can display for a Brocade HBA by viewing HTML files that are loaded to your system as the driver package installs. These files contain all message information that can display on system logs for the Brocade HBA. View these files through your internet browser.

Table 8 provides the location of the Brocade HBA message files for each supported system.

Table 9 lists the message file names and content.

Linux Messages Log /var/log/message dmesg command1VmWare Messages Log /var/log/message* ,

/var/log/vmkernel*, /var/log/vmkwarning*,/proc/vmware/log

dmesg command

1. For ESX Server Console operating system. For Guest system, refer to information in Windows or Linux.

TABLE 8 Message file locationOperating System File Location

Linux /opt/bfa

VMware /opt/bfa

Solaris /opt/bfa

Windows aen.zip loaded to your driver installation directory. Unzip this file to obtain all message catalog files.

TABLE 9 Event message filesEvent Catalog File Content

bfal_aen_adapter.html Adapter events, such as adapter added or removed.

bfal_aen_audit.html Audit events, such as authentication enabled or disabled for base port.

bfal_aen_ioc.html IO controller (IOC) events, such as IOC heartbeat status and IOC enabled or disabled.

bfal_aen_itnim.html Initiator-target nexus events, such as target is online for initiator, has lost connection with initiator, or is disconnected by initiator.

bfal_aen_lport.html Logical port events, such as a logical port is created, deleted, online, offline, has lost fabric connectivity, or the ports login has failed.

bfal_aen_port.html Physical port events, such as port online or offline, SFP removed from port, or authentication successful or unsuccessful for base port.

TABLE 7 System Event LogsOperating System Log Name Location Viewing Message Log Brocade Fibre Channel HBA Troubleshooting Guide 3153-1000885-02

Event logs3Adjust the logging level, or the types of messages logged to your system log that relate to HBA driver operatio

HBA Troubleshooting Guide

Documents

Transcript of HBA Troubleshooting Guide