EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer...

22
1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub. We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback. _________________ ___________________________ Abstract This guide helps you troubleshoot OneFS upgrade failures and error messages when upgrading from OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x. June 15, 2017 EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE TROUBLESHOOT FAILURES WHEN UPGRADING FROM ONEFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x OneFS 6.5.x - 7.2.x

Transcript of EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer...

Page 1: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Abstract

This guide helps you troubleshoot OneFS upgrade failures and error messages when upgrading from OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x.

June 15, 2017

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT FAILURES WHEN UPGRADING FROM ONEFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

OneFS 6.5.x - 7.2.x

Page 2: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

2 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Contents and overview

Before you begin

Page 3

Appendix A

If you need further assistance

Start troubleshooting

Page 5

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these

steps.

2. Perform

troubleshooting

steps in order.

3. Appendixes

Appendix B

How to use this flowchart

Simultaneous upgrade

Page 11

Rolling upgrade

Page 12

Analysis

Page 6

Restart the upgrade

Page 14

Post-upgrade checks

Page 16

Appendix C

Output of the isi_for_array -s "uname -a"

command

Appendix D

Output of the isi status -q command

Best practices and useful information

Page 4

Page 3: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session.

This log file can be shared with Isilon Technical Support , if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,

you can configure logging by using your local SSH client 's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be

preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster , you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial console

connection available. This way, if you are unable to connect through the network, you

will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.

Page 4: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

4 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Best practices and useful information

Most upgrade problems occur during rolling upgrades that are initiated from the OneFS web

administration interface (WebUI).

For best results, do the following:

Use the command-line interface (CLI) to perform upgrades.

Initiate the upgrade from the highest-numbered node in the cluster, unless the

highest-numbered node is an Accelerator.

If the highest-numbered node is an Accelerator, then initiate the upgrade from node 1.

Use the command-line interface

It is best to initiate the upgrade from the command-line interface. The CLI displays more detailed

information than the web interface, and is not reliant on the WebUI services running in order to

function. You can also launch a screen session, which enables you to resume from where you left off

if you get disconnected.

Initiate the upgrade from the highest-numbered node

The node that you initiate the upgrade from is called the "master node."

During an upgrade, each node is upgraded and rebooted in turn, in ascending numerical order,

starting with the lowest-numbered node.

When the master node is the highest-numbered node, the upgrade starts with node 1, and the last

node to be rebooted is the master node.

The system should always upgrade and reboot the master node last , regardless of which numbered

node it is, but this does not always happen. Sometimes, when the master node is not the

highest-numbered node, the system starts upgrading with node 1 as usual, but when it reaches the

master node, it upgrades and reboots that node in its numerical order . This stops the upgrade

process because, after it is rebooted, the master node can no longer tell the rest of the nodes to

upgrade. Therefore, you should always initiate the upgrade from the highest-numbered node in the

cluster (unless, as stated above, the highest-numbered node is an Accelerator; in this case, you

should initiate the upgrade from node 1).

When to use this troubleshooting guide

Use this guide if you are upgrading from OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x to a supported version of

OneFS.

If you experience problems when upgrading from OneFS 8.0.x, refer to:

EMC Isilon Customer Troubleshooting Guide: Upgrades from OneFS 8.x to a Later Version.

For information on supported upgrade paths, see the "OneFS Upgrade Compatibility" section of the

Isilon Supportability and Compatibility Guide.___________________________________

_________________________________________________________________________

Page 5: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

5 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Start troubleshooting

IntroductionStart troubleshooting here. For an overview

of the conventions used in this flowchart, see

Appendix B: How to use this flowchart.

Start

Note Most upgrade problems occur

during rolling upgrades that are

initiated from the OneFS web

administration interface. Therefore,

we will use the command-line

interface exclusively to troubleshoot

your issue and get your upgrade

restarted. For more information, see

"Best practices and useful

information" on page 4.

Did the

upgrade fail with a

specific error displayed

on the screen?

Follow the prompts and onscreen instructions.

Yes

Can the

upgrade be completed

successfully now?No

End troubleshooting

Yes

No Go to Page 6

Go to Page 6

If you have not done so already, log in to

the cluster and configure logging through

SSH, as described on page 3.

Related information Isilon OneFS 7.x Upgrade Planning and Process Guide

Current Isilon OneFS Patches

____________________________________________

________________________

Page 6: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

6 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Analysis

Page

6

After

running the command,

do you see this error?

ERROR Client connected from

an unprivileged port number

50230. Refusing the connection

[Errno 54] RPC session

disconnected

NoInstall a patch as described in the

following article:

OneFS: After a failed or paused

upgrade, commands sent from nodes

that are not yet upgraded might fail,

article 198906.

Then continue troubleshooting.

Yes

Go to Page 7

Run the following command to see which nodes were successfully upgraded.

isi_for_array -s "uname -a"

The output provides a list of all the nodes and indicates which version of

OneFS each is running. For an example of the output, see Appendix C.

Note: If the node did not fully reboot or is down, it will not show up. Also, if

the upgrade was a rolling upgrade, an error might appear stating a node did

not come back online.

_________

You could have arrived here from:

Page 5 - Start troubleshooting

Page 8 - Analysis (3)

________________________

_________________

Page 7: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

7 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Analysis (2)

You could have arrived here from:

Page 6 - Analysis Page

7

Using the

output of the

isi_for_array -s "uname -a"

command from Page 6,

are all the nodes running

the new version

of OneFS?

No

Yes

No

Yes Go to Page 8

Go to Page 9

End troubleshooting

Run the following command:

isi status -q

In the output, look at the Health DASR column to see if

any nodes report -D- (Down). For an example of the

output, see Appendix D.__________

Do any

nodes report as down?

A down node means that it

failed to join the cluster

following the

upgrade.

Page 8: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

8 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Analysis (3)

You could have arrived here from:

Page 7 - Analysis (2)

Has it been

at least 15 minutes since the

nodes rebooted as part of

the upgrade?

Yes

Wait 15 minutes

Go back to Page 6

No

Page

8

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Page 9: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

9 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Analysis (4)

You could have arrived here from:

Page 7 - Analysis (2)

No

Go to Page 10

Yes

Did you

follow the steps in the

"Planning an Upgrade" and

"Completing pre-upgrade tasks"

sections of the OneFS Upgrade

Planning and Process Guide

before beginning the

upgrade?

Follow the steps in the "Planning an

Upgrade" and the "Completing pre-

upgrade tasks" sections of the

OneFS Upgrade Planning and

Process Guide.

Page

9

Page 10: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

10 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Analysis (5)

You could have arrived here from:

Page 9 - Analysis (4)

Page

10

Did you

perform a simultaneous

upgrade or a rolling

upgrade?

RollingSimultaneous

Go to Page 11 Go to Page 12

Page 11: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

11 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Simultaneous upgrade

You could have arrived here from:

Page 10 - Analysis (5) Page

11

Do any

nodes report as down?

A down node means that it

failed to join the cluster

following the

upgrade.

NoYes

Go to Page 14

Run the following command:

isi status -q

In the output, look at the Health DASR column to see if

any nodes report -D- (Down). For an example of the

output, see Appendix D._________

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Page 12: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

12 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Rolling upgrade

You could have arrived here from:

Page 10 - Analysis (5)

Page

12

For each node that did not get upgraded, run the following command to check

that node's /var/log/messages file to see if there are errors with a timestamp

that occurred during the upgrade. In the command, replace <YYYY-MM-DD>

with the date of the upgrade:

grep '^<YYYY-MM-DD>' /var/log/update_engine*

For example:

grep '^2015-04-15' /var/log/update_engine*

Yes

Are there

errors on a node that did

not get upgraded? No Go to Page 14

Is the

following error present?

Unable to claim upgrade

daemon on one or

more nodes.

No

Yes

Go to Page 13

Go to Page 14

Run the following command to determine which nodes did not get upgraded:

isi_for_array -s "uname -a"

The output provides a list of all the nodes and indicates which version of

OneFS each is running. For an example of the output, see Appendix C. _________

Page 13: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

13 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Rolling upgrade (2)

You could have arrived here from:

Page 12 - Rolling upgradePage

13

Run the following command:

isi services -a isi_upgrade_d

Is the

service enabled or

disabled?

Disabled

You are still in the middle of an upgrade and unable

to proceed.

Disable the service by running the following

command:

isi services -a isi_upgrade_d disable

Go to Page 14

Enabled

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Page 14: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

14 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Restart the upgrade

You could have arrived here from:

Page 11 - Simultaneous upgrade

Page 12 - Rolling upgrade

Page 13 - Rolling upgrade (2)

Page

14

Open a screen session by running the following command, where <session name> is a name that

you provide. Record the name in case you need to use it later. The screen session enables you to

easily reconnect to the upgrade process if the session gets disconnected during the upgrade.

screen -S <session name>

If you get disconnected, you can use the following command to reconnect:

screen -x <session name>

Note: If you are running OneFS 7.1.1.2 or 7.1.0.6, skip this step. The screen session feature

does not work in OneFS 7.1.1.2 or 7.1.0.6.

Open an SSH connection to the

highest-numbered node in the cluster,

and log in using the root account.

Restart the upgrade by running one of the following commands:

For a rolling upgrade:

isi update --rolling

For a simultaneous upgrade:

isi update

Did the

upgrade

restart?No

Go to Page 15

Wait for the upgrade

to complete.

Yes

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

__________________________

____________________

_______________________

Page 15: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

15 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Restart the upgrade (2)

You could have arrived here from:

Page 14 - Restart the upgrade

Page

15

Have all

of the nodes been

upgraded?NoYes

Go to Page 16

Run the following command to determine whether

any more nodes were upgraded:

isi_for_array -s "uname -a"

The output provides a list of all the nodes and indicates which version of

OneFS each is running. For an example of the output, see Appendix C. _________

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Page 16: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

16 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Post-upgrade checks

You could have arrived here from:

Page 15 - Restart the upgrade (2)

Page

16

NoYes

Go to Page 17 End troubleshooting

Do any

nodes report as down?

A down node means that it

failed to join the cluster

following the

upgrade.

Run the following command:

isi status -q

In the output, look at the Health DASR column to see if

any nodes report -D- (Down). For an example of the

output, see Appendix D._________

Page 17: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

17 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Post-upgrade checks (2)

You could have arrived here from:

Page 16 - Post-upgrade checks

Page

17

Yes No

End troubleshooting

Reboot each down node as follows:

1. If possible, use a serial console to connect to the node.

Otherwise, log in to the node by using SSH.

For instructions about connecting through a serial console, see

article 16744 on the EMC Online Support site.

2. After you are connected to the node, run the following command

to reboot the node:

shutdown -r now

3. Wait for the rebooted nodes to come back online.

___________

Do any

nodes report as

down? A down node means

that it failed to join the cluster

following the

reboot.

Run the following command:

isi status -q

In the output, look at the Health DASR column to see if any nodes

report -D- (Down). For an example of the output, see Appendix D.__________

If you want to determine root cause, please contact Isilon Technical

Support before continuing. If you do not want root cause analysis,

then continue.

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Page 18: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

18 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Contact Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to Isilon Technical Support

1. When troubleshooting is complete, in the command-line interface, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful , refer to article 304567 on the EMC Online Support site for

directions on how to upload files over FTP.

___________

Page 19: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

19 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page

# Note Provides context and additional

information. Sometimes a note is linked

to a process step with a colored dot.

CAUTION!Caution boxes warn that

a particular step needs

to be performed with

great care, to prevent

serious consequences.

End point Document ShapeCalls out supporting documentation

for a process step. When possible,

these shapes contain links to the

reference document.

Sometimes linked to a process step

with a colored dot.

Optional process step

Directional arrows indicate

the path through the

process flow.

IntroductionDescribes what the section helps you to

accomplish.

You could have arrived here from:

Page 4 - Start Troubleshooting

Appendix B: How to use this flowchart

Page 20: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

20 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

You could have arrived here from:

Page 6 - Analysis

Page 12 - Rolling upgrade

Page 15 - Restart the upgrade (2)

Appendix C: Output of the isi_for_array -s "uname -a" command

Example output for isi_for_array -s "uname -a"

cluster-1: Isilon OneFS cluster-1 v7.0.2.5 Isilon OneFS v7.0.2.5

B_7_0_2_216(RELEASE): 0x7000250005000D8:Mon Nov 25 20:16:16 PST 2013

[email protected]:/build/mnt/obj.RELEASE/build/mnt/src/sys/

IQ.amd64.release amd64

______________

_____________________

___________________________

Page 21: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

21 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

You could have arrived here from:

Page 7 - Analysis (2)

Page 11 - Simultaneous upgrade

Page 16 - Post-upgrade checks

Page 17 - Post-upgrade checks (2)

Appendix D: Output of the isi status -q command

Example output for isi status -q

Cluster Name: mycluster

Cluster Health: [ ATTN ]

Cluster Storage: HDD SSD

Size: 11G (23G Raw) 0 (0 Raw)

VHS Size: 11G

Used: 573M (5%) 0 (n/a)

Avail: 11G (95%) 0 (n/a)

Health Throughput (bps) HDD Storage SSD Storage

ID |IP Address |DASR | In Out Total| Used / Size |Used / Size

-------------------+-----+-----+-----+-----+-----------------+-----------------

1|192.168.100.100|-A-- | 396K| 828K| 1.2M| 144M/ 2.8G( 5%)| (No SSDs)

2|192.168.101.101|OK | 49K| 3.2M| 3.2M| 145M/ 2.8G( 5%)| (No SSDs)

3|192.168.102.102|OK | 3.5K| 162K| 165K| 142M/ 2.8G( 5%)| (No SSDs)

4|192.168.103.103|OK | 49K| 356K| 405K| 143M/ 2.8G( 5%)| (No SSDs)

-------------------+-----+-----+-----+-----+-----------------+-----------------

Cluster Totals: | 498K| 4.5M| 5.0M| 573M/ 11G( 5%)| (No SSDs)

Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

_________________

___________________________

_____________________________________________________

Page 22: EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures … · 3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from OneFS 6.5.x, 7.0.x,

22 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Failures when Upgrading from

OneFS 6.5.x, 7.0.x, 7.1.x, or 7.2.x

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Copyright © 2017 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS-IS. DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com