EMC Isilon Customer Troubleshooting Guide: Troubleshoot ......1 - EMC Isilon Customer...

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.

___________________________

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues with SNMP on Your Isilon Cluster

___________________

Abstract This guide helps you troubleshoot issues with SNMP on your Isilon cluster.

April 29, 2019

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT ISSUES WITH SNMP ON YOUR ISILON CLUSTER

OneFS 7.2 - 8.x

https://community.emc.com/docs/DOC-49017

http://bit.ly/isi-docfeedback

http://bit.ly/isilon-docfeedback



___________________________


___________________

Contents and overview

Before you beginPage 3

Appendix A If you need further assistance

Start troubleshootingPage 4

Unable to view SNMP dataPage 5

Incorrect SNMP dataPage 14

Appendix C Example output

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these steps.

2. Perform troubleshooting steps in order.

3. Appendixes

Appendix BHow to use this flowchart

Appendix D Instructions for downloading and using

PortQry






___________________________


___________________

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.

Configure screen logging through SSHWe recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

1. Open an SSH connection to the cluster and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.

2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:

cd / i f s/ dat a/ I si l on_Suppor t

3. Run the following command to capture all input and output from the session:

scr een - L

This will create a file named scr eenl og. 0 that will be appended to during your session.

4. Perform troubleshooting.




https://support.emc.com/kb/304071



___________________________


___________________

Start troubleshooting

Start

IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.

If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.

Perform an SNMPWalk to gather SNMP information from your SNMP enabled devices by running the following command for your version of SNMP:

SNMPv3Where <engi neI D> is the SNMP EngineID and <user name> is the name of a valid SNMP user.

Refer to the following article to locate the EngineID: [OneFS] How to get the SNMP EngineID of the Isilon, article 303918. <-- 7.x only

snmpwal k - v3 - e <engi neI D> - l noaut hnopr i v - u <user name> - mALL l ocal host | head

SNMPv2Where <commname> is your SNMP community name:

snmpwal k - v2c - c <commname> l ocal host | l ess

Did the snmpwalk return

output?

Does the SNMP data appear to be

correct?

Yes

No Go to Page 5

No Go to Page 14Yes

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support, as instructed in Appendix A.






___________________________


___________________

Unable to view SNMP data

Verify that the SNMP service is enabled on the cluster by running the following command for your version of OneFS:

OneFS 8.xi si ser vi ces - a snmp

OneFS 7.2i si ser vi ces - a snmpd

Is the service enabled? No

Enable the SNMP service by running the following command for your version of OneFS:

OneFS 8.xi si ser vi ces - a snmp enabl e

OneFS 7.2i si ser vi ces - a snmpd enabl e

Go to Page 6

Yes

Did this resolve the

issue?

End troubleshooting

Yes

No

You could have arrived here from:

- Page 4 - Start troubleshooting






___________________________


___________________

Unable to view SNMP data (2)

Verify that the SNMP services is running on all nodes by running the following command for your version of OneFS,

Note: "- l " is a dash and lower case "L".

OneFS 8.xi si _f or _ar r ay - sX ' pgr ep snmp | wc - l '

OneFS 7.2i si _f or _ar r ay - sX ' pgr ep snmpd | wc - l '

If the SNMP service is running on every node, the output will look similar to this example:

cl ust er - 2# i si _f or _ar r ay - sX ' pgr ep snmp| wc - l 'cl ust er - 1: 1cl ust er - 2: 1cl ust er - 3: 1


- Page 5 - Unable to view SNMP data

Is the service running

on all nodes?

Refer toFor clusters running OneFS 8.x, see the "Configure the cluster for SNMP monitoring" section of the OneFS 8.0.0 CLI Administration Guide to verify the SNMP-specific security level.

Yes

No Go to Page 7

Go to Page 10




http://www.emc.com/collateral/TechnicalDocument/docu65065.pdf



___________________________


___________________



- Page 6 - Unable to view SNMP data (2)

Verify that the isi_mcp service is running on all nodes by running the following command:

i si _f or _ar r ay - sX ' pgr ep i si _mcp | wc - l '

If the isi_mcp service is running, clusters running OneFS 8.x will return a value of 3 and clusters running OneFS 7.2 will

return a value of 2.

Is isi_mcp running on all

nodes?

Yes

On the nodes that are not running SNMP, try to locate relevant error messages by

doing the following:

- Review the / var / l og/ i si _mcp log file.

- Review the Messages log by running:gr ep - i snmp / var / l og/ messages

No


Upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

Go to Page 8






___________________________


___________________




Start the service on the nodes that isi_mcp is not running on by running the following command, where <nodes> is a

comma-separated list of the nodes that the service is not running on:

i si _f or _ar r ay - n <nodes> ' ki l l al l i si _mcp; i si _mcp'

Check to see if the SNMP services are now running on all nodes by running the following command for your version of OneFS,

OneFS 8.xi si _f or _ar r ay - sX ' pgr ep snmp | wc - l '

OneFS 7.2i si _f or _ar r ay - sX ' pgr ep snmpd | wc - l '

If the SNMP service is running on every node, the output will look similar to this example:

cl ust er - 2# i si _f or _ar r ay - sX ' pgr ep snmp| wc - l 'cl ust er - 1: 1cl ust er - 2: 1cl ust er - 3: 1

Is the service running

on all nodes?

No

Go to Page 9

Yes Did this resolve the issue? Yes End troubleshooting

Go to Page 10

No






___________________________


___________________




Check to see that the isi_mcp service is now running on all nodes by running the following command:

i si _f or _ar r ay - sX ' pgr ep i si _mcp | wc - l '

If the isi_mcp service is running, clusters running OneFS 8.x will return a value of 3 and clusters

running OneFS 7.2 will return a value of 2.

Is isi_mcp running on all

nodes?No

Start the isi_mcp process on the nodes that it is not running on. Run the following command where <nodes> is a

comma-separated list of the nodes that isi_mcp is not running on:

i si _f or _ar r ay - n <nodes> ' ki l l al l i si _mcp; i si _mcp'

On the nodes that are not running SNMP, try to locate relevant error messages by

doing the following:

- Review the / var / l og/ i si _mcp log file.

- Review the Messages log by running:gr ep - i snmp / var / l og/ messages



Appendix A.

Yes


issue?

Yes

End troubleshooting

No






___________________________


___________________



- Page 6 - Unable to view SNMP data (2)- Page 8 - Unable to view SNMP data (4)

Determine if SNMP is using a custom port by running the following command for your version of OneFS:

OneFS 8.xcat / et c/ snmpd. conf i g | gr ep SnmpdPor t

Example output, the port is indicated by the bold text:begemot SnmpdPor t St at us. 0. 0. 0. 0. 161 = 1begemot SnmpdPor t v6St at us. 0: 0: 0: 0: 0: 0: 0: 0. 161 = 1

OneFS 7.2gr ep - i " agent addr ess" / et c/ mcp/ t empl at es/ snmp_snmpd. conf

If SNMP is using the default port 161, there will be no output from this command.

If the output looks similar to the following, SNMP is using the custom port specified in the output. In this example, the custom port is 171:

agent Addr ess udp: 171

To verify that the SNMP service is listening on all nodes, run the following command where <por t > is the SNMP port number that you

identified in the previous step.

Note: The default is 161 unless you have configured a custom port.

i si _f or _ar r ay - sX ' net st at - al n | gr ep " * . <por t >" '

For example: i si _f or _ar r ay - sX ' net st at - al n | gr ep " * . 161" '

See Appendix C for example output.

Is the SNMP service listening

on all nodes?



Go to Page 11YesNo

______________________________________________________________

_________






___________________________


___________________




From the SNMP host, try to ping the IP address of the node that you

are attempting to connect to:

pi ng <I Paddr ess>

Is the SNMP host able to ping the

node IP address?No



Appendix A.

Yes

From the SNMP host, verify that you can connect to the SNMP port. Run the following command, where <I P> is the IP address of the node, and <por t > is the SNMP port number that you identified on page 10:

For Linux/BSD SNMP hostsnc - u - z <I P> <por t >

For example:nc - u - z 10. 1. 1. 11 161

For Windows hostsThere is no native tool on Windows to test this. Optionally, you may

install the Microsoft® PortQry Command Line Port Scanner (see Appendix D for instructions on downloading and using the tool). If you

are unable to use the Microsoft® tool, you will need to use a Linux machine. If you do not have access to a Linux machine, have your Networking team verify whether the SNMP port can be reached.

Can you connect to the SNMP port?

No

Yes

Go to Page 12

This indicates a potential routing or

firewall issue.

_______

_________






___________________________


___________________




Obtain your SNMP configuration from your SNMP host.

From the Isilon cluster, run the following command:

i si snmp set t i ngs vi ew

Compare your SNMP configuration to the configuration

on the Isilon cluster.

Do the SNMP host and cluster settings match?

No

Go to Page 13

Yes


issue?Yes End

troubleshooting

No

You will need to adjust the SNMP configuration on either the host or the cluster so that they

match.

For more information on editing SNMP settings on the Isilon cluster, see the Isilon OneFS CLI

Administration Guide for your version of OneFS.



_______________________________








___________________________


___________________




Initiate a packet capture on the cluster and the client and reproduce the issue.

Cluster packet capture: For instructions, see OneFS: How to gather Isilon cluster diagnostics using the

isi_netlogger command, article 304448.

Client packet capture: Perform the packet capture on your client, using the method appropriate for the operating

system and configuration.



Appendix A.

Gather a full log set that includes the cluster packet capture:

i si _gat her _i nf o - f / i f s/ net l og

___________








___________________________


___________________

Incorrect SNMP data


- Page 4 - Start troubleshooting

Are you using Isilon-MIB for SNMP

monitoring?Yes

No

Download the Isilon MIB from the cluster web administration interface:

Click Cluster Management > General Settings > SNMP Monitoring. Click the Download link for ISILON-MIB.

After downloading and using Isilon MIB, retest and if the data still appears incorrect, engage Isilon Technical Support.

Go to Page 15


issue?

Yes

End troubleshooting

No

Gather the following information and continue to page 15:

- An example of the inaccurate reporting you are experiencing.

- What you expect the report to show.- Your SNMP host configuration.






___________________________


___________________

Incorrect SNMP data (2)


- Page 14 - Incorrect SNMP data

Perform a full snmpwalk and save it. Depending on which SNMP version you are using, run the following two commands, where <commname> is the

community name.

Note: for each version of SNMP, there are two full commands. Copy and paste the commands into the cluster CLI and remove any hard line breaks. Press Enter to execute the command.

SNMP v2snmpwal k - v2c - c <commname> - m / usr / l ocal / shar e/ snmp/ mi bs/ I SI LON- MI B. t xt l ocal host >

/ var / cr ash/ snmpwal k. out

snmpwal k - v2c - c <commname> - m / usr / l ocal / shar e/ snmp/ mi bs/ I SI LON- MI B. t xt l ocal host I si l on >> / var / cr ash/ snmpwal k. out

SNMP v3snmpwal k ?v3 ?e <engi ne I D> - l noaut hnopr i v ?u <user name> - mALL l ocal host > / var / cr ash/ snmpv3wal k. out

snmpwal k ?v3 ?e <engi ne I D> - l noaut hnopr i v ?u <user name> - mALL l ocal host I si l on >> / var / cr ash/ snmpv3wal k. out

Gather a full log set that includes the snmpwalk

i si _gat her _i nf o - f / var / cr ash/ snmpwal k. out

Gather the following information:

- An example of the inaccurate reporting you are experiencing.- What you expect the report to show.- Your SNMP host configuration.



Appendix A.






___________________________


___________________

Appendix A: If you need further assistance

___________

Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Upload node log files and the screen log file to Isilon Technical Support1. When troubleshooting is complete, in the command-line interface, type exi t to end your screen session.2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for

your method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

FTP:

i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

HTTP:

i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SMTP:

i si _gat her _i nf o - - emai l - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SupportIQ:Copy and past the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.

i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \ - - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p

3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the Dell EMC Online Support site for directions on how to upload files over FTP.





http://www.emc.com/contact/contact-us.htm



___________________________


___________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page # Note

Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.

CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.

End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.

Optional process step

IntroductionDescribes what the section helps you to accomplish.

Appendix B: How to use this flowchart


- Page 4 - Start Troubleshooting

Directional arrows indicate the path through

the process flow.






___________________________


___________________

Appendix C: Example output



Example outputIf the SNMP service is listening, the output will look similar to the following for your version of OneFS.

OneFS 8.xEach node should have two lines of output, with the correct SNMP port listed:Cl ust er 2- 1: udp6 0 0 * . 161 * . *Cl ust er 2- 1: udp4 0 0 * . 161 * . * Cl ust er 2- 2: udp6 0 0 * . 161 * . *Cl ust er 2- 2: udp4 0 0 * . 161 * . * Cl ust er 2- 3: udp6 0 0 * . 161 * . *Cl ust er 2- 3: udp4 0 0 * . 161 * . *

OneFS 7.2Each node should have one line of output, with the correct SNMP port listed:

cl ust er - 1: udp4 0 0 * . 161 * . *cl ust er - 2: udp4 0 0 * . 161 * . *cl ust er - 3: udp4 0 0 * . 161 * . *






___________________________


___________________

Appendix D: Instructions for downloading and using PortQry



Using the Microsoft PortQry Command Line Port Scanner

Download the tool here:http://www.microsoft.com/en-us/download/details.aspx?id=17148

The command syntax for the tool is:por t qr y - n <I P> - p udp - e <por t > - nr | f i ndst r / B " UDP"

If the port is reachable, the response is similar to one of the following:UDP por t 161 ( snmp ser vi ce) : LI STENI NGUDP por t 161 ( snmp ser vi ce) : FI LTERED

If the port is not reachable, the response is similar to the following:UDP por t 161 ( snmp ser vi ce) : NOT LI STENI NG

____________________________________________________




http://www.microsoft.com/en-us/download/details.aspx?id=17148



___________________________


___________________

Copyright 2019 © Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

EMC Isilon Customer Troubleshooting Guide: Troubleshoot ......1 - EMC Isilon Customer...

Documents

Transcript of EMC Isilon Customer Troubleshooting Guide: Troubleshoot ......1 - EMC Isilon Customer...