EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues … · 2020-03-05 · Event Log...

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.

___________________________

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshooting Issues with the OneFS Cluster Event Log

___________________

Abstract This guide helps you troubleshoot issues with the OneFS cluster event log (CELOG) on your Isilon cluster.

April 30, 2019

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT ISSUES WITH THE OneFS CLUSTER EVENT LOG

OneFS 8.x

https://community.emc.com/docs/DOC-49017

http://bit.ly/isi-docfeedback

http://bit.ly/isilon-docfeedback



___________________________


___________________

Contents and overview

Before you beginPage 3

Appendix A If you need further assistance

Start troubleshootingPage 4

RoutingPage 5

Test messagePage 6

Verify processes are runningPage 7

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these steps.

2. Perform troubleshooting steps in order.

3. Appendixes

Appendix BHow to use this flowchart

Enable ESRSPage 11

DebugPage 17

Check for subscribed alertsPage 13






___________________________


___________________

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

1. Open an SSH connection to the cluster and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.

2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:

cd / i f s/ dat a/ I si l on_Suppor t

3. Run the following command to capture all input and output from the session:

scr een - L

This will create a file named scr eenl og. 0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.




https://support.emc.com/kb/304071



___________________________


___________________

Note Below is a list of log locations and what type of content each log contains:

- / var / l og/ i si _cel og_moni t or . l og This log contains system monitoring and event creation details.

- / var / l og/ i si _cel og_capt ur e. l og This log contains recorded events and attachments.

- / var / l og/ i si _cel og_anal ysi s. l og This log is responsible for assigning events to eventgroups.

- / var / l og/ i si _cel og_r epor t i ng. l og This log evaluates alert conditions and manages sending alert events.

- / var / l og/ i si _cel og_al er t i ng. l og This log is responsible for sending alerts.

Start troubleshooting

Start

IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.

If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.

Run the following command and record the output for use later in this guide:

i si event event s l i st

Did that command return output, indicating that events

are being generated?Go to Page 6

Go to Page 5

Yes

No






___________________________


___________________

Routing

Should you be receiving ESRS/SRS

dial homes but are not?

You could have arrived here from:

- Page 4 - Start troubleshooting

No

Are you having a problem receiving

SNMP notifications or email alerts?

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

Yes

Go to Page 11Yes

No

Are you receiving incorrect

alerts, or alerts with unexpected

content?

No

Go to Page 17

Yes






___________________________


___________________

Test message


- Page 4 - Start troubleshooting

Run the following command:

i si event s t est cr eat e - - message=" t est message"

Do not omit the quotation marks.

Wait five minutes and then run:

i si event event s l i st | gr ep t est

This should return output that appears similar to this:

cl ust er - 1# i si event event s l i st | gr ep t est1. 82662 04/ 09 14: 54 W 1 93594 t est1. 82661 04/ 09 14: 51 W 1 93594 t est message

Does the message titled "test message"

appear?

No

Review the following log locations. Search for errors related to the test message that you created. Record

any of your findings.

/ var / l og/ i si _cel og_moni t or . l og / var / l og/ i si _cel og_capt ur e. l og/ var / l og/ i si _cel og_anal ysi s. l og

YesAlerts are being recorded on the cluster. Continue to monitor the

cluster and contact Isilon Technical Support if the issue persists.

Go to Page 7






___________________________


___________________

Verify processes are running


- Page 6 - Test message- Page 10 - Verify processes are running (4)

Verify that processes are running by running the following command:

i si _f or _ar r ay - s " ps - auwwx | gr ep cel og | gr ep - v gr ep | wc - l "

There are typically 15 to 30 processes running for CELOG. The average is 20 to 24 processes. If any nodes are on the lower end of the process count, investigate those

particular nodes. Record the running processes for comparison later.

In this example, node 1 is showing a less-than-average amount of processes running and should be investigated:

cl ust er - 1# i si _f or _ar r ay - s " ps - auwwx| gr ep cel og| gr ep - v gr ep| wc - l "cl ust er - 1: 18 <- - l ow endcl ust er - 2: 20cl ust er - 3: 24

Restart the MCP process on the nodes with less-than-average process counts:

ps - auwwx | gr ep mcp | gr ep - v gr ep

Takes notes of which processes are running, there should be three processes running as shown in this example:

cl ust er - 1# ps - auwwx| gr ep mcp| gr ep - v gr epr oot 1540 3. 0 0. 9 140840 9140 - I s 10: 23AM 1: 10. 15 i si _mcp: mast er ( i si _mcp)r oot 1368 0. 0 0. 8 137508 7748 - I s 10: 23AM 0: 00. 02 i si _mcp: f ai l saf e ( i si _mcp)r oot 1370 0. 0 0. 8 137508 7884 - I s 10: 23AM 0: 00. 12 i si _mcp: f or ker ( i si _mcp)

Go to Page 8

Did you locate any errors related to creating

the test message?

No


Upload log files and contact Isilon Technical Support, as instructed in Appendix A.

Yes

_____________________________________________________






___________________________


___________________

Verify processes are running (2)


- Page 7 - Verify processes are running

Stop all MCP processes by running the following command:

ki l l al l - 9 i si _mcp

Verify that all processes were stopped by running:


If there is no output, all the processes were stopped successfully.

Restart the processes by running:

i si _mcp

Verify that all processes were restarted by running:


There should be three processes running as shown in this example:

cl ust er - 1# ps - auwwx| gr ep mcp| gr ep - v gr epr oot 1540 3. 0 0. 9 140840 9140 - I s 10: 23AM 1: 10. 15 i si _mcp: mast er ( i si _mcp)r oot 1368 0. 0 0. 8 137508 7748 - I s 10: 23AM 0: 00. 02 i si _mcp: f ai l saf e ( i si _mcp)r oot 1370 0. 0 0. 8 137508 7884 - I s 10: 23AM 0: 00. 12 i si _mcp: f or ker ( i si _mcp)

Go to Page 9






___________________________


___________________



- Page 8 - Verify processes are running (2)

Disable CELOG by running the following command:

i si ser vi ces - a cel og di sabl e

Wait 60 to 120 seconds and run the following command to verify that the CELOG service stopped:

i si _f or _ar r ay - s " ps - auwx | gr ep ' [ c] el og' "

Did the service stop? No

Run the following command:

i si _f or _ar r ay - sX ' ki l l al l - 6 $<pgr ep i si _cel og) '

Yes

Enable CELOG by running:

i si ser vi ces - a cel og enabl e

Go to Page 10






___________________________


___________________



- Page 9 - Verify processes are running (3)

Verify that services are running:

i si _f or _ar r ay - s " ps - auwwx| gr ep cel og| gr ep - v gr ep| wc - l "

Compare the output to the output on page 7.

Is there now an average amount of

services running per the guidelines on

page 7?

Yes Has the original issue been fixed? Yes End

troubleshooting

Refer to:{ISILON} OneFS 8.x: Unable to

restart celog, article 519875

No



Appendix A.

Did this article fix the issue? No

Yes

End troubleshooting

_____

_____







___________________________


___________________

Enable ESRS


- Page 5 - Routing

Verify that ESRS is enabled by running the following command for your version of OneFS:

OneFS 8.1i si esr s vi ew

OneFS 8.0 and earlieri si r emot esuppor t connect emc vi ew

Is ESRS enabled? No

Enable ESRS by running:

i si ser vi ces - a i si _esr s_d enabl e

Yes

Go to Page 12






___________________________


___________________

Enable ESRS (2)


- Page 11 - Enable ESRS- Page 14 - Check for subscribed alerts (2)

Verify that the Remot eSuppor t (ESRS) channel is enabled by running the following command:

i si event channel s l i st

See the note box on this page for example output.

Example outputcl ust er - 1# i s i event channel s l i s tI D Name Type Enabl ed- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 Remot eSuppor t connect emc Yes2 Hear t beat Sel f - Test hear t beat Yes- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Is the channel enabled? No

Enable the Remot eSuppor t channel by running:

i si event channel s modi f y Remot eSuppor t - - enabl ed=t r ue

Yes

Go to Page 13

Take note of the value in the I D column of the Remot eSuppor t channel for use later

in this guide.

_____________________________________________________






___________________________


___________________

Check for subscribed alerts


- Page 12 - Enable ESRS (2)- Page 16 - Check for subscribed alerts (4)

Check for alerts subscribed to Remot eSuppor t by running the

following command:

i si event al er t s l i st


Example outputcl ust er - 1# i si event al er t s l i stName Sev Channel Condi t i on Descr i pt i on- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Connect EMC New * Remot eSuppor t NEW al l cat egor i esConnect EMC Resol ved * Remot eSuppor t RESOLVED al l cat egor i esConnect EMC Wor sened * Remot eSuppor t SEVERI TY I NCREASE al l cat egor i esHear t beat * Hear t beat Sel f - Test ONGOI NG 1 event gr oup( s)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Tot al : 4

Are there alerts subscribed to the Remot eSuppor t

channel?

No

Subscribe to alerts by running the following command, where <al er t > is the name of the alert and <channel > is the name of the ESRS

channel (default channel name is: Remot eSuppor t ):

i si event al er t s modi f y <al er t > add- channel =<channel >

Go to Page 14Yes

Did this resolve the

issue?NoYesEnd troubleshooting

_______________________________________________________






___________________________


___________________

Check for subscribed alerts (2)


- Page 13 - Check for subscribed alerts

Check the alerts directories by running the following command, where <I D> is the channel ID that you located on page 12:

cd / i f s/ . i f svar / db/ cel og_al er t i ng/ <I D>

List the contents of the alerts directory by running:

l s - l


Example outputcl ust er - 1# cd / i f s/ . i f svar / db/ cel og_al er t i ng/ 1cl ust er - 1# l s - lt ot al 72- r wxr - xr - x 1 r oot wheel 0 Dec 15 10: 56 f ai l . l og- r w- r - - r - - 1 r oot wheel 13 Mar 23 00: 01 l ast _bat ch_sentdr wxr - xr - x 2 r oot wheel 0 Mar 23 00: 01 r equest sdr wxr - xr - x 2 r oot wheel 431 Mar 23 00: 01 sent- r wxr - xr - x 1 r oot wheel 379 Mar 23 00: 01 sent . l og.

Review the contents of the r equest s and sent directories to determine what is waiting to be sent and what has already been sent.

Go to Page 15

_______






___________________________


___________________



- Page 14 - Check for subscribed alerts (2)

Are any recent or new events showing up in

the r equest s or sent directories?

Go to Page 16

No,neither

Yes,sent

Yes,r equest s

If recent or new events are showing up in the r equest s directory, this indicates

that the cluster is having problems relating the events to the gateway.



Appendix A.



Appendix A.






___________________________


___________________



- Page 15 - Check for subscribed alerts (3)

Is this your second time on

this page?No

Check page 13 to ensure events are being generated, and to ensure alerts are configured correctly to trigger an alert.

Yes

Are the alerts configured properly?

Yes



Appendix A.

No

Configure the alerts properly.

Are events now being routed to either the r equest s or the

sent directories?

NoEnd troubleshooting Yes





We appreciate your help in improving this document. Submit your feedback at http://bit.ly/Isilon-docfeedback.


______________________________________________

Debug


- Page 5 - Routing

Enable debug logging by copying and pasting the following command into the command-line interface.This is a single command and line breaks will need to be removed before executing:

i si _f or _ar r ay " / usr / bi n/ i si _cel og_moni t or r est ar t debug; pki l l - SI GUSR1 i si _cel og_al er t i ng; pki l l - SI GUSR1 i si _cel og_r epor t i ng; pki l l - SI GUSR1 i si _cel og_capt ur e; pki l l - SI GUSR1 i si _cel og_anal ysi s"

Reproduce the issue you are trying to fix.

Go to Page 18





We appreciate your help in improving this document. Submit your feedback at http://bit.ly/Isilon-docfeedback.


______________________________________________

Debug (2)


- Page 17 - Debug

Once the issue has been reproduced successfully, disable debug logging by copying and pasting the following command into the command-line interface.

This is a single command and line breaks will need to be removed before executing.

i si _f or _ar r ay " / usr / bi n/ i si _cel og_moni t or r est ar t ; pki l l - SI GUSR1 i si _cel og_al er t i ng; pki l l - SI GUSR1 i si _cel og_r epor t i ng; pki l l - SI GUSR1 i si _cel og_capt ur e; pki l l - SI GUSR1 i si _cel og_anal ysi s"



Appendix A.

CAUTION!Do not forget to disable debug logging once you have completed your debug. Keeping debug logging enabled causes OneFS to log more messages in the log files. Over time, the additional log files could fill the / var partition. In addition, the / var partition is stored on a node's boot drive and excessive logging could adversely affect the wear life of those drives.






___________________________


___________________

Appendix A: If you need further assistance

___________

Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Upload node log files and the screen log file to Isilon Technical Support1. When troubleshooting is complete, in the command-line interface, type exi t to end your screen session.2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for

your method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

FTP:

i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

HTTP:

i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SMTP:

i si _gat her _i nf o - - emai l - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SupportIQ:Copy and past the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.

i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \ - - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p

3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the Dell EMC Online Support site for directions on how to upload files over FTP.





http://www.emc.com/contact/contact-us.htm



___________________________


___________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page # Note

Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.

CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.

End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.

Optional process step

IntroductionDescribes what the section helps you to accomplish.

Appendix B: How to use this flowchart


- Page 4 - Start Troubleshooting

Directional arrows indicate the path through

the process flow.






___________________________


___________________

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues … · 2020-03-05 · Event Log...

Documents

Transcript of EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues … · 2020-03-05 · Event Log...