EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE...1 - EMC Isilon Customer Troubleshooting Guide:...

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Kerberos issues on your Isilon

cluster

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Abstract

This troubleshooting guide helps you with Kerberos authentication issues on your Isilon cluster.

February 13, 2018

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT KERBEROS ISSUES ON YOUR ISILON CLUSTER

OneFS 7.2.0 - 8.1.0

http://bit.ly/isi-docfeedback


https://community.emc.com/docs/DOC-49017


cluster



___________________________

Contents and overview

Page 3 Before you begin

Appendix A If you need further assistance

Page 4 Start troubleshooting

Page 5 Verify the SmartConnect zone name

Page 6 Active Directory

Page 7 Check for time skew

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these

steps.

2. Perform

troubleshooting

steps in order.

3. Appendixes

Appendix B How to use this flowchart

Page 8 Verify the domain

Page 9 Check for missing SPNs

Page 10 Add missing SPNs

Page 11 Verify SPNs

Page 12 Retrieve a Kerberos ticket

Page 13 Verify the Kerberos ticket

Page 14 Domain controllers

Page 15 Check for duplicate SPNs

Page 16 Remove duplicate SPNs

Page 17 Verify the Kerberos ticket (2)

Page 18 Packet capture





cluster



___________________________

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session.

This log file can be shared with EMC Isilon Technical Support, if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,

you can configure logging by using your local SSH client's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account .

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be

preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster, you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial cable

available. This way, if you are unable to connect through the network, you will still be

able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 16744 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.




https://support.emc.com/kb/16744


cluster



___________________________

Start troubleshooting

Go to Page 5

Start

IntroductionStart troubleshooting here. If you need

help to understand the flowchart

conventions that are used in this guide,

see Appendix B: How to use this

flowchart.

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on page 3.

Open an SSH connection to any node connected to the

network with access to the domain controllers (DCs) for

the domain.

During the troubleshooting process, we will troubleshoot each

problematic SmartConnect zone name, one at a time.

You will begin troubleshooting the first zone name on the next page.





cluster



___________________________

Verify the SmartConnect zone name

Page

5

You could have arrived here from:

Page 4 - Start troubleshooting

Begin troubleshooting the first problematic SmartConnect zone name.

Verify the name of the SmartConnect zone that is having issues by running the following command :

OneFS 8.0.0 - 8.1.0

isi network pools list -v

OneFS 7.2.0

isi networks list pools -v

From the output for the problematic pool, record the Access Zone, SmartConnect Zone, and

SmartConnect Zone Aliases. See Appendix C for example output.

Go to Page 6

__________





cluster



___________________________

Active Directory

Are you using NFS with

Kerberos in a non-Active

Directory environment?

Page

6


Page 5 - Verify the SmartConnect zone

name

Yes

No

Were you directed to

this guide from:

EMC Isilon Customer

Troubleshooting Guide:

Troubleshoot Windows Active

Directory Authentication?

Go to Page 7Yes

Go to:

EMC Isilon Customer



Directory Authentication

No

Note the page number that you are

currently on.

Upload log files and contact Isilon

Technical Support, as instructed in

Appendix A.




http://www.emc.com/collateral/TechnicalDocument/docu63151.pdf


cluster



___________________________

Check for time skew

A time skew on the cluster can cause authentication issues. Verify that the

time on the cluster is accurate by running the following command, where

<dcIP> is the IP address of the DC:

ntpdate -q -u <dcIP>

The example output at the bottom of this page shows a large time offset.

Page

7


Page 6 - Active Directory

How many

seconds is the time

off by, if any?

More than

300 seconds299 seconds

or less

Go to Page 8

Verify the IP address of the domain controller on the domain that you are troubleshooting by running

the following command:

isi auth ads trusts list --provider=domain.com | egrep -i 'domain|DC|---'

See Appendix D for example output.

This indicates a timekeeping problem.

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Example ntpdate -q -u <dcIP> output

ntpdate -q -u 1.1.1.1

server 1.1.1.1, stratum 1, offset -25220.651826, delay 0.04146

24 Mar 23:10:49 ntpdate[3078]: step time server 1.1.1.1 offset -25220.651826 sec

__________





cluster



___________________________

Verify which domain you are joined to by running the following

command:

isi auth ads list

The domain name is shown in bold in the example output at the

bottom of this page.

Verify the domain

Page

8


Page 7 - Check for time skew

Example isi auth ads list output

Cluster-1# isi auth ads list

Name Authentication Status Site

---------------------------------------------------------------------------

DOMAIN.LOCAL.isilon.com Yes online Default-First-Site-Name

---------------------------------------------------------------------------

Go to Page 9

Is the domain reporting

as online?

Yes

No

Go to:

EMC Isilon Customer



Directory Authentication




http://www.emc.com/collateral/TechnicalDocument/docu63151.pdf


cluster



___________________________

Check for missing SPNs

Page

9


Page 8 - Verify the domain

Check the SmartConnect zone for any missing SPNs by

running the following command, where <FQDN> is the fully

qualified domain name:

isi auth ads spn check --domain=<FQDN>

For SMB access, a Service Principal Name (SPN) is constructed like this:

host/cluster.isilon.com

Host is the prefix and a type of SPN.

Cluster is the short name.

Cluster.isilon.com is the fully qualified domain name (FQDN).

If the following two SPNs are not available, or are not listed in this format,

you will add them in the next few steps.

host/cluster

host/cluster.isilon.com

Note This command is not zone aware

which means if the cluster is

joined to multiple trusted

domains, there may be

unnecessary SPNs. If this is the

case, contact Isilon Technical

Support.

Are there any

missing SPNs for this

SmartConnect zone

name?

Go to Page 10

Yes

Go to Page 15No

Refer toFor more information on SPNs,

see OneFS Service Principal

Names for Kerberos

Authentication, article 336420






cluster



___________________________

Add missing SPNs

Page

10


Page 9 - Check for missing SPNs

Page 11 - Verify SPNs

Add any missing SPNs back by running the following command, where:

<FQDN> is the short name that the cluster is joined to

<domainadmin> is the user name for a domain administrative user

isi auth ads spn create --spn=HOST/<FQDN.domain.com> --user=<domainadmin>

Note that when adding SPNs, the replication

process between DCs can take between 5 and 15

minutes. For large environments, replication can

take over one hour to complete.

Go to Page 11

Note If the following error message appears when

you attempt to add SPNs:

LDAP CONSTRAINT VIOLATION

Verify that the user that is trying to add the

SPNs is a domain administrator. If the add

operation continues to fail, contact Isilon

Technical Support.

___________________________

__________________

If you want to use the short name, you must add the short name SPN by running the following command, where:

<shortname> is the SmartConnect zone name without the domain suffix

<FQDN> is the fully qualified domain name that the cluster is joined to

<domainadmin> is the user name for a domain administrative user

isi auth ads spn create --spn=HOST/<shortname> --domain="<FQDN>" --user=<domainadmin>

Note Service Principal Names (SPNs)

must be propagated to all DCs to

make them available to clients.





cluster



___________________________

Verify SPNs

Page

11


Page 10 - Add missing SPNs

Recheck for any missing SPNs by running the following

command, where <FQDN> is the fully qualified domain

name that the cluster is joined to:

isi auth ads spn check --domain=<FQDN>

Are there any

missing SPNs for this

SmartConnect zone

name?

Note If this is the second time that you have

checked the same SmartConnect zone

name for missing SPNs, and if you are

unable to add those SPNs, upload log

files and contact Isilon Technical


Return to Page 10Yes

Go to Page 12

No





cluster



___________________________

Retrieve a Kerberos ticket

Page

12


Page 11 - Verify SPNs

This completes the check for the first SmartConnect zone name. Return

to page 5 and complete the steps for all the remaining affected zones. If

there are no more affected zones, continue with troubleshooting.

Using the same client, reconnect to the cluster

to trigger retrieval of a Kerberos ticket.

Go to Page 13

Purge a Windows® client of tickets by running the

following three commands on the client:

klist purge

ipconfig /flushdns

net use * /delete

If DNS is not properly configured,

Kerberos tickets cannot be granted.

Refer to:

OneFS: How to configure Windows

DNS for a SmartConnect zone, article

442679






cluster



___________________________

Verify the Kerberos ticket

Page

13


Page 12 - Retrieve a Kerberos ticket

Verify the Kerberos ticket by running the

following command on a Windows client:

klist

Locate the SPN in the output. See the

bold line in the example output at the


Example output

The output has been truncated to display the specific Kerberos ticket for the client and domain that is in question .

#9> Client: user @ LOCAL.COM

Server: host/domain.local.com @ LOCAL.COM

KerbTicket Encryption Type: XXX-###-XXX-XXXX-XXXX-##

Ticket Flags 0x40a00000 -> forwardable renewable pre_authent

Start Time: 2/17/2016 9:24:52 (local)

End Time: 2/17/2016 19:24:51 (local)

Renew Time: 2/24/2016 9:24:51 (local)

Session Key Type: XXX-###-XXX-XXXX-XXXX-##

Is the SPN listed in

the klist output?

Kerberos is now

working for this

particular zone name.

If you have additional

zone names to

troubleshoot, return to

page 5.

Go to Page 14

No

Yes YesHas the issue

been solved?


are currently on.



No





cluster



___________________________

Domain controllers

Page

14


Page 13 - Verify the Kerberos ticket

Page 15 - Check for duplicate SPNs

Page 17 - Verify the Kerberos ticket (2)

Determine which DCs your nodes are connected to by running the following command , where

<zone> is the name of the access zone that you obtained on page 5:

isi auth ads trusts list --provider=domain.com | egrep -i 'domain|DC|---'

See Appendix D for example output.

Are any of the

DCs read-only?

This can be verified with

your Active Directory

administrator.

Microsoft's best practices indicate

that read-only DCs should not be

mixed with read/write DCs in the

same site.

Yes

No

Go to Page 18

_____________________________

_____________________________

_______________________________

__________





cluster



___________________________

Check for duplicate SPNs

Page

15


Page 9 - Check for missing SPNs

Check for duplicate SPNs on the DC by running the following

command from a Windows command prompt, where <sczonename>

is the name of the SmartConnect zone that you obtained on page 5:

setspn -Q */ <sczonename>

Are any duplicate

SPNs listed?Return to Page 14No

Go to Page 16

Yes





cluster



___________________________

Remove duplicate SPNs

Page

16


Page 15 - Check for duplicate SPNs

Remove duplicate SPNs by running the following command, where <spn>

is the SPN name and <AccountName> is the machine account name.

Make sure you do not remove duplicate SPNs from the Isilon cluster.

setspn -D <spn> <AccountName>

This step can only be performed by a domain admin.

Note that when adding SPNs, the replication process between DCs can

take between 5 and 15 minutes. For large environments, replication can

take over one hour to complete.

Using the same client, reconnect to the cluster to

trigger retrieval of a Kerberos ticket.

Go to Page 17

Purge a Windows client of tickets by running the

following three commands on the client:

klist purge

ipconfig /flushdns

net use * /delete





cluster



___________________________

Verify the Kerberos ticket (2)

Page

17


Page 16 - Remove duplicate SPNs

Verify the Kerberos ticket by running the

following command on a Windows client:

klist

Locate the SPN in the output. See the

bold line in the example output at the


Example output

The output has been truncated to display the specific Kerberos ticket for the client and domain that is in question .

#9> Client: user @ LOCAL.COM

Server: host/domain.local.com @ LOCAL.COM

KerbTicket Encryption Type: XXX-###-XXX-XXXX-XXXX-##

Ticket Flags 0x40a00000 -> forwardable renewable pre_authent

Start Time: 2/17/2016 9:24:52 (local)

End Time: 2/17/2016 19:24:51 (local)

Renew Time: 2/24/2016 9:24:51 (local)

Session Key Type: XXX-###-XXX-XXXX-XXXX-##

Is the SPN listed in

the klist output?

Return to Page 14

Yes

No

Is the issue

resolved?


are currently on.



Yes

No

Kerberos is now

working for this

particular zone name.

If you have additional

zone names to

troubleshoot, return to

page 5.





cluster



___________________________

Packet capture

Page

18


Page 14 - Domain controllers

Initiate a PCAP on the cluster by running the following command,

where <IP> is the IP address of the client:

isi_netlogger -c -i all -s 800 -- host <IP>

Note A packet capture (PCAP) will help you to

determine whether there is a problem with an

intermediary device that is between your cluster

and the resource that you are trying to access.

Best practice is to keep the PCAP as short as

possible to keep the diagnostic file a

manageable size.

Reproduce the issue.

End the PCAP by

pressing CTRL+C.

Upload the PCAP to Isilon Technical Support by running the following command:

isi_gather_info --nologs -n1 -f $(ls -1dtr /ifs/netlog/bundled/* | tail -n 1)

Record the resulting .tgz file name that is generated.


are currently on.







cluster



___________________________

Contact EMC Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to EMC Isilon Technical Support

1. When troubleshooting is complete, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful , refer to article 16759 on the EMC Online Support site for

directions on how to upload files over FTP.___________




http://www.emc.com/contact/contact-us.htm



cluster



___________________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page

# Note Provides context and additional

information. Sometimes a note is linked

to a process step with a colored dot.

CAUTION!Caution boxes warn that

a particular step needs

to be performed with

great care, to prevent

serious consequences.

End point Document ShapeCalls out supporting documentation

for a process step. When possible,

these shapes contain links to the

reference document.

Sometimes linked to a process step

with a colored dot.

Optional process step

Directional arrows indicate

the path through the

process flow.

IntroductionDescribes what the section helps you to

accomplish.


Page # - Page title

Appendix B: How to use this flowchart





cluster



___________________________


Page 5 - Verify the SmartConnect zone name

Appendix C: Example output

OneFS 7.2.0 example output

cluster-1# isi networks list pools -v

subnet0:pool0 - Default ext-1 pool

In Subnet: subnet0

Allocation: Static

Ranges: 1

1.1.1.25-1.1.1.30

Pool Membership: 2

1:ext-1 (up)

2:ext-1 (up)

Aggregation Mode: Link Aggregation Control Protocol (LACP)

Access Zone: System (1)

SmartConnect:

Suspended Nodes : 2

Auto Unsuspend ... 0

Zone : cluster.test.local

Zone Aliases : 1

clusteralias.test2.local

Time to Live : 0

Service Subnet : subnet0 Connection Policy: Round Robin

OneFS 8.1.0 example output

cluster-1# isi network pools list -v

ID: groupnet0.example.pool1

Groupnet: groupnet0

Subnet: subnet0

Name: pool1

Rules: -

Access Zone: System

Allocation Method: static

Aggregation Mode: lacp

SC Suspended Nodes: -

Description: -

Ifaces: 1:ext-1

IP Ranges: 1.1.1.25-1.1.1.30

Rebalance Policy: auto

SC Auto Unsuspend Delay: 0

SC Connect Policy: round_robin

SC Zone: cluster.test.local

SC DNS Zone Aliases: 1

clusteralias.test2.local

SC Failover Policy: round_robin

SC Subnet: -

SC TTL: 0

Static Routes: -





cluster



___________________________


Page 7 - Check for time skew

Page 14 - Domain controllers

Appendix C: Example output

_______________________

_______________________

Example output

cluster-1# isi auth ads trusts list --provider=bamf.int | egrep -i 'domain|DC|---'

Domain: domain.com

DC Site: yourcity

DC Name: dcname.example.com

DC Address: 1.1.1.23

------------------------------------------------------------------------

Domain: domain2.com

DC Site: yourcity

DC Name: dcname1.example1.com

DC Address: 1.1.1.35

------------------------------------------------------------------------





cluster



___________________________

Copyright © 2018 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS-IS. DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE...1 - EMC Isilon Customer Troubleshooting Guide:...

Documents

Transcript of EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE...1 - EMC Isilon Customer Troubleshooting Guide:...