8/11/2019 OneFS Upgrade Planning and Process Guide
1/52
Isilon
OneFS
Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
2/52
Copyright 2013-2014 EMC Corporation. All rights reserved. Published in USA.
Published March, 2014
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change
without notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind withrespect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a
particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.
EMC, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and othercountries. All other trademarks used herein are the property of their respective owners.
For the most up-to-date regulatory document for your product line, go to EMC Online Support (https://support.emc.com). Fordocumentation on EMC Data Domain products, go to the EMC Data Domain Support Portal (https://my.datadomain.com).
EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 In North America 1-866-464-7381www.EMC.com
2 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
3/52
Introduction to this guide 5
About this guide..............................................................................................6Isilon scale-out NAS overview..........................................................................6Where to go for support...................................................................................6Documentation resources............................................................................... 6
Planning an upgrade 9
Upgrade planning overview .......................................................................... 10Types of OneFS upgrades.............................................................................. 10Verify the upgrade path.................................................................................11Check system requirements.......................................................................... 11
Assess upgrade impact................................................................................. 12Consider upgrade limitations........................................................... 12Review feature changes and known issues.......................................12
Plan an upgrade schedule.............................................................................13Upgrade a test cluster................................................................................... 13
Completing pre-upgrade tasks 15
About pre-upgrade tasks...............................................................................16Check cluster readiness................................................................................ 16
Restart thecluster............................................................................16Check hardware health.................................................................... 16Check for free node space................................................................18
Resolve outstanding events and errors.............................................19Verify configurations and settings................................................................. 20
Verify global namespace requirements.............................................20Check diskpool configurations........................................................ 21Verify the disk pool policy................................................................ 21Configure the LDAP service...............................................................22Check SmartLock mode....................................................................22Reconfigure unsupported SMB settings............................................22Configure Kerberos settings............................................................. 23Install supported version of InsightIQ...............................................23Install supported version of Isilon for vCenter...................................23
Upgrade compatibility check utility............................................................... 24
Upgrade compatibility checks.......................................................... 24Download the OneFS installation image........................................... 26Run the upgrade compatibility check utility......................................27
Back up data.................................................................................................28SyncIQ backup.................................................................................28NDMP backup.................................................................................. 28Back up custom settings..................................................................29
Collect cluster data....................................................................................... 31Collect cluster information............................................................... 31Collect cluster status........................................................................33Collect cluster logs...........................................................................33
Complete or stop jobs in progress.................................................................33Complete NDMP backup...................................................................33
Chapter 1
Chapter 2
Chapter 3
CONTENTS
OneFS Upgrade Planning and Process Guide 3
8/11/2019 OneFS Upgrade Planning and Process Guide
4/52
Complete system jobs......................................................................34Pause or cancel SyncIQ jobs .............................................................34
Disconnect client connections.......................................................................35Activate SupportIQ or ESRS Gateway............................................................. 36
Performing the OneFS upgrade 37Upgrade process overview.............................................................................38Upgrade OneFS through the command-line interface.....................................38Upgrade OneFS through the web administration interface............................. 39Verify the OneFS installation......................................................................... 41
Completing post-upgrade tasks 43
About post-upgrade tasks............................................................................. 44Allow upgrade jobs to run..............................................................................44Verify operational status............................................................................... 44Re-establish user privileges...........................................................................45
Restore client connections and test your workflow........................................ 46Restore custom settings................................................................................46Re-install Aspera........................................................................................... 46Modify custom scripts................................................................................... 47Implement the Platform API...........................................................................47
Troubleshooting your upgrade 49
Troubleshooting overview............................................................................. 50Review the upgradelogs............................................................................... 50Common issues.............................................................................................50
Adjust the SMB client-credit minimum............................................. 50
Adjust NFS performance...................................................................51Verify a 64-bit system ......................................................................51
Chapter 4
Chapter 5
Chapter 6
CONTENTS
4 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
5/52
CHAPTER 1
Introduction to this guide
This section contains the following topics:
u About this guide......................................................................................................6u Isilon scale-out NAS overview..................................................................................6u Where to go for support...........................................................................................6u Documentation resources........................................................................................6
Introduction to this guide 5
8/11/2019 OneFS Upgrade Planning and Process Guide
6/52
About this guideThis guide provides planning information for upgrading to a later version of OneFS andincludes instructions for assessing an EMC Isilon cluster to ensure it is ready for an
upgrade. This guide also provides upgrade installation instructions and describes post-upgrade restoration and change management procedures.
Isilon scale-out NAS overviewThe EMC Isilon scale-out NAS storage platform combines modular hardware with unifiedsoftware to harness unstructured data. Powered by the distributed OneFS operatingsystem, an EMC Isilon cluster delivers a scalable pool of storage with a globalnamespace.
The platform's unified software provides centralized web-based and command-lineadministration to manage the following features:
u A symmetrical cluster that runs a distributed file system
u Scale-out nodes that add capacity and performance
u Storage options that manage files, block data, and tiering
u Flexible data protection and high availability
u Software modules that control costs and optimize resources
Where to go for supportYou can contact EMC Isilon Technical Support for any questions about EMC Isilonproducts.
Online Support Live Chat
Create a Service Request
Telephone Support United States: 800-782-4362 (1-800-SVC-4EMC)
Canada: 800-543-4782
Worldwide: +1-508-497-7901
For local phone numbers in your country, see EMC CustomerSupport Centers.
Help with online
support
For questions specific to EMC Online Support registration or
access, email [email protected].
Documentation resourcesThere are several documents that supplement the information in this guide. Make surethat the document versions that you reference are applicable to the OneFS upgradeversion. Some documents are applicable to multiple versions.
Gather the following documents to begin planning your upgrade:
u Isilon OneFS Release Notes
Introduction to this guide
6 OneFS Upgrade Planning and Process Guide
http://www.emc.com/collateral/contact-us/h4165-csc-phonelist-ho.pdfmailto:[email protected]://www.emc.com/collateral/contact-us/h4165-csc-phonelist-ho.pdfhttp://www.emc.com/collateral/contact-us/h4165-csc-phonelist-ho.pdfhttps://support.emc.com/servicecenter/createSR/https://support.emc.com/servicecenter/liveChat/8/11/2019 OneFS Upgrade Planning and Process Guide
7/52
u Isilon Supportability and Compatibility Guide
u Isilon Third-Party Software and Hardware Compatibility Guide(only available in 7.0and later.)
u Isilon OneFS Upgrade Readiness Checklist
u Isilon OneFS Web Administration Guide(previously known as the Isilon OneFS UserGuide)
u Isilon OneFS CLI Administration Guide(previously known as the Isilon OneFSCommand Reference)
u Isilon OneFS Event Reference
You can search for Isilon OneFS documentation and knowledge base articles on the EMCOnline Supportpage.
Introduction to this guide
Documentation resources 7
https://support.emc.com/products/15209_Isilon-OneFShttps://support.emc.com/products/15209_Isilon-OneFS8/11/2019 OneFS Upgrade Planning and Process Guide
8/52
8/11/2019 OneFS Upgrade Planning and Process Guide
9/52
CHAPTER 2
Planning an upgrade
This section contains the following topics:
u Upgrade planning overview .................................................................................. 10u Types of OneFS upgrades...................................................................................... 10u Verify the upgrade path.........................................................................................11u Check system requirements.................................................................................. 11u Assess upgrade impact......................................................................................... 12u Plan an upgrade schedule.....................................................................................13
u Upgrade a test cluster........................................................................................... 13
Planning an upgrade 9
8/11/2019 OneFS Upgrade Planning and Process Guide
10/52
Upgrade planning overviewAn upgrade plan is helpful in determining what tasks you need to complete before andafter installing a new OneFS image.
Your upgrade plan helps you estimate the time it will take to complete tasks anddetermine who needs to complete them.
Create an upgrade plan by evaluating how the upgrade will affect users and workflow,assessing the impact of the new version of OneFS on your EMC Isilon cluster, andanalyzing upgrade risks.
Types of OneFS upgradesThere are two options available for upgrading the OneFS operating system: a rollingupgrade or a simultaneous upgrade.
Rolling upgradeA rolling upgrade individually upgrades and restarts each node in the EMC Isiloncluster so that only one node is offline at a time.
Note
Rolling upgrades are not available between all OneFS versions. See the Isilon
Supportability and Compatibility Guidefor more information.
A rolling upgrade takes longer to complete than a simultaneous upgrade. Rollingupgrades must be performed sequentially by node number, beginning with thelowest number node. Each node is upgraded and restarted in turn. The upgradeprocess can be monitored from the starting node.
During a rolling upgrade, the remaining nodes in the cluster remain online and cancontinue serving clients; however, all clients connections will be interrupted at somepoint and the user can experience one of the following behaviors:
u There is an observable lag when the clients reconnect to an available node.
u SMB clients do not automatically reconnect to an available node; someconnection resets may occur and the user must establish a new clientconnection.
u Dynamic NFS clients will automatically reconnect to an available node ifconfigured as recommended. Refer to EMC Isilon knowledge base article 90041,Best practices for NFS client settings, for more information.
Planning an upgrade
10 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
11/52
8/11/2019 OneFS Upgrade Planning and Process Guide
12/52
l Isilon for vCenter
l Hadoop/HDFS
l Web browsers
l Windows versions
l OS X versions
l Apache HTTP Server
l Aspera Enterprise Server
l File transfer applications
l Anti-virus scan applications
Protocols l File system protocols
l Isilon with iSCSI
l Kerberos
Assess upgrade impactAssessing upgrade impact is a critical exercise to help ensure that upgrading does notdisrupt your EMC Isilon cluster, workflow, or users.
Consider all the areas of your environment that might be affected by upgrading to a newversion and plan a strategy to handle tasks and issues that might occur.
Consider upgrade limitations
The process for upgrading OneFS on an EMC Isilon cluster has certain limitations that youshould understand.
If the upgrade process cannot be completed for any reason, such as insufficient space inkey directories or a stalled drive, the system will revert to the existing version and theupgrade will be cancelled. The OneFS upgrade process does not allow you to pause orhalt so that you can attend to the blocking issue. Preparing your cluster as recommendedin "Completing pre-upgrade tasks" will help avoid situations that result in a cancelledupgrade.
You should be aware that after you upgrade to a new major version of OneFS, you cannotrestore the previous version. A OneFS version upgrade consists of re-imaging the kerneland file system and copying user changes from the old file system to the new one. Oncethe OneFS kernel is re-imaged, there is no mechanism for rolling back to the previous
major version. For example, if you upgrade from OneFS 6.5 to OneFS 7.0.2, you cannotrevert back to OneFS 6.5.
Review feature changes and known issues
You should review feature changes and known issues introduced in the upgrade versionof OneFS.
Familiarize yourself with new, modified, and deprecated features in the upgrade versionand evaluate whether the new version is right for your environment. Because of theinherent differences between major versions of OneFS, some of your requirements mightnot be supported in exactly the same way, the upgrade version might impact the
Planning an upgrade
12 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
13/52
performance, configuration, or functionality of your cluster after upgrading, or the newversion might override some of your custom settings with default settings.
Also familiarize yourself with known issues and limitations in the upgrade version tomake sure that the issues do not affect your requirements, workflow, or users. If the newversion contains an issue that affects your environment, you should consider waiting toupgrade until after EMC Isilon resolves the issue in a later version of OneFS.
See the OneFS Release Notesfor a summary of new features, feature changes, and knownissues in the upgrade version of OneFS. Refer to the OneFS Web Administration Guideorthe OneFS CLI Administration Guideto learn detailed information about new and changedfeatures.
Plan an upgrade scheduleConsider all the factors that will go in to preparing and executing your upgrade and createan upgrade schedule.
An upgrade schedule can help your upgrade go smoothly. Your schedule should estimate
time for each stage of the upgrade process.The upgrade process begins with ensuring your EMC Isilon cluster is ready to beupgraded. Cluster preparation is important to minimize upgrade errors or failure. It isrecommended that you set aside two weeks to check cluster health and resolve issuesbefore performing the upgrade itself. The two-week period allows time for shipping andreplacing degraded hardware.
The main factors you should take into consideration when estimating a schedule include:
Data back up and information collection
Estimate the time it will take to back up our data. Consider cluster size, number offiles, types of files, and file size. Also include time to collect information about thecluster such as status, logs, and settings.
Cluster preparation and readiness
Schedule time for node and drive health checks and replacement of bad hardware.Include time to update configurations and settings that are not supported in the newversion.
Upgrade maintenance window
Estimate the time it will take to run the upgrade. Consider upgrade type (rolling orsimultaneous) and cluster size. Build in time to let the upgrade jobs run and to re-establish permissions and connections. A best practice is to upgrade the clusterduring an off-hours maintenance window. Schedule time to inform users when theupgrade will take place and how they might be affected; client connections may beslow, file access may be affected, or clients may be disconnected altogether.
TroubleshootingSchedule time or extend the maintenance window for post-upgrade tasks such asreconfiguring settings, re-installing software, and troubleshooting.
Upgrade a test clusterIf available, upgrade a test cluster that has the same version of OneFS and the samesoftware configurations as your production EMC Isilon cluster.
Upgrading a test cluster before a production cluster exposes issues that slow down theupgrade of your production system.
Planning an upgrade
Plan an upgrade schedule 13
8/11/2019 OneFS Upgrade Planning and Process Guide
14/52
8/11/2019 OneFS Upgrade Planning and Process Guide
15/52
CHAPTER 3
Completing pre-upgrade tasks
This section contains the following topics:
u About pre-upgrade tasks.......................................................................................16u Check cluster readiness........................................................................................ 16u Verify configurations and settings......................................................................... 20u Upgrade compatibility check utility....................................................................... 24u Back up data.........................................................................................................28u Collect cluster data............................................................................................... 31
u Complete or stop jobs in progress.........................................................................33u Disconnect client connections...............................................................................35u Activate SupportIQ or ESRS Gateway..................................................................... 36
Completing pre-upgrade tasks 15
8/11/2019 OneFS Upgrade Planning and Process Guide
16/52
About pre-upgrade tasksEnsure that the EMC Isilon cluster is ready to upgraded to a later version of OneFS.
One purpose of the pre-upgrade tasks is to ensure that the cluster hardware, OneFSfeatures and modules, and file system do not have issues that might interrupt theupgrade process. Another purpose of the pre-upgrade tasks is to collect cluster data,record configuration settings, and reduce the potential for lost data.
Preparing the cluster for upgrade includes the following pre-upgrade tasks:
u Check for hardware issues such as drive stalls.
u Ensure storage space requirements are met.
u Ensure OneFS feature and module configurations are supported by the later versionof OneFS.
u Collect pre-upgrade cluster data including custom configurations.
u
Back up cluster data.
Check cluster readinessEnsure that the EMC Isilon cluster hardware and file system do not have issues that mightinterrupt the OneFS upgrade.
Completing pre-upgrade tasks such as checking the health of the hardware componentson the cluster, ensuring that storage space requirements are met, and managingoutstanding cluster events and errors are recommended for a successful upgrade.
Restart the cluster
It is recommended that you restart the EMC Isilon cluster before you upgrade the OneFSsoftware. Restarting the cluster can expose problems that might interfere with theupgrade process.
Restarting the cluster flushes the caches, frees up memory, clears unused connections,and exposes problems in hardware or firmware that must be fixed before upgrading. Forexample, restarting a node might expose degraded drives that must be smartfailed andreplaced, or stalled drives that must be resolved.
It is recommended that you begin a process of rolling restarts two weeks before youupgrade in order to leave enough time to resolve any issues and, if necessary, order andreplace hardware. Unresolved issues will likely reoccur during the upgrade process andmight interrupt or block the upgrade.
Check hardware health
Check the health of your hardware components to confirm that the EMC Isilon cluster cansupport an upgrade.
You can run the following commands to evaluate the health of each node and determineif it is ready for an upgrade.
Command Description- -isi status -D Displays detailed information about running
protection operations, failed protection
operations, and operation errors.
Completing pre-upgrade tasks
16 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
17/52
Command Description- -
Check jobs or devices that report a status of
ATTENTION.
isi devices | grep -v healthy Displays information about the current status
of all nodes and drives in the cluster, includingdrive health.
Check any drives that do not report a status of
HEALTHY.
status advanced
(Run from the isi configcommand shell)
Indicates the health status of all nodes in the
cluster.
Check any drives that do not report a status of
OK.
isi_for_array -s 'gmirror status | grep -i
degraded'
Checks the cluster for degraded drives.
If a drive is degraded, do not proceed with the
upgrade.
For more information, see EMC Isilon
knowledge base article 88746, Cannot perform
upgrade with degraded boot drive.
isi_for_array -s 'isi_dmilog' Checks the data management interface (DMI)
log for a record of errors reported by the
individual memory modules (DIMM/RAM).
Do not proceed with the upgrade if the
command returns any errors reported as
correctableor fatalin the Totals
section. First, see the following EMC Isilon
knowledge base articles, assess whether the
articles are related to the errors, and follow the
instructions in the articles:
l Article 89662, ECC error policy violation
alert does not clear after replacement
l Article 89205, Physical memory low
l Article 89614, Temperature sensitivity in
the DIMM module used in the Isilon IQ
10000X-SSD, IQ 5000S-SSD, and IQ
If the errors cannot be resolved, contact EMC
Isilon Technical Support to determine whether
the DIMM must be replaced. DIMM replacementmust be performed by an EMC Isilon technician
and requires that the node be powered down
during the service.
isi status -q |grep -v OK Displays nodes that report a status other than
OK.
Check any nodes that return a status of
ATTENTION, SMARTFAIL, or DOWN.
For more information, see EMC Isilon
knowledge base article 16497, How to
determine why a node is in an attention state.
Completing pre-upgrade tasks
Check hardware health 17
8/11/2019 OneFS Upgrade Planning and Process Guide
18/52
Command Description- -isi_for_array "ps aux | grep opensm" | grep
master
Checks whether a node owns the OpenSM
(subnet manager) master role.
One node in the cluster must own the OpenSM
master role. If the command does not return
any output then the Infiniband switch oranother device has assumed the OpenSM
master role. In that case, confirm that
Infiniband cables from another cluster are not
connected to the cluster's primary or secondary
Infiniband switch, and then run the command
again.
If you proceed with an upgrade where the
OpenSM master role is not owned by a node on
the cluster, the upgrade results in a cluster
split.
If any of these commands indicate a drive failure on the cluster, it is recommended thatyou enable the Virtual Hot Spare (VHS) feature. VHS ensures the cluster has enough freespace available in order to smartfail the drive and re-protect its data. See EMC Isilonknowledge base article 88964, How to enable and configure Virtual Hot Spare (VHS)forinstructions.
Check for free node space
Ensure minimum available-space requirements for node directories and overall capacityof the EMC Isilon cluster are met before you upgrade OneFS.
The amount of total available space on the cluster must be at least 10 percent. Do not
proceed with the OneFS upgrade if the minimum available-space requirement for thecluster is not met.
The directories in the following table must meet a minimum available-space requirement.If any of these directories exceed the minimum available-space requirement, you mustmake more space available before upgrading. Otherwise, the upgrade process will fail,but it will not return an error indicating space issues as the cause.
Directory Requirement- -root partition
(/)
Cannot be more than 97 percent full.
See the following resources for resolution instructions if this directory is at or near
the minimum available-space requirement:
l EMC Isilon knowledge base article 89552, Node reached 95% used capacityon the root filesystem.
l Event ID 100010003, "The /(root) partition is near capacity," in the OneFS
Event Reference.
/ifs Cannot be more than 95 percent full.
See the following resources for resolution instructions if this directory is at or near
the minimum available-space requirement:
l EMC Isilon knowledge base article 167267, There is at least one smartpool at
or over capacity or the SmartPool is near or over capacity.
Completing pre-upgrade tasks
18 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
19/52
Directory Requirement- -
l EMC Isilon white paper, Best Practices Guide for Maintaining Enough Free
Space on Isilon Clusters and Pools.
l Event ID 100010004, "The cluster's /ifs partition is near capacity," in the
OneFS Event Reference.
/var Cannot be more than 90 percent full.
See the following resources for resolution instructions if this directory is at or near
the minimum available-space requirement:
l EMC Isilon knowledge base article 89848, The /var partition is near capacity
(95% used).
l Event ID 100010001, "The /var partition is near capacity," in the OneFS Event
Reference.
/var/crash Cannot be more than 90 percent full.
See the following resources for resolution instructions if this directory is at or nearthe minimum available-space requirement:
l EMC Isilon knowledge base article 89823, The crash partition of a node in the
cluster has reached 90% capacity alert.
l Event ID 100010002, "The /var/crash partition is near capacity," in the OneFS
Event Reference.
Run the following command to check the available space capacity of each instance ofthese directories on the cluster:
isi_for_array 'df -h'
The system returns output similar to the following example:
virtual-demo-3: Filesystem Size Used Avail Capacity Mounted on--------------------------------------------------------------------virtual-demo-3: /dev/imdd0a 495M 418M 38M 92% /virtual-demo-3: devfs 1.0K 1.0K 0B 100% /devvirtual-demo-3: /dev/imdd1a 495M 6.7M 449M 1% /varvirtual-demo-3: /dev/imdd2a 496M 5.9M 451M 1% /var/crashvirtual-demo-3: OneFS 40G 283M 20G 1% /ifs
virtual-demo-2: Filesystem Size Used Avail Capacity Mounted on--------------------------------------------------------------------virtual-demo-2: /dev/imdd0a 495M 417M 38M 92% /virtual-demo-2: devfs 1.0K 1.0K 0B 100% /devvirtual-demo-2: /dev/imdd1a 495M 6.1M 449M 1% /varvirtual-demo-2: /dev/imdd2a 496M 5.9M 451M 1% /var/crashvirtual-demo-2: OneFS 40G 283M 20G 1% /ifs
Resolve outstanding events and errors
Before you upgrade, resolve outstanding critical events, errors, and failures. Unresolvedevents and errors can disrupt the OneFS upgrade process.
Check log files in the following directories on each node for errors and failures:
u /var/log
u /var/log/messages
u /var/log/crash
Completing pre-upgrade tasks
Resolve outstanding events and errors 19
8/11/2019 OneFS Upgrade Planning and Process Guide
20/52
Note
If the logs contain a dynamic sector recovery (DSR) failure or an Isilon Data Integrity (IDI)
failure (most likely in /var/log/idi.log), contact Isilon Technical Support before you
upgrade.
Run the following command to view all events on the cluster:
isi events list
Run the following command to view critical events on the cluster:
isi events list --severity=critical | grep "\-\-"
See the OneFS Event Reference Guidefor information about specific event details. Seethe following EMC Isilon knowledge base articles that address specific errors, and if thelogs contain any of these errors, complete the resolution instructions provided by thearticle:
u Article 167267, There is at least one smartpool at or over capacity
u Article 89373, Detected IDI failure, attempting DSR
u Article 89479, DSR - Dynamic Sector Recovery Failure
u Article 89475, IDI error. Shallow verification failure in block
u Article 89464, Detected IDI network checksum error on path
u Article 88683, Cluster needs to be restriped but FlexProtect is not running
Cancel non-critical events before upgrading OneFS to prevent recurrence of notificationsthat are being addressed by the upgrade or that you know to be harmless. See thefollowing knowledge base articles for more information:
u Article 16597, How to quiet or cancel an event (alert) and article
u Article 16586, How to reset the celog database and clear all historical alerts
Verify configurations and settingsOneFS configurations and settings on the EMC Isilon cluster must be supported by theversion of OneFS that you want to upgrade to.
Verify global namespace requirements
Verify that your EMC Isilon cluster meets the minimum requirements for GlobalNamespace Acceleration (GNA).
If you are upgrading to OneFS 7.0 or later and GNA is enabled, the EMC Isilon cluster mustmeet the following requirements before you upgrade:
u 20% or more of the nodes in the cluster must contain at least one SSD
u SSDs must make up at least 1.5% of the total storage capacity on the cluster; 2%storage is recommended
If the cluster does not meet these system requirements before you upgrade the cluster toOneFS 7.0 or later, the upgrade process will fail and GNA is disabled.
Run the following command to view size and capacity of SSDs on the cluster:
isi status -q
Completing pre-upgrade tasks
20 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
21/52
See EMC Isilon knowledge base article 90371, Upgrade to OneFS 7.0.1.2 and later ishalted if Global Namespace Acceleration is enabled and SSD storage is less than theenforced minimum of 1.5 percent of total storage, for more information.
Check disk pool configurations
The disk pool configuration on your EMC Isilon cluster must be supported by the versionof OneFS that you are upgrading to.
OneFS 7.0 and later does not support certain disk-pool configurations that were allowedbut not recommended in earlier versions of OneFS. If you are upgrading to OneFS 7.0 andlater, any small disk pools (pools of two or fewer nodes) of the same node type must bemerged into a single, larger disk pool consisting of nodes of the same type orequivalence class. Pools with fewer than three nodes remain without provisioning andcannot participate in the cluster. You do not need to run a SmartPools job after mergingpools.
Run the following command to list all disk pool configurations on a node:
isi storagepool list --verbose
See the Isilon Supportability and Compatibility Guidefor additional details about nodepools, node equivalence classes, and autoprovisioning.
Verify the disk pool policy
Confirm that the disk pool policy on the EMC Isilon cluster is supported by the version ofOneFS that you are upgrading to.
If any of the following conditions apply to your cluster, you must set the /ifs/.ifsvar
directory to use the system:anydisk pool policy.
u The first version of OneFS installed on the cluster was earlier than 6.0 and the globalprotection policy was set to Advanced.
u The SmartPools module is not licensed.
u Every node contains one or more SSDs.
If you upgrade OneFS without setting the directory to system:any, the upgrade fails
without an error message, the ifsdirectory becomes read-only, and services such as
authentication, SMB file sharing, and the web administration interface cannot be started.
See EMC Isilon knowledge base article 90487, Upgrade to OneFS 7.0.x fails or is halted ifthe /ifs/.ifsvar directory does not use the system : any disk pool policy, for additionalinformation.
Procedure
1. Run the following command to determine the disk pool policy of the /ifs/.ifsvar/directory:
isi get -Dd /ifs/.ifsvar
If the disk pool policy is system:any, the first disk pools entry appears similar to the
following:
* Disk pools: policy system : any -> target()
2. If the disk pool policy is not system:any, run the following command to set the
directory policy to system:any:
isi set -FR --diskpool system:any /ifs/.ifsvar
Completing pre-upgrade tasks
Check disk pool configurations 21
8/11/2019 OneFS Upgrade Planning and Process Guide
22/52
Configure the LDAP service
Ensure the Lightweight Directory Access Protocol (LDAP) service configured on the EMCIsilon cluster is supported by the OneFS version you are upgrading to.
The OneFS operating system was updated with a newer version of the LDAP provider inversion 6.5. Any existing LDAP settings were reclassified as "legacy LDAP." You candetermine which LDAP service is configured on the cluster through the OneFS webadministration interface.
The following table displays the OneFS versions that support legacy and standard LDAP.
Version LDAP service supported- -6.0 and earlier Legacy LDAP only
6.5.x.x Both legacy LDAP and standard LDAP
7.0 Standard LDAP only
7.1 Standard LDAP only
If you are upgrading to OneFS version 7.0 or 7.1 and the legacy LDAP service is enabledon the cluster, disable the legacy LDAP service, and then enable and configure thestandard LDAP service as described in the OneFS User Guidefor OneFS version 6.5.4.
Check SmartLock mode
Before you upgrade, verify that the SmartLock mode that is configured on the EMC Isiloncluster is supported by the OneFS version you are upgrading to.
OneFS can operate in either SmartLock enterprise or SmartLock compliance mode.SmartLock compliance mode enables you to protect your data in compliance with the
regulations defined by U.S. Securities and Exchange Commission rule 17a-4.
u If you are upgrading to OneFS 7.0 or later from either OneFS 6.0 or OneFS 6.5,enterprise mode is supported. Upgrading to OneFS 7.0 or later in enterprise mode isfully supported from OneFS 6.0 and OneFS 6.5.
u Clusters running OneFS 6.0 or OneFS 6.5 cannot be upgraded to compliance modeand, after being upgraded to enterprise mode, cannot be converted to compliancemode.
Reconfigure unsupported SMB settings
Ensure that SMB settings on the EMC Isilon cluster are supported by the later verison of
OneFS.If the SMB settings on the EMC Isilon cluster are not supported by the later OneFS versionthat you are upgrading to, the upgrade will fail. Remove or modify unsupported SMBsettings before you upgrade by editing the /etc/mcp/override/smbd.xml file or
the /etc/mcp/override/smbd_shares.xml file. After you modify SMB settings,
test the workflow.
See the following EMC Isilon knowledge base articles for more information:
u Article 88910, Unsupported SMB parameters in OneFS 6.5.x
u Article 90167, Upgrading to 6.5 can fail due to a pre-upgrade SMB configurationcheck
Completing pre-upgrade tasks
22 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
23/52
Configure Kerberos settings
Configure the Kerberos settings file to before you upgrade OneFS.
If Kerberos with NFS is configured on the EMC Isilon cluster, upgrading OneFS might
delete the local /etc/nfs.keytab file from each node. The following methods willsave the keytab file during the upgrade:
u If you want to preserve the keytab file in each local /etcdirectory, add the following
text to the /etc/mcp/override/user_preserve_files.xml file of each
node on the cluster:
......
u Copy the keytab file to the /ifsdirectory where it can be accessed by all nodes on
the cluster instead of saved locally on each node.
See EMC Isilon knowledge base article 16584, How to configure the Isilon cluster to useKerberos with NFS in a non-Active Directory environment, for more configurationinformation.
Install supported version of InsightIQ
Ensure the installed version of InsightIQ is supported by the later version of OneFS before
you upgrade.OneFS 6.5 supports InsightIQ 1.0 and later and OneFS 7.0 or later supports InsightIQ 2.5or later.
If you are upgrading to OneFS 7.0 or later, you must install InsightIQ 2.5 or 3.0 on theEMC Isilon cluster before you proceed. See the following documents for more informationabout InsightIQ:
u InsightIQ 2.5 Installation and Setup Guide
u InsightIQ 3.0 Installation and Setup Guide
Run the following command to determine the version of InsightIQ installed on the cluster:
isi license | grep -i insightiq
Install supported version of Isilon for vCenter
Ensure the installed version of Isilon for vCenter is supported by the later version ofOneFS before you upgrade.
OneFS 6.5 supports Isilon for vCenter 1.0 and later and OneFS 7.0 or later supports Isilonfor vCenter 1.3
If you are upgrading to OneFS 7.0 or later, you must install Isilon for vCenter 1.3 on theEMC Isilon cluster before you proceed. See the Isilon for vCenter 1.3 Installation Guideformore information.
Completing pre-upgrade tasks
Configure Kerberos settings 23
8/11/2019 OneFS Upgrade Planning and Process Guide
24/52
Run the following command to determine the version of Isilon for vCenter installed on thecluster:
isi license |grep -i vCenter
Upgrade compatibility check utilityThe upgrade compatibility check utility performs a number of checks to verify whether theEMC Isilon cluster is compatible with the new version of OneFS and ready for an upgrade.
The utility examines cluster settings, activities, or statuses that might interrupt theupgrade process or cause the upgrade to fail. If the upgrade compatibility check utilityfinds any issues, investigate and resolve the issues, and then run the utility again forverification. It is important that you do not rely only on the checks run by the utility toensure that your cluster is ready for an upgrade. You should review cluster readiness andconfiguration verification tasks described in this guide as well.
The upgrade compatibility check utility is included in the OneFS installation package andcan be run without starting the actual upgrade process.
Upgrade compatibility checks
The upgrade compatibility check utility examines the following areas of the EMC Isiloncluster and returns warnings or errors if an area is not compatible with the version thatyou want to upgrade to.
Check Description and recommendation- -Disk load Checks the cluster usage level and returns a
warning if the disk load is greater than 50
percent.
It is recommended that you disconnect all
client connections and stop all system jobsbefore upgrading. A disk load that is greater
than 50 percent might indicate that multiple
clients are connected to the cluster or that
system jobs are running.
Free space Checks the free space and returns a warning or
an error if certain capacity thresholds are
exceeded for the following cluster partitions or
node pools:
/
Returns a warning if the root partition
exceeds 97 percent of its capacity. Theutility returns an error if the partition
reaches 100 percent capacity.
/var
Returns a warning if the /varpartition is
more than 90 percent full. The utility
returns an error if the partition reaches
100 percent full.
Completing pre-upgrade tasks
24 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
25/52
Check Description and recommendation- -
/ifs
Returns a warning if the /ifspartition
exceeds 95 percent of its capacity. The
utility returns an error if the partition
reaches 100 percent capacity.
Node pools
Returns a warning if one or more node
pools exceeds 90 percent of its capacity.
The utility returns an error if all the node
pools reach 90 percent capacity.
If the utility reports an error, do not proceed
with the upgrade until you reduce the size of
the partition or node pool so that the free space
for that partition is below the threshold.
See "Check for free node space" in this guidefor more information.
Drive stalls Checks the health of the drives in the cluster
and returns a warning if the cluster contains
stalled drives.
If the utility reports a stalled drive, do not
proceed with the upgrade until you smartfail
and replace the stalled drive or resolve the
stall.
See EMC Isilon knowledge base article 89477,
Introduction to drive stalls, for information
about resolving drive stalls.
Smartfail operation status Determines whether a smartfail operation is
running on any drives or nodes in the cluster.
If a smartfail operation is running, the utility
returns an error. Wait for the smartfail
operation to complete before proceeding with
the upgrade.
Integrity scan job status Determines whether the integrity scan job is
running.
If the integrity scan job is running, wait for the
job to complete before proceeding with the
upgrade.
Outstanding critical events Checks for critical events on the cluster.
If the utility returns any critical events, do not
proceed with the upgrade until you resolve the
issues.
See the Isilon OneFS Event Referencefor more
information.
Unsupported SMB configuration Checks for deprecated SMB settings.
If the utility reports that the SMB configuration
is not supported, do not proceed with the
Completing pre-upgrade tasks
Upgrade compatibility checks 25
8/11/2019 OneFS Upgrade Planning and Process Guide
26/52
Check Description and recommendation- -
upgrade until you remove the unsupported
settings and reconfigure SMB.
See "Reconfigure unsupported SMB settings" in
this guide for more information.
Legacy LDAP support enabled Checks the compatibility of the LDAP service.
If the utility reports that the cluster is
configured with an LDAP service deprecated in
the version you are upgrading to, do not
proceed with the upgrade until you reconfigure
the cluster with a supported LDAP service.
See "Configure the LDAP service" in this guide
for more information.
Unsupported SmartPools configuration Checks the compatibility of the SmartPools
configuration.
If the utility reports that the cluster contains aSmartPools configuration deprecated in the
version you are upgrading to, do not proceed
with the upgrade until you reconfigure
SmartPools to a supported configuration.
See "Check disk pool configurations" and
"Verify the disk pool policy" in this guide for
more information.
Download the OneFS installation image
Before running the upgrade compatibility check utility or upgrading the OneFS software,you must download the installation image of the OneFS upgrade version.
Procedure
1. From the EMC Online Support site, click the download link for the version of the OneFSimage that you want to install.
https://support.emc.com/downloads/15209_Isilon-OneFS
2. Save the installation files to the /ifs/datadirectory on the EMC Isilon cluster.
3. From the EMC Online Support site, next to the download link, click Checksumandrecord the MD5 checksum value.
4. Open a secure shell (SSH) connection to any node in the cluster and log in using theroot account.
5. In the OneFS command-line interface, run the following command, specifying theinstallation image file name:
md5
The command returns an MD5 checksum value.
6. Compare the MD5 checksum value recorded from the EMC Online Support site againstthe MD5 checksum generated from the command-line interface.
Completing pre-upgrade tasks
26 OneFS Upgrade Planning and Process Guide
https://support.emc.com/downloads/15209_Isilon-OneFS8/11/2019 OneFS Upgrade Planning and Process Guide
27/52
8/11/2019 OneFS Upgrade Planning and Process Guide
28/52
Note
Although the prompt indicates that the image will be updated, because the --
check-onlyoption is specified the command will only run the upgrade
compatibility check utility.
After you finish
If the command returns errors, you must resolve the errors before proceeding with theupgrade. Warnings are informational and do not prevent an upgrade.
Back up dataIt is recommended that you back up all files and data on your EMC Isilon clusterimmediately before you upgrade.
Allow sufficient time for a full and an incremental backup, if needed. Depending on thesize of your cluster and the file types stored, a back up can take one or more days.
It is recommended that you back up the following directories:u Back up modified files from:
l /usr/
l /usr/local/
l /usr/libexec/
l /var/
l /var/crash/
u Back up all files from/ec/mcp/override/. Some files in this directory may need
to be reconfigured after upgrading.
You should validate that you can restore the data from your backup system before youupgrade.
SyncIQ backup
You can back up the data on an EMC Isilon cluster by replicating the data from one clusterto another with SyncIQ.
SyncIQ enables you to retain a consistent backup copy of your data on another Isiloncluster. You must configure a SyncIQ license on both Isilon clusters before you canreplicate data between them.
You can replicate data at the directory level while optionally excluding specific files andsub-directories from being replicated. SyncIQ creates and references snapshots toreplicate a consistent point-in-time image of a root directory. Metadata such as accesscontrol lists (ACLs) and alternate data streams (ADS) are replicated along with data.
See the OneFS Web Administration Guideor the OneFS CLI Administration Guideforinformation on replicating data with SyncIQ.
NDMP backup
You can back up the data on your EMC Isilon cluster through the Network DataManagement Protocol (NDMP).
From a backup server, you can direct backup and recovery processes between a clusterand backup devices such as tape devices, media servers, and virtual tape libraries
(VTLs).
Completing pre-upgrade tasks
28 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
29/52
You can perform both NDMP three-way backup and NDMP two-way backup. During anNDMP three-way backup operation, a data management application (DMA) on a backupserver instructs the cluster to start backing up data to a tape media server that is eitherattached to the LAN or directly attached to the DMA. During a two-way NDMP backup, aDMA on a backup server instructs a backup accelerator node on the cluster to startbacking up data to a tape media server that is attached to the backup accelerator node.
NDMP two-way backup is the most efficient method in terms of cluster resourceconsumption. However, NDMP two-way backup requires that you attach one or moreBackup Accelerator nodes to the cluster. In both backup models, file history data istransferred from the cluster to a backup server.
See the OneFS Web Administration Guideor the OneFS CLI Administration Guideforinformation on backing up data with NDMP.
Back up custom settings
Document and back up any custom settings you have on your EMC Isilon cluster.
Most settings are transferred over to the new version of OneFS during the upgrade.
Backing up custom settings enables you to reapply any settings that were not transferredduring the upgrade process.
Back up the following custom settings:
Setting Description
- -SMB audit logging If you have an audit log directory in /ifsfor
example, /ifs/auditlogyou might have
custom SMB logging settings.
After upgrading, you must reconfigure SMB
audit logging.
See the following EMC Isilon knowledge basearticles for more information about SMB audit
logging:
l Article 88882, SMB audit procedure
l Article 16545, How to enable SMB or SMB2
logging and auditing in OneFS 6.5
Passwords for local user accounts Make a list of your local accounts and their
passwords before you upgrade.
After you upgrade, you might have to reset the
passwords of the local user accounts that you
configured on the cluster.
Other users should be prepared to reset the
passwords of their local accounts after the
upgrade.
Changes to system control (sysctl) values If you overrode a default sysctlvalue either
for the cluster in /etc/mcp/override/
sysctl.confor for a local node in /etc/
local/sysctl.conf, you might need to
reset the sysctlvalue after upgrading.
Completing pre-upgrade tasks
Back up custom settings 29
8/11/2019 OneFS Upgrade Planning and Process Guide
30/52
Setting Description
- -A sysctlvalue that was modified in another
file, such as /etc/sysctl.conf, will not be
copied over when you upgrade.
It is recommended that you do not modifysysctlvalues unless you are instructed to do
so by Isilon Technical Support. However, if you
must change a sysctlvalue, set it in the
override file to preserve it when you upgrade a
node or a cluster.
Document your sysctlvalues and back up
the override files before you proceed.
See EMC Isilon knowledge base article 89232,
Configuring sysctls and making sysctl changes
persist through node and cluster reboots and
upgradesfor more information.
Certificates If you configured a certificate for your cluster,
you must configure a certificate again after you
upgrade. Make sure you have the certificate
information on hand so that you can set up the
certificate after the upgrade.
Static routes If you are upgrading a cluster running a version
of OneFS earlier than 6.5, you must reconfigure
the static routes that were set in rc.conf.
Back up the rc.conffile so that you can
reconfigure the routes with Flexnet after you
upgrade.
If you are upgrading a cluster running OneFS
6.5 or later, static routes that you set with
Flexnet or the isi networks modify
pool --add-static-routescommand
should transfer over when you upgrade.
If you are upgrading a cluster running OneFS
6.5 or later, static routes that you set in
rc.confshould be reconfigured with Flexnet
before you upgrade.
See the following EMC Isilon knowledge base
articles for more information about staticroutes:
l Article 16590, How to add and delete static
IP routes on OneFS 6.5.x.x and later
l Article 16531, How to add and delete static
IP routes on OneFS 5.0-6.5.4 and later
Aspera for Isilon Before you upgrade, back up your Aspera
configuration files. If your Aspera settings are
not transferred over after the upgrade, you
must reinstall Aspera.
Completing pre-upgrade tasks
30 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
31/52
Setting Description
- -Back up configuration files in the following
directories:
l /ifs/.ifsvar/aspera/etc/
l /ifs/.ifsvar/aspera/www/
l /usr/local/aspera/var/aspera-
prepost
You should also back up your custom Aspera
scripts. For more information, see the Aspera
Enterprise Server for Isilon guide at the
Asperasoft web site.
Cron jobs Cron jobs settings that were not configured
through /etc/mcp/ override/
crontab.smbtimeare lost after an upgrade.
Document and back up these cron jobs or re-configure them through /etc/mcp/
override/crontab.smbtimebefore you
upgrade.
Check for syntax changes of OneFS commands
that may affect your cron job. You might have to
modify a cron job that contains changed OneFS
commands so that the job works after the
upgrade.
Collect cluster dataYou should collect information about your EMC Isilon cluster before you upgrade.
Collecting information helps both you and Isilon Technical Support troubleshoot if aproblem occurs.
Collect cluster information
Before you upgrade, record key information about your EMC Isilon cluster.
Refer to the recorded information to verify that your settings have been preserved afteryou upgrade. If there is an issue with your upgrade, share the information with Isilon
Technical Support.
It is recommended that you collect and record the following information:
u Upgrade date
u Cluster name
u Current OneFS version
u Upgrade OneFS version
u Physical installation location
u List of all nodes types and quantity of each
u Software licenses:
Completing pre-upgrade tasks
Collect cluster data 31
8/11/2019 OneFS Upgrade Planning and Process Guide
32/52
l SmartConnect Advanced
l SmartQuotas
l SmartPools
l SmartLock
l
SyncIQl SnapShotIQ
l InsightIQ
l SmartDedupe
l iSCSI
l Isilon for vCenter
l HDFS
u SyncIQ usage (destination or source)
u Cluster usage (production, test, development, evaluation)
u FlexProtect level of the cluster (+1, +2, +2:1, +3, +3:1, +4, 2x, 3x, 4x, 5x, 6x, 7x, 8x)
u Custom protection levels for directories, files or pools?
u IB switch type (dual or single)
u MTU (9,000 or 1,500)
u File sharing protocols:
l SMB
l NFS
l HTTP
l FTP
l SSH
l HDFS
l iSCSI
u Cluster access method for administration (serial console, SSH, web interface
u Is SupportIQ or ESRS Gateway support enabled?
u List of customizations:
l Local accounts and passwords
l SMB audit logging
l
Custom scriptsl Cron jobs
l API calls
l Static routes
l Modifications tosysctlvalues
l Certificates
Completing pre-upgrade tasks
32 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
33/52
Collect cluster status
Before you upgrade, get the status of your EMC Isilon cluster.
Run the isi statuscommand with the following options to get the status of your
cluster, events, and jobs:isi status -D -w
These options display non-truncated, detailed information on running protectionoperations, including a list of worker processes. These options also display moreinformation on failed protection operations, including a list of errors.
Save the results to help Isilon Technical Support troubleshoot if a problem occurs duringor after the upgrade.
For more information, see the OneFS CLI Administration Guide.
Collect cluster logs
Collect and save EMC Isilon cluster logs before you upgrade.
Collecting the information in the logs before you upgrade can help Isilon TechnicalSupport troubleshoot if a problem occurs during or after the upgrade.
You can collect cluster logs by running the following root command:
isi_gather_info
Logs generated by this command are stored in the /ifsdirectory. The default temporary
directory is /ifs/data/Isilon_Support/ .
You must have root access to run isi_gather_info.
For more information, see the OneFS CLI Administration Guide.
Complete or stop jobs in progressYou should ensure that there are no jobs running on your EMC Isilon cluster beforebeginning the OneFS upgrade.
Wait for jobs to fully complete or stop jobs before upgrading.
Complete NDMP backup
Before you upgrade, you must wait for Network Data Management Protocol (NDMP)backups to finish so you have saved copies of your data.
If you cannot wait for the backups to finish, stop the active NDMP backups 30 60minutes before the upgrade. The NDMP backup process requires this additional time tocome to a halt.
For instructions on how to stop backup jobs, see the documentation for your backupapplication.
Completing pre-upgrade tasks
Collect cluster status 33
8/11/2019 OneFS Upgrade Planning and Process Guide
34/52
Complete system jobs
You must complete or cancel all system jobs running on your EMC Isilon cluster beforeyou upgrade the OneFS operating system.
Before you begin
See the OneFS Web Administration Guideor the OneFS CLI Administration Guidefor moreextensive information on managing system jobs.
OneFS performs system jobs through a service that runs in the background. The followingsystem jobs must be allowed to run to completion; the information they provide is usefulin determining cluster readiness:
FlexProtect
Following a node failure, performs a pass on the file system to ensure that all filesremain protected. FlexProtect is most efficient in clusters that contain only HDDs.
FlexProtectLin
Following a node failure, performs a pass on the file system to ensure that all filesremain protected. FlexProtectLin is most efficient if file system metadata is stored onSSDs.
IntegrityScan
Verifies file system integrity.
You can either wait for other system jobs to finish or cancel them before you upgrade.
Note
If a FlexProtect, FlexProtectLin, or IntegrityScan system job takes longer than expected,
contact Isilon Technical Support. Do not stop one of these system jobs unless instructed
to do so by support.
Procedure
1. Run the following command to check for running system jobs:
isi job status
2. Run the following command for each running job, replacing with a job name or ajob instance ID :
isi job cancel
3. Run the following command 20 to 30 minutes before you upgrade to stop the systemjob service:
isi services -a isi_job_d disable
4. Run the following command to re-enable the system job service before you upgrade;upgrading relies on the system job service:
isi services -a isi_job_d enable
Pause or cancel SyncIQ jobs
Pause or cancel any SyncIQ jobs that are running on your EMC Isilon cluster beforeupgrading the OneFS operating system.
Pause or cancel SyncIQ jobs 30 to 60 minutes before upgrading to reduce the likelihoodof failed backups or incomplete SyncIQ jobs during the upgrade.
Completing pre-upgrade tasks
34 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
35/52
Run the following command to determine if a license for SyncIQ is activated on thecluster:
isi license |grep -l synciq
Run the following command to view a list of SyncIQ policies on the cluster:
isi sync policy list -v
Pausing a replication job temporarily stops data from being replicated, but does not freethe cluster resources replicating the data. You can pause a running SyncIQ job and thenresume the job later
Cancelling a SyncIQ job stops data from being replicated and frees the cluster resourcesthat were replicating data. You cannot resume a cancelled SyncIQ job; to restartreplication, you must start the policy again.
u Pause a SyncIQ job by running the following command, replacing withthe policy that you want to pause:
isi sync jobs pause
u Cancel a SyncIQ job by running the following command, replacing with
the policy that you want to cancel:isi sync jobs cancel
It is recommended that you set SyncIQ jobs to run only when manually started in order toensure that a job cannot run again. You might need to reproduce the original schedulesafter the upgrade. You can set the policy to run manually in the individual policy settingsfrom the web administration interface.
Disconnect client connectionsDisconnect client connections to the EMC Isilon cluster before you upgrade OneFS.
Client disconnection is required for simultaneous upgrades and highly recommended for
rolling upgrades to deliver the best upgrade experience. You can terminate connectionsimmediately or specify an amount of time to wait for connections to terminate after youstart the upgrade process through the drain time option available when you install thenew OneFS image.
Disconnect the following connection protocols at least 5 to 10 minutes before youupgrade to avoid leaving clients with stalled mounts:
u NFS
u SMB
u HTTP
u Hadoop/HDFS
u iSCSI
Note
If there are any iSCSI connections to the cluster, you can only perform a simultaneous
upgrade. After all the initiator connections are stopped, set the iSCSI LUN to offline
until the upgrade is complete.
Completing pre-upgrade tasks
Disconnect client connections 35
8/11/2019 OneFS Upgrade Planning and Process Guide
36/52
Activate SupportIQ or ESRS GatewayYou can enable SupportIQ or the ESRS Gateway, both of which are features that can helpIsilon Technical Support troubleshoot if a problem occurs.
When SupportIQ or the ESRS Gateway is enabled, Isilon Technical Support personnel canrequest logs through scripts that gather EMC Isilon cluster data and then upload the datato a secure location. Both features can also enable remote access, which allows IsilonTechnical Support personnel to troubleshoot your cluster remotely and run additionaldata-gathering scripts.
You must enable and configure either SupportIQ or ESRS Gateway support before IsilonTechnical Support can gather data or access clusters remotely.
ESRS Gateway support is only available on clusters running OneFS 7.1 or later.
See the OneFS Web Administration Guideor the OneFS CLI Administration Guidefor moreinformation on SupportIQ and the ESRS Gateway.
Completing pre-upgrade tasks
36 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
37/52
CHAPTER 4
Performing the OneFS upgrade
This section contains the following topics:
u Upgrade process overview.....................................................................................38u Upgrade OneFS through the command-line interface.............................................38u Upgrade OneFS through the web administration interface..................................... 39u Verify the OneFS installation................................................................................. 41
Performing the OneFS upgrade 37
8/11/2019 OneFS Upgrade Planning and Process Guide
38/52
Upgrade process overviewThe OneFS upgrade process consists of installing the upgrade image, starting theupgrade, and verifying that the upgrade completed.
You must first download the installation image from the EMC Support website. See"Download the OneFS image" for more information.
You can upgrade OneFS form the command-line interface or the web administrationinterface.
Before upgrading OneFS, be sure that you have completed all pre-upgrade tasks.
Upgrade OneFS through the command-line interfaceYou can upgrade the OneFS version that is running on an EMC Isilon cluster through thecommand-line interface.
Before you begin
You must download the OneFS installation image from the EMC Support website. TheOneFS installation image is used to run compatibility checks. If you have not downloadedthe image and run the checks, you must do so before installing the new version of OneFS.
You must resolve any compatibility issues found by the checks before installing the newversion of OneFS. See "Run compatibility checks" for more information.
Procedure
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster andlog in with the root account.
2. Choose one of the following options for upgrading OneFS.
Options Description
Simultaneous
upgrade
Perform a simultaneous upgrade and restart all of the nodes in thecluster at the same time by running the following command:
isi update
The cluster prompts you to specify the location of the OneFSinstallation image that you downloaded.
Rolling upgrade Perform a rolling upgrade, which upgrades and restarts each nodein the cluster sequentially, by running the following command:
isi update --rolling
You can include the following options:l Specify a timeout period for TCP clients to disconnect with the
--drain-timeoption.
This option allows time for TCP clients to disconnect from thecluster before the upgrade process restarts the node. Thedefault timeout is zero seconds.
Note
If any client connections are not terminated within the specified
drain time, the upgrade will halt or fail. You must manually
disconnect client connections and restart the upgrade.
Performing the OneFS upgrade
38 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
39/52
Options Description
l Display a confirmation prompt before each node restarts byincluding the --manualoption.
If you do not include this option, each upgraded node isautomatically restarted without a prompt.
Note
This option requires that you stay connected to the session to
confirm each reboot. If you disconnect, the upgrade will halt
until you can connect to the node to confirm the reboot.
Rolling upgrades are not available for some OneFS versions.Review the Isilon Supportability and Compatibility Guidefor a listof versions that support rolling upgrades.
Results
After the upgrade, a number of upgrade-related jobs will continue to run for some time.During this time, the cluster is accessible, but you might experience a decrease in clusterperformance. After the jobs complete, performance will return to normal.
Upgrade OneFS through the web administration interfaceYou can upgrade the OneFS version that is running on an EMC Isilon cluster through theweb administration interface.
Before you begin
You must download the OneFS installation image from the EMC Support website. The
OneFS installation image is used to run compatibility checks. If you have not downloadedthe image and run the checks, you must do so before installing the new version of OneFS.
You must resolve any compatibility issues found by the checks before installing the newversion of OneFS. See "Run compatibility checks" for more information.
Procedure
1. Log in to the lowest-numbered node in the cluster through the OneFS webadministration interface with the root account.
2. Navigate to the Upgrade OneFSpage.
l For OneFS 6.5 or earlier, click Cluster> Cluster Management> Upgrade Summary.
l For OneFS 7.0 or later, click Help>About This Cluster, and then click Upgrade.
3. Browse to the location of the installation image that you want to install, and then clickSubmit.
4. In the Upgrade Modearea, select one of the following upgrade modes, and then clickContinue.
Options Description
Simultaneous
upgrade
Upgrades and restarts all the nodes in the cluster at the sametime. While the nodes are restarting, the cluster is unavailablefor client connections.
Performing the OneFS upgrade
Upgrade OneFS through the web administration interface 39
8/11/2019 OneFS Upgrade Planning and Process Guide
40/52
Options Description
Rolling upgrade Upgrades and restarts each node in the cluster sequentially sothat clients can access the cluster during the upgrade.
Rolling upgrades are not available for some OneFS versions.Review the Isilon Supportability and Compatibility Guidefor alist of versions that support rolling upgrades.
The cluster displays status updates and a prompt to continue the upgrade process.
5. Optional: For rolling upgrades, specify how to terminate TCP client connections beforeupgrading the node.
Options Description
To immediately
terminate client
connections
Click Immediately terminate TCP connections.
To allow client
connections to terminate
after a wait period
a. Click Wait for TCP connections to terminate.
b. In the Wait timefield, type an integer that representshow long to wait in seconds, minutes, hours, days,weeks, or months.
Note
If any client connections are not terminated within the
specified wait time, the upgrade will halt or fail. You must
manually disconnect client connections and restart the
upgrade.
6. Optional: For rolling upgrades, specify whether you want to confirm restarting eachnode during the upgrade process.
Options Description
To display a notification and a
prompt before each upgraded node
is restarted
Click Confirm before rebooting nodes.
Note
This option requires that you stay connected to
the session to confirm each reboot. If you
disconnect, the upgrade will halt until you can
connect to the node to confirm the reboot.
To automatically restart each
upgraded node without a prompt
Click Reboot nodes without confirmation.
7. Click yesto begin the upgrade process.
The cluster might display several confirmation messages. You must confirm eachmessage to continue the upgrade process.
After the upgrade is complete, the cluster restarts, and then the web administrationinterface login page appears. If the login page does not appear, clear your web
browser cache and reload the page.
Performing the OneFS upgrade
40 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
41/52
8/11/2019 OneFS Upgrade Planning and Process Guide
42/52
8/11/2019 OneFS Upgrade Planning and Process Guide
43/52
CHAPTER 5
Completing post-upgrade tasks
This section contains the following topics:
u About post-upgrade tasks..................................................................................... 44u Allow upgrade jobs to run......................................................................................44u Verify operational status....................................................................................... 44u Re-establish user privileges...................................................................................45u Restore client connections and test your workflow.................................................46u Restore custom settings........................................................................................46
u Re-install Aspera...................................................................................................46u Modify custom scripts........................................................................................... 47u Implement the Platform API ...................................................................................47
Completing post-upgrade tasks 43
8/11/2019 OneFS Upgrade Planning and Process Guide
44/52
About post-upgrade tasksAfter an upgrade, you should perform a number of restoration and change managementtasks to ensure your EMC Isilon cluster performs and behaves as expected.
Once the new OneFS image has been installed, the system will run a series of upgradejobs on the system that must be allowed to finish, and then you should verify that yourcluster is operational.
It is important that you build time into your upgrade plan to re-establish custom settingsand privileges, and re-enable connections and features. You should also make time tomodify settings for new and changed features.
Allow upgrade jobs to runYou must allow upgrade-related jobs to run to completion.
After the upgrade, a number of upgrade-related jobsincluding an important job thatredistributes, or re-stripes, data more efficientlywill continue to run for a while. Youmust allow the jobs to run.
Although the EMC Isilon cluster remains accessible while the upgrade jobs are running,the jobs might temporarily decrease the cluster's performance.
Verify operational statusYou can run a series of commands as root to help verify that the EMC Isilon cluster isworking correctly after an upgrade.
If you find an unresolvable issue, contact Isilon Technical Support.
Procedure
1. Check the new version number of the cluster:
uname -a
2. View the status of the cluster and make sure all your nodes are operational:
isi status -D -w
3. Ping all of the cluster's internal and external interfaces to verify network connectivityand to help verify that SmartConnect works correctly.
4. Review the list of events and address any critical events:
isi events list -w
5. Check the status of jobs and resume the jobs that you paused for the upgrade:
isi job status view
6. Verify your network interfaces:
isi networks list interfaces
7. Verify your subnets:
isi networks list subnets --verbose
8. Verify your pools:
isi networks list pools --verbose
9. Review the cluster's other log files to check for stray problems:
cat /var/log/messages
Completing post-upgrade tasks
44 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
45/52
10.Review the list of SyncIQ jobs:
isi sync jobs list
11.Check the SyncIQ job reports:
isi sync reports list
12.Review the list of your scheduled snapshots:
isi snapshot schedules list
13.Check the cluster's input and output; type Ctrl-C when you are done:
isi statistics system --nodes --top
14.Check the devices in the nodes to validate the status of your drives:
isi_for_array -s "isi devices | grep -iv healthy"
15.Check your global SMB settings:
isi smb settings global view
16.Check the status of the firmware to ensure that the firmware is consistent acrossnodes:
isi firmware status
17.Make sure that all your licenses carried over and remain up to date:
isi license
18.Check the status of your authentication providers to make sure they remain active:
isi auth status --verbose
Re-establish user privilegesAfter you upgrade, you should re-establish user privileges and roles.
After upgrading, you can log in to the EMC Isilon cluster through SSH only as root. Otheradministrator accounts can log in only after you give them administrative access andassign them roles.
Even though you can log in to the web administration interface with an administrativeaccount other than root, OneFS limits the changes that you can make until you assignadministrative roles. Similarly, some of the commands in the command-line interfacerequire privileges.
OneFS 7.0 and later includes role-based access control (RBAC) for administration. Inplace of a root or administrator account, RBAC lets you manage administrative access byrole. A role limits privileges to an area of administration. For example, you can createseparate administrator roles for security, auditing, storage, and backup. AdministrativeRBAC helps improve security and compliance.
If your system used RBAC before your upgrade, custom roles are still in place. However, ifthe privileges assigned to built-in roles have changed in the new version, your usersassigned to those built-in roles will now have those new privileges.
See the OneFS Web Administration Guideor the OneFS CLI Administration Guidefor moreinformation.
Completing post-upgrade tasks
Re-establish user privileges 45
8/11/2019 OneFS Upgrade Planning and Process Guide
46/52
Restore client connections and test your workflowAfter an upgrade, ensure that you EMC Isilon cluster works correctly, and then restoreclient connections and confirm that all your users, clients, and applications can access
the cluster.
If you disconnected iSCSI initiators before you upgraded, you should reconnect them andset the cluster's iSCSI LUN to online in the web administration interface.
Test your workflows to ensure that they function correctly.
Restore custom settingsYou should restore the custom settings that you backed up from your EMC Isilon cluster.
Certain custom settings might not been transferred during the upgrade. Reapply thecustom settings that you recorded and backed up before the upgrade. The custom
settings include:u SMB audit logging
u Passwords for local user accounts
u Changes to system controls
u Certificates
u Static routes
u Aspera for Isilon
u Cron jobs
Re-install AsperaIf you were running Aspera for Isilon before you upgraded your EMC Isilon cluster, youmust reinstall Aspera after you upgrade to restore the Aspera service.
Before you begin
See EMC Isilon knowledge base article 91221, Aspera Enterprise Server application isstopped after upgrading to OneFS 6.5 or later, for information about restarting the Asperaserver after upgrading. Also, see the Aspera Enterprise Server for Isilonguide atAsperasoft.com.
Procedure
1. Connect to a node by SSH and log in as root.2. To install Aspera on the node, run the following command:
/usr/local/aspera/var/install.sh
3. To install Aspera on all the nodes in the cluster, run the following command:
isi_for_array /usr/local/aspera/var/install.sh
4. Verify that the installation script enabled the Aspera Central service and the Asperanode daemon by running the following command:
isi services
Completing post-upgrade tasks
46 OneFS Upgrade Planning and Process Guide
8/11/2019 OneFS Upgrade Planning and Process Guide
47/52
8/11/2019 OneFS Upgrade Planning and Process Guide
48/52
8/11/2019 OneFS Upgrade Planning and Process Guide
49/52
CHAPTER 6
Troubleshooting your upgrade
This section contains the following topics:
u Troubleshooting overview..................................................................................... 50u Review the upgrade logs....................................................................................... 50u Common issues.....................................................................................................50
Troubleshooting your upgrade 49
8/11/2019 OneFS Upgrade Planning and Process Guide
50/52
Troubleshooting overviewYou can check upgrade logs and review common upgrade issues if you experienceproblems with your OneFS upgrade.
You can search for Isilon OneFS documentation and knowledge base articles on the EMCOnline Support page. Go to EMC Online Supportto begin a search.
If you need help troubleshooting, contact Isilon Technical Support.
Review the upgrade logsYou can inspect logs from the upgrade to check for issues that occurred during theupgrade process.
An upgrade generates several logs and places them in the /var/crashdirectory.
update_engine
Generates two logs. One log contains general information about the upgrade andone contains information about the after-upgrade reboot. The logs names are similarto the following:
update_engine_2011-06-26_08_56_48.txtupdate_engine_2011-06-26_09_00_50.txt
update_handler
Generates a log that provides an overall view of the upgrade on the EMC Isiloncluster. You will usually find more specific information about errors in this log. Thelog name is similar to the following:
update_handler_2011-11-20_05_54_59.txt
update_proxy
Generates a log that provides a view of the new OneFS image running on the nodes.The log name is similar to the following:
update_proxy_2011-11-20_05_54_59.txt
Common issuesCheck to see if problems with your upgrade are due to common issues.
Adjust the SMB client-credit minimum
SMB2 client connections might stop responding because of an increase in the client-credit minimum.
Adjust the SMB client-credit minimum if the following errors occur:
u The SMB clients show system "error 51" or "error 0x80070033."
u The Microsoft Windows event log contains the following entry:
Operating system error 64 (The specified network name is nolonger available.)
Troubleshooting your upgrade
50 OneFS Upgrade Planning and Process Guide
https://support.emc.com/products/15209_Isilon-OneFShttps://support.emc.com/products/15209_Isilon-OneFS8/11/2019 OneFS Upgrade Planning and Process Guide
51/52
u An error similar to the following appears in the packet captures on the servers:
58031 9.752678000 10.232.21.62 146.168.81.131 SMB2 131 WriteResponse, Error: STATUS_INSUFF_SERVER_RESOURCES
See EMC Isilon knowledge base article 88645, SMB2 connections stop respondingbecause client credit minimum value is too high, for more information.
Adjust NFS performance
Fix the system controls for Network File System (NFS) server threads
NFS performance might be lower than expected because of a configuration change. Thedefault number of NFS server threads was changed to address a potential issue in whichthe NFS server monopolizes node resources.
It is recommended that you set threads_min and threads_max to the same value.Increasing the number of threads can improve performance at the expense of stability.Before you change the number of threads, contact Isilon Technical Support to determinethe values that work best for your EMC Isilon cluster; the values vary by CPUs, memory,the number of nodes, and other factors.
After you determine the number of threads for your cluster by consulting with IsilonTechnical Support, you can adjust the number of NFS threads by running the followingcommands, where x is an integer.
Procedure
1. To modify the minimum number of threads, run the following command:
isi_sysctl_cluster vfs.nfsrv.rpc.threads_min=x
2. To modify the maximum number of threads, run the following command:
isi_sysctl_cluster vfs.nfsrv.rpc.threads_max=x
Verify a 64-bit system
You might need to migrate to a 64-bit system.
The upgrade version of OneFS may require a 64-bit U.S. system, and applications andtools that you want to run in the user space must be compiled for a 64-bit system. If atool fails to detect that it must run as a 64-bit application, you must recompile it for a 64-bit system.
Note
Running your own applications on
Top Related