IBM Spectrum Scale 4.2.3: Administration Guide · IBM Spectrum Scale V ersion 4 Release 2.3...

IBM Spectrum ScaleVersion 4 Release 2.3

Administration Guide

SA23-1455-19

IBM

NoteBefore using this information and the product it supports, read the information in “Notices” on page 653.

This edition applies to version 4 release 2 modification 3 of the following products, and to all subsequent releasesand modifications until otherwise indicated in new editions:v IBM Spectrum Scale ordered through Passport Advantage® (product number 5725-Q01)v IBM Spectrum Scale ordered through AAS/eConfig (product number 5641-GPF)v IBM Spectrum Scale for Linux on Z (product number 5725-S28)v IBM Spectrum Scale for IBM ESS (product number 5765-ESS)

Significant changes or additions to the text and illustrations are indicated by a vertical line (|) to the left of thechange.

IBM welcomes your comments; see the topic “How to send your comments” on page xxiii. When you sendinformation to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believesappropriate without incurring any obligation to you.

© Copyright IBM Corporation 2014, 2019.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Tables . . . . . . . . . . . . . . . ix

About this information . . . . . . . . xiPrerequisite and related information . . . . . xxiiConventions used in this information . . . . . xxiiHow to send your comments . . . . . . . . xxiii

Summary of changes . . . . . . . . xxv

Chapter 1. Configuring the GPFS cluster 1Creating your GPFS cluster . . . . . . . . . 1Displaying cluster configuration information . . . 1

Basic configuration information . . . . . . . 1Information about protocol nodes . . . . . . 2

Adding nodes to a GPFS cluster . . . . . . . . 2Deleting nodes from a GPFS cluster . . . . . . 3Changing the GPFS cluster configuration data . . . 4Security mode . . . . . . . . . . . . . 15Running IBM Spectrum Scale commands withoutremote root login . . . . . . . . . . . . 17

Configuring sudo . . . . . . . . . . . 17Configuring the cluster to use sudo wrapperscripts . . . . . . . . . . . . . . . 18Configuring IBM Spectrum Scale GUI to usesudo wrapper . . . . . . . . . . . . 19Configuring a cluster to stop using sudo wrapperscripts . . . . . . . . . . . . . . . 19

Node quorum considerations . . . . . . . . 19Node quorum with tiebreaker considerations . . . 20Displaying and changing the file system managernode. . . . . . . . . . . . . . . . . 20Determining how long mmrestripefs takes tocomplete . . . . . . . . . . . . . . . 21Starting and stopping GPFS . . . . . . . . . 22Shutting down an IBM Spectrum Scale cluster . . . 23

Chapter 2. Configuring the CES andprotocol configuration . . . . . . . . 25Configuring Cluster Export Services . . . . . . 25

Setting up Cluster Export Services shared rootfile system. . . . . . . . . . . . . . 25Configuring Cluster Export Services nodes . . . 26Configuring CES protocol service IP addresses . 26CES IP aliasing to network adapters on protocolnodes . . . . . . . . . . . . . . . 27Deploying Cluster Export Services packages onexisting IBM Spectrum Scale 4.1.1 and later nodes 32Verifying the final CES configurations . . . . 32

Creating and configuring file systems and filesetsfor exports. . . . . . . . . . . . . . . 33Configuring with the installation toolkit . . . . . 33Deleting a Cluster Export Services node from anIBM Spectrum Scale cluster . . . . . . . . . 33

Setting up Cluster Export Services groups in an IBMSpectrum Scale cluster . . . . . . . . . . . 34

Chapter 3. Configuring and tuning yoursystem for GPFS . . . . . . . . . . 37General system configuration and tuningconsiderations . . . . . . . . . . . . . 37

Clock synchronization . . . . . . . . . . 37GPFS administration security . . . . . . . 37Cache usage . . . . . . . . . . . . . 38Access patterns . . . . . . . . . . . . 40Aggregate network interfaces . . . . . . . 40Swap space . . . . . . . . . . . . . 41

Linux configuration and tuning considerations . . 41updatedb considerations . . . . . . . . . 41Memory considerations . . . . . . . . . 41GPFS helper threads . . . . . . . . . . 42Communications I/O . . . . . . . . . . 42Disk I/O . . . . . . . . . . . . . . 42

AIX configuration and tuning considerations . . . 43GPFS use with Oracle . . . . . . . . . . 43

Chapter 4. Parameters for performancetuning and optimization . . . . . . . 45Tuning parameters change history . . . . . . . 47

Chapter 5. Configuring and tuning yoursystem for cloud services . . . . . . 53Designating the cloud services nodes . . . . . . 53Pre-validating the cloud account settings . . . . 54Associating a file system with cloud services nodes 55Creating a cloud storage account . . . . . . . 56Enabling a Transparent cloud tiering policy. . . . 57Enabling a policy for Cloud data sharing exportservice . . . . . . . . . . . . . . . . 58Tuning cloud services parameters . . . . . . . 58Integrating cloud services metrics with theperformance monitoring tool . . . . . . . . 60

GPFS-based configuration . . . . . . . . 60File-based configuration . . . . . . . . . 61

Configuring cloud services with SKLM . . . . . 62Deploying WORM solutions . . . . . . . . . 63

Creating immutable filesets and files . . . . . 63Setting up Transparent cloud tiering for WORMsolutions . . . . . . . . . . . . . . 64

Chapter 6. Configuring Active FileManagement . . . . . . . . . . . . 73Configuration parameters for AFM . . . . . . 73Parallel I/O configuration parameters for AFM . . 76

Chapter 7. Configuring AFM-based DR 79Configuration parameters for AFM-based DR . . . 79

© Copyright IBM Corp. 2014, 2019 iii

|||||||

Parallel I/O configuration parameters forAFM-based DR . . . . . . . . . . . . . 80

Chapter 8. Tuning for Kernel NFSbackend on AFM and AFM DR . . . . 83Tuning the gateway node on the NFS client . . . 83Tuning on both the NFS client (gateway) and theNFS server (the home/secondary cluster) . . . . 83Tuning the NFS server on the home/secondarycluster or the NFS server . . . . . . . . . . 84

Chapter 9. Performing GPFSadministration tasks . . . . . . . . . 87Requirements for administering a GPFS file system 87

adminMode configuration attribute . . . . . 88Common GPFS command principles . . . . . . 89

Specifying nodes as input to GPFS commands. . 89Stanza files . . . . . . . . . . . . . 90Listing active IBM Spectrum Scale commands . . 92

Chapter 10. Verifying network operationwith the mmnetverify command . . . . 93

Chapter 11. Managing file systems. . . 95Mounting a file system . . . . . . . . . . 95

Mounting a file system on multiple nodes . . . 96Mount options specific to IBM Spectrum Scale. . 96Mounting a file system through GUI . . . . . 97Changing a file system mount point on protocolnodes . . . . . . . . . . . . . . . 98

Unmounting a file system . . . . . . . . . 98Unmounting a file system on multiple nodes . . 98Unmounting a file system through GUI . . . . 99

Deleting a file system . . . . . . . . . . . 99Determining which nodes have a file systemmounted . . . . . . . . . . . . . . . 100Checking and repairing a file system . . . . . 100Dynamic validation of descriptors on disk. . . . 102Listing file system attributes . . . . . . . . 102Modifying file system attributes . . . . . . . 103Querying and changing file replication attributes 104

Querying file replication. . . . . . . . . 104Changing file replication attributes . . . . . 104

Using Direct I/O on a file in a GPFS file system 105File compression . . . . . . . . . . . . 105Setting the Quality of Service for I/O operations(QoS) . . . . . . . . . . . . . . . . 111Restriping a GPFS file system . . . . . . . . 113Querying file system space . . . . . . . . . 114Querying and reducing file system fragmentation 115

Querying file system fragmentation . . . . . 116Reducing file system fragmentation . . . . . 116

Protecting data in a file system using backup. . . 117Protecting data in a file system using themmbackup command. . . . . . . . . . 117Backing up a file system using the GPFS policyengine . . . . . . . . . . . . . . . 123Backing up file system configurationinformation . . . . . . . . . . . . . 123

Using APIs to develop backup applications . . 124Scale Out Backup and Restore (SOBAR) . . . . 124Scheduling backups using IBM Spectrum Protectscheduler . . . . . . . . . . . . . . . 125Configuration reference for using IBM SpectrumProtect with IBM Spectrum Scale . . . . . . . 125

Options in the IBM Spectrum Protectconfiguration file dsm.sys . . . . . . . . 126Options in the IBM Spectrum Protectconfiguration file dsm.opt . . . . . . . . 127Base IBM Spectrum Protect client configurationfiles for IBM Spectrum Scale usage . . . . . 128

Restoring a subset of files or directories from alocal file system snapshot . . . . . . . . . 129Restoring a subset of files or directories from alocal fileset snapshot . . . . . . . . . . . 130Restoring a subset of files or directories from localsnapshots using the sample script . . . . . . 132

Chapter 12. File system formatchanges between versions of IBMSpectrum Scale . . . . . . . . . . 133

Chapter 13. Managing disks . . . . . 137Displaying disks in a GPFS cluster . . . . . . 137Adding disks to a file system . . . . . . . . 138Deleting disks from a file system . . . . . . . 138Replacing disks in a GPFS file system . . . . . 140Additional considerations for managing disks . . 142Displaying GPFS disk states . . . . . . . . 142

Disk availability . . . . . . . . . . . 142Disk status . . . . . . . . . . . . . 142

Changing GPFS disk states and parameters . . . 143Changing your NSD configuration . . . . . . 145Changing NSD server usage and failback . . . . 146Enabling and disabling Persistent Reserve . . . . 146

Chapter 14. Managing protocolservices. . . . . . . . . . . . . . 149Configuring and enabling SMB and NFS protocolservices . . . . . . . . . . . . . . . 149Configuring and enabling the Object protocolservice. . . . . . . . . . . . . . . . 150

Performance tuning for object services . . . . 151Configuring and enabling the BLOCK service . . 151Disabling protocol services . . . . . . . . . 153

Chapter 15. Managing protocol userauthentication . . . . . . . . . . . 155Setting up authentication servers to configureprotocol user access . . . . . . . . . . . 155

Integrating with AD server . . . . . . . . 155Integrating with LDAP server . . . . . . . 156Integrating with Keystone Identity Service . . 161

Configuring authentication and ID mapping for fileaccess . . . . . . . . . . . . . . . . 162

AD-based authentication for file access . . . . 162Configuring LDAP-based authentication for fileaccess . . . . . . . . . . . . . . . 169

iv IBM Spectrum Scale 4.2.3: Administration Guide

||

||

Configuring NIS-based authentication . . . . 174Managing user-defined authentication . . . . . 175Configuring authentication for object access . . . 178

Configuring local authentication for objectaccess . . . . . . . . . . . . . . . 179Configuring an AD-based authentication forobject access . . . . . . . . . . . . . 181Configuring an LDAP-based authentication forobject access . . . . . . . . . . . . . 183Configuring object authentication with anexternal keystone server . . . . . . . . . 186Creating object accounts . . . . . . . . . 186Managing object users, roles, and projects . . . 188Deleting expired tokens . . . . . . . . . 191

Deleting the authentication and the ID mappingconfiguration . . . . . . . . . . . . . 191Listing the authentication configuration . . . . 193Verifying the authentication services configured inthe system . . . . . . . . . . . . . . 193Modifying the authentication method . . . . . 194Authentication limitations . . . . . . . . . 195

Chapter 16. Managing protocol dataexports . . . . . . . . . . . . . . 199Managing SMB shares . . . . . . . . . . 199

Creating SMB share . . . . . . . . . . 199Changing SMB share configuration . . . . . 200Creating SMB share ACLs . . . . . . . . 200Removing SMB shares . . . . . . . . . 200Listing SMB shares . . . . . . . . . . 200Managing SMB shares using MMC . . . . . 201

Managing NFS exports . . . . . . . . . . 205Creating NFS exports. . . . . . . . . . 205Changing NFS export configuration . . . . . 206Removing NFS exports . . . . . . . . . 206Listing NFS exports . . . . . . . . . . 206GUI navigation for NFS exports . . . . . . 207Making bulk changes to NFS exports . . . . 207

Multiprotocol exports . . . . . . . . . . 209Multiprotocol export considerations . . . . . . 210

Chapter 17. Managing object storage 211Understanding and managing Object services . . 211Understanding the mapping of OpenStackcommands to IBM Spectrum Scale administratorcommands . . . . . . . . . . . . . . 213Changing Object configuration values . . . . . 214Changing the object base configuration to enableS3 API. . . . . . . . . . . . . . . . 214Configuring OpenStack EC2 credentials . . . . 214Managing OpenStack access control lists using S3API. . . . . . . . . . . . . . . . . 215Managing object capabilities . . . . . . . . 216Managing object versioning . . . . . . . . 217

Enabling object versioning . . . . . . . . 217Disabling object versioning . . . . . . . . 217Creating a version of an object: Example . . . 217

Mapping of storage policies to filesets . . . . . 218Administering storage policies for object storage 219

Creating storage policy for object compression 220

Creating storage policy for object encryption 220Adding a region in a multi-region objectdeployment . . . . . . . . . . . . . . 221Administering a multi-region object deploymentenvironment. . . . . . . . . . . . . . 223Unified file and object access in IBM SpectrumScale . . . . . . . . . . . . . . . . 224

Identity management modes for unified file andobject access . . . . . . . . . . . . . 224Authentication in unified file and object access 229Validating shared authentication ID mapping 229The objectizer process . . . . . . . . . 230File path in unified file and object access . . . 231Administering unified file and object access . . 233In-place analytics using unified file and objectaccess . . . . . . . . . . . . . . . 240Limitations of unified file and object access . . 241Constraints applicable to unified file and objectaccess . . . . . . . . . . . . . . . 242Data ingestion examples. . . . . . . . . 243curl commands for unified file and object accessrelated user tasks . . . . . . . . . . . 244

Configuration files for IBM Spectrum Scale forobject storage . . . . . . . . . . . . . 244Backing up and restoring object storage . . . . 248

Backing up the object storage . . . . . . . 248Restoring the object storage . . . . . . . 250

Configuration of object for isolated node andnetwork groups . . . . . . . . . . . . 253Enabling access to existing filesets . . . . . . 254Enabling the object heatmap policy . . . . . . 256

Chapter 18. Managing GPFS quotas 259Enabling and disabling GPFS quota management 259Default quotas . . . . . . . . . . . . . 260Implications of quotas for different protocols . . . 262Explicitly establishing and changing quotas . . . 263Setting quotas for users on a per-project basis . . 264Checking quotas . . . . . . . . . . . . 266Listing quotas . . . . . . . . . . . . . 267Activating quota limit checking . . . . . . . 268Deactivating quota limit checking . . . . . . 269Changing the scope of quota limit checking . . . 269Creating file system quota reports . . . . . . 269Restoring quota files . . . . . . . . . . . 270

Chapter 19. Managing GUI users . . . 273

Chapter 20. Managing GPFS accesscontrol lists . . . . . . . . . . . . 277Traditional GPFS ACL administration . . . . . 277

Setting traditional GPFS access control lists . . 278Displaying traditional GPFS access control lists 279Applying an existing traditional GPFS accesscontrol list . . . . . . . . . . . . . 279Changing traditional GPFS access control lists 280Deleting traditional GPFS access control lists 280

NFS V4 ACL administration . . . . . . . . 281NFS V4 ACL Syntax . . . . . . . . . . 281NFS V4 ACL translation . . . . . . . . . 283

Contents v

Setting NFS V4 access control lists . . . . . 284Displaying NFS V4 access control lists . . . . 284Applying an existing NFS V4 access control list 284Changing NFS V4 access control lists . . . . 284Deleting NFS V4 access control lists . . . . . 285Considerations when using GPFS with NFS V4ACLs . . . . . . . . . . . . . . . 285

NFS and GPFS . . . . . . . . . . . . . 285Exporting a GPFS file system using NFS . . . 285NFS usage of GPFS cache . . . . . . . . 289Synchronous writing using NFS . . . . . . 289Unmounting a file system after NFS export . . 289NFS automount considerations . . . . . . 289Clustered NFS and GPFS on Linux . . . . . 289

Authorizing protocol users . . . . . . . . . 290Authorizing file protocol users . . . . . . 290Authorizing object users. . . . . . . . . 299Authorization limitations . . . . . . . . 305

Chapter 21. Considerations for GPFSapplications . . . . . . . . . . . . 307Exceptions to Open Group technical standards . . 307Determining if a file system is controlled by GPFS 307GPFS exceptions and limitations to NFS V4 ACLs 308

Linux ACLs and extended attributes. . . . . 308General CES NFS Linux exceptions and limitations 309Considerations for the use of direct I/O(O_DIRECT). . . . . . . . . . . . . . 310NFS protocol node limitations . . . . . . . . 311

Chapter 22. Accessing a remote GPFSfile system . . . . . . . . . . . . 313Remote user access to a GPFS file system . . . . 315Using NFS/SMB protocol over remote clustermounts . . . . . . . . . . . . . . . 316

Configuring protocols on a separate cluster . . 317Managing multi-cluster protocol environments 318Upgrading multi-cluster environments . . . . 319Limitations of protocols on remotely mountedfile systems . . . . . . . . . . . . . 319

Mounting a remote GPFS file system . . . . . 320Managing remote access to a GPFS file system . . 322Using remote access with public and private IPaddresses. . . . . . . . . . . . . . . 322Using multiple security levels for remote access 324Changing security keys with remote access . . . 325NIST compliance . . . . . . . . . . . . 326Important information about remote access . . . 327

Chapter 23. Information lifecyclemanagement for IBM Spectrum Scale . 329Storage pools . . . . . . . . . . . . . 329

Internal storage pools . . . . . . . . . 330External storage pools . . . . . . . . . 335

Policies for automating file management . . . . 336Overview of policies . . . . . . . . . . 336Policy rules . . . . . . . . . . . . . 337The mmapplypolicy command and policy rules 356Policy rules: Examples and tips . . . . . . 360Managing policies . . . . . . . . . . . 365

Working with external storage pools. . . . . 368Backup and restore with storage pools . . . . 373ILM for snapshots . . . . . . . . . . . 374

Filesets . . . . . . . . . . . . . . . 375Fileset namespace . . . . . . . . . . . 376Filesets and quotas . . . . . . . . . . 376Filesets and storage pools . . . . . . . . 377Filesets and global snapshots . . . . . . . 377Fileset-level snapshots . . . . . . . . . 378Filesets and backup . . . . . . . . . . 378Managing filesets . . . . . . . . . . . 379

Immutability and appendOnly features . . . . . 382

Chapter 24. Creating and maintainingsnapshots of file systems . . . . . . 387Creating a snapshot . . . . . . . . . . . 387Listing snapshots . . . . . . . . . . . . 388Restoring a file system from a snapshot . . . . 389Reading a snapshot with the policy engine . . . 390Linking to a snapshot . . . . . . . . . . 390Deleting a snapshot . . . . . . . . . . . 391Managing snapshots with IBM Spectrum Scale GUI 392

Chapter 25. Creating and managingfile clones . . . . . . . . . . . . . 395Creating file clones . . . . . . . . . . . 395Listing file clones . . . . . . . . . . . . 396Deleting file clones . . . . . . . . . . . 397Splitting file clones from clone parents . . . . . 397File clones and disk space management . . . . 397File clones and snapshots . . . . . . . . . 397File clones and policy files . . . . . . . . . 398

Chapter 26. Scale Out Backup andRestore (SOBAR). . . . . . . . . . 399Backup procedure with SOBAR . . . . . . . 399Restore procedure with SOBAR . . . . . . . 401

Chapter 27. Data Mirroring andReplication . . . . . . . . . . . . 405General considerations for using storage replicationwith GPFS . . . . . . . . . . . . . . 406Data integrity and the use of consistency groups 406Handling multiple versions of IBM Spectrum Scaledata . . . . . . . . . . . . . . . . 406Continuous Replication of IBM Spectrum Scaledata . . . . . . . . . . . . . . . . 407

Synchronous mirroring with GPFS replication 407Synchronous mirroring utilizing storage basedreplication . . . . . . . . . . . . . 417Point In Time Copy of IBM Spectrum Scale data 425

Chapter 28. Implementing a clusteredNFS environment on Linux . . . . . 429NFS monitoring . . . . . . . . . . . . 429NFS failover . . . . . . . . . . . . . . 429NFS locking and load balancing . . . . . . . 429CNFS network setup . . . . . . . . . . . 430CNFS setup . . . . . . . . . . . . . . 430

vi IBM Spectrum Scale 4.2.3: Administration Guide

||||||||||||

CNFS administration . . . . . . . . . . . 431

Chapter 29. Implementing ClusterExport Services . . . . . . . . . . 433CES features. . . . . . . . . . . . . . 433

CES cluster setup . . . . . . . . . . . 433CES network configuration . . . . . . . . 434CES address failover and distribution policies 435CES protocol management . . . . . . . . 436CES management and administration . . . . 436

CES NFS support . . . . . . . . . . . . 437CES SMB support . . . . . . . . . . . . 440CES OBJ support . . . . . . . . . . . . 441Migration of CNFS clusters to CES clusters . . . 443

Chapter 30. Identity management onWindows . . . . . . . . . . . . . 447Auto-generated ID mappings . . . . . . . . 447Installing Windows IDMU . . . . . . . . . 447Configuring ID mappings in IMU . . . . . . 448

Chapter 31. Protocols cluster disasterrecovery . . . . . . . . . . . . . 451Protocols cluster disaster recovery limitations andprerequisites. . . . . . . . . . . . . . 451Example setup for protocols disaster recovery . . 452Setting up gateway nodes to ensure clustercommunication during failover . . . . . . . 453Creating the inband disaster recovery setup . . . 453Creating the outband disaster recovery setup. . . 455Performing failover for protocols cluster whenprimary cluster fails . . . . . . . . . . . 457

Re-create file export configuration . . . . . 457Restore file export configuration . . . . . . 457

Performing failback to old primary for protocolscluster . . . . . . . . . . . . . . . . 458

Re-create file protocol configuration for oldprimary . . . . . . . . . . . . . . 458Restore file protocol configuration for oldprimary . . . . . . . . . . . . . . 459

Performing failback to new primary for protocolscluster . . . . . . . . . . . . . . . . 461

Re-create file protocol configuration for newprimary . . . . . . . . . . . . . . 461Restore file protocol configuration for newprimary . . . . . . . . . . . . . . 464

Backing up and restoring protocols and CESconfiguration information . . . . . . . . . 467Updating protocols and CES configurationinformation . . . . . . . . . . . . . . 468Protocols and cluster configuration data requiredfor disaster recovery . . . . . . . . . . . 468

Object data required for protocols cluster DR 468SMB data required for protocols cluster DR . . 474NFS data required for protocols cluster DR . . 475Authentication related data required forprotocols cluster DR . . . . . . . . . . 476CES data required for protocols cluster DR . . 478

Chapter 32. File Placement Optimizer 481Distributing data across a cluster . . . . . . . 485FPO pool file placement and AFM . . . . . . 485Configuring FPO . . . . . . . . . . . . 485

Configuring IBM Spectrum Scale Clusters . . . 486Basic Configuration Recommendations . . . . 490Configuration and tuning of Hadoop workloads 501Configuration and tuning of databaseworkloads . . . . . . . . . . . . . 502Configuring and tuning SparkWorkloads . . . 502

Ingesting data into IBM Spectrum Scale clusters 503Exporting data out of IBM Spectrum Scale clusters 503Upgrading FPO . . . . . . . . . . . . 503Monitoring and administering IBM Spectrum ScaleFPO clusters. . . . . . . . . . . . . . 506

Rolling upgrades . . . . . . . . . . . 507The IBM Spectrum Scale FPO cluster . . . . 509Failure detection . . . . . . . . . . . 511Disk Failures . . . . . . . . . . . . 511Node failure. . . . . . . . . . . . . 513Handling multiple nodes failure . . . . . . 514Network switch failure . . . . . . . . . 515Data locality. . . . . . . . . . . . . 515Disk Replacement . . . . . . . . . . . 524

Auto recovery . . . . . . . . . . . . . 526Failure and recovery . . . . . . . . . . 526QoS support for autorecovery . . . . . . . 528

Restrictions . . . . . . . . . . . . . . 528

Chapter 33. Encryption . . . . . . . 529Encryption keys . . . . . . . . . . . . 529Encryption policies . . . . . . . . . . . 530Encryption policy rules . . . . . . . . . . 530Preparation for encryption . . . . . . . . . 535Establishing an encryption-enabled environment 540

Simplified setup: Using SKLM with aself-signed certificate . . . . . . . . . . 540Simplified setup: Valid and invalidconfigurations . . . . . . . . . . . . 547Simplified setup: Accessing a remote file system 550Simplified setup: Doing other tasks . . . . . 554Regular setup: Using SKLM with a self-signedcertificate . . . . . . . . . . . . . . 560Regular setup: Using SKLM with a certificatechain . . . . . . . . . . . . . . . 568Configuring encryption with SKLM v2.7 . . . 577Configuring encryption with the Vormetric DSMkey server . . . . . . . . . . . . . 580

Secure deletion . . . . . . . . . . . . . 587Encryption and standards compliance . . . . . 588

Encryption and FIPS-140-2 certification . . . . 588Encryption and NIST SP800-131A compliance 589

Encryption in a multicluster environment . . . . 589Encryption in a Disaster Recovery environment 589Encryption and backup/restore . . . . . . . 589Encryption and snapshots . . . . . . . . . 589Encryption requirements and limitations . . . . 590

Contents vii

|||||||||||||

||||||

Chapter 34. Managing certificates tosecure communications between GUIweb server and web browsers . . . . 591

Chapter 35. Securing protocol data 593Planning for protocol data security . . . . . . 595Configuring protocol data security . . . . . . 595

Enabling secured connection between the IBMSpectrum Scale system and authenticationserver . . . . . . . . . . . . . . . 595Securing data transfer . . . . . . . . . 598Securing NFS data transfer . . . . . . . . 598Securing SMB data transfer. . . . . . . . 601Secured object data transfer . . . . . . . 601

Data security limitations. . . . . . . . . . 601

Chapter 36. Cloud services:Transparent cloud tiering and Clouddata sharing . . . . . . . . . . . . 603Administering Transparent cloud tiering and Clouddata sharing services . . . . . . . . . . . 603

Starting cloud services . . . . . . . . . 603Stopping cloud services . . . . . . . . . 603Checking the cloud services version . . . . . 604

Testing a cloud storage account . . . . . . . 604Listing cloud storage accounts. . . . . . . . 605Administering files for Transparent cloud tiering 605

Applying a policy on a Transparent cloudtiering node . . . . . . . . . . . . . 606Migrating files to the cloud storage tier. . . . 608Pre-migrating files to the cloud storage tier . . 608Recalling files from the cloud storage tier . . . 610Reconciling files between IBM Spectrum Scalefile system and cloud storage tier. . . . . . 611Cleaning up files transferred to the cloudstorage tier . . . . . . . . . . . . . 611Deleting cloud objects . . . . . . . . . 612Listing files migrated to the cloud storage tier 613Restoring files . . . . . . . . . . . . 613

Cloud data sharing . . . . . . . . . . . 614Listing files exported to the cloud . . . . . 615

Deleting a file system association . . . . . . . 618Deleting a cloud storage account . . . . . . . 618Backing up configuration data for Transparentcloud tiering. . . . . . . . . . . . . . 619Manual recovery of Transparent cloud tieringdatabase . . . . . . . . . . . . . . . 619Known limitations of cloud services . . . . . . 620

Chapter 37. Highly available writecache (HAWC) . . . . . . . . . . . 623Applications that can benefit from HAWC. . . . 624Restrictions and tuning recommendations forHAWC . . . . . . . . . . . . . . . 624Using HAWC . . . . . . . . . . . . . 625

Chapter 38. Local read-only cache 627

Chapter 39. Miscellaneous advancedadministration topics . . . . . . . . 629Changing IP addresses and host names. . . . . 629Enabling a cluster for IPv6 . . . . . . . . . 630Using multiple token servers . . . . . . . . 630Exporting file system definitions between clusters 631IBM Spectrum Scale port usage . . . . . . . 631Securing the IBM Spectrum Scale system usingfirewall . . . . . . . . . . . . . . . 634

Firewall recommendations for the IBMSpectrum Scale installation . . . . . . . . 634Firewall recommendations for internalcommunication among nodes . . . . . . . 635Firewall recommendations for protocol access 636Firewall recommendations for IBM SpectrumScale GUI . . . . . . . . . . . . . 640Firewall recommendations for IBM SKLM . . . 641Firewall recommendations for Vormetric DSM 641Firewall recommendations for the REST API . . 642Firewall recommendations for PerformanceMonitoring tool . . . . . . . . . . . 642Firewall considerations for Active FileManagement (AFM) . . . . . . . . . . 643Firewall considerations for remote mounting offile systems . . . . . . . . . . . . . 643Firewall recommendations for using IBMSpectrum Protect with IBM Spectrum Scale . . 644Firewall considerations for using IBM SpectrumArchive with IBM Spectrum Scale . . . . . 644Examples of how to open firewall ports . . . 644

Logging file system activities . . . . . . . . 646Supported web browser versions and web browsersettings for GUI . . . . . . . . . . . . 646

Chapter 40. GUI limitations . . . . . 649

Accessibility features for IBMSpectrum Scale . . . . . . . . . . 651Accessibility features . . . . . . . . . . . 651Keyboard navigation . . . . . . . . . . . 651IBM and accessibility . . . . . . . . . . . 651

Notices . . . . . . . . . . . . . . 653Trademarks . . . . . . . . . . . . . . 654Terms and conditions for product documentation 655IBM Online Privacy Statement. . . . . . . . 655

Glossary . . . . . . . . . . . . . 657

Index . . . . . . . . . . . . . . . 663

viii IBM Spectrum Scale 4.2.3: Administration Guide

||

||

|||

Tables

1. IBM Spectrum Scale library information units xii2. Conventions . . . . . . . . . . . xxiii3. List of changes in documentation . . . . xxxii4. Configuration attributes on the mmchconfig

command . . . . . . . . . . . . . 65. Configuration parameters at cache and their

default values at the cache cluster . . . . . 736. Configuration parameters at cache and their

default values at the cache cluster - Validvalues . . . . . . . . . . . . . . 75

7. Configuration parameters at cache for parallelI/O . . . . . . . . . . . . . . . 76

8. Configuration parameters at cache for parallelI/O - valid values . . . . . . . . . . 77

9. Configuration parameters at primary and theirdefault values . . . . . . . . . . . . 79

10. Configuration parameters at primary and theirdefault values - Valid values . . . . . . . 79

11. Configuration parameters at cache for parallelI/O . . . . . . . . . . . . . . . 80

12. Configuration parameters at cache for parallelI/O - valid values . . . . . . . . . . 80

13. NFS server parameters . . . . . . . . . 8414. COMPRESSED and illCompressed flags . . . . 10815. Set QoS classes to unlimited . . . . . . 11216. Allocate the available IOPS . . . . . . . 11217. Authentication requirements for each file

access protocol. . . . . . . . . . . . 17718. Object services and object protocol nodes 21319. Object input behavior in unified_mode 22620. Configuration options for [swift-constraints]

in swift.conf . . . . . . . . . . . . 24221. Configurable options for [DEFAULT] in

object-server-sof.conf . . . . . . . . . 24522. Configurable options for [capabilities] in

spectrum-scale-object.conf . . . . . . . 24623. Configuration options for [DEFAULT] in

spectrum-scale-objectizer.conf . . . . . . 24624. Configuration options for

[IBMOBJECTIZER-LOGGER] inspectrum-scale-objectizer.conf . . . . . . 247

25. Configuration options for object-server.conf 24726. Configuration options for

/etc/sysconfig/memcached. . . . . . . 24727. Configuration options for proxy-server.conf 24728. Removal of a file with ACL entries DELETE

and DELETE_CHILD . . . . . . . . . 28329. Mapping from SMB Security Descriptor to

NFSv4 ACL entry . . . . . . . . . . 29130. Mapping from NFSv4 ACL entry to SMB

Security Descriptor. . . . . . . . . . 29231. ACL permissions required to work on files

and directories, while using SMB protocol(table 1 of 2) . . . . . . . . . . . . 294

32. ACL permissions required to work on filesand directories, while using SMB protocol(table 2 of 2) . . . . . . . . . . . . 295

33. ACL permissions required to work on filesand directories, while using NFS protocol(table 1 of 2) . . . . . . . . . . . . 296

34. ACL permissions required to work on filesand directories, while using NFS protocol(table 2 of 2) . . . . . . . . . . . . 296

35. Commands and reference to manage ACLtasks . . . . . . . . . . . . . . 298

36. ACL options that are available to manipulateobject read ACLs . . . . . . . . . . 303

37. Summary of commands to set upcross-cluster file system access.. . . . . . 322

38. The effects of file operations on an immutablefile or an appendOnly file . . . . . . . 383

39. IAM modes and their effect on immutable fileoperations . . . . . . . . . . . . 384

40. Example for retention period . . . . . . 39241. Example - Time stamp of snapshots that are

retained based on the retention policy . . . 39242. Valid EncParamString values . . . . . . 53143. Valid combine parameter string values 53144. Valid wrapping parameter string values 53145. Required version of IBM Spectrum Scale 53646. Remote Key Management servers . . . . . 53647. The RKM.conf file . . . . . . . . . . 53848. The client keystore directory . . . . . . 54049. Configuring a node for encryption in the

simplified setup. . . . . . . . . . . 54350. Setup of Cluster1 and Cluster2. . . . . . 55051. Managing another key server . . . . . . 55552. Security features that are used to secure

authentication server . . . . . . . . . 59353. Sample policy list . . . . . . . . . . 60754. IBM Spectrum Scale port usage . . . . . 63255. Firewall related information. . . . . . . 63456. Recommended port numbers that can be used

for installation . . . . . . . . . . . 63557. Recommended port numbers that can be used

for internal communication . . . . . . . 63558. Recommended port numbers for NFS access 63659. Recommended port numbers for SMB access 63760. Recommended port numbers for iSCSI access 63761. Port numbers for object access . . . . . . 63862. Port numbers for object authentication 63863. Port numbers for Postgres database for object

protocol . . . . . . . . . . . . . 63964. Consolidated list of recommended ports for

different functions . . . . . . . . . . 63965. Firewall recommendations for GUI . . . . 64066. Firewall recommendations for SKLM 64167. Firewall recommendations for SKLM 64168. Firewall recommendations for REST API 642

© Copyright IBM Corp. 2014, 2019 ix

||

|

||||||

||||

69. Recommended port numbers that can be usedfor Performance Monitoring tool . . . . . 642

70. Required port number for mmbackup andHSM connectivity to IBM Spectrum Protectserver . . . . . . . . . . . . . . 644

x IBM Spectrum Scale 4.2.3: Administration Guide

About this information

This edition applies to IBM Spectrum Scale™ version 4.2.3 for AIX®, Linux, and Windows.

IBM Spectrum Scale is a file management infrastructure, based on IBM® General Parallel File System(GPFS™) technology, which provides unmatched performance and reliability with scalable access tocritical file data.

To find out which version of IBM Spectrum Scale is running on a particular AIX node, enter:lslpp -l gpfs\*

To find out which version of IBM Spectrum Scale is running on a particular Linux node, enter:rpm -qa | grep gpfs (for SLES and Red Hat Enterprise Linux)

dpkg -l | grep gpfs (for Ubuntu Linux)

To find out which version of IBM Spectrum Scale is running on a particular Windows node, openPrograms and Features in the control panel. The IBM Spectrum Scale installed program name includesthe version number.

Which IBM Spectrum Scale information unit provides the information you need?

The IBM Spectrum Scale library consists of the information units listed in Table 1 on page xii.

To use these information units effectively, you must be familiar with IBM Spectrum Scale and the AIX,Linux, or Windows operating system, or all of them, depending on which operating systems are in use atyour installation. Where necessary, these information units provide some background information relatingto AIX, Linux, or Windows. However, more commonly they refer to the appropriate operating systemdocumentation.

Note: Throughout this documentation, the term “Linux” refers to all supported distributions of Linux,unless otherwise specified.

© Copyright IBM Corp. 2014, 2019 xi

Table 1. IBM Spectrum Scale library information units

Information unit Type of information Intended users

IBM Spectrum Scale: Concepts,Planning, and Installation Guide

This guide provides the followinginformation:

Product overview

v Overview of IBM Spectrum Scalev GPFS architecturev Protocols support overview:

Integration of protocol accessmethods with GPFS

v Active File Managementv AFM-based Asynchronous Disaster

Recovery (AFM DR)

v Data protection and disaster recoveryin IBM Spectrum Scale

v Introduction to IBM Spectrum ScaleGUI

v IBM Spectrum Scale management APIv Introduction to Cloud servicesv IBM Spectrum Scale in an OpenStack

cloud deployment

v IBM Spectrum Scale product editionsv IBM Spectrum Scale license

designation

v Capacity based licensingv IBM Spectrum Storage™ Suite

Planning

v Planning for GPFSv Planning for protocolsv Considerations for GPFS applicationsv Firewall recommendationsv Planning for cloud services

System administrators, analysts,installers, planners, and programmers ofIBM Spectrum Scale clusters who arevery experienced with the operatingsystems on which each IBM SpectrumScale cluster is based

xii IBM Spectrum Scale 4.2.3: Administration Guide

Table 1. IBM Spectrum Scale library information units (continued)


IBM Spectrum Scale: Concepts,Planning, and Installation Guide

Installing and upgrading

v Steps for establishing and startingyour IBM Spectrum Scale cluster

v Installing IBM Spectrum Scale onLinux nodes and deploying protocols

v Installing IBM Spectrum Scale on AIXnodes

v Installing IBM Spectrum Scale onWindows nodes

v Installing cloud services on IBMSpectrum Scale nodes

v Installing and configuring IBMSpectrum Scale management API

v Installing Active File Managementv Installing and upgrading AFM-based

Disaster Recovery

v Installing call homev Migration, coexistence and

compatibility

v Steps to permanently uninstall GPFSand/or Protocols

System administrators, analysts,installers, planners, and programmers ofIBM Spectrum Scale clusters who arevery experienced with the operatingsystems on which each IBM SpectrumScale cluster is based

About this information xiii



IBM Spectrum Scale:Administration Guide


Configuring

v Configuring the GPFS clusterv Configuring the CES and protocol

configuration

v Configuring and tuning your systemfor GPFS

v Parameters for performance tuningand optimization

v Configuring and tuning your systemfor Cloud services

v Configuring Active File Managementv Configuring AFM-based DRv Tuning for Kernel NFS backend on

AFM and AFM DR

Administering

v Performing GPFS administration tasksv Verifying network operation with the

mmnetverify command

v Managing file systemsv File system format changes between

versions of IBM Spectrum Scale

v Managing disksv Managing protocol servicesv Managing protocol user

authentication

v Managing protocol data exportsv Managing object storagev Managing GPFS quotasv Managing GUI usersv Managing GPFS access control listsv Considerations for GPFS applicationsv Accessing a remote GPFS file system

System administrators or programmersof IBM Spectrum Scale systems

xiv IBM Spectrum Scale 4.2.3: Administration Guide



IBM Spectrum Scale:Administration Guide

v Information lifecycle management forIBM Spectrum Scale

v Creating and maintaining snapshotsof file systems

v Creating and managing file clonesv Scale Out Backup and Restore

(SOBAR)

v Data Mirroring and Replicationv Implementing a clustered NFS

environment on Linux

v Implementing Cluster Export Servicesv Identity management on Windowsv Protocols cluster disaster recoveryv File Placement Optimizerv Encryptionv Managing certificates to secure

communications between GUI webserver and web browsers

v Securing protocol datav Cloud services: Transparent cloud

tiering and Cloud data sharing

v Highly-available write cache (HAWC)v Local read-only cachev Miscellaneous advanced

administration

v GUI limitations

System administrators or programmersof IBM Spectrum Scale systems

About this information xv



IBM Spectrum Scale: ProblemDetermination Guide


Monitoring

v Performance monitoringv Monitoring system health through the

IBM Spectrum Scale GUI

v Monitoring system health by usingthe mmhealth command

v Monitoring events through callbacksv Monitoring capacity through GUIv Monitoring AFM and AFM DRv GPFS SNMP supportv Monitoring the IBM Spectrum Scale

system by using call home

v Monitoring the health of cloudservices

Troubleshooting

v Best practices for troubleshootingv Understanding the system limitationsv Collecting details of the issuesv Managing deadlocksv Installation and configuration issuesv Upgrade issuesv Network issuesv File system issuesv Disk issuesv Security issuesv Protocol issuesv Disaster recovery issuesv Performance issuesv GUI issuesv AFM issuesv AFM DR issuesv Transparent cloud tiering issuesv Recovery proceduresv Support for troubleshootingv References

System administrators of GPFS systemswho are experienced with thesubsystems used to manage disks andwho are familiar with the conceptspresented in the IBM Spectrum Scale:Concepts, Planning, and Installation Guide

xvi IBM Spectrum Scale 4.2.3: Administration Guide



IBM Spectrum Scale: Commandand Programming Reference


Command reference

v gpfs.snap commandv mmaddcallback commandv mmadddisk commandv mmaddnode commandv mmadquery commandv mmafmconfig commandv mmafmctl commandv mmafmlocal commandv mmapplypolicy commandv mmauth commandv mmbackup commandv mmbackupconfig commandv mmblock commandv mmbuildgpl commandv mmcallhome commandv mmces commandv mmcesdr commandv mmchattr commandv mmchcluster commandv mmchconfig commandv mmchdisk commandv mmcheckquota commandv mmchfileset commandv mmchfs commandv mmchlicense commandv mmchmgr commandv mmchnode commandv mmchnodeclass commandv mmchnsd commandv mmchpolicy commandv mmchpool commandv mmchqos commandv mmclidecode commandv mmclone commandv mmcloudgateway commandv mmcrcluster commandv mmcrfileset commandv mmcrfs commandv mmcrnodeclass commandv mmcrnsd commandv mmcrsnapshot command

v System administrators of IBMSpectrum Scale systems

v Application programmers who areexperienced with IBM Spectrum Scalesystems and familiar with theterminology and concepts in theXDSM standard

About this information xvii




v mmdefedquota commandv mmdefquotaoff commandv mmdefquotaon commandv mmdefragfs commandv mmdelacl commandv mmdelcallback commandv mmdeldisk commandv mmdelfileset commandv mmdelfs commandv mmdelnode commandv mmdelnodeclass commandv mmdelnsd commandv mmdelsnapshot commandv mmdf commandv mmdiag commandv mmdsh commandv mmeditacl commandv mmedquota commandv mmexportfs commandv mmfsck commandv mmfsctl commandv mmgetacl commandv mmgetstate commandv mmhadoopctl commandv mmhealth commandv mmimgbackup commandv mmimgrestore commandv mmimportfs commandv mmkeyserv commandv mmlinkfileset commandv mmlsattr commandv mmlscallback commandv mmlscluster commandv mmlsconfig commandv mmlsdisk command



xviii IBM Spectrum Scale 4.2.3: Administration Guide




v mmlsfileset commandv mmlsfs commandv mmlslicense commandv mmlsmgr commandv mmlsmount commandv mmlsnodeclass commandv mmlsnsd commandv mmlspolicy commandv mmlspool commandv mmlsqos commandv mmlsquota commandv mmlssnapshot commandv mmmigratefs commandv mmmount commandv mmnetverify commandv mmnfs commandv mmnsddiscover commandv mmobj commandv mmperfmon commandv mmpmon commandv mmprotocoltrace commandv mmpsnap commandv mmputacl commandv mmquotaoff commandv mmquotaon commandv mmremotecluster commandv mmremotefs commandv mmrepquota commandv mmrestoreconfig commandv mmrestorefs commandv mmrestripefile commandv mmrestripefs commandv mmrpldisk commandv mmsdrrestore commandv mmsetquota commandv mmshutdown commandv mmsmb commandv mmsnapdir commandv mmstartup commandv mmtracectl commandv mmumount commandv mmunlinkfileset commandv mmuserauth commandv mmwinservctl commandv spectrumscale command



About this information xix




Programming reference

v IBM Spectrum Scale DataManagement API for GPFSinformation

v GPFS programming interfacesv GPFS user exitsv IBM Spectrum Scale management API

commands



xx IBM Spectrum Scale 4.2.3: Administration Guide



IBM Spectrum Scale: Big Data andAnalytics Guide


IBM Spectrum Scale support forHadoop

v HDFS transparencyv Supported IBM Spectrum Scale

storage modes

v Hadoop cluster planningv Installation and configuration of

HDFS transparency

v Application interaction with HDFStransparency

v Upgrading the HDFS Transparencycluster

v Rolling upgrade for HDFSTransparency

v Securityv Advanced featuresv Hadoop distribution supportv Limitations and differences from

native HDFS

v Problem determination

BigInsights® 4.2.5 and HortonworksData Platform 2.6

v Planning– Hardware requirements

– Preparing the environment

– Preparing a stanza file

v Installation– Set up

– Installation of software stack

– BigInsights value-add services onIBM Spectrum Scale

v Upgrading software stack– Migrating from BI IOP to HDP

– Upgrading IBM Spectrum Scaleservice MPack

– Upgrading HDFS Transparency

– Upgrading IBM Spectrum Scale filesystem

– Upgrading to BI IOP 4.2.5



About this information xxi

||

|

|



IBM Spectrum Scale: Big Data andAnalytics Guide

v Configuration– Setting up High Availability [HA]

– IBM Spectrum Scale configurationparameter checklist

– Dual-network deployment

– Manually starting services inAmbari

– Setting up local repository

– Configuring LogSearch

– Setting IBM Spectrum Scaleconfiguration for BigSQL

v Administration– IBM Spectrum Scale-FPO

deployment

– Ranger

– Kerberos

– Short-circuit read (SSR)

– Disabling short circuit write

– IBM Spectrum Scale servicemanagement

– Ambari node management

– Restricting root access

– IBM Spectrum Scale managementGUI

– IBM Spectrum Scale versus NativeHDFS

v Troubleshooting– Snap data collection

v Limitations– Limitations and information

v FAQ– General

– Service fails to start

– Service check failures



Prerequisite and related informationFor updates to this information, see IBM Spectrum Scale in IBM Knowledge Center(www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html).

For the latest support information, see the IBM Spectrum Scale FAQ in IBM KnowledgeCenter(www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html).

Conventions used in this informationTable 2 on page xxiii describes the typographic conventions used in this information. UNIX file nameconventions are used throughout this information.

xxii IBM Spectrum Scale 4.2.3: Administration Guide

|

||

|

|

|

|

http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html

Note: Users of IBM Spectrum Scale for Windows must be aware that on Windows, UNIX-style filenames need to be converted appropriately. For example, the GPFS cluster configuration data is stored inthe /var/mmfs/gen/mmsdrfs file. On Windows, the UNIX namespace starts under the %SystemDrive%\cygwin64 directory, so the GPFS cluster configuration data is stored in the C:\cygwin64\var\mmfs\gen\mmsdrfs file.

Table 2. Conventions

Convention Usage

bold Bold words or characters represent system elements that you must use literally, such ascommands, flags, values, and selected menu options.

Depending on the context, bold typeface sometimes represents path names, directories, or filenames.

bold underlined bold underlined keywords are defaults. These take effect if you do not specify a differentkeyword.

constant width Examples and information that the system displays appear in constant-width typeface.

Depending on the context, constant-width typeface sometimes represents path names,directories, or file names.

italic Italic words or characters represent variable values that you must supply.

Italics are also used for information unit titles, for the first use of a glossary term, and forgeneral emphasis in text.

Angle brackets (less-than and greater-than) enclose the name of a key on the keyboard. Forexample, refers to the key on your terminal or workstation that is labeled with theword Enter.

\ In command examples, a backslash indicates that the command or coding example continueson the next line. For example:

mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \-E "PercentTotUsed < 85" -m p "FileSystem space used"

{item} Braces enclose a list from which you must choose an item in format and syntax descriptions.

[item] Brackets enclose optional items in format and syntax descriptions.

The notation indicates a control character sequence. For example, meansthat you hold down the control key while pressing .

item... Ellipses indicate that you can repeat the preceding item one or more times.

| In synopsis statements, vertical lines separate a list of choices. In other words, a vertical linemeans Or.

In the left margin of the document, vertical lines indicate technical changes to theinformation.

Note: CLI options that accept a list of option values delimit with a comma and no space between values.As an example, to display the state on three nodes use mmgetstate -N NodeA,NodeB,NodeC. Exceptions tothis syntax are listed specifically within the command.

How to send your commentsYour feedback is important in helping us to produce accurate, high-quality information. If you have anycomments about this information or any other IBM Spectrum Scale documentation, send your commentsto the following e-mail address:

[email protected]

About this information xxiii

Include the publication title and order number, and, if applicable, the specific location of the informationabout which you have comments (for example, a page number or a table number).

To contact the IBM Spectrum Scale development organization, send your comments to the followinge-mail address:

[email protected]

xxiv IBM Spectrum Scale 4.2.3: Administration Guide

Summary of changes

This topic summarizes changes to the IBM Spectrum Scale licensed program and the IBM Spectrum Scalelibrary. Within each information unit in the library, a vertical line (|) to the left of text and illustrationsindicates technical changes or additions that are made to the previous edition of the information.

Summary of changesfor IBM Spectrum Scale version 4 release 2.3as updated, June 2018

This release of the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library includes thefollowing improvements:

Added support for OpenStack Mitaka packagesSupport for OpenStack Mitaka packages has been added for the object protocol. For moreinformation, see Protocols support overview: Integration of protocol access methods with GPF in IBMSpectrum Scale: Concepts, Planning, and Installation Guide.

Authentication considerations changesThe following changes are done:v Authentication support matrix has been divided to separate out the File and object protocols

and accordingly, the corresponding explanation is modified.v The matrix is further divided based on the authentication service that is used.v A diagram is added to explain the high-level flow of authentication for File protocols.v "Authentication for file access" topic is renamed to "Authentication and ID mapping for file

access".

For more information, see the Authentication considerations topic in the IBM Spectrum Scale:Concepts, Planning, and Installation Guide.

Big data and analytics changesFor information on changes in IBM Spectrum Scale Big Data and Analytics support, see Big Dataand Analytics - summary of changes.

Directory preallocationIn environments in which many files are added to and removed from a directory in a short time,you can improve performance by setting the minimum compaction size of the directory. Theminimum compaction size is the number of directory slots, including both full and empty slots,that a directory is allowed to retain when it is compacted. For more information, seegpfs_prealloc() subroutine, mmchattr command, and mmlsattr command in IBM Spectrum Scale:Command and Programming Reference.

Express Edition no longer availableIBM Spectrum Scale Express Edition is no longer available. For information on migrating fromIBM Spectrum Scale Express Edition 4.2.2.x or earlier to IBM Spectrum Scale Standard Edition4.2.3.x, see Migrating from Express Edition to Standard Edition in IBM Spectrum Scale: Concepts,Planning, and Installation Guide.

FPO enhancementsFPO performs the following functions:v Provides QoS support for auto recoveryv Supports locality-aware data copyv Uses the mmrestripefile command to check whether the replicas of data blocks are matched

for one file

© Copyright IBM Corp. 2014, 2019 xxv

|||

||

||||

||

||

|

|

||

||

|||

|||||||

|||||

||

|

|

||

https://www.ibm.com/support/knowledgecenter/STXKQY_BDA_SHR/bl1bda_soc.htm?cp=STXKQY_4.2.3https://www.ibm.com/support/knowledgecenter/STXKQY_BDA_SHR/bl1bda_soc.htm?cp=STXKQY_4.2.3

Installation toolkit support for gpfs.adv and gpfs.crypto packagesThe installation toolkit now supports installation, deployment, and upgrade of gpfs.adv andgpfs.crypto packages.

Installation toolkit support for populating cluster definition fileThe installation toolkit now supports populating the cluster definition file with the current clusterstate. For more information, see Populating cluster definition file with current cluster state using theinstallation toolkit in IBM Spectrum Scale: Concepts, Planning, and Installation Guide.

Installation toolkit support for Red Hat Enterprise Linux 7.4 and 7.5The installation toolkit now also supports Red Hat Enterprise Linux 7.4 and 7.5 on x86_64,PPC64, and PPC64LE architectures. For more information, see Installation prerequisites in IBMSpectrum Scale: Concepts, Planning, and Installation Guide.

IBM Spectrum Scale GUI changesThe following main changes are made in the IBM Spectrum Scale GUI:v Supports mounting and unmounting of file systems on selected nodes or group of nodes using

GUI. For more information, see Mounting a file system through GUI and Unmounting a file systemthrough GUI topics in IBM Spectrum Scale: Administration Guide.

v Added new Storage > Pools page. The Pools page provides details about configuration, health,capacity, and performance aspects of storage pools.

v Added new Files > Active File Management page. This new GUI page helps to view theconfiguration, health status, and performance of AFM, AFM DR, and gateway nodes.

v Added new Monitoring > Tips page. The tip events give recommendations to the user toavoid certain issues that might occur in the future. A tip disappears from the GUI when theproblem behind the tip event is resolved.

v Added option to select events of type “tip” in the Settings > Event Notifications > EmailRecipients page. You can configure whether to send email to the recipients if a tip event isreported in the system.

v Added detailed view in the Files > Filesets page. You can access the detailed view ofindividual filesets either by double-clicking the individual filesets or by selecting View Detailsoption.

v Modified the Storage > NSDs page to list the rack, position, and node of the NSD in anFPO-enabled environment. This helps to sort the NSDs based on these parameters. The failuregroup definition is also modified to accommodate these new parameters.

v Added the Customize the number of replicas option in the Files > Information Lifecycle pageto specify the number of replicas in a file placement rule.

v Modified the Settings > Event Notifications page to accept both IP address and host name forthe email server.

v Added Nodes and File Systems tabs in the detailed view that is available in the Files >Transparent Cloud Tiering page.

v Added a separate Properties tab in the detailed view that is available in the Monitoring >Nodes , Files > File Systems , and Storage > NSDs pages.

IBM Spectrum Scale functionality to support GDPR requirementsTo understand the requirements of EU General Data Protection Regulation (GDPR) compliancethat are applicable to unstructured data storage and how IBM Spectrum Scale helps to addressthem, see the IBM Spectrum Scale functionality to support GDPR requirements technote.

Introduction of IBM Spectrum Scale management API Version 2The architecture and syntax of IBM Spectrum Scale management API is changed. The newimplementation is based on the GUI stack. The GUI server is managing and processing the APIrequests and commands. Version 2 has the following features:v Reuses the GUI deployment's backend infrastructure, which makes introduction of new API

commands easier.

xxvi IBM Spectrum Scale 4.2.3: Administration Guide

|||

||||

||||

||

|||

||

||

|||

|||

|||

|||

||

||

||

||

||||

||||

||

http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD106420

v No separate configuration is required as the GUI installation takes care of the basicdeployment.

v Fixes scalability issues and introduces new features such as filter parameter, field parameter,and paging.

v Supports large clusters with thousands of nodes.v All POST, PUT, and DELETE requests are completed asynchronously. A "jobs" object is created

immediately when such a request is submitted.v The APIs are driven by the same WebSphere® server and object cache that is used by the IBM

Spectrum Scale GUI.v The mmrest command is no longer required for configuring the management API. The IBM

Spectrum Scale GUI installation and configuration takes care of the API infrastructureconfiguration. For more information on how to configure IBM Spectrum Scale management APIVersion 2, see Configuring IBM Spectrum Scale management API in IBM SpectrumScale:Administration Guide.

As the syntax and architecture are changed for the API, modified the entire set of commands,which were available in the Version 1. New API commands are also added for improvedflexibility. For more information about the available commands, see IBM Spectrum Scalemanagement API commands in IBM Spectrum Scale: Command and Programming Reference. You canalso access the documentation corresponding to each API command from the GUI itself. The APIdocumentation is available in the GUI at: https://:/ibm/api/explorer/. For example: https://scalegui.ibm.com:443/ibm/api/explorer.

Linux on Z enhancementsThe following changes are made:v IBM Spectrum Scale for Linux on Z now supports Remote Cluster Mount (Multi-cluster).v SLES 12.2 and RHEL 7.3 are now supported by IBM Spectrum Scale for Linux on Z.

mmcallhome command: Addition of --long option to mmcallhome group list commandThe --long option displays the long admin node names. For more information, see mmcallhomecommand in IBM Spectrum Scale: Command and Programming Reference.

mmchconfig command: Setting an InfiniBand partition keyThe --verbsRdmaPkey attribute specifies an InfiniBand partition key for a connection between anode and an InfiniBand server that is included in an InfiniBand partition. For more information,see mmchconfig command in the IBM Spectrum Scale: Command and Programming Reference.

mmdiag command: Status and queue statistics for NSD queuesThe --nsd parameter displays the status and queue statistics for NSD queues.

For more information, see mmdiag command in IBM Spectrum Scale: Command and ProgrammingReference.

mmfsck command: Severity of errorsThe command displays a summary of the errors that were found that includes the severity ofeach error: CRITICAL, NONCRITICAL, or HARMLESS. You must specify the verbose orsemi-verbose parameter to get this output. For more information, see mmfsck command in IBMSpectrum Scale: Command and Programming Reference.

mmhealth command: Addition of new options to commandAddition of AFM and THRESHOLD options to the mmhealth node show and mmhealth clustershow commands. The AFM option displays the health status of a gateway node or cluster. TheTHRESHOLD option monitors whether the node-related thresholds rules evaluation is runningas expected, and if the health status has changed as a result of the threshold limits being crossed.

Addition of --clear option to the mmhealth node eventlog command. This option clears the eventlog's database.

Summary of changes xxvii

||

||

|

||

||

|||||

|||||||

||

|

|

|||

||||

||

||

|||||

|||||

||

Addition of threshold add and threshold delete option to the mmhealth command. This optionallows users to create and delete threshold rule.

Addition of event hide, event unhide, and list hidden options to the mmhealth command. Theevent hide option hides the specified TIP events, while the event unhide option reveals all TIPevents that were previously hidden. The list hidden option shows all the TIP events that areadded to the list of hidden events.

Addition of config interval option to the mmhealth command. The config interval option allowsyou to set the interval for monitoring the whole cluster.

For more information, see mmhealth command in IBM Spectrum Scale: Command and ProgrammingReference.

mmkeyserv command: Updating a certificate or a connectionYou can now get a fresh certificate from an Remote Key Management (RKM) server withoutrebuilding the connection. You can also temporarily update a connection by adding backupservers, reordering the list of backup servers, or changing the timeout, number of retries, or retryinterval. For more information, see mmkeyserv command in IBM Spectrum Scale: Command andProgramming Reference.

mmlslicense command: Displaying disk and cluster size informationYou can now get information about disk and cluster size with the mmlslicense command. Formore information, see mmlslicense command in IBM Spectrum Scale: Command and ProgrammingReference.

mmnetverify command: EnhancementsSeveral enhancements increase the capabilities of the mmnetverify command. Network checks areadded to measure the total bandwidth, to check connectivity with the CTDB port, and to checkconnectivity with servers that are used with the Object protocol. If there are multiple local nodes,the command is run on all the local nodes in parallel. The lists of local nodes and target nodesaccept node classes. The --ces-override parameter causes the command to consider all the nodesin the configuration to be CES-enabled. For more information, see mmnetverify command in IBMSpectrum Scale: Command and Programming Reference.

mmrestripefile command: Fix inconsistencies between file data and replicasThe -c option compares the data of individual files with their replicas and attempts to fix anyinconsistencies. For more information, see mmrestripefile command in IBM Spectrum Scale: Commandand Programming Reference.

Monitoring of AFM and AFM DR Using commands:v Functionality added to mmhealth,mmdiag, and mmperfmon.

Using IBM Spectrum Scale GUI:v Added new Files > Active File Management page. This new GUI page helps to view the

configuration, health status, and performance of AFM, AFM DR, and gateway nodes.

Mount options specific to IBM Spectrum Scale: syncnfs is now the default on Linux nodesIn the mount options specific to IBM Spectrum Scale, syncnfs is now the default on Linux nodes.On AIX nodes, nosyncnfs is the default. For more information, see Mount options specific to IBMSpectrum Scale in IBM Spectrum Scale: Command and Programming Reference.

Protocol support on remotely mounted file systemsYou can create an NFS/SMB export on a file system that is mounted from a remote cluster. Formore information, see the Using NFS/SMB protocol over remote cluster mounts topic in the IBMSpectrum Scale: Administration Guide.

Tip added to event status to inform users when a configuration is not optimalA new event type TIP is added to system health monitoring. A Tip is similar to a state-changingevent, but can be hidden by the user. Like state-changing events, a tip is removed automatically if

xxviii IBM Spectrum Scale 4.2.3: Administration Guide

||

||||

||

||

||||||

||||

||||||||

||||

||

|

|

||

||||

||||

|||

the problem is resolved. For more information on Tip, see Event type and monitoring status forsystem health in the IBM Spectrum Scale: Problem Determination Guide

Quality of Service for I/O operations (QoS): Detailed statisticsYou can now display more detailed statistics about IOPS rates for the QoS programs that arerunning on each node. The statistics are intended to be used as input for programs that analyzeand display data. For more information, see mmchqos command and mmlsqos command in IBMSpectrum Scale: Command and Programming Reference.

Support for Samba 4.5

Transparent cloud tiering enhancements.The following changes are done:v Support for configuring and deploying WORM solutions. Your files will remain

WORM-compliant, both in the file system and on the cloud object storage. For moreinformation, see the Deploying WORM solutions topic in the IBM Spectrum Scale: AdministrationGuide.

v Support for configuring Transparent cloud tiering with a proxy server.v Support for configuring cloud retention time, which overrides the default value.v Support for restoring only the file stubs from the cloud storage tier in situations where files are

deleted from the local file system.v Support for Power8 Little Endian platform.

Note: This feature is available from 4.2.3.1 onwards.v Substantial improvement in the performance when files are transparently recalled from the

storage tier.v Support for manually deleting orphaned cloud objects before retention time expires. For more

information, see the Manually deleting cloud objects before retention time topic in the IBM SpectrumScale: Administration Guide.

v Support for migrating files in the co-resident state, by which applications can directly accessdata without performing any recall operation. For more information, see the Pre-migrating filesto the cloud storage tier topic in the IBM Spectrum Scale: Administration Guide

-Y option

Added the -Y option to the following commands:

v mmblockv mmcloudgatewayv mmdfv mmdiagv mmgetstate

v mmhealthv mmkeyservv mmlsclusterv mmlsconfigv mmlsdisk

v mmlsfilesetv mmlsfsv mmlslicensev mmlsmgrv mmlsmount

v mmlsnodeclassv mmlsnsdv mmlspolicyv mmlsquotav mmlssnapshot

v mmnetverifyv mmnfsv mmrepquotav mmsmbv mmuserauth

Documented commands, structures, and subroutinesThe following lists the modifications to the documented commands, structures, and subroutines:

New commandsThe following commands are new:v mmclidecode

New structuresThere are no new structures.

New subroutinesThere are no new subroutines.

Summary of changes xxix

||

|||||

|

||

||||

|

|

||

|

|

||

|||

|||

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

||

||

||

|

||

||

Changed commandsThe following commands were changed:v mmadqueryv mmbackupv mmblockv mmcallhomev mmcesv mmcesdrv mmchattrv mmchconfigv mmchqosv mmcloudgatewayv mmcrnsdv mmdfv mmdiagv mmfsckv mmgetstatev mmhadoopctlv mmhealthv mmimgbackupv mmimgrestorev mmkeyservv mmlsattrv mmlsclusterv mmlsconfigv mmlsdiskv mmlsfilesetv mmlsfsv mmlslicensev mmlsmgrv mmlsmountv mmlsnodeclassv mmlsnsdv mmlspolicyv mmlsqosv mmlsquotav mmlssnapshotv mmnetverifyv mmnfsv mmprotocoltracev mmrepquotav mmsmbv mmuserauthv spectrumscale

xxx IBM Spectrum Scale 4.2.3: Administration Guide

||

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

|

Changed structuresThe following structures were changed:v gpfs_iattr64_t

Changed subroutinesThe following subroutines were changed:v gpfs_prealloc

Deleted commandsmmrest

Deleted structuresThere are no deleted structures.

Deleted subroutinesThere are no deleted subroutines.

MessagesThe following are the new, changed, and deleted messages:

New messages6027-1525, 6027-1756, 6027-2392, 6027-2393, 6027-2503, 6027-2504, and 6027-3258

Changed messages6027-1023, 6027-1725

Deleted messagesNone.

Changes in documentation

Big data and analytics support

Moved the entire big data and analytics support information to a new section. See thetopic Big data and analytics support in IBM Spectrum Scale: Big Data and Analytics Guide.

Restructured events pageThe events page was split up into 19 different pages, with a separate page for eachcomponent. See the topic .Events in the IBM Spectrum Scale: Problem Determination Guide.

Renamed “REST API” to “IBM Spectrum Scale management API” in the documentation.

List of documentation changes in product guides and respective Knowledge Center sections

The following is a list of documentation changes including changes in topic titles,changes in placement of topics, and deleted topics:

Summary of changes xxxi

||

|

||

|

||

||

||

||

||

||

||

|

|

||

|||

|

|

||

Table 3. List of changes in documentation

GuideKnowledge centersection List of changes

Concepts, Planning,and Installation Guide

Product overview Under IBM Spectrum Scale management API

v Moved the IBM Spectrum Scale management API topics from theAdministering section to the IBM Spectrum Scale management APIsection.

Planning Under Planning for protocols → Authentication considerations

v Changed the title Authentication for file access to Authentication andID mapping for file access

Under Planning for protocols → Planning for SMB

v The SMB share limitations topic under Administering → Managingprotocol data exports → Managing SMB shares has been removed.

Limitations from the SMB share limitations topic have been addedin the SMB limitations topic.

Installing andupgrading

v Removed Installing the Scale Management server (REST API)section.

v Moved Manually upgrading pmswift and Manually upgrading thePerformance Monitoring tool from Manually installing thePerformance Monitoring tool to the Migration, coexistence andcompatibility section.

v Moved Upgrading IBM Spectrum Scale components with theinstallation toolkit from Using the spectrumscale installation toolkit toperform installation tasks: Explanations and examples to theMigration, coexistence and compatibility section.

v Created the Upgrading Object packages section:– Moved the Upgrading Object packages to version 4.2.2.x from

4.2.2.x topic from Migrating to IBM Spectrum Scale 4.2.2.x fromIBM Spectrum Scale 4.2.0.x or later to the Upgrading Objectpackages section.

– Added the Upgrading Object packages to version 4.2.3.x from4.2.2.x topic.

Administration Guide Configuring v Removed the Configuring and starting the Scale Management server(REST API) section.

v Removed the Enabling Cloud services performance monitoring metricson the GUI topic under the Configuring → Configuring and tuningyour system for Cloud services section.

Administering Under File Placement Optimizer

v Added Data locality based copy in IBM Spectrum Scale:Administration Guide section.

v Added mmgetlocation in IBM Spectrum Scale: Administration Guidev Data locality restore section is renamed to Data locality in IBM

Spectrum Scale: Administration Guide.

v Added the maintenance steps for IBM Spectrum Scale FPO.v Added the performance tuning steps for IBM Spectrum Scale

Sharing Nothing Cluster.

xxxii IBM Spectrum Scale 4.2.3: Administration Guide

||

||||

||||

|||

||

||

|

||

||

||||

||||

||||

|

||||

||

||||

|||

||

||

|

||

|

||

Table 3. List of changes in documentation (continued)

GuideKnowledge centersection List of changes

ProblemDetermination Guide

Monitoring Under Monitoring AFM and AFM DR, renamed the following topics:

v Fileset states for AFM to Monitoring fileset states for AFMv Fileset states for AFM DR to Monitoring fileset states for AFM DRv Callback events for AFM and AFM DR to Monitoring callback events

for AFM and AFM DR

v Prefetch to Monitoring prefetchv Moved the Monitoring callback events for AFM and AFM DR topic

under Monitoring health and events

v Moved the Monitoring with mmpmon topic under Monitoringperformance

Restructured the following topics:

v Monitoring system health by using the mmhealth commandThis topic has been split into four topics:

– Monitoring the health of a node

– Event type and monitoring status for system health

– Threshold monitoring for system health

– Use cases

v Monitoring the IBM Spectrum Scale system by using call homeThis topic has been split into four topics:

– Understanding call home

– Configuring call home to enable manual and automated dataupload

– Monitoring, uploading, and sharing collected data with IBMSupport

– Use cases

v List of performance metricsThis topic has been split into three topics:

– Linux metrics

– GPFS metrics

– Protocol metrics

Troubleshooting v Under SMB issues, added a new topic Slow access to SMB causedby contended access to files or directories.

Command andProgrammingReference

Command reference v Removed the mmrest command man page.

Programmingreference

v Added documentation for each IBM Spectrum Scale managementAPI Version 2 command.

Changes in the Library and related publications section

v Under Library and related publications, the following topics were updated:– Redbooks, Redpapers™, and Blueprints: Six new links added.– ISV links– Applying IBM Spectrum Scale → Using AFM with object

v Under AFM-based Disaster Recovery section, the Failback of multiple filesets use case wasadded.

Summary of changes xxxiii

|

||||

||||

|

|

||

|

||

||

|

|

|

|

|

|

|

|

|

|

||

||

|

|

|

|

|

|

|||

|||

||

|||||

|

|

|

|

|

||

xxxiv IBM Spectrum Scale 4.2.3: Administration Guide

Chapter 1. Configuring the GPFS cluster

There are several tasks involved in managing your GPFS cluster. This topic points you to the informationyou need to get started.

GPFS cluster management tasks include the following.v “Creating your GPFS cluster”v “Displaying cluster configuration information”v “Specifying nodes as input to GPFS commands” on page 89v “Adding nodes to a GPFS cluster” on page 2v “Deleting nodes from a GPFS cluster” on page 3v “Changing the GPFS cluster configuration data” on page 4v “Node quorum considerations” on page 19v “Node quorum with tiebreaker considerations” on page 20v “Displaying and changing the file system manager node” on page 20v “Determining how long mmrestripefs takes to complete” on page 21v “Starting and stopping GPFS” on page 22

Note: In IBM Spectrum Scale V4.1.1 and later, many of these tasks can also be handled by the installationtoolkit configuration options. For more information on the installation toolkit, see Using the spectrumscaleinstallation toolkit to perform installation tasks: Explanations and examples topic in the IBM Spectrum Scale:Concepts, Planning, and Installation Guide.

For information on RAID administration, see IBM Spectrum Scale RAID: Administration.

Creating your GPFS clusterYou must first create a GPFS cluster by issuing the mmcrcluster command.

For more information, see mmcrcluster command in IBM Spectrum Scale: Command and ProgrammingReference.

For details on how GPFS clusters are created and used, see GPFS cluster creation considerations topic inIBM Spectrum Scale: Concepts, Planning, and Installation Guide.

Displaying cluster configuration informationUse the mmlscluster command to display cluster configuration information.

“Basic configuration information”“Information about protocol nodes” on page 2

For more usage information, see mmlscluster command in IBM Spectrum Scale: Command and ProgrammingReference.

Basic configuration informationTo display basic cluster configuration information, enter the mmcluster command with no parameters:mmlscluster

The command displays information like the following example:

© Copyright IBM Corp. 2014, 2019 1

|

GPFS cluster information========================

GPFS cluster name: cluster1.kgn.ibm.comGPFS cluster id: 680681562214606028GPFS UID domain: cluster1.kgn.ibm.comRemote shell command: /usr/bin/sshRemote file copy command: /usr/bin/scpRepository type: CCR

Node Daemon node name IP address Admin node name Designation--------------------------------------------------------------------------------------

1 k164n04.kgn.ibm.com 198.117.68.68 k164n04.kgn.ibm.com quorum2 k164n05.kgn.ibm.com 198.117.68.69 k164n05.kgn.ibm.com quorum3 k164n06.kgn.ibm.com 198.117.68.70 k164sn06.kgn.ibm.com quorum-manager

If the cluster uses a server-based repository, the command also displays the following information:v The primary GPFS cluster configuration serverv The secondary GPFS cluster configuration server

Information about protocol nodesTo display information about the protocol nodes, enter the following command:mmlscluster --ces

The command displays information like the following example:GPFS cluster information========================GPFS cluster name: cluster1.kgn.ibm.comGPFS cluster id: 4708497829760395040

Cluster Export Services global parameters-----------------------------------------Shared root directory: /gpfs/ces/cesEnabled Services: OBJ SMB NFSLog level: 0Address distribution policy: even-coverage

Node Daemon node name IP address CES IP address list-----------------------------------------------------------------------4 k16n07.kgn.ibm.com 192.168.4.4 10.18.64.235 k16n08.kgn.ibm.com 192.168.4.5 10.18.64.246 k16n09.kgn.ibm.com 192.168.4.6 10.18.64.267 k16n10.kgn.ibm.com 192.168.4.11 Node suspended, Node starting up8 k16n11.kgn.ibm.com 192.168.4.12 Node suspended, Node starting up

Adding nodes to a GPFS clusterYou can add nodes to an existing GPFS cluster by issuing the mmaddnode command. The new nodes areavailable immediately after the successful completion of this command.

You must follow these rules when adding nodes to a GPFS cluster:v You may issue the command only from a node that already belongs to the GPFS cluster.v A node may belong to only one GPFS cluster at a time.v The nodes must be available for the command to be successful. If any of the nodes listed are not

available when the command is issued, a message listing those nodes is displayed. You must correctthe problem on each node and reissue the command to add those nodes.

v After the nodes are added to the cluster, you must use the mmchlicense command to designateappropriate GPFS licenses to the new nodes.

To add node k164n06 to the GPFS cluster, enter:

2 IBM Spectrum Scale 4.2.3: Administration Guide

mmaddnode -N k164n06

The system displays information similar to:Mon Aug 9 21:53:30 EDT 2004: 6027-1664 mmaddnode: Processing node k164n06.kgn.ibm.commmaddnode: Command successfully completedmmaddnode: 6027-1371 Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

To confirm the addition of the nodes, enter:mmlscluster

The system displays information similar to:GPFS cluster information========================GPFS cluster name: cluster1.kgn.ibm.comGPFS cluster id: 680681562214606028GPFS UID domain: cluster1.kgn.ibm.comRemote shell command: /usr/bin/sshRemote file copy command: /usr/bin/scpRepository type: server-based

GPFS cluster configuration servers:-----------------------------------Primary server: k164sn06.kgn.ibm.comSecondary server: k164n05.kgn.ibm.com

Node Daemon node name IP address Admin node name Designation-------------------------------------------------------------------------------------1 k164n04.kgn.ibm.com 198.117.68.68 k164n04.kgn.ibm.com quorum2 k164n05.kgn.ibm.com 198.117.68.69 k164n05.kgn.ibm.com quorum3 k164n06.kgn.ibm.com 198.117.68.70 k164sn06.kgn.ibm.com quorum-manager

You can also use the installation toolkit to add nodes. For more information, see Adding nodes, NSDs, orfile systems to an installation process in IBM Spectrum Scale: Concepts, Planning, and Installation Guide.

For complete usage information, see mmaddnode command, mmlscluster command and mmchlicense commandin IBM Spectrum Scale: Command and Programming Reference.

Deleting nodes from a GPFS clusterYou can delete nodes from a GPFS cluster by issuing the mmdelnode command.

You must follow these rules when deleting nodes:v A node being deleted cannot be the primary or secondary GPFS cluster configuration server unless you

intend to delete the entire cluster. Verify this by issuing the mmlscluster command. If a node to bedeleted is one of the servers and you intend to keep the cluster, issue the mmchcluster command toassign another node as a configuration server before deleting the node.

v A node that is being deleted cannot be designated as an NSD server for any disk in the GPFS cluster,unless you intend to delete the entire cluster. Verify this by issuing the mmlsnsd command. If a nodethat is to be deleted is an NSD server for one or more disks, move the disks to nodes that will remainin the c

IBM Spectrum Scale 4.2.3: Administration Guide · IBM Spectrum Scale V ersion 4 Release 2.3...

Documents

Transcript of IBM Spectrum Scale 4.2.3: Administration Guide · IBM Spectrum Scale V ersion 4 Release 2.3...