Red Hat Storage Console 2.0 Administration Guide · Red Hat Storage Console is a web-based storage...

Rakesh Ghatvisave Red Hat

Red Hat Storage Console2.0Administration Guide

Administering Red Hat Storage Console

Red Hat Storage Console 2.0 Administration Guide

Administering Red Hat Storage Console

Rakesh GhatvisaveRed Hat Customer Content [email protected]

Legal Notice

Copyright © 2017 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinitylogo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and othercountries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related toor endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and other countriesand are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed orsponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

AbstractThis guides provides comprehensive information to manage and administer Red Hat Ceph Storage2.0 cluster environment using the Red Hat Storage Console 2.0.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. OVERVIEW

CHAPTER 2. ABOUT THIS GUIDE

CHAPTER 3. CONSOLE NAVIGATION3.1. DASHBOARD VIEW3.2. CLUSTERS TAB3.3. HOSTS TAB3.4. STORAGE TAB

CHAPTER 4. CLUSTER ADMINISTRATION4.1. CLUSTER DETAILS4.2. CLUSTER EXPANSION4.3. CLUSTER DETACHMENT

CHAPTER 5. HOST ADMINISTRATION5.1. HOST METRICS

CHAPTER 6. STORAGE ADMINISTRATION6.1. MANAGING OBJECT STORAGE POOLS6.2. MANAGING RBDS

CHAPTER 7. USER ADMINISTRATION7.1. ADMINISTERING USERS7.2. USER AUTHENTICATION AND MAIL SERVER CONFIGURATION

CHAPTER 8. TASK MANAGEMENT8.1. TASK LIST8.2. TASK FILTERING

CHAPTER 9. EVENT MANAGEMENT9.1. EVENTS VIEW9.2. EVENTS FILTERING9.3. DISMISSING EVENTS

CHAPTER 10. NOTIFICATIONS AND SHORTCUTS MENUBAR10.1. EVENTS PREVIEW AND SHORTCUT10.2. DISCOVERED HOSTS SHORTCUT10.3. TASKS PREVIEW AND SHORTCUT10.4. CONSOLE VERSION10.5. USER SIGNOUT

CHAPTER 11. CONSOLE JOURNAL MAPPING11.1. CONSOLE JOURNAL MAPPING MECHANISM

CHAPTER 12. TROUBLESHOOTING12.1. TROUBLESHOOTING NODES CONFIGURATION12.2. TROUBLESHOOTING MONITOR NODES CONFIGURATION12.3. TROUBLESHOOTING OSD NODES CONFIGURATION12.4. TROUBLESHOOTING IMPORT CLUSTER NODES12.5. TROUBLESHOOTING POST CLUSTER CREATION FAILURES12.6. TROUBLESHOOTING MISCELLANEOUS

3

4

55

111213

15151922

2525

323234

383843

494949

52525254

575758596061

6363

64646667686969

Table of Contents

1


2

CHAPTER 1. OVERVIEW

Red Hat Storage Console is a web-based storage management platform specially designed toefficiently manage the software-defined storage technologies offered by Red Hat. The Consolesimplifies the installation, configuration, monitoring, and management of Red Hat’s software-basedstorage systems. Red Hat Storage Console 2.0 provides seamless management of the Red HatCeph Storage ecosystem by providing functionalities such as creating new Ceph storage clusters,monitoring the cluster health and managing the storage capacity through an intuitive and simplifiedinterface.

CHAPTER 1. OVERVIEW

3

CHAPTER 2. ABOUT THIS GUIDE

The Red Hat Storage Console 2.0 Administration Guide is intended for all Storage administratorsand users who plan to manage their Red Hat Ceph Storage cluster environment using Red HatStorage Console in a simplified and GUI-based approach.


4

CHAPTER 3. CONSOLE NAVIGATION

Red Hat Storage Console provides a vertical navigation bar available at the left hand side of theconsole to conveniently access the core administrative actions.

Figure 3.1. Console Navigation Bar

The Console navigation bar provides access to the following administrative tabs:

1. Dashboard: default landing interface

2. Clusters: clusters view and administration

3. Hosts : hosts view and administration

4. Storage : storage view and administration

5. Admin: events, tasks, user and mail server administration

3.1. DASHBOARD VIEW

The Dashboard view is the default landing interface after logging into the Console. The ConsoleDashboard provides a high-level summary of the defined clusters and its associated components.

Figure 3.2. Dashboard View


5

The Dashboard view provides summary for the following components in a visual data cards format.

3.1.1. Aggregated Cluster Status

The cluster status card displays the number of cluster(s) defined in the Console and the aggregatedstatus. The cluster status is denoted in the form of on-screen status icons.

3.1.2. Disk Utilization Status

The aggregated storage utilization percentage is depicted through a donut chart in color codes. Thedisk utilization of the last 24 hours is depicted in the form of a sparkline chart, progressively showingthe variation and the utilization over a period of time.

Figure 3.3. Aggregate Storage Utilization


6

The threshold breach conditions are reflected by the following color indicators:

Table 3.1. Threshold Status Indicators

Threshold Status Color Code

No Threshold Blue

When a threshold exists and the percentage hasnot surpassed any thresholds

Green

When the utilization percentage has surpassed thewarning threshold, but not the critical threshold

Orange

When the utilization percentage has surpassed thecritical threshold

Red

3.1.3. Storage Profiles

The storage profiles card depicts the utilization of the most used storage profiles.

Figure 3.4. Most Used Storage Profiles


7

For thresholds breach condition indicators, see Table 3.1 Threshold Status Indicators.

3.1.4. Cluster Elements Status

The associated elements of a cluster is depicted in an aggregated status card format.

Figure 3.5. Cluster Elements

The aggregated cluster elements displayed in the card along with their current status are:

The number of hosts participating in the clusters

The number of Monitor hosts

The number of Placement Groups

The number of storage pools

The number of OSDs and Object block devices

3.1.5. Most Used Pools

The Most Used Pools card lists the most used pools in the cluster with their utilization values.


8

Figure 3.6. Most Used Pools

For thresholds breach condition indicators, see Table 3.1 Threshold Status Indicators.

For example, in the following screen, the storage pools are displaying full capacity utilization,depicted in red. It means that the utilization percentage has surpassed the critical threshold.


9

3.1.6. System and Memory Performance

The System and Disk Performance card displays the cross-cluster system and disk performance.

Figure 3.7. System and Network Performance

The aggregated CPU and Memory utilization is displayed in the form of a donut chart. To view theavailable CPU and memory percentage, hover the cursor anywhere on the grey area of the donutchart. A tooltip will appear showing you the available CPU and memory in percentage unit.

The utilization is also depicted in the form of a sparkline chart, progressively showing the variationand the utilization over a period of time. The sparkline charts are placed before the donut charts.

The aggregated disk and the network performance values are displayed with the correspondingsparkline charts. Storage administrators can use the data available here as a baseline to monitor theperformance of the disks in the cluster. The system calculates the following disk and networkperformance metrics:

IOPS (Input/Output Operations Per Second): shows the average read/writes requested on thedisk with a sparkline chart

Network Throughput: shows the throughput in kilobytes per second (KB/s) with a sparklinechart

Network Latency: shows the network latency in milliseconds (ms) with a sparkline chart

Figure 3.8. Disk and Network Performance


10

3.2. CLUSTERS TAB

The Clusters tab in the navigation bar lists all the defined clusters in the Console in a rows format.Each row shows the individual cluster attributes. At a glance, for a given cluster, the viewable visualitems are:

The cluster status

The disk utilization summary

The number of hosts

The number of pools part of the cluster

The number of alerts triggered

Figure 3.9. Cluster Attributes

The clickable attributes in the cluster row are the cluster name, the number of hosts and pools, andthe alerts. Clicking on any of these attributes will redirect you to the specific administration page.


11

The various administrative operations such as create and import cluster are invoked from theClusters tab.

Note

For detailed instructions to create a new cluster, see the Initial Cluster Creation section ofthe Red Hat Storage Console 2.0 Quick Start Guide.

3.3. HOSTS TAB

The Hosts tab in the navigation bar lists all the accepted hosts assigned to different clusters.Unaccepted hosts which are not part of any cluster appear in the events notifications bar. The Hostscan be filtered by the host name and the host role. To filter host names, enter the name of the hostin the Search field at the right hand side of the hosts view interface.

Figure 3.10. Hosts View

The Hosts view displays the following labels at a glance for individual hosts:

The host name

The storage utilization and system performance.

The name of the assigned cluster

The role of the host whether Monitor or OSD.

The number of alerts

Note

For detailed instructions on accepting hosts, see the Accepting Hosts section of the RedHat Storage Console 2.0 Quick Start Guide.


12

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#initial_cluster_creation

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#accepting_hosts

3.4. STORAGE TAB

The Storage tab in the navigation bar lists the created Object storage pools and allows theadministrator to create new Object storage pools and RADOS Block Devices (RBD).

The Object storage pool view is the default view after clicking on the Storage tab.

3.4.1. Object Storage Pools View

Users can view the list of all object storage pools created in the Storage pools view. Theconfiguration attributes and the metrics of each Object storage is displayed.

Figure 3.11. Storage Pools View

For a given Object storage, the following attributes are displayed:

The storage pool name along with the current status

The Storage pool utilization data in donut chart format

The number of configured replicas, OSDs and PGs

Quotas whether enabled or disabled.

Active alerts for the pool

The vertical ellipsis icon for administrative actions

3.4.2. RBD View

To view the RBDs, navigate to Storage > RBDs from the console navigation bar.

Figure 3.12. RBD View


13

Users can view a list of all RBDs created in the system in the RBDs view. The configurationsummary and the metrics of each RBD is displayed. For a given individual RBD, the followingsummary is displayed:

The RBD label and the cluster name

The RBD storage size

The Backing Pool name

The Alerts configuration status

The vertical ellipsis icon for administrative actions

Note

For detailed instructions on creating storage pools, see the Provisioning Storage sectionof the Red Hat Storage Console 2.0 Quick Start Guide.


14

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#provisioning_storage

CHAPTER 4. CLUSTER ADMINISTRATION

Red Hat Storage Console provides efficient administration of Ceph clusters. The variousadministrative tasks available are outlined below.

4.1. CLUSTER DETAILS

4.1.1. Cluster Overview

To get an overview of an individual cluster, follow these steps:

1. Click on the Clusters tab from the console navigation bar.

2. In the Clusters view, click on the name of the cluster.

3. The entire cluster information is displayed in a tabbed structure with Overview being thefirst tab.

4.1.2. Cluster Hosts

To view the hosts part of an individual cluster, follow these steps:

1. Click on the Clusters tab from the Console navigation bar.


3. Click Hosts from the tabs.


15

4.1.3. Cluster Pools

To view the object storage pools part of an individual cluster, follow these steps:



3. Click Pools from the tabs.

4.1.4. Cluster RBDs

To view the object storage pools part of an individual cluster, follow these steps:



3. Click RBDs from the tabs.


16

4.1.5. Cluster OSDs

To view the OSDs part of an individual cluster, follow these steps:

1. Click on the Clusters tab in the console navigation bar.


3. Click OSDs from the tabs.

4.1.6. Cluster Configuration Metadata

To view the Ceph cluster configuration details, follow these steps:

1. Click on the Clusters tab from the console navigation bar.


3. Click Configuration from the tabbed view.

In the Configurations tab, the following information can be viewed:

Ceph Cluster Configuration

The general cluster settings and default values.


17

Notifications

The Notifications box displays the general notifications for the different events in the Console.

Utilization Thresholds


18

The Utilization box displays the thresholds values to alert the user when there is a breach.

4.2. CLUSTER EXPANSION

An existing cluster in the Console can be expanded by provisioning additional nodes through theexpand cluster mechanism. The user can add additional Monitor or OSD nodes depending on theneed. The cluster will expand automatically after adding the nodes.

In order for the hosts to appear in the list of available hosts, Red Hat Storage Console agent must beinstalled, configured and activated in each participating Ceph Storage nodes. For instructions onconfiguring the Red Hat Storage Console agent, see the Installing and Configuring the Red HatStorage Console Agent section of the Red Hat Ceph Storage Installation Guide for Red HatEnterprise Linux available on the Red Hat Customer Portal.

The following procedure outlines the steps to expand an existing cluster:

Procedure. Expanding a Cluster

1. Log into Red Hat Storage Console using the default username and password as admin, atthe following URL:

http://<hostname>/skyring

2. Click on the Clusters tab in the navigation bar. The list of clusters are listed.


19

https://access.redhat.com/documentation/en/red-hat-ceph-storage/2/single/installation-guide-for-red-hat-enterprise-linux#installing_and_configuring_the_red_hat_storage_console_agent

3. Locate the cluster to be expanded. Click on the vertical ellipsis icon at the right hand side. Acallout menu opens up. Click Expand.

4. The unaccepted hosts dialogue appears. Click View and Accept.

Note

The steps 4 -7 are not applicable if the required hosts for cluster expansion arealready accepted and initialized.

5. The list of unaccepted hosts is displayed. To accept the hosts individually, click Accept onthe right of the corresponding hostnames. Alternatively, the Accept All button can be usedto accept all the displayed hosts at once.


20

6. The selected hosts initialize.

7. Once the hosts are initialized, a green check mark appears before the hostnames. ClickContinue.

8. Select the hosts from the list of initialized hosts by checking the check-box. The role of thehosts, whether Monitor or OSD can be selected from the drop-down menu. After selectingthe required hosts, click Expand Cluster.


21

9. The expand cluster request is submitted. Click Close.

Note

After closing the dialogue box, the Tasks page opens, displaying the progress of thecluster expansion. Alternatively, the detailed cluster expansion progress can be viewed inthe Tasks page.

Cluster Expansion Status

To view the cluster expansion status:

1. From the Console navigation bar click Admin > Tasks.

2. The latest submitted task is displayed at the top. Locate your submitted task and click on itto display the Task metadata and the completion status.

4.3. CLUSTER DETACHMENT

A defined cluster can be removed or detached from the Console. The detaching cluster from theConsole undergoes two stages:

Unmanage Cluster


22

The unmanage cluster action is triggered from the Cluster view page of the Console. By invoking theUnmanage action, the user intends to detach the specific cluster from the Console management.Once the cluster is unmanaged, its monitoring is disabled and no further actions can be performedon the cluster.

Forget Cluster

Forget cluster is the second stage to remove a cluster from the Console. Once the cluster isforgotten, it will not be viewable in the Clusters view of the Console. However, the cluster can beviewed in the Calamari API and the cluster data is archived in the Console repository.

The following procedure outlines the steps to detach a cluster from the Console:

Procedure: Detaching a cluster from the Console

1. Log into Red Hat Storage Console using the default username and password as admin, atthe following URL:

http://<hostname>/skyring

2. Click on the Clusters tab in the navigation bar. The list of clusters are listed.

3. Locate the cluster to be unmanaged and click on the vertical ellipsis icon at the right handside. A callout menu appears. Click Unmanage.

Note

The unmanage operation is triggered in the backend after clicking Unmanage.After a while, a notification pops out on the notifications bar at the top right handcorner confirming the cluster is successfully unmanaged. The cluster status iconturns from the current status to a spanner icon indicating that the cluster isunmanaged. To see the detailed cluster unmamage status, navigate to the Taskspage and locate the recent unmanage cluster task.

4. After the cluster is unmanaged, click on the vertical ellipsis icon at the right hand side.


23

5. Click Forget from the callout menu.

The cluster is successfully removed from the Console.

Cluster Detachment Status

To view the cluster unmanage and forget status:

1. Click Tasks from the Console navigation bar. The latest submitted task is displayed at thetop.

2. To view task details of Unmanage cluster, locate and click on the Unmanage Cluster task.The task box is displayed with the task metadata and the completion status.

3. To view the task details of the Forget cluster request, locate and click on the Forget Clustertask. The task box is displayed with the task metadata and the completion status.


24

CHAPTER 5. HOST ADMINISTRATION

5.1. HOST METRICS

The detailed individual host metrics can be viewed in the Console. To view individual host metrics,navigate to the Hosts view from the console navigation bar. In the hosts view, select the host bylocating it or by searching it using the search field provided. Once the Host is located, click on thehost name to view the detailed host metrics.

The Individual hosts view is divided into three tabs: Overview, Configuration and OSDs.

Note

For a Monitor host, only Overview and Configuration tabs are available.

5.1.1. Host Overview

The Overview tab is selected by default. It provides comprehensive detail of the selected host indata cards format. The viewable details are described below

Host Summary

The Host role - whether OSD or Monitor

The operation time - shows the time elapsed in days and hours since the start

The Cluster group - shows the cluster name on which the host is assigned.

Utilization

The host node CPU and Memory utilization is displayed in form of a donut chart. The CPU usageand Memory utilization is depicted in percentage unit. The utilization data is also depicted in theform of a sparkline chart, progressively showing the variation and the utilization over a period oftime.


25

In case there is no data to load, the chart will display a placeholder saying No data available likeshown in the below screen:

Viewing Host CPU Utilization

For example, in the below donut chart, the CPU utilization percentage is given within the donut chartand highlighted by a green curve on the donut chart. Alongside, the CPU usage percentage label isshown.


26

To view, the available processing power, hover the cursor anywhere on the grey area of the donutchart. A tooltip will appear showing you the available CPU in percentage unit.


27

Viewing Host Memory Utilization

For example, in the below donut chart, the Memory utilization percentage is highlighted in a greencolored curve on the donut chart.

To view the available memory, hover the cursor anywhere on the grey area of the donut chart. Atooltip will appear showing you the available memory in percentage format.


28

Viewing Network Utilization

The overall network utilization is depicted in the form of a donut chart, displaying the network usagein percentage unit. Additionally, the cluster network address and the Public network address isdisplayed with the current status.

At the top right hand side of the Network utilization card, a drop-down menu allows the user to viewthe network utilization in the elapsed hours. By default, the past 1 hour network utilization metrics isdisplayed.


29

Performance

The overall performance is displayed in the Performance data card. A storage administrator canuse the data available here as a baseline to monitor the performance of the disks in the host. Thesystem calculates the following disk performance metrics:

IOPS (Input/Output Operations Per Second) - shows the average read/writes requested onthe disk

Network Throughput - shows the throughput in bytes per second (B/s)

Network Latency - shows the network latency in milliseconds (ms)

Note

The network throughput value shows the summary of read/write average in bytescompounded from all the interfaces.

At the top right hand side of the Network Performance card, a drop-down menu allows the user toview the network performance metrics in the elapsed hours. By default, the past 1 hour networkutilization metrics is displayed.

5.1.2. Host Configuration


30

5.1.2. Host Configuration

The configuration tab displays the essential network, hardware and the OS information in a tabularformat.

5.1.3. Host OSDs

The OSDs tab lists all the OSDs in the selected host. The OSDs can be filtered by entering the OSDname. Additionally, a sidebar is provided at the left side to filter OSDs by OSD Status, PG Statusand by Utilization.

To view the OSDs belonging to a particular storage profile, a Storage Profile drop-down is provided.Select the storage profile to list the OSDs part of the selected storage profile.

To view the OSD summary, select the OSD from the list and the OSD summary is displayed at theright hand side of the interface.


31

CHAPTER 6. STORAGE ADMINISTRATION

Red Hat Storage Console allows storage administrators to efficiently manage the object storagepools and block storage devices that are part of the nodes participating in the clusters. The Storageview page provides configuration summary for each pool in the cluster and supports creation of newstorage pools.

Note

For detailed instructions on creating storage pools, see the Provisioning Storage sectionof the Red Hat Storage Console 2.0 Quick Start Guide.

6.1. MANAGING OBJECT STORAGE POOLS

The console allows administrators to perform storage specific operations such as editing anddeleting storage pools.

6.1.1. Editing Object Storage Pools

The following procedure outlines the steps to edit an Object Storage pool:

Procedure. Editing an Object Storage Pool

1. Navigate to Storage Pools view by clicking Storage > Pools from the console navigation.

2. Locate the storage pool to be edited from the storage pools list. Click on the vertical ellipsisicon at the right hand side of the storage pool.

3. A callout menu opens up. Click Edit.

4. The individual pool view expands below exposing the editable pool attributes. The editablepool attributes exposes actions such as renaming a pool, setting the number of objectreplicas and tweaking the quota settings.


32

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#provisioning_storage

5. Enter the required values in the attribute fields and click the check mark icon at the righthand side of the expanded panel to save the new settings.

6. A pop-up notification appears over the notification bar at the top right hand side corner of theinterface confirming the saved changes.

6.1.2. Removing Object Storage Pools

The Red Hat Storage Console allows administrators to remove storage pools from the cluster.Object storage pools can be disassociated from the cluster and then removed from the console.

The following procedure outlines the steps to remove an object storage pool:

Procedure. Removing an Object Storage Pool

1. Navigate to Storage Pools view by clicking Storage > Pools in the console navigation.

2. Locate the storage pool from the storage pools list. Click on the vertical ellipsis icon at theright hand side of the storage pool.

3. A callout menu opens up. Click Remove.


33

4. A confirmation dialogue box appears. Click Ok to remove the selected storage pool.

Object Storage Removal Status

To view the object storage removal status:

1. Navigate to the Admin tab at the left hand side of the Red Hat Storage Console interface andclick Tasks.

2. The latest submitted task is displayed at the top. Locate your submitted task and click on it.

3. The completion status is displayed.

6.2. MANAGING RBDS


34

The configured RBDs that are part of clusters in the console can be reconfigured and re provisionedto the clusters. Additionally, RBDs can be dissociated and removed from a pool.

6.2.1. Resizing RBD

The following procedure outlines the steps to resize a RBD:

Procedure. Resizing a RBD

1. Navigate to RBD view by clicking Storage > RBDs from the console navigation.

2. The RBD view is displayed. Locate the RBD to be resized from the list.

3. Click on the vertical ellipsis icon at the right hand side of the RBD.

4. A callout menu opens up. Click Resize.

5. The individual RBD view expands below. The resizable attribute available is Target Size.

6. Enter the required storage size and select the storage capacity unit from the drop-downmenu. Click Save.


35

A pop-up notification appears over the events sidebar at the top right hand side corner of the consoleconfirming the saved changes.

6.2.2. Removing RBD

The following procedure outlines the steps to remove a RBD:

Procedure. Removing a RBD

1. Navigate to RBD view by clicking Storage > RBDs from the console navigation bar.

2. The RBD view is displayed. Locate the RBD to be removed from the list.

3. Click on the vertical ellipsis icon at the right hand side of the RBD.

4. A callout menu opens up. Click Remove.

5. A confirmation dialogue box appears. Click Ok to remove the RBD.


36

RBD Removal Status

To view the RBD removal status:

1. Navigate to the Admin tab at the left hand side of the Red Hat Storage Console interface andclick Tasks.

2. The latest submitted task is displayed at the top. Locate your submitted task and click on it.

3. The completion status is displayed.


37

CHAPTER 7. USER ADMINISTRATION

Red Hat Storage Console provides efficient and user-friendly way of managing console users. Theconsole provides user administrative tasks such as adding, editing and deleting a console user. Theconsole also allows user management and authentication through directory services such asLightweight Directory Access Protocol (LDAP) and Active Directory (AD).

7.1. ADMINISTERING USERS

7.1.1. Adding User

To add a new user, in the Console navigation bar, click Admin > Users. The list of created users isdisplayed in the user view. If there are no prior users created, only the default admin user is listed.

The following procedure outlines the steps to add a new user:

Procedure: Adding a New User

1. On the right hand side, click Add User button highlighted in blue.

2. Enter the user credentials: User ID, the First Name and Last Name, the Email address andthe Password. Click Save.


38

Note

To enable events and alerts notifications via email, check the check-box for EmailNotifications

3. The new user is added successfully.

7.1.2. Editing User

To edit an existing user:

1. Navigate to the user view by clicking Admin > Users from the console navigation.

2. Locate the user to be edited and click Actions at the right hand side.

3. From the callout menu, click Edit user.

4. The user editing interface is opened. The editable attributes are the email address and thepassword.


39

5. Click Save after the editing is complete.

7.1.3. Enable and Disable User

The console user operations can be enabled and disabled in the User view of the console.

To disable a created user, perform these steps:

1. In the User view, locate the user to be disabled and click Actions at the right hand side ofthe user row.

2. From the callout menu, click Disable user.

3. The status icon for the user turns X indicating that the user is disabled.


40

4. To enable the user for console operations, from the callout menu, click Enable user.

5. The status icon for the user turns from X to a green check mark indicating that the user isenabled

7.1.4. Enable and Disable Notifications

Email notifications for console alerts can be enabled and disabled for a console user.

Note

For mail server configuration instructions, see the Mail Server Configuration section of thisguide.

To enable notifications perform the following steps:

1. In the User view, locate the user and click Actions at the right hand side.

2. From the callout menu, click Enable Notification.


41

3. The icon in the notification attribute for the user changes from X to a green check mark.

4. To disable the notifications, from the callout menu, click Disable Notification.

5. The icon in the notification column attribute for the user changes from green check mark toX.

7.1.5. Deleting User

The users created for console operations can be deleted from the console.

To delete a user, in the User view, locate the user to be deleted and click Actions > Delete user.


42

7.2. USER AUTHENTICATION AND MAIL SERVER CONFIGURATION

7.2.1. LDAP and AD Configuration

Before configuring the LDAP/AD, perform the following steps:

1. Stop the skyring service by executing the following command:

# systemctl stop skyring

2. Change the variable providerName: localauthprovider to providerName:ldapauthprovider in the following file:

/etc/skyring/skyring.conf

3. Start the skyring service by executing the following command:

# systemctl start skyring

After the above steps are performed, the LDAP/AD settings can be configured in the console.

The following example procedure outlines the steps to configure the LDAP/AD in the console:

Procedure. Configuring LDAP Example

1. Navigate to the LDAP/AD settings page by clicking Admin > LDAP/AD Settings.

2. The LDAP configuration input fields are displayed. Select the authentication method,whether AD or LDAP. For this instance, LDAP is selected.


43

3. Enter the following information in the input fields:

LDAP/AD Host: The LDAP server address

Port: The port number

Base Path Name: The base path to search accounts

Bind Account: The LDAP Admin account

Bind Password: The LDAP Admin account password

4. After entering the details, click Save.


44

5. The configuration is saved successfully.

Note

Perform the same set of instructions for configuring AD by selecting the AD radio button.

7.2.2. Adding LDAP/AD User

After LDAP/AD is successfully configured, a new LDAP/AD user can be added.

The following procedure outlines the steps to add a new LDAP user:

Procedure. Adding LDAP User

1. Navigate to the User view by clicking Admin > Users from the console navigation bar.

2. The Console users list is displayed. Click Add LDAP/AD User at the right hand side of theUser view.

3. Enter the User ID in the Name or User ID input field and click Search.


45

Note

In Red Hat Storage Console 2.0, users can search for LDAP accounts only byusing the User ID search parameter.

4. The system will search for existing users from the configured LDAP directory. Locate theuser and click Import.

5. The User is successfully added to the console MongoDB database.

7.2.3. Mail Server Configuration

Configure a mail server to send email notifications to registered console users. The mail server willsend email notifications to users about state changes and any threshold breaches. To receivenotifications, register a valid email address.

Note

An SMTP server without SSL/TLS encryption can be configured only through ConsoleAPI.

The following example procedure outlines the steps to configure mailing server in the console:

Procedure. Configuring Mail Server

1. Navigate to the mail settings page by clicking Admin > Mail Settings.

2. The LDAP configuration input fields are displayed. Enable email notifications by checkingthe Email Notifications check box. Upon checking the check box, it changes to Enabled.


46

3. Enter the following settings in the input fields:

Outgoing Mail Server SMTP Address: The SMTP server address

Port: The port number of the SMTP server depending on the encryption type selected

Note

The port number depends on the type of encryption selected. For SSLencryption, the port number is 465 and for TLS encryption, the port number is587

User Name: The valid email address to be registered with the mail server

Password: The valid password of the email account to be registered

From: The valid email address to be registered with the mail server

Subject Prefix: The incoming email subject prefix.

4. Validate the settings by testing it using the test process. Enter a test email address in theTest Message Recipient input field. In order to determine that the settings entered arecorrect, click TEST.


47

5. If the entered settings are correct and the test is successful, a sample test email is receivedby the test email address from the registered email address. Click Save to save the emailnotification configuration.


48

CHAPTER 8. TASK MANAGEMENT

Red Hat Storage Console consists of sizeable number of user initiated actions to accomplishoperations such as creating clusters, creation of new storage pools, and importing cluster to name afew. It is crucial for the administrators or the users to monitor and view the status of the actions theyinitiated in a visual data format. The benefit of providing an advance monitoring mechanism of theinitiated actions is to remedy any failed operation by analysing the execution steps and actuatetroubleshooting.

Red Hat Storage Console offers an advanced and intuitive task monitoring mechanism for eachsignificant action initiated in the console by the user. A storage administrator can view and performthe following actions in the console:

The status of an initiated task whether successful or failed

The details of all past and present cluster-wide initiated actions

The timestamp of the initiated task with the unique task ID

Retrieve a specific task by adding available filters

8.1. TASK LIST

To view the tasks list in the console:

1. Log into Red Hat Storage Console using the user login credentials.

2. From the navigate bar, navigate to Admin > Tasks.

3. The console wide initiated tasks are displayed in a page view with the most recent task atthe top.

8.2. TASK FILTERING

Task filtering makes it easy to retrieve a specific initiated task. A specific task can be retrieved byapplying the available filters: The task name, the period between which the task was initiated andthe status check boxes.

Task Filtering Example


49

For example, an administrator wants to retrieve all the successful tasks initiated for creating astorage block device in the console.

The following procedure will help the administrator to retrieve the required tasks:

1. Navigate to the Tasks view by clicking Admin > Tasks from the console navigation. TheTasks view is displayed.

2. At the top of the page, the filtering widgets are available. Enter the keyword block device inthe first search field.

Note

As the individual filters are applied, they are reflected in the Active Filters field inthe form of labels.

3. Set the search period from the calendar widget. For this example, the start date is set as FriJuly 01 2016 00:00 Hours and the end date is set as Sun July 10 2016 23:55 Hours.

4. Select the task status as Completed from the check box options. All the applied filter labelsappear in the Active filters field with the number of retrieved results.


50

5. Once all the necessary filters are applied, the system retrieves all the tasks related to blockdevices and displays the list.

Note

At any point of time, to remove an applied filter, click x on the filer labels. In order to clearall the applied filters, click Clear All Filters.

Note

A task cannot be paused, resumed or deleted from the console. All created tasks remainsin the console history for an indefinite period.

Task Details

1. The individual task detail for an initiated operation can be viewed by clicking on the Taskname in the Tasks view.

The task detail page displays the following information:

Task Name: The name of the initiated operation

Task ID: The unique task ID generated by the system

Time Submitted: The time and date when the task was initiated

Status: Progressive execution steps with status whether failed or success.


51

CHAPTER 9. EVENT MANAGEMENT

The Events management system in Red Hat Storage Console provides comprehensive informationof all the events raised by the console.

An Event is a record of a voluntary action carried out by a console user or any action triggered bythe system. Events can be informative in nature such as user login records, creating and deleting aconsole user, configuring LDAP user and other similar administrative actions. Non-informativeevents are events related to critical system warnings and console health status to name a few.

An Alert is a visual notification to draw the attention of a user to a single event or multiple eventstriggered by the console.

The Events management system provides the following notifications and visual data to the user:

Notifies the administrator of any critical event occurred in the console

Provides summary of all severe and critical events occurred in the console

Provides detailed information and an event

Provides a comprehensive list of all active and dismissed events.

Provides advanced event search filters based on filter inputs such as hostname and clustername, time-period range and severity level.

9.1. EVENTS VIEW

To view the Events generated in the console, follow these steps:

1. Log into Red Hat Storage Console using the user credentials.

2. From the navigate pane, navigate to Admin > Events.

3. The console wide triggered alerts are displayed in a page view with the most recent event atthe top.

9.2. EVENTS FILTERING

Events filtering makes it convenient to retrieve a specific event occurred in the console. A specificevent can be retrieved by applying the available filters: The Object, the time period between whichthe task was initiated and the status check boxes - Active, Dismissed and All.


52

Active events are events which are raised in the console but are not yet acknowledged by the user.Whereas, Dismissed events are events which are acknowledged by the user and is dismissedmanually.

Note

Non-informative Events are dismissed manually by voluntary user action and alsoautomatically by the system if it detects a corresponding event generated.

Event Filtering Example

For example, an administrator wants to retrieve all the events generated by the system related touser settings. The following filtering procedure will help the administrator to retrieve the neededevents:

1. Navigate to the Events view by clicking Admin > Events from the console navigation bar.The Events view is displayed.

2. At the top of the page, the events filtering options are given. In the first filtering option, selectthe search criteria as Description from the drop-down and enter user setting in the searchinput field.

3. Set the search time period from the calendar widget. For this example, the From date is setas Fri July 01 2016 00:00 Hours and the To date is set as Tue July 12 2016 23:55 Hours.

4. Select the type of events from the available options: Active, Dismissed and All. By default,the Active type is selected.

5. All the applied filter labels appear in the Active filters field with the number of retrievedevents.

6. The system retrieves all the events matching the applied filters and displays the list.


53

Note

At any point of time, to remove an applied filter, click x on the filer labels. In orderto clear all the applied filters, click Clear All Filters.

9.3. DISMISSING EVENTS

For example, to dismiss a warning event, follow these steps:

1. In the Events view, click on the warning icon in the Active filters row.

2. The Active filter is applied as severity: warning and the events matching the severity levelsare retrieved.

3. Click Details at the right hand side to take you to the event detail page.


54

4. Enter an appropriate comment in the comments field and click Dismiss.

5. The event is successfully dismissed and can be retrieved by checking the Dismissed statuscheckbox.

Alternatively, the events icon in the Notifications Menu Bar can also be used to direct the user to theevent dismissal page.


55

After the events are dismissed, they will no longer appear in the Notifications Menu Bar.


56

CHAPTER 10. NOTIFICATIONS AND SHORTCUTSMENUBAR

The Notifications and Shortcuts menubar is a utility embedded in the Red Hat Storage Console toissue visual alerts to notify the user of any critical events occurring in the console.

Figure 10.1. Notifications and Shortcuts Menubar

Additionally, Notifications and Shortcuts menubar also provides previews and shortcuts to thefollowing console related operations, starting from left to right:

Events Preview

Discovered Hosts Preview

Tasks Preview

About Version

User Signout

10.1. EVENTS PREVIEW AND SHORTCUT

The Events preview and shortcut option displays a summary of all new and unread alerts. To viewthe alerts summary, click on the events icon on the menubar. A drop-down menu is openeddisplaying all the non-informative events.

Figure 10.2. Events Preview

CHAPTER 10. NOTIFICATIONS AND SHORTCUTS MENUBAR

57

Click on a specific event and you will be redirected to the Events page where you can view anddismiss the event.

10.2. DISCOVERED HOSTS SHORTCUT

The discovered hosts shortcut option displays a numerical alert of the number of discovered hostsby the console. To redirect to the View and Accept Hosts page, click on the Discovered Hostsicon.

Figure 10.3. Discovered Hosts Shortcut


58

10.3. TASKS PREVIEW AND SHORTCUT

To view the tasks that are in progress, click on the tasks icon from the menubar. To view the Taskview, click on See All and you will be redirected to the Tasks view of the console.

Figure 10.4. Tasks Preview and Shortcut


59

10.4. CONSOLE VERSION

The About icon on the sidebar displays the version information of the Red Hat Storage Console, thestorage provider, the Graphite-web and the MongoDB database. To view the version information,click on the About icon and a modal About box is opened.

Figure 10.5. Console Version


60

10.5. USER SIGNOUT

The user can sign out and change the user password using this option.

To signout from the console:

1. Click on the user icon from the menu bar.

2. Click the green signout button.

Figure 10.6. User Signout

To change the user password:

1. Click on the user icon from the menu bar.

2. Click change password.

Figure 10.7. Change Password


61

3. A dialog window is opened. Enter the new password twice and click Save.


62

CHAPTER 11. CONSOLE JOURNAL MAPPING

In Red Hat Ceph Storage, when you create a OSD, you also need to create its Journal, which iswhere data is initially written before it is flushed to an OSD. Each OSD needs a Journal which is on adifferent disk than the OSD.

The OSD to journal mapping in Red Hat Storage Console happens automatically based on variousaspects such as the type of the disk whether SSD or rotational and the available size on the disks.

This section describes the logic involved within Red Hat Storage Console for mapping journal disks.This will help storage administrators to map their knowledge with the output from Red Hat StorageConsole during creating OSDs for clusters.

11.1. CONSOLE JOURNAL MAPPING MECHANISM

The Journal mapping mechanism is based on a simple algorithm that maps disks with lower storagecapacity as journals for disks with higher storage capacity. The selection of journals is not random.To achieve this, the algorithm first sorts the list of disks based on size in descending order and thenstarts the mapping of disks with their journal. The logic used for journal mapping is listed below:

Disks with higher storage capacity is preferred to be used as a data disk. Whereas, disks withlower storage capacity is used as Journal disks.

Mapping of data disks to their journal disk happens at per node basis. So a disk cannot have itsjournal disk on a separate node.

In the case wherein all the disks are rotational in nature, a disk can serve as a journal for onlyone disk. In the case wherein all the disks are SSDs, one SSD can be used as a journal for amaximum of 4 disks.

In the case wherein there are rotational and SSD disks both available on the node, the SSDs aregiven preference to be used as a journal. Also, one SSD can work as journal for a maximum of 4disks as long as space is available. In this case, the disks are mapped to SSDs to be used asjournal and is continued till the capacity is full or till the number of mapped disks reaches four. If aSSD already reaches the limit, the next SSD is used for journal mapping.

In the case wherein all the rotational disks are exhausted with only SSDs left, SSDs tries to usejournals among themselves and within them smaller ones would be used as journals. To achievethis, the SSD disks are sorted in their descending order of size and then this logic is applied tomap the data disks to the journal disks.

CHAPTER 11. CONSOLE JOURNAL MAPPING

63

CHAPTER 12. TROUBLESHOOTING

This Troubleshooting section is intended for users who are responsible for maintaining,administering, or supporting Red Hat Storage Console environment.

There could be various reasons for a cluster creation operation to fail in the console. Few reasonsthat may cause cluster creation to fail at different stages are but not limited to:

Incorrect storage nodes configuration

Storage nodes not bootstrapped well with ceph-installer.

Raw disks added to the storage nodes

This section endeavors to list few commonly faced failure scenarios and the troubleshootingmethods.

About MongoDB Database

Red Hat Storage Console 2.0 uses MongoDB as its database to store and maintain the configurationand entities detail under the database named skyring. Under the skyring database, there is acollection table called storage_nodes from where the specific records can be removed.

The following commands are used with MongoDB:

1. To open MongoDB shell:

mongo 127.0.0.1:27017/skyring -u admin -p <passwd>

Note

User authentication is enabled for MongoDB databass with the default user nameas admin. Locate the password in the /etc/skyring/skyring.conf file.

2. To switch database to skyring:

use skyring

3. To list all the collections(tables):

show collections

4. To list all the contents of the storage_nodes collection(table):

db.storage_nodes.find()

5. To remove a specific record:

db.storage_nodes.remove({"hostname": "<FQDN of the node>"})

12.1. TROUBLESHOOTING NODES CONFIGURATION


64

Issue 1: Package Installation step becomes unresponsive for nodes

Cause

This can happen if ceph-installer is not behaving well or a prior task was unresponsive in ceph-installer. If a task is unresponsive in ceph-installer, any further tasks will not succeed. The timeoutfor nodes installation is 15 minutes. After 15 minutes is elapsed, the node will time out. If ceph-installer become unresponsive, then most likely all the nodes would fail to install in Red Hat StorageConsole one by one.

Resolution

To resolve this issue, restart the ceph-installer service on the Storage Console server node usingthe following command:

# systemctl restart ceph-installer

After the ceph-installer service is restarted, retry the cluster creation with the same set of nodes.

Issue 2: Package Installation fails on the nodes

Cause This can happen if the repositories configured on the storage nodes are not the requiredones. Based on the role of the node in Storage Console, different repositories should be enabled onthe node.

Resolution

To resolve this issue, cross-check all the repositories enabled for the storage nodes and ensurevalid repositories are configured. For detailed instructions on package installation and the requiredrepositories, see the Red Hat Storage Console Package Installation section of the Red Hat StorageConsole 2.0 Quick Start Guide.

Issue 3: Storage nodes is accepted but fails to initialize

Cause 1

The main cause for this to happen is a probable mismatch of salt-minion key on the storage nodesand the salt key maintained for the same node on the Storage Console server running salt-master.Due to this, nodes are accepted normally, as this step does not need nodes to be reachable and theavailability of the keys at salt-master is enough.

Storage Console starts the node initialization task based on a start event reaching salt-master for anode. This would not happen if salt-minion key in storage node does not match with salt key knownto the salt-master about the storage node. In this case, start event would never reach salt-masterwhich in turn fails to initialize the nodes.

Resolution

To resolve this issue, perform the following steps:

1. Remove the salt-minion keys on the storage node using the following command:

rm -r /etc/salt/pki/minion/*

2. Remove the salt key of the node in Storage Console server using the following command:


65

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#red_hat_storage_console_package_installation

salt-key -d <fqdn name of node>

3. Restart the salt-minion on the storage node using the following command:

service salt-minion restart

Reinitialize the nodes using the reinitialize option.

Cause 2

Another cause for failed initialization can be due to misconfiguration of salt-minion and its incorrectpointing to Storage Console server salt-master. In this case, salt-minion service might be down onthe storage nodes.

Resolution


1. Remove the specific records from the table storage_nodes of the MongoDB database.

2. Remove the salt-minion keys on the storage nodes using the following command:

rm -r /etc/salt/pki/minion/*

3. Remove the minion keys from salt-master on Storage Console server node using thefollowing command:

rm -r /etc/salt/pki/master/*

4. Restart salt-minion service on storage nodes using the following command:

service salt-minion restart

5. Restart the salt-master service on Storage Console server node using the follwingcommand:

service salt-master restart

After performing these steps, nodes will reappear as unaccepted nodes in Storage Console. Acceptthe nodes and continue with the cluster creation flow.

12.2. TROUBLESHOOTING MONITOR NODES CONFIGURATION

Issue 1: MON configuration fails for a storage MON node

Cause

This scenario can arise if the OS on the storage nodes are not freshly installed. It is recommendedto use freshly installed OS nodes for cluster creation.

Resolution



66

1. Remove /var/lib/ceph directory if it exists.

2. Remove all the ceph and calamari RPMs from the node using the yum remove command.

3. Reboot the node.

12.3. TROUBLESHOOTING OSD NODES CONFIGURATION

Issue 1: OSD configuration fails for a storage OSD node

Cause 1

This can happen due to improper size of the disk, or required ports not opened for OSD creation. Inthis scenario, OSD activation frequently times out leading to a failed OSD creation. Another probablecause for OSD creation failure is insufficient space during partitioning of the disk.

Resolution

To resolve this issue, remove the disk partitions using the partprobe command. Next, unmount allthe mounted partitions of the disk and its journal disk and then add a fresh disk to the node.

For example, out of 5 disks, 2 disks failed to create an OSD. Let the cluster creation progress andcomplete if it does. Next, remove the details of only the failed disks. Next, add new disks to the nodewhich is automatically added to the cluster through the cluster auto expansion feature.

Cause 2 Another probable cause is when the cluster creation process is stopped due to timeout ofOSD activation. This can happen if the firewalld and the required ports are not opened.

Resolution

Verify if the required ports are open on the node, and remove and add the new disk. Once it is done,follow the below steps:

1. Remove /var/lib/ceph directory if it exists.

2. Use partprobe command to remove partitions of the disks.

3. Remove all the mounts from the disks and their respective partitions.

4. Remove and add new disk.

5. Ensure all the required ports are opened on the storage node.

6. Restart the storage node.

7. Remove the node records from the MongoDB database.

8. Restart salt-minion service on storage node.

9. Initiate the cluster creation operation from Storage Console and accept the nodes.

Issue 2: Create Cluster operation fails when adding OSD

To resolve this issue, clean the partitions created during OSD creation and manually remove thecluster from the MongoDB database. Remove all the partitions created from the disk used whileOSD creation using partprobe command.

Additionally, perform the following steps:


67

1. Remove any directory that exists with the name /var/lib/ceph on OSD storage node rm -rf/var/lib/ceph/

2. Restart the OSD storage node to ensure all the mounts for the disk partitions are not ineffect.

3. Remove the cluster from the MongoDB database.

Issue 3: OSD Creation Failure

Cause

In a few cases, if firewalld service is running on storage nodes, the OSD activation consumesindefinite period of time and finally fails to create OSD successfully.

Resolution

To resolve this issue, open the required ports for OSD activation to complete.

Note

For the list of ports, see the Firewall Configuration section in the Red Hat Storage Console2.0 Quick Start Guide.

Issue 4: Timeout Failures

Installation of binaries fail on storage nodes when create cluster flow is initiated and the task timesout.

Resolution

View the task ID logged in /var/log/skyring/skyring.log for node installation and check the ceph-installer logs at /var/log/messages for more details.

Most of the cases occur in repositories that are not configured properly. The task either fails toinstall bits on storage node or times out.

If previous task of node installation has become unresponsive due to some reason in ceph-installercomponent, no further tasks succeeds. In this case, restart the ceph-installer service by using theservice ceph-installer restart command.

12.4. TROUBLESHOOTING IMPORT CLUSTER NODES

Issue: Nodes fail during import cluster operation

The import cluster mechanism is developed in such a way that:

If nodes are not accepted, they would get accepted during import cluster process.

If a node fails to be accepted during the import cluster process due to slow network, the importcluster task status will show the node as failed.

During the import cluster process, few clusters fail to initialize and are reported as failed in theimport cluster task status. Reinitializing from the Console will not help as the nodes might getaccepted/initialized but there is no way to show that the nodes are participating in the cluster.


68

https://access.redhat.com/documentation/en/red-hat-storage-console/2.0/single/quick-start-guide/#firewall_configuration

Resolution

1. Detach the cluster in the Console and import it back after restarting the salt-minion serviceson the storage nodes.

2. Prior to initiating import cluster operation, if the number of nodes in the cluster are high, thenodes are already listed as unaccepted in the Console. Accept the nodes before startingimport cluster and then initiate import cluster operation.

In resolution 2, if the number of nodes are high, best approach is to accept the nodes in batches toprevent nodes time out due to salt-master level limitation of serial nodes acceptance.

Follow these steps to eliminate nodes acceptance failures and for an uninterrupted import clusteroperation.

12.5. TROUBLESHOOTING POST CLUSTER CREATION FAILURES

Issue: Post cluster creation actions failure

After the cluster is created, there are a few set of actions that occur to assist the cluster. Theseactions use Calamari running on one of the MON nodes of the cluster to make changes in theunderlying cluster. Failure in any of these actions is attributed to Calamari.

The actions involved post cluster creation are:

1. Updating latest cluster status

2. Removal of default created pool (RBD)

3. Creation of default erasure coded profiles

4. Updating CRUSH map

Out of these actions, updating CRUSH map is important as cluster can be created successfully withOSDs but CRUSH map would be out of date.

These issues occur only if there is a calamari start failure during create cluster request. This isseen only in cases where MON node is not freshly installed and has some lint from previousinstallation.

Under such scenarios, remove the cluster details from the MongoDB database. If cluster records arethere in the Console, use the Unmanage and Forget cluster feature. Clear all the nodes andrecreate the cluster in Storage Console.

12.6. TROUBLESHOOTING MISCELLANEOUS

Issue: Unable to start skyring service

Cause

The socket file /var/run/.skyring-event is created after the skyring service is started. Oncethe skyring service is gracefully stopped, the /var/run/.skyring-event socket file isautomatically removed. But in the event where the skyring service is killed abruptly, the socket file isnot removed automatically.


69

When the user attempts to start the skyring service again, the startup fails by displaying thefollowing log error:

CRITICAL listener.go:146 StartListener] listen error:listen unix /var/run/.skyring-event: bind: address already in use

Resolution

To resolve this issue, follow these steps:

1. Manually remove the socket file by running the following command:

rm -f /var/run/.skyring-event

2. Start the skyring service by running the following command:

systemctl start skyring


70

Red Hat Storage Console 2.0 Administration Guide · Red Hat Storage Console is a web-based storage...

Documents

Transcript of Red Hat Storage Console 2.0 Administration Guide · Red Hat Storage Console is a web-based storage...