OpenShift Container Platform 3 - Red Hat Customer …...2.6.4. Example Inventory Files 2.6.4.1....

OpenShift Container Platform 3.9

Installation and Configuration

OpenShift Container Platform 3.9 Installation and Configuration

Last Updated: 2019-12-03

Legal Notice

Copyright © 2019 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United Statesand other countries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by theofficial Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and othercountries and are used with the OpenStack Foundation's permission. We are not affiliated with,endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

OpenShift Installation and Configuration topics cover the basics of installing and configuringOpenShift in your environment. Use these topics for the one-time tasks required to get OpenShiftup and running.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. OVERVIEW

CHAPTER 2. INSTALLING A CLUSTER2.1. PLANNING

2.1.1. Initial Planning2.1.2. Installation Methods2.1.3. Sizing Considerations2.1.4. Environment Scenarios

2.1.4.1. Single Master and Node on One System2.1.4.2. Single Master and Multiple Nodes2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes2.1.4.4. Multiple Masters Using Native HA with Co-located Clustered etcd2.1.4.5. Multiple Masters Using Native HA with External Clustered etcd2.1.4.6. Stand-alone Registry

2.1.5. RPM Versus Containerized2.2. PREREQUISITES

2.2.1. System Requirements2.2.1.1. Red Hat Subscriptions2.2.1.2. Minimum Hardware Requirements2.2.1.3. Production Level Hardware Requirements2.2.1.4. Storage management2.2.1.5. Red Hat Gluster Storage Hardware Requirements2.2.1.6. Optional: Configuring Core Usage2.2.1.7. SELinux2.2.1.8. Red Hat Gluster Storage

Optional: Using OverlayFS2.2.1.9. Security Warning

2.2.2. Environment Requirements2.2.2.1. DNS

2.2.2.1.1. Configuring Hosts to Use DNS2.2.2.1.2. Configuring a DNS Wildcard

2.2.2.2. Network Access2.2.2.2.1. NetworkManager2.2.2.2.2. Configuring firewalld as the firewall2.2.2.2.3. Required Ports

2.2.2.3. Persistent Storage2.2.2.4. Cloud Provider Considerations

2.2.2.4.1. Overriding Detected IP Addresses and Host Names2.2.2.4.2. Post-Installation Configuration for Cloud Providers

2.3. HOST PREPARATION2.3.1. Setting PATH2.3.2. Operating System Requirements2.3.3. Host Registration2.3.4. Installing Base Packages2.3.5. Installing Docker2.3.6. Configuring Docker Storage

2.3.6.1. Configuring OverlayFS2.3.6.2. Configuring Thin Pool Storage2.3.6.3. Reconfiguring Docker Storage2.3.6.4. Enabling Image Signature Support2.3.6.5. Managing Container Logs

21

222222222323232323242425252525252527282929303030313131323334343434373838393939394041

42424343464647

Table of Contents

1

2.3.6.6. Viewing Available Container Logs2.3.6.7. Blocking Local Volume Usage

2.3.7. Ensuring Host Access2.3.8. Setting Proxy Overrides2.3.9. What’s Next?

2.4. INSTALLING ON CONTAINERIZED HOSTS2.4.1. RPM Versus Containerized Installation2.4.2. Install Methods for Containerized Hosts2.4.3. Required Images2.4.4. Starting and Stopping Containers2.4.5. File Paths2.4.6. Storage Requirements2.4.7. Open vSwitch SDN Initialization

2.5. QUICK INSTALLATION2.5.1. Overview2.5.2. Before You Begin2.5.3. Running an Interactive Installation2.5.4. Defining an Installation Configuration File2.5.5. Running an Unattended Installation2.5.6. Verifying the Installation2.5.7. Uninstalling OpenShift Container Platform2.5.8. What’s Next?

2.6. ADVANCED INSTALLATION2.6.1. Overview2.6.2. Before You Begin2.6.3. Configuring Ansible Inventory Files

Image Version Policy2.6.3.1. Configuring Cluster Variables2.6.3.2. Configuring Deployment Type2.6.3.3. Configuring Host Variables2.6.3.4. Configuring Project Parameters2.6.3.5. Configuring Master API Port2.6.3.6. Configuring Cluster Pre-install Checks2.6.3.7. Configuring System Containers

2.6.3.7.1. Running Docker as a System Container2.6.3.7.2. Running etcd as a System Container

2.6.3.8. Configuring a Registry Location2.6.3.9. Configuring a Registry Route2.6.3.10. Configuring the Registry Console2.6.3.11. Configuring Router Sharding2.6.3.12. Configuring Red Hat Gluster Storage Persistent Storage

2.6.3.12.1. Configuring Container-Native Storage2.6.3.12.2. Configuring Container-Ready Storage

2.6.3.13. Configuring an OpenShift Container Registry2.6.3.13.1. Configuring Registry Storage

Option A: NFS Host GroupOption B: External NFS HostOption C: OpenStack PlatformOption D: AWS or Another S3 Storage SolutionOption E: Container-Native StorageOption F: Google Cloud Storage (GCS) bucket on Google Compute Engine (GCE)

2.6.3.14. Configuring Global Proxy Options2.6.3.15. Configuring the Firewall

48484950515151515252535353535354555557575858585859596060666768707072737474757677777778797979798080808181

83


2

2.6.3.16. Configuring Schedulability on Masters2.6.3.17. Configuring Node Host Labels

2.6.3.17.1. Configuring Dedicated Infrastructure Nodes2.6.3.18. Configuring Session Options2.6.3.19. Configuring Custom Certificates2.6.3.20. Configuring Certificate Validity2.6.3.21. Configuring Cluster Metrics

2.6.3.21.1. Configuring Metrics StorageOption A: DynamicOption B: NFS Host GroupOption C: External NFS HostUpgrading or Installing OpenShift Container Platform with NFS

2.6.3.22. Configuring Cluster Logging2.6.3.22.1. Configuring Logging Storage

Option A: DynamicOption B: NFS Host GroupOption C: External NFS HostUpgrading or Installing OpenShift Container Platform with NFS

2.6.3.23. Customizing Service Catalog Options2.6.3.23.1. Configuring the OpenShift Ansible Broker

2.6.3.23.1.1. Configuring Persistent Storage for the OpenShift Ansible Broker2.6.3.23.1.2. Configuring the OpenShift Ansible Broker for Local APB Development

2.6.3.23.2. Configuring the Template Service Broker2.6.3.24. Configuring Web Console Customization

2.6.4. Example Inventory Files2.6.4.1. Single Master Examples

Single Master, Single etcd, and Multiple NodesSingle Master, Multiple etcd, and Multiple Nodes

2.6.4.2. Multiple Masters ExamplesMultiple Masters Using Native HA with External Clustered etcdMultiple Masters Using Native HA with Co-located Clustered etcd

2.6.5. Running the Advanced Installation2.6.5.1. Running the RPM-based Installer2.6.5.2. Running the Containerized Installer

2.6.5.2.1. Running the Installer as a System Container2.6.5.2.2. Running Other Playbooks2.6.5.2.3. Running the Installer as a Docker Container2.6.5.2.4. Running the Installation Playbook for OpenStack

2.6.5.3. Running Individual Component Playbooks2.6.6. Verifying the Installation

Verifying Multiple etcd HostsVerifying Multiple Masters Using HAProxy

2.6.7. Optionally Securing Builds2.6.8. Uninstalling OpenShift Container Platform

2.6.8.1. Uninstalling Nodes2.6.9. Known Issues2.6.10. What’s Next?

2.7. DISCONNECTED INSTALLATION2.7.1. Overview2.7.2. Prerequisites2.7.3. Required Software and Components

2.7.3.1. Syncing Repositories2.7.3.2. Syncing Images

84858586878788888889898990909090919191

92929494949696969799

100102103104105105106106108108110110111111111

112113113114114114115115116

Table of Contents

3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.7.3.3. Preparing Images for Export2.7.4. Repository Server

2.7.4.1. Placing the Software2.7.5. OpenShift Container Platform Systems

2.7.5.1. Building Your Hosts2.7.5.2. Connecting the Repositories2.7.5.3. Host Preparation

2.7.6. Installing OpenShift Container Platform2.7.6.1. Importing OpenShift Container Platform Component Images2.7.6.2. Running the OpenShift Container Platform Installer2.7.6.3. Creating the Internal Docker Registry

2.7.7. Post-Installation Changes2.7.7.1. Re-tagging S2I Builder Images2.7.7.2. Configuring a Registry Location2.7.7.3. Creating an Administrative User2.7.7.4. Modifying the Security Policies2.7.7.5. Editing the Image Stream Definitions2.7.7.6. Loading the Container Images

2.7.8. Installing a Router2.8. INSTALLING A STAND-ALONE DEPLOYMENT OF OPENSHIFT CONTAINER REGISTRY

2.8.1. About OpenShift Container Registry2.8.2. Minimum Hardware Requirements2.8.3. Supported System Topologies2.8.4. Host Preparation2.8.5. Stand-alone Registry Installation Methods

2.8.5.1. Quick Installation for Stand-alone OpenShift Container Registry2.8.5.2. Advanced Installation for Stand-alone OpenShift Container Registry

CHAPTER 3. SETTING UP THE REGISTRY3.1. REGISTRY OVERVIEW

3.1.1. About the Registry3.1.2. Integrated or Stand-alone Registries

3.2. DEPLOYING A REGISTRY ON EXISTING CLUSTERS3.2.1. Overview3.2.2. Deploying the Registry3.2.3. Deploying the Registry as a DaemonSet3.2.4. Registry Compute Resources3.2.5. Storage for the Registry

3.2.5.1. Production Use3.2.5.1.1. Use Amazon S3 as a Storage Back-end

3.2.5.2. Non-Production Use3.2.6. Enabling the Registry Console

3.2.6.1. Deploying the Registry Console3.2.6.2. Securing the Registry Console3.2.6.3. Troubleshooting the Registry Console

3.2.6.3.1. Debug Mode3.2.6.3.2. Display SSL Certificate Path

3.3. ACCESSING THE REGISTRY3.3.1. Viewing Logs3.3.2. File Storage3.3.3. Accessing the Registry Directly

3.3.3.1. User Prerequisites3.3.3.2. Logging in to the Registry

119120121121121121122122122122123123123123124125125126126127127127128128128128130

133133133133133133133134134134135135136137137138139139139140140140142142143


4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3.3.3. Pushing and Pulling Images3.3.4. Accessing Registry Metrics

3.4. SECURING AND EXPOSING THE REGISTRY3.4.1. Overview3.4.2. Manually Securing the Registry3.4.3. Manually Exposing a Secure Registry3.4.4. Manually Exposing a Non-Secure Registry

3.5. EXTENDED REGISTRY CONFIGURATION3.5.1. Maintaining the Registry IP Address3.5.2. Whitelisting Docker Registries3.5.3. Setting the Registry Hostname3.5.4. Overriding the Registry Configuration3.5.5. Registry Configuration Reference

3.5.5.1. Log3.5.5.2. Hooks3.5.5.3. Storage3.5.5.4. Auth3.5.5.5. Middleware

3.5.5.5.1. S3 Driver Configuration3.5.5.5.2. CloudFront Middleware3.5.5.5.3. Overriding Middleware Configuration Options3.5.5.5.4. Image Pullthrough3.5.5.5.5. Manifest Schema v2 Support

3.5.5.6. OpenShift3.5.5.7. Reporting3.5.5.8. HTTP3.5.5.9. Notifications3.5.5.10. Redis3.5.5.11. Health3.5.5.12. Proxy3.5.5.13. Cache

3.6. KNOWN ISSUES3.6.1. Overview3.6.2. Image Push Errors with Scaled Registry Using Shared NFS Volume3.6.3. Pull of Internally Managed Image Fails with "not found" Error3.6.4. Image Push Fails with "500 Internal Server Error" on S3 Storage3.6.5. Image Pruning Fails

CHAPTER 4. SETTING UP A ROUTER4.1. ROUTER OVERVIEW

4.1.1. About Routers4.1.2. Router Service Account

4.1.2.1. Permission to Access Labels4.2. USING THE DEFAULT HAPROXY ROUTER

4.2.1. Overview4.2.2. Creating a Router4.2.3. Other Basic Router Commands4.2.4. Filtering Routes to Specific Routers4.2.5. HAProxy Strict SNI4.2.6. TLS Cipher Suites4.2.7. Highly-Available Routers4.2.8. Customizing the Router Service Ports4.2.9. Working With Multiple Routers

143144145145145148150151151152153153155155156156157157158159160160161

162163163163164164164164165165165165166166

168168168168168169169170170171172172172172173

Table of Contents

5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2.10. Adding a Node Selector to a Deployment Configuration4.2.11. Using Router Shards

4.2.11.1. Creating Router Shards4.2.11.2. Modifying Router Shards

4.2.12. Finding the Host Name of the Router4.2.13. Customizing the Default Routing Subdomain4.2.14. Forcing Route Host Names to a Custom Routing Subdomain4.2.15. Using Wildcard Certificates4.2.16. Manually Redeploy Certificates4.2.17. Using Secured Routes4.2.18. Using Wildcard Routes (for a Subdomain)4.2.19. Using the Container Network Stack4.2.20. Exposing Router Metrics4.2.21. ARP Cache Tuning for Large-scale Clusters4.2.22. Protecting Against DDoS Attacks

4.3. DEPLOYING A CUSTOMIZED HAPROXY ROUTER4.3.1. Overview4.3.2. Obtaining the Router Configuration Template4.3.3. Modifying the Router Configuration Template

4.3.3.1. Background4.3.3.2. Go Template Actions4.3.3.3. Router Provided Information4.3.3.4. Annotations4.3.3.5. Environment Variables4.3.3.6. Example Usage

4.3.4. Using a ConfigMap to Replace the Router Configuration Template4.3.5. Using Stick Tables4.3.6. Rebuilding Your Router

4.4. CONFIGURING THE HAPROXY ROUTER TO USE THE PROXY PROTOCOL4.4.1. Overview4.4.2. Why Use the PROXY Protocol?4.4.3. Using the PROXY Protocol

4.5. USING THE F5 ROUTER PLUG-IN4.5.1. Overview4.5.2. Prerequisites and Supportability

4.5.2.1. Configuring the Virtual Servers4.5.3. Deploying the F5 Router4.5.4. F5 Router Partition Paths4.5.5. Setting Up F5 Native Integration

CHAPTER 5. DEPLOYING RED HAT CLOUDFORMS5.1. DEPLOYING {MGMT-APP} ON OPENSHIFT CONTAINER PLATFORM

5.1.1. Introduction5.2. REQUIREMENTS FOR RED HAT CLOUDFORMS ON OPENSHIFT CONTAINER PLATFORM5.3. CONFIGURING ROLE VARIABLES

5.3.1. Overview5.3.2. General Variables5.3.3. Customizing Template Parameters5.3.4. Database Variables

5.3.4.1. Containerized (Podified) Database5.3.4.2. External Database

5.3.5. Storage Class Variables5.3.5.1. NFS (Default)

173173176178179180180181181

182183189189191

192193193194194194194195

200200201

202203205205205205206210210210211212213214

216216216217218218218219219219219

220221


6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.3.5.2. NFS External5.3.5.3. Cloud Provider5.3.5.4. Preconfigured (Advanced)

5.4. RUNNING THE INSTALLER5.4.1. Deploying Red Hat CloudForms During or After OpenShift Container Platform Installation5.4.2. Example Inventory Files

5.4.2.1. All Defaults5.4.2.2. External NFS Storage5.4.2.3. Override PV Sizes5.4.2.4. Override Memory Requirements5.4.2.5. External PostgreSQL Database

5.5. ENABLING CONTAINER PROVIDER INTEGRATION5.5.1. Adding a Single Container Provider

5.5.1.1. Adding Manually5.5.1.2. Adding Automatically

5.5.2. Multiple Container Providers5.5.2.1. Preparing the Script

5.5.2.1.1. Example5.5.2.2. Running the Playbook

5.5.3. Refreshing Providers5.6. UNINSTALLING RED HAT CLOUDFORMS

5.6.1. Running the Uninstall Playbook5.6.2. Troubleshooting

CHAPTER 6. MASTER AND NODE CONFIGURATION6.1. INSTALLATION DEPENDENCIES6.2. CONFIGURING MASTERS AND NODES6.3. MAKING CONFIGURATION CHANGES USING ANSIBLE

6.3.1. Using the htpasswd commmand6.4. MAKING MANUAL CONFIGURATION CHANGES6.5. MASTER CONFIGURATION FILES

6.5.1. Admission Control Configuration6.5.2. Asset Configuration6.5.3. Authentication and Authorization Configuration6.5.4. Controller Configuration6.5.5. etcd Configuration6.5.6. Grant Configuration6.5.7. Image Configuration6.5.8. Image Policy Configuration6.5.9. Kubernetes Master Configuration6.5.10. Network Configuration6.5.11. OAuth Authentication Configuration6.5.12. Project Configuration6.5.13. Scheduler Configuration6.5.14. Security Allocator Configuration6.5.15. Service Account Configuration6.5.16. Serving Information Configuration6.5.17. Volume Configuration6.5.18. Basic Audit6.5.19. Advanced Audit6.5.20. Specifying TLS ciphers for etcd

6.6. NODE CONFIGURATION FILES6.6.1. Pod and Node Configuration

221222222222222223223223223224224224224225225225225226227227227227228

229229229229231

233233233234236236236238238239239240241

243244245245246247248249252254255

Table of Contents

7

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.6.2. Docker Configuration6.6.3. Setting Node Queries per Second (QPS) Limits and Burst Values6.6.4. Parallel Image Pulls with Docker 1.9+

6.7. PASSWORDS AND OTHER SENSITIVE DATA6.8. CREATING NEW CONFIGURATION FILES6.9. LAUNCHING SERVERS USING CONFIGURATION FILES6.10. CONFIGURING LOGGING LEVELS6.11. RESTARTING OPENSHIFT CONTAINER PLATFORM SERVICES

CHAPTER 7. OPENSHIFT ANSIBLE BROKER CONFIGURATION7.1. OVERVIEW7.2. MODIFYING THE OPENSHIFT ANSIBLE BROKER CONFIGURATION7.3. REGISTRY CONFIGURATION

7.3.1. Production or Development7.3.2. Storing Registry Credentials7.3.3. Mock Registry7.3.4. Dockerhub Registry7.3.5. APB Filtering7.3.6. Local OpenShift Container Registry7.3.7. Red Hat Container Catalog Registry7.3.8. ISV Registry7.3.9. Multiple Registries

7.4. DAO CONFIGURATION7.5. LOG CONFIGURATION7.6. OPENSHIFT CONFIGURATION7.7. BROKER CONFIGURATION7.8. SECRETS CONFIGURATION7.9. RUNNING BEHIND A PROXY

7.9.1. Registry Adapter Whitelists7.9.2. Configuring the Broker Behind a Proxy Using Ansible7.9.3. Configuring the Broker Behind a Proxy Manually7.9.4. Setting Proxy Environment Variables in Pods

CHAPTER 8. ADDING HOSTS TO AN EXISTING CLUSTER8.1. OVERVIEW8.2. ADDING HOSTS USING THE QUICK INSTALLER TOOL8.3. ADDING HOSTS

Procedure8.4. ADDING ETCD HOSTS TO EXISTING CLUSTER8.5. REPLACING EXISTING MASTERS WITH ETCD COLOCATED8.6. MIGRATING THE NODES

CHAPTER 9. LOADING THE DEFAULT IMAGE STREAMS AND TEMPLATES9.1. OVERVIEW9.2. OFFERINGS BY SUBSCRIPTION TYPE

9.2.1. OpenShift Container Platform Subscription9.2.2. xPaaS Middleware Add-on Subscriptions

9.3. BEFORE YOU BEGIN9.4. PREREQUISITES9.5. CREATING IMAGE STREAMS FOR OPENSHIFT CONTAINER PLATFORM IMAGES9.6. CREATING IMAGE STREAMS FOR XPAAS MIDDLEWARE IMAGES9.7. CREATING DATABASE SERVICE TEMPLATES9.8. CREATING INSTANT APP AND QUICKSTART TEMPLATES9.9. WHAT’S NEXT?

255256256256257258258262

264264265265266267269269269270271271271

272272272273274274274275275276

277277277278278280281282

284284284284285285285286286286287288


8

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTER 10. CONFIGURING CUSTOM CERTIFICATES10.1. OVERVIEW10.2. CONFIGURING A CERTIFICATE CHAIN10.3. CONFIGURING CUSTOM CERTIFICATES DURING INSTALLATION10.4. CONFIGURING CUSTOM CERTIFICATES FOR THE WEB CONSOLE OR CLI10.5. CONFIGURING A CUSTOM MASTER HOST CERTIFICATE10.6. CONFIGURING A CUSTOM WILDCARD CERTIFICATE FOR THE DEFAULT ROUTER10.7. CONFIGURING A CUSTOM CERTIFICATE FOR THE IMAGE REGISTRY10.8. CONFIGURING A CUSTOM CERTIFICATE FOR A LOAD BALANCER10.9. RETROFIT CUSTOM CERTIFICATES INTO A CLUSTER

10.9.1. Retrofit Custom Master Certificates into a Cluster10.9.2. Retrofit Custom Router Certificates into a Cluster

10.10. USING CUSTOM CERTIFICATES WITH OTHER COMPONENTS

CHAPTER 11. REDEPLOYING CERTIFICATES11.1. OVERVIEW11.2. CHECKING CERTIFICATE EXPIRATIONS

11.2.1. Role Variables11.2.2. Running Certificate Expiration Playbooks

Other Example Playbooks11.2.3. Output Formats

HTML ReportJSON Report

11.3. REDEPLOYING CERTIFICATES11.3.1. Redeploying All Certificates Using the Current OpenShift Container Platform and etcd CA11.3.2. Redeploying a New or Custom OpenShift Container Platform CA11.3.3. Redeploying a New etcd CA11.3.4. Redeploying Master Certificates Only11.3.5. Redeploying etcd Certificates Only11.3.6. Redeploying Node Certificates Only11.3.7. Redeploying Registry or Router Certificates Only

11.3.7.1. Redeploying Registry Certificates Only11.3.7.2. Redeploying Router Certificates Only

11.3.8. Redeploying Custom Registry or Router Certificates11.3.8.1. Redeploying Registry Certificates Manually11.3.8.2. Redeploying Router Certificates Manually

CHAPTER 12. CONFIGURING AUTHENTICATION AND USER AGENT12.1. OVERVIEW12.2. IDENTITY PROVIDER PARAMETERS12.3. CONFIGURING IDENTITY PROVIDERS

12.3.1. Configuring identity providers with Ansible12.3.2. Configuring identity providers in the master configuration file12.3.3. Configuring an identity provider or method

12.3.3.1. Manually provisioning a user when using the lookup mapping method12.3.4. Allow all12.3.5. Deny all12.3.6. HTPasswd12.3.7. Keystone

12.3.7.1. Configuring authentication on the master12.3.7.2. Creating Users with Keystone Authentication12.3.7.3. Verifying Users

12.3.8. LDAP authentication

289289289289290291292293294295295295296

297297297297298299299299299300301301302303303304304304304304304306

309309309310311311312312313313314315316317318318

Table of Contents

9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12.3.9. Basic authentication (remote)12.3.9.1. Configuring authentication on the master12.3.9.2. Troubleshooting

12.3.10. Request header12.3.11. GitHub

12.3.11.1. Registering the application on GitHub12.3.11.2. Configuring authentication on the master12.3.11.3. Creating users with GitHub authentication12.3.11.4. Verifying users

12.3.12. GitLab12.3.13. Google12.3.14. OpenID connect

12.4. TOKEN OPTIONS12.5. GRANT OPTIONS12.6. SESSION OPTIONS12.7. PREVENTING CLI VERSION MISMATCH WITH USER AGENT

CHAPTER 13. SYNCING GROUPS WITH LDAP13.1. OVERVIEW13.2. CONFIGURING LDAP SYNC

13.2.1. LDAP Client Configuration13.2.2. LDAP Query Definition13.2.3. User-Defined Name Mapping

13.3. RUNNING LDAP SYNC13.4. RUNNING A GROUP PRUNING JOB13.5. SYNC EXAMPLES

13.5.1. RFC 230713.5.1.1. RFC2307 with User-Defined Name Mappings

13.5.2. RFC 2307 with User-Defined Error Tolerances13.5.3. Active Directory13.5.4. Augmented Active Directory

13.6. NESTED MEMBERSHIP SYNC EXAMPLE13.7. LDAP SYNC CONFIGURATION SPECIFICATION

13.7.1. v1.LDAPSyncConfig13.7.2. v1.StringSource13.7.3. v1.LDAPQuery13.7.4. v1.RFC2307Config13.7.5. v1.ActiveDirectoryConfig13.7.6. v1.AugmentedActiveDirectoryConfig

CHAPTER 14. CONFIGURING LDAP FAILOVER14.1. PREREQUISITES FOR CONFIGURING BASIC REMOTE AUTHENTICATION14.2. GENERATING AND SHARING CERTIFICATES WITH THE REMOTE BASIC AUTHENTICATION SERVER

14.3. CONFIGURING SSSD FOR LDAP FAILOVER14.4. CONFIGURING APACHE TO USE SSSD14.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM TO USE SSSD AS THE BASIC REMOTEAUTHENTICATION SERVER

CHAPTER 15. CONFIGURING THE SDN15.1. OVERVIEW15.2. AVAILABLE SDN PROVIDERS

Installing VMware NSX-T (™) on OpenShift Container Platform15.3. CONFIGURING THE POD NETWORK WITH ANSIBLE

321322324324332332332335335335336337340341341

342

345345345345346347347348348349351353355357360363363365366367368369

371371

371372374

377

379379379379379


10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15.4. CONFIGURING THE POD NETWORK ON MASTERS15.5. CONFIGURING THE POD NETWORK ON NODES15.6. MIGRATING BETWEEN SDN PLUG-INS

15.6.1. Migrating from ovs-multitenant to ovs-networkpolicy15.7. EXTERNAL ACCESS TO THE CLUSTER NETWORK15.8. USING FLANNEL

CHAPTER 16. CONFIGURING NUAGE SDN16.1. NUAGE SDN AND OPENSHIFT CONTAINER PLATFORM16.2. DEVELOPER WORKFLOW16.3. OPERATIONS WORKFLOW16.4. INSTALLATION

CHAPTER 17. CONFIGURING FOR AMAZON WEB SERVICES (AWS)17.1. OVERVIEW17.2. PERMISSIONS17.3. CONFIGURING A SECURITY GROUP

17.3.1. Overriding Detected IP Addresses and Host Names17.4. CONFIGURING AWS VARIABLES17.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM FOR AWS

17.5.1. Configuring OpenShift Container Platform for AWS with Ansible17.5.2. Manually Configuring OpenShift Container Platform Masters for AWS17.5.3. Manually Configuring OpenShift Container Platform Nodes for AWS17.5.4. Manually Setting Key-Value Access Pairs

17.6. APPLYING CONFIGURATION CHANGES17.7. LABELING CLUSTERS FOR AWS

17.7.1. Resources That Need Tags17.7.2. Tagging an Existing Cluster

CHAPTER 18. CONFIGURING FOR OPENSTACK18.1. OVERVIEW18.2. PERMISSIONS18.3. CONFIGURING A SECURITY GROUP18.4. CONFIGURING OPENSTACK VARIABLES18.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR OPENSTACK

18.5.1. Configuring OpenShift Container Platform for OpenStack with Ansible18.5.2. Manually Configuring OpenShift Container Platform Masters for OpenStack18.5.3. Manually Configuring OpenShift Container Platform Nodes for OpenStack18.5.4. Installing OpenShift Container Platform by Using an Ansible Playbook

18.6. APPLYING CONFIGURATION CHANGES

CHAPTER 19. CONFIGURING FOR GCE19.1. OVERVIEW19.2. PERMISSIONS19.3. CONFIGURING MASTERS

19.3.1. Configuring OpenShift Container Platform Masters for GCE with Ansible19.3.2. Manually Configuring OpenShift Container Platform Masters for GCE

19.4. CONFIGURING NODES19.5. CONFIGURING MULTIZONE SUPPORT IN A GCE DEPLOYMENT19.6. APPLYING CONFIGURATION CHANGES

CHAPTER 20. CONFIGURING FOR AZURE20.1. OVERVIEW20.2. PERMISSIONS

380381381382383383

386386386386386

389389389389390391

392392393393394394394395395

397397397397398398398399400400401

402402402402402403404404404

406406406

Table of Contents

11

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20.3. PREREQUISITES20.4. THE AZURE CONFIGURATION FILE20.5. CONFIGURING MASTERS20.6. CONFIGURING NODES20.7. APPLYING CONFIGURATION CHANGES

CHAPTER 21. CONFIGURING FOR VMWARE VSPHERE21.1. OVERVIEW21.2. ENABLING VMWARE VSPHERE CLOUD PROVIDER21.3. CONFIGURING OPENSHIFT CONTAINER PLATFORM FOR VSPHERE USING ANSIBLE21.4. THE VMWARE VSPHERE CONFIGURATION FILE21.5. CONFIGURING MASTERS21.6. CONFIGURING NODES21.7. APPLYING CONFIGURATION CHANGES21.8. BACKUP OF PERSISTENT VOLUMES

CHAPTER 22. CONFIGURING FOR LOCAL VOLUME22.1. OVERVIEW

22.1.1. Enable Local Volumes22.1.2. Mount Local Volumes22.1.3. Configure Local Provisioner22.1.4. Deploy Local Provisioner22.1.5. Adding New Devices

CHAPTER 23. CONFIGURING PERSISTENT VOLUME CLAIM PROTECTION23.1. OVERVIEW

23.1.1. Enable PVC Protection

CHAPTER 24. CONFIGURING PERSISTENT STORAGE24.1. OVERVIEW24.2. PERSISTENT STORAGE USING NFS

24.2.1. Overview24.2.2. Provisioning24.2.3. Enforcing Disk Quotas24.2.4. NFS Volume Security

24.2.4.1. Group IDs24.2.4.2. User IDs24.2.4.3. SELinux24.2.4.4. Export Settings

24.2.5. Reclaiming Resources24.2.6. Automation24.2.7. Additional Configuration and Troubleshooting

24.3. PERSISTENT STORAGE USING RED HAT GLUSTER STORAGE24.3.1. Overview

24.3.1.1. Container-Native Storage24.3.1.2. Container-Ready Storage24.3.1.3. Standalone Red Hat Gluster Storage24.3.1.4. GlusterFS Volumes24.3.1.5. gluster-block Volumes24.3.1.6. Gluster S3 Storage

24.3.2. Considerations24.3.2.1. Software Prerequisites24.3.2.2. Hardware Requirements24.3.2.3. Storage Sizing

406406407408408

410410410412413414414415415

416416416417417418419

421421421

422422422422423424424425426427427428429429429429430430430431431

432432432432433


12

24.3.2.4. Volume Operation Behaviors24.3.2.5. Volume Security

24.3.2.5.1. POSIX Permissions24.3.2.5.2. SELinux

24.3.3. Support Requirements24.3.4. Installation

24.3.4.1. Container-Ready Storage: Installing Red Hat Gluster Storage Nodes24.3.4.2. Using the Advanced Installer

24.3.4.2.1. Example: Basic Container-Native Storage Installation24.3.4.2.2. Example: Basic Container-Ready Storage Installation24.3.4.2.3. Example: Container-Native Storage with an Integrated OpenShift Container Registry24.3.4.2.4. Example: Container-Native Storage for OpenShift Logging and Metrics24.3.4.2.5. Example: Container-Native Storage for Applications, Registry, Logging, and Metrics24.3.4.2.6. Example: Container-Ready Storage for Applications, Registry, Logging, and Metrics

24.3.5. Uninstall Container-Native Storage24.3.6. Provisioning

24.3.6.1. Static Provisioning24.3.6.2. Dynamic Provisioning

24.4. PERSISTENT STORAGE USING OPENSTACK CINDER24.4.1. Overview24.4.2. Provisioning Cinder PVs

24.4.2.1. Creating the Persistent Volume24.4.2.2. Cinder PV format24.4.2.3. Cinder volume security

24.5. PERSISTENT STORAGE USING CEPH RADOS BLOCK DEVICE (RBD)24.5.1. Overview24.5.2. Provisioning

24.5.2.1. Creating the Ceph Secret24.5.2.2. Creating the Persistent Volume

24.5.3. Ceph Volume Security24.6. PERSISTENT STORAGE USING AWS ELASTIC BLOCK STORE

24.6.1. Overview24.6.2. Provisioning

24.6.2.1. Creating the Persistent Volume24.6.2.2. Volume Format24.6.2.3. Maximum Number of EBS Volumes on a Node

24.7. PERSISTENT STORAGE USING GCE PERSISTENT DISK24.7.1. Overview24.7.2. Provisioning

24.7.2.1. Creating the Persistent Volume24.7.2.2. Volume Format

24.8. PERSISTENT STORAGE USING ISCSI24.8.1. Overview24.8.2. Provisioning

24.8.2.1. Enforcing Disk Quotas24.8.2.2. iSCSI Volume Security24.8.2.3. iSCSI Multipathing24.8.2.4. iSCSI Custom Initiator IQN

24.9. PERSISTENT STORAGE USING FIBRE CHANNEL24.9.1. Overview24.9.2. Provisioning

24.9.2.1. Enforcing Disk Quotas24.9.2.2. Fibre Channel Volume Security

434434434435435436436436438438439440441

443445446446449450450450451452452453453454454454456457457457458459459459459460460461461461461

462462462463463463464464464

Table of Contents

13

24.10. PERSISTENT STORAGE USING AZURE DISK24.10.1. Overview24.10.2. Prerequisites24.10.3. Provisioning24.10.4. Configuring Azure Disk for regional cloud

24.10.4.1. Creating the Persistent Volume24.10.4.2. Volume Format

24.11. PERSISTENT STORAGE USING AZURE FILE24.11.1. Overview24.11.2. Before you begin24.11.3. Example configuration files24.11.4. Configuring Azure File for regional cloud24.11.5. Creating the PV24.11.6. Creating the Azure Storage Account secret

24.12. PERSISTENT STORAGE USING FLEXVOLUME PLUG-INS24.12.1. Overview24.12.2. FlexVolume drivers

24.12.2.1. FlexVolume drivers with master-initiated attach/detach24.12.2.2. FlexVolume drivers without master-initiated attach/detach

24.12.3. Installing FlexVolume drivers24.12.4. Consuming storage using FlexVolume drivers

24.13. USING VMWARE VSPHERE VOLUMES FOR PERSISTENT STORAGE24.13.1. Overview

Prerequisites24.13.2. Provisioning VMware vSphere volumes

24.13.2.1. Creating persistent volumes24.13.2.2. Formatting VMware vSphere volumes

24.14. PERSISTENT STORAGE USING LOCAL VOLUME24.14.1. Overview24.14.2. Provisioning24.14.3. Creating Local Persistent Volume Claim24.14.4. Feature Status

24.15. DYNAMIC PROVISIONING AND CREATING STORAGE CLASSES24.15.1. Overview24.15.2. Available dynamically provisioned plug-ins24.15.3. Defining a StorageClass

24.15.3.1. Basic StorageClass object definition24.15.3.2. StorageClass annotations24.15.3.3. OpenStack Cinder object definition24.15.3.4. AWS ElasticBlockStore (EBS) object definition24.15.3.5. GCE PersistentDisk (gcePD) object definition24.15.3.6. GlusterFS object definition24.15.3.7. Ceph RBD object definition24.15.3.8. Trident object definition24.15.3.9. VMware vSphere object definition24.15.3.10. Azure Disk object definition

24.15.4. Changing the default StorageClass24.15.5. Additional information and examples

24.16. VOLUME SECURITY24.16.1. Overview24.16.2. SCCs, Defaults, and Allowed Ranges24.16.3. Supplemental Groups24.16.4. fsGroup

465465465465465466467467467467469469469470471471471472475476476477477477478478479479479480480480481481481

483483483484484485485487488488488489490490490491

494497


14

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24.16.5. User IDs24.16.6. SELinux Options

24.17. SELECTOR-LABEL VOLUME BINDING24.17.1. Overview24.17.2. Motivation24.17.3. Deployment

24.17.3.1. Prerequisites24.17.3.2. Define the Persistent Volume and Claim24.17.3.3. Deploy the Persistent Volume and Claim

24.18. ENABLING CONTROLLER-MANAGED ATTACHMENT AND DETACHMENT24.18.1. Overview24.18.2. Determining What Is Managing Attachment and Detachment24.18.3. Configuring Nodes to Enable Controller-managed Attachment and Detachment

24.19. PERSISTENT VOLUME SNAPSHOTS24.19.1. Overview24.19.2. Features24.19.3. Installation and Setup

24.19.3.1. Starting the External Controller and Provisioner24.19.3.2. Managing Snapshot Users

24.19.4. Lifecycle of a Volume Snapshot and Volume Snapshot Data24.19.4.1. Persistent Volume Claim and Persistent Volume

24.19.4.1.1. Snapshot Promoter24.19.4.2. Create Snapshot24.19.4.3. Restore Snapshot24.19.4.4. Delete Snapshot

CHAPTER 25. PERSISTENT STORAGE EXAMPLES25.1. OVERVIEW25.2. SHARING AN NFS MOUNT ACROSS TWO PERSISTENT VOLUME CLAIMS

25.2.1. Overview25.2.2. Creating the Persistent Volume25.2.3. Creating the Persistent Volume Claim25.2.4. Ensuring NFS Volume Access25.2.5. Creating the Pod25.2.6. Creating an Additional Pod to Reference the Same PVC

25.3. COMPLETE EXAMPLE USING CEPH RBD25.3.1. Overview25.3.2. Installing the ceph-common Package25.3.3. Creating the Ceph Secret25.3.4. Creating the Persistent Volume25.3.5. Creating the Persistent Volume Claim25.3.6. Creating the Pod25.3.7. Defining Group and Owner IDs (Optional)25.3.8. Setting ceph-user-secret as Default for Projects

25.4. USING CEPH RBD FOR DYNAMIC PROVISIONING25.4.1. Overview25.4.2. Creating a pool for dynamic volumes25.4.3. Using an existing Ceph cluster for dynamic persistent storage25.4.4. Setting ceph-user-secret as the default for projects

25.5. COMPLETE EXAMPLE USING GLUSTERFS25.5.1. Overview25.5.2. Prerequisites25.5.3. Static Provisioning

499501

503503503503503504505505505505506506506507507507510511511511511512513

514514514514514515516517521

523523523523524525526527527528528528529532532533533534

Table of Contents

15

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25.5.4. Using the Storage25.6. COMPLETE EXAMPLE USING GLUSTERFS FOR DYNAMIC PROVISIONING

25.6.1. Overview25.6.2. Prerequisites25.6.3. Dynamic Provisioning25.6.4. Using the Storage

25.7. MOUNTING VOLUMES ON PRIVILEGED PODS25.7.1. Overview25.7.2. Prerequisites25.7.3. Creating the Persistent Volume25.7.4. Creating a Regular User25.7.5. Creating the Persistent Volume Claim25.7.6. Verifying the Setup

25.7.6.1. Checking the Pod SCC25.7.6.2. Verifying the Mount

25.8. SWITCHING AN INTEGRATED OPENSHIFT CONTAINER REGISTRY TO GLUSTERFS25.8.1. Overview25.8.2. Prerequisites25.8.3. Manually Provision the GlusterFS PersistentVolumeClaim25.8.4. Attach the PersistentVolumeClaim to the Registry

25.9. BINDING PERSISTENT VOLUMES BY LABELS25.9.1. Overview

25.9.1.1. Assumptions25.9.2. Defining Specifications

25.9.2.1. Persistent Volume with Labels25.9.2.2. Persistent Volume Claim with Selectors25.9.2.3. Volume Endpoints25.9.2.4. Deploy the PV, PVC, and Endpoints

25.10. USING STORAGE CLASSES FOR DYNAMIC PROVISIONING25.10.1. Overview25.10.2. Scenario 1: Basic Dynamic Provisioning with Two Types of StorageClasses25.10.3. Scenario 2: How to enable Default StorageClass behavior for a Cluster

25.11. USING STORAGE CLASSES FOR EXISTING LEGACY STORAGE25.11.1. Overview

25.11.1.1. Scenario 1: Link StorageClass to existing Persistent Volume with Legacy Data25.12. CONFIGURING AZURE BLOB STORAGE FOR INTEGRATED DOCKER REGISTRY

25.12.1. Overview25.12.2. Before You Begin25.12.3. Overriding Registry Configuration

CHAPTER 26. WORKING WITH HTTP PROXIES26.1. OVERVIEW26.2. CONFIGURING NO_PROXY26.3. CONFIGURING HOSTS FOR PROXIES26.4. CONFIGURING HOSTS FOR PROXIES USING ANSIBLE26.5. PROXYING DOCKER PULL26.6. USING MAVEN BEHIND A PROXY26.7. CONFIGURING S2I BUILDS FOR PROXIES26.8. CONFIGURING DEFAULT TEMPLATES FOR PROXIES26.9. SETTING PROXY ENVIRONMENT VARIABLES IN PODS26.10. GIT REPOSITORY ACCESS

CHAPTER 27. CONFIGURING GLOBAL BUILD DEFAULTS AND OVERRIDES

537538539539539540542542543543543544545545545545545545546549549549550550550551551551

552552552555559559559561561561562

564564564565566566567567567568568

569


16

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27.1. OVERVIEW27.2. SETTING GLOBAL BUILD DEFAULTS

27.2.1. Configuring Global Build Defaults with Ansible27.2.2. Manually Setting Global Build Defaults

27.3. SETTING GLOBAL BUILD OVERRIDES27.3.1. Configuring Global Build Overrides with Ansible27.3.2. Manually Setting Global Build Overrides

CHAPTER 28. CONFIGURING PIPELINE EXECUTION28.1. OVERVIEW28.2. OPENSHIFT JENKINS CLIENT PLUGIN28.3. OPENSHIFT JENKINS SYNC PLUGIN

CHAPTER 29. CONFIGURING ROUTE TIMEOUTS

CHAPTER 30. CONFIGURING NATIVE CONTAINER ROUTING30.1. NETWORK OVERVIEW30.2. CONFIGURE NATIVE CONTAINER ROUTING30.3. SETTING UP A NODE FOR CONTAINER NETWORKING30.4. SETTING UP A ROUTER FOR CONTAINER NETWORKING

CHAPTER 31. ROUTING FROM EDGE LOAD BALANCERS31.1. OVERVIEW31.2. INCLUDING THE LOAD BALANCER IN THE SDN31.3. ESTABLISHING A TUNNEL USING A RAMP NODE

31.3.1. Configuring a Highly-Available Ramp Node

CHAPTER 32. AGGREGATING CONTAINER LOGS32.1. OVERVIEW32.2. PRE-DEPLOYMENT CONFIGURATION32.3. SPECIFYING LOGGING ANSIBLE VARIABLES32.4. DEPLOYING THE EFK STACK32.5. UNDERSTANDING AND ADJUSTING THE DEPLOYMENT

32.5.1. Ops Cluster32.5.2. Elasticsearch

32.5.2.1. Persistent Elasticsearch Storage32.5.2.1.1. Using NFS as a persistent volume32.5.2.1.2. Using NFS as local storage32.5.2.1.3. Changing the Scale of Elasticsearch32.5.2.1.4. Expose Elasticsearch as a Route

32.5.3. Fluentd32.5.4. Kibana32.5.5. Curator

32.5.5.1. Creating the Curator Configuration32.6. CLEANUP32.7. TROUBLESHOOTING KIBANA32.8. SENDING LOGS TO AN EXTERNAL ELASTICSEARCH INSTANCE32.9. SENDING LOGS TO AN EXTERNAL SYSLOG SERVER32.10. PERFORMING ADMINISTRATIVE ELASTICSEARCH OPERATIONS32.11. REDEPLOYING EFK CERTIFICATES32.12. CHANGING THE AGGREGATED LOGGING DRIVER32.13. MANUAL ELASTICSEARCH ROLLOUTS

32.13.1. Performing an Elasticsearch Rolling Cluster Restart32.13.2. Performing an Elasticsearch Full Cluster Restart

569569570571

572572573

575575576576

578

579579579580580

581581581581

584

585585585586595596596596598599601602602603614615617618618

620620623624625626627628

Table of Contents

17

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTER 33. AGGREGATE LOGGING SIZING GUIDELINES33.1. OVERVIEW33.2. INSTALLATION

33.2.1. Large Clusters33.3. SYSTEMD-JOURNALD AND RSYSLOG33.4. SCALING UP EFK LOGGING33.5. STORAGE CONSIDERATIONS

CHAPTER 34. ENABLING CLUSTER METRICS34.1. OVERVIEW34.2. BEFORE YOU BEGIN34.3. METRICS PROJECT34.4. METRICS DATA STORAGE

34.4.1. Persistent Storage34.4.2. Capacity Planning for Cluster Metrics

Known Issues and Limitations34.4.3. Non-Persistent Storage

34.5. METRICS ANSIBLE ROLE34.5.1. Specifying Metrics Ansible Variables34.5.2. Using Secrets

34.5.2.1. Providing Your Own Certificates34.6. DEPLOYING THE METRIC COMPONENTS

34.6.1. Metrics Diagnostics34.7. SETTING THE METRICS PUBLIC URL34.8. ACCESSING HAWKULAR METRICS DIRECTLY

34.8.1. OpenShift Container Platform Projects and Hawkular Tenants34.8.2. Authorization

34.9. SCALING OPENSHIFT CONTAINER PLATFORM CLUSTER METRICS PODS34.10. INTEGRATION WITH AGGREGATED LOGGING34.11. CLEANUP34.12. PROMETHEUS ON OPENSHIFT CONTAINER PLATFORM

34.12.1. Setting Prometheus Role Variables34.12.2. Deploying Prometheus Using Ansible Installer

34.12.2.1. Additional Methods for Deploying Prometheus34.12.2.2. Accessing the Prometheus Web UI34.12.2.3. Configuring Prometheus for OpenShift Container Platform

34.12.3. OpenShift Container Platform Metrics via Prometheus34.12.3.1. Current Metrics

34.12.4. Undeploying Prometheus

CHAPTER 35. CUSTOMIZING THE WEB CONSOLE35.1. OVERVIEW35.2. LOADING EXTENSION SCRIPTS AND STYLESHEETS

35.2.1. Setting Extension Properties35.3. EXTENSION OPTION FOR EXTERNAL LOGGING SOLUTIONS35.4. CUSTOMIZING AND DISABLING THE GUIDED TOUR35.5. CUSTOMIZING DOCUMENTATION LINKS35.6. CUSTOMIZING THE LOGO35.7. CUSTOMIZING THE MEMBERSHIP WHITELIST35.8. CHANGING LINKS TO DOCUMENTATION35.9. ADDING OR CHANGING LINKS TO DOWNLOAD THE CLI

35.9.1. Customizing the About Page35.10. CONFIGURING NAVIGATION MENUS

630630630632632633633

635635635635635636636638638638639643643643644644645645646646646646647647648649649649650650652

653653653654655655655655656656656657658


18

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35.10.1. Top Navigation Dropdown Menus35.10.2. Application Launcher35.10.3. System Status Badge35.10.4. Project Left Navigation

35.11. CONFIGURING FEATURED APPLICATIONS35.12. CONFIGURING CATALOG CATEGORIES35.13. CONFIGURING QUOTA NOTIFICATION MESSAGES35.14. CONFIGURING THE CREATE FROM URL NAMESPACE WHITELIST35.15. DISABLING THE COPY LOGIN COMMAND

35.15.1. Enabling Wildcard Routes35.16. CUSTOMIZING THE LOGIN PAGE

35.16.1. Example Usage35.17. CUSTOMIZING THE OAUTH ERROR PAGE35.18. CHANGING THE LOGOUT URL35.19. CONFIGURING WEB CONSOLE CUSTOMIZATIONS WITH ANSIBLE35.20. CHANGING THE WEB CONSOLE URL PORT AND CERTIFICATES

CHAPTER 36. DEPLOYING EXTERNAL PERSISTENT VOLUME PROVISIONERS36.1. OVERVIEW36.2. BEFORE YOU BEGIN

36.2.1. External Provisioners Ansible Role36.2.2. External Provisioners Ansible Variables36.2.3. AWS EFS Provisioner Ansible Variables

36.3. DEPLOYING THE PROVISIONERS36.3.1. Deploying the AWS EFS Provisioner

36.3.1.1. AWS EFS Object Definition36.4. CLEANUP

658659659660661

662663664664664664665665665666667

668668668668668669670670670671

Table of Contents

19


20

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/developer_guide/#dev-guide-secrets

CHAPTER 1. OVERVIEWOpenShift Container Platform Installation and Configuration topics cover the basics of installing andconfiguring OpenShift Container Platform in your environment. Configuration, management, andlogging are also covered. Use these topics for the one-time tasks required to quickly set up yourOpenShift Container Platform environment and configure it based on your organizational needs.

For day to day cluster administration tasks, see Cluster Administration .

CHAPTER 1. OVERVIEW

21

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-index

CHAPTER 2. INSTALLING A CLUSTER

2.1. PLANNING

2.1.1. Initial Planning

For production environments, several factors influence installation. Consider the following questions asyou read through the documentation:

Which installation method do you want to use? The Installation Methods section provides someinformation about the quick and advanced installation methods.

How many pods are required in your cluster? The Sizing Considerations section provides limitsfor nodes and pods so you can calculate how large your environment needs to be.

How many hosts do you require in the cluster? The Environment Scenarios section providesmultiple examples of Single Master and Multiple Master configurations.

Is high availability required? High availability is recommended for fault tolerance. In this situation,you might aim to use the Multiple Masters Using Native HA example as a basis for yourenvironment.

Which installation type do you want to use: RPM or containerized? Both installations provide aworking OpenShift Container Platform environment, but you might have a preference for aparticular method of installing, managing, and updating your services.

Which identity provider do you use for authentication? If you already use a supported identityprovider, it is a best practice to configure OpenShift Container Platform to use that identityprovider during advanced installation.

Is my installation supported if integrating with other technologies? See the OpenShift ContainerPlatform Tested Integrations for a list of tested integrations.

2.1.2. Installation Methods

IMPORTANT

As of OpenShift Container Platform 3.9, the quick installation method is deprecated. In afuture release, it will be removed completely. In addition, using the quick installer toupgrade from version 3.7 to 3.9 is not supported.

Both the quick and advanced installation methods are supported for development and productionenvironments. If you want to quickly get OpenShift Container Platform up and running to try out for thefirst time, use the quick installer and let the interactive CLI guide you through the configuration optionsrelevant to your environment.

For the most control over your cluster’s configuration, you can use the advanced installation method.This method is particularly suited if you are already familiar with Ansible. However, following along withthe OpenShift Container Platform documentation should equip you with enough information to reliablydeploy your cluster and continue to manage its configuration post-deployment using the providedAnsible playbooks directly.

If you install initially using the quick installer, you can always further tweak your cluster’s configurationand adjust the number of hosts in the cluster using the same installer tool. If you wanted to later switch


22

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-high-availabilityhttps://access.redhat.com/articles/2176281

to using the advanced method, you can create an inventory file for your configuration and carry on thatway.

2.1.3. Sizing Considerations

Determine how many nodes and pods you require for your OpenShift Container Platform cluster.Cluster scalability correlates to the number of pods in a cluster environment. That number influences theother numbers in your setup. See Cluster Limits for the latest limits for objects in OpenShift ContainerPlatform.

2.1.4. Environment Scenarios

This section outlines different examples of scenarios for your OpenShift Container Platformenvironment. Use these scenarios as a basis for planning your own OpenShift Container Platform cluster,based on your sizing needs.

NOTE

Moving from a single master cluster to multiple masters after installation is notsupported.

For information on updating labels, see Updating Labels on Nodes.

2.1.4.1. Single Master and Node on One System

OpenShift Container Platform can be installed on a single system for a development environment only.An all-in-one environment is not considered a production environment.

2.1.4.2. Single Master and Multiple Nodes

The following table describes an example environment for a single master (with etcd installed on thesame host) and two nodes:

Host Name Infrastructure Component to Install

master.example.com Master, etcd, and node

node1.example.com Node

node2.example.com

2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes

The following table describes an example environment for a single master, three etcd hosts, and twonodes:


master.example.com Master and node


23

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-cluster-limitshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#updating-labels-on-nodeshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#node

etcd1.example.com etcd

etcd2.example.com

etcd3.example.com


node2.example.com


2.1.4.4. Multiple Masters Using Native HA with Co-located Clustered etcd

The following describes an example environment for three masters with co-located clustered etcd, oneHAProxy load balancer, and two nodes using the native HA method:


master1.example.com Master (clustered using native HA) and node andclustered etcd

master2.example.com

master3.example.com

lb.example.com HAProxy to load balance API master endpoints


node2.example.com

2.1.4.5. Multiple Masters Using Native HA with External Clustered etcd

The following describes an example environment for three masters, one HAProxy load balancer, threeexternal clustered etcd hosts, and two nodes using the native HA method:


master1.example.com Master (clustered using native HA) and node

master2.example.com

master3.example.com

lb.example.com HAProxy to load balance API master endpoints


24

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#node

etcd1.example.com Clustered etcd

etcd2.example.com

etcd3.example.com


node2.example.com


2.1.4.6. Stand-alone Registry

You can also install OpenShift Container Platform to act as a stand-alone registry using the OpenShiftContainer Platform’s integrated registry. See Installing a Stand-alone Registry for details on thisscenario.

2.1.5. RPM Versus Containerized

An RPM installation installs all services through package management and configures services to runwithin the same user space, while a containerized installation installs services using container images andruns separate services in individual containers.

See the Installing on Containerized Hosts topic for more details on configuring your installation to usecontainerized services.

2.2. PREREQUISITES

2.2.1. System Requirements

The following sections identify the hardware specifications and system-level requirements of all hostswithin your OpenShift Container Platform environment.

2.2.1.1. Red Hat Subscriptions

You must have an active OpenShift Container Platform subscription on your Red Hat account toproceed. If you do not, contact your sales representative for more information.

2.2.1.2. Minimum Hardware Requirements

The system requirements vary per host type:


25

MastersPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: RHEL 7.3 or later with the "Minimal" installation option and the latestpackages from the Extras channel, or RHEL Atomic Host 7.4.5 or later.

Minimum 4 vCPU (additional are strongly recommended).

Minimum 16 GB RAM (additional memory is strongly recommended, especially if etcdis co-located on masters).

Minimum 40 GB hard disk space for the file system containing /var/.

Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

Minimum 1 GB hard disk space for the file system containing the system’s temporary

directory.

Masters with a co-located etcd require a minimum of 4 cores. 2 core systems will notwork.

NodesPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: link:RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host7.4.5 or later.

NetworkManager 1.0 or later.

1 vCPU.

Minimum 8 GB RAM.

Minimum 15 GB hard disk space for the file system containing /var/.

Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

Minimum 1 GB hard disk space for the file system containing the system’s temporary

directory.

An additional minimum 15 GB unallocated space per system running containers forDocker’s storage back end; see Configuring Docker Storage. Additional space mightbe required, depending on the size and number of containers that run on the node.

ExternaletcdNodes

Minimum 20 GB hard disk space for etcd data.

See the Hardware Recommendations section of the CoreOS etcd documentation forinformation how to properly size your etcd nodes.

Currently, OpenShift Container Platform stores image, build, and deploymentmetadata in etcd. You must periodically prune old resources. If you are planning toleverage a large number of these resources, place etcd on machines with largeamounts of memory and fast SSD drives.


26

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/installation_guide/indexhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/installation_and_configuration_guide/https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/installation_guide/indexhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/installation_and_configuration_guide/https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendationshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-pruning-resources

AnsibleController

The host that you run the Ansible playbook on must have at least 75MiB of free memory perhost in the inventory.

Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires making changesto the default configuration. See Managing Storage with Docker-formatted Containers for instructionson configuring this during or after installation.

The system’s temporary directory is determined according to the rules defined in the tempfilemodule in Python’s standard library.

IMPORTANT

OpenShift Container Platform only supports servers with x86_64 architecture.

You must configure storage for each system that runs a container daemon. For containerizedinstallations, you need storage on masters. Also, by default, the web console is run in containers onmasters, and storage is needed on masters to run the web console. Containers are run on nodes, sostorage is always required on the nodes. The size of storage depends on workload, number ofcontainers, the size of the containers being run, and the containers' storage requirements. Containerizedetcd also needs container storage configured.

2.2.1.3. Production Level Hardware Requirements

Test or sample environments function with the minimum requirements. For production environments,the following recommendations apply:

Master Hosts

In a highly available OpenShift Container Platform cluster with external etcd, a master host shouldhave, in addition to the minimum requirements in the table above, 1 CPU core and 1.5 GB of memoryfor each 1000 pods. Therefore, the recommended size of a master host in an OpenShift ContainerPlatform cluster of 2000 pods would be the minimum requirements of 2 CPU cores and 16 GB ofRAM, plus 2 CPU cores and 3 GB of RAM, totaling 4 CPU cores and 19 GB of RAM.

A minimum of three etcd hosts and a load-balancer between the master hosts are required.

See Recommended Practices for OpenShift Container Platform Master Hosts for performanceguidance.

Node Hosts

The size of a node host depends on the expected size of its workload. As an OpenShift ContainerPlatform cluster administrator, you will need to calculate the expected workload, then add about 10percent for overhead. For production environments, allocate enough resources so that a node hostfailure does not affect your maximum capacity.

For more information, see Sizing Considerations and Cluster Limits.

IMPORTANT

Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.


27

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/managing_containers/managing_storage_with_docker_formatted_containershttps://docs.python.org/2/library/tempfile.html#tempfile.tempdirhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-capacity-host-practices-masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-cluster-limitshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#disabling-swap-memory

2.2.1.4. Storage management

Table 2.1. The main directories to which OpenShift Container Platform components write data

Directory Notes Sizing Expected Growth

/var/lib/openshift Used for etcd storageonly when in singlemaster mode and etcd isembedded in theatomic-openshift-master process.

Less than 10GB. Will grow slowly with theenvironment. Onlystoring metadata.

/var/lib/etcd Used for etcd storagewhen in Multi-Mastermode or when etcd ismade standalone by anadministrator.

Less than 20 GB. Will grow slowly with theenvironment. Onlystoring metadata.

/var/lib/docker When the run time isdocker, this is the mountpoint. Storage used foractive containerruntimes (includingpods) and storage oflocal images (not usedfor registry storage).Mount point should bemanaged by docker-storage rather thanmanually.

50 GB for a Node with16 GB memory.

Additional 20-25 GB forevery additional 8 GB ofmemory.

Growth is limited by thecapacity for runningcontainers.

/var/lib/containers When the run time isCRI-O, this is the mountpoint. Storage used foractive containerruntimes (includingpods) and storage oflocal images (not usedfor registry storage).

50 GB for a Node with16 GB memory.

Additional 20-25 GB forevery additional 8 GB ofmemory.

Growth limited bycapacity for runningcontainers

/var/lib/origin/openshift.local.volumes

Ephemeral volumestorage for pods. Thisincludes anythingexternal that is mountedinto a container atruntime. Includesenvironment variables,kube secrets, and datavolumes not backed bypersistent storage PVs.

Varies Minimal if pods requiringstorage are usingpersistent volumes. Ifusing ephemeralstorage, this can growquickly.


28

/var/log Log files for allcomponents.

10 to 30 GB. Log files can growquickly; size can bemanaged by growingdisks or managed usinglog rotate.

Directory Notes Sizing Expected Growth

2.2.1.5. Red Hat Gluster Storage Hardware Requirements

Any nodes used in a Container-Native Storage or Container-Ready Storage cluster are consideredstorage nodes. Storage nodes can be grouped into distinct cluster groups, though a single node can notbe in multiple groups. For each group of storage nodes:

A minimum of three storage nodes per group is required.

Each storage node must have a minimum of 8 GB of RAM. This is to allow running the Red HatGluster Storage pods, as well as other applications and the underlying operating system.

Each GlusterFS volume also consumes memory on every storage node in its storage cluster,which is about 30 MB. The total amount of RAM should be determined based on how manyconcurrent volumes are desired or anticipated.

Each storage node must have at least one raw block device with no present data or metadata.These block devices will be used in their entirety for GlusterFS storage. Make sure the followingare not present:

Partition tables (GPT or MSDOS)

Filesystems or residual filesystem signatures

LVM2 signatures of former Volume Groups and Logical Volumes

LVM2 metadata of LVM2 physical volumes

If in doubt, wipefs -a should clear any of the above.

IMPORTANT

It is recommended to plan for two clusters: one dedicated to storage for infrastructureapplications (such as an OpenShift Container Registry) and one dedicated to storage forgeneral applications. This would require a total of six storage nodes. Thisrecommendation is made to avoid potential impacts on performance in I/O and volumecreation.

2.2.1.6. Optional: Configuring Core Usage

By default, OpenShift Container Platform masters and nodes use all available cores in the system theyrun on. You can choose the number of cores you want OpenShift Container Platform to use by settingthe GOMAXPROCS environment variable. See the Go Language documentation for more information,including how the GOMAXPROCS environment variable works.


29

https://golang.org/pkg/runtime/#GOMAXPROCS

For example, run the following before starting the server to make OpenShift Container Platform onlyrun on one core:

# export GOMAXPROCS=1

2.2.1.7. SELinux

Security-Enhanced Linux (SELinux) must be enabled on all of the servers before installing OpenShiftContainer Platform or the installer will fail. Also, configure SELINUX=enforcing and SELINUXTYPE=targeted in the /etc/selinux/config file:

# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=enforcing# SELINUXTYPE= can take one of these three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPE=targeted

2.2.1.8. Red Hat Gluster Storage

To access GlusterFS volumes, the mount.glusterfs command must be available on all schedulablenodes. For RPM-based systems, the glusterfs-fuse package must be installed:

# yum install glusterfs-fuse

This package comes installed on every RHEL system. However, it is recommended to update to thelatest available version from Red Hat Gluster Storage. To do this, the following RPM repository must beenabled:

# subscription-manager repos --enable=rh-gluster-3-client-for-rhel-7-server-rpms

If glusterfs-fuse is already installed on the nodes, ensure that the latest version is installed:

# yum update glusterfs-fuse

Optional: Using OverlayFSOverlayFS is a union file system that allows you to overlay one file system on top of another.

As of Red Hat Enterprise Linux 7.4, you have the option to configure your OpenShift Container Platformenvironment to use OverlayFS. The overlay2 graph driver is fully supported in addition to the older overlay driver. However, Red Hat recommends using overlay2 instead of overlay, because of its speedand simple implementation.

Comparing the Overlay Versus Overlay2 Graph Drivers has more information about the overlay andoverlay2 drivers.

See the Overlay Graph Driver section of the Atomic Host documentation for instructions on how toenable the overlay2 graph driver for the Docker service.


30

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#comparing-overlay-graph-drivershttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/managing_containers/#using_the_overlay_graph_driver

2.2.1.9. Security Warning

OpenShift Container Platform runs containers on hosts in the cluster, and in some cases, such as buildoperations and the registry service, it does so using privileged containers. Furthermore, those containersaccess the hosts' Docker daemon and perform docker build and docker push operations. As such,cluster administrators should be aware of the inherent security risks associated with performing docker run operations on arbitrary images as they effectively have root access. This is particularly relevant for docker build operations.

Exposure to harmful containers can be limited by assigning specific builds to nodes so that any exposureis limited to those nodes. To do this, see the Assigning Builds to Specific Nodes section of theDeveloper Guide. For cluster administrators, see the Configuring Global Build Defaults and Overridessection of the Installation and Configuration Guide.

You can also use security context constraints to control the actions that a pod can perform and what ithas the ability to access. For instructions on how to enable images to run with USER in the Dockerfile,see Managing Security Context Constraints (requires a user with cluster-admin privileges).

For more information, see these articles:

http://opensource.com/business/14/7/docker-security-selinux

https://docs.docker.com/engine/security/security/

2.2.2. Environment Requirements

The following section defines the requirements of the environment containing your OpenShiftContainer Platform configuration. This includes networking considerations and access to externalservices, such as Git repository access, storage, and cloud infrastructure providers.

2.2.2.1. DNS

OpenShift Container Platform requires a fully functional DNS server in the environment. This is ideally aseparate host running DNS software and can provide name resolution to hosts and containers runningon the platform.

IMPORTANT

Adding entries into the /etc/hosts file on each host is not enough. This file is not copiedinto containers running on the platform.

Key components of OpenShift Container Platform run themselves inside of containers and use thefollowing process for name resolution:

1. By default, containers receive their DNS configuration file (/etc/resolv.conf) from their host.

2. OpenShift Container Platform then inserts one DNS value into the pods (above the node’snameserver values). That value is defined in the /etc/origin/node/node-config.yaml file by thednsIP parameter, which by default is set to th

OpenShift Container Platform 3 - Red Hat Customer …...2.6.4. Example Inventory Files 2.6.4.1....

Documents

Transcript of OpenShift Container Platform 3 - Red Hat Customer …...2.6.4. Example Inventory Files 2.6.4.1....