OpenShift + Container Native Storage (CNS)

31 May 2017

Michael Holzerland, Solution Architect

OpenShift supports Persistent Storage

AUTOMATED CONFIGURATION

SINGLE CONTROL PANEL

CHOICE OF PERSISTENT STORAGE

Ceph RBD

Amazon EBS

Azure Disk

GCE Disk

iSCSI

NFS

GlusterFS

GlusterFS Support in OpenShift

CONTAINER NODE 1

CONTAINER NODE 2

CONTAINER NODE 3

CONTAINER NODE 4

NGINX Container

NGINX Container

NGINX Container

NGINX Container

Scalable Highly-Available Automated Elastic

● Red Hat Gluster Storage runs inside OpenShift in a container (in kubernetes pods)

● Application and storage containers can be co-located

● There’s exactly one RHGS container per host

● Uses host networking, not overlay

● Use OpenShift templates to deploy

● Use a dynamic volume allocator (heketi) to automate volume creation

Launch Container-Native Storage July 2016

RHGS is supported Everywhere!

Red Hat Confidential - NDA Required

RED HAT GLUSTER STORAGE

concepts

VOLUMEA namespace

presented as a POSIX mount point and is

comprised of bricks.

BRICK

The basic unit of storage, represented by an export directory on a

server

SERVER/NODES

Contain the bricks

GLUSTER STORAGE CONCEPTS

GAJAH ANNUAL REPORT 2015 |

BRICKS

/export1

/export2

/export3

/export4

/export5

/export6

/export7

/export8

/export9

/export10

/export11

/export12

/export13

/export14

/export15

STORAGE NODE STORAGE NODE STORAGE NODE

A Brick is the combination of a node and file system (hostname:/dir)

Each brick inherits limits of underlying file system (XFS)

Red Hat Gluster Storage operates at the brick level, not the node level

Ideally, each brick in a volume should be the same size

GAJAH ANNUAL REPORT 2015 |

ELASTIC HASH ALGORITHM

#

BRICK BRICK

0 1 1 0 1 0 1 0

No Central Metadata Server• Suitable for unstructured data storage• No single point of failure

Elastic Hashing• Files assigned to virtual volumes• Virtual volumes assigned to multiple bricks• Volumes easily reassigned on-the-flyLocation Hashed on Filename• No performance bottleneck• Eliminates risk scenarios

GAJAH ANNUAL REPORT 2015 |

VOLUMES

Bricks taken from multiple hostsbecome one addressable unit

High availability as needed

Load balanced data

Managed by Gluster

CLIENTS

SERVER SERVER

BRICK BRICK BRICK BRICK

VOLUMES

OpenShift + Storage - How it all works

● CNS: providing dynamic persistent storage for openshift with GlusterFS in a hyper-converged fashion

● Heketi: the high-level service interface to gluster to manage the lifecycle of volumes in multiple Gluster clusters.

Openshift ⇐⇒ Heketi ⇐⇒ Gluster

What is Container native Storage … and what is Heketi?

● openshift/kubernetes○ dynamic glusterfs provisioner○ glusterfs plugin

● Heketi○ high-level service interface for gluster volume lifecycle

management● glusterfs:

○ one or more glusterfs clusters○ running hyper-converged in openshift

● cns-deploy:○ tool to deploy gluster and heketi into an existing openshift

cluster

Components of CNS

● pod: group of one or more containers that form an entity● persistent volume (PV): to be mounted by application pod● provisioner: to provide PVs upon request● plugin: mechanism to mount the PV, referenced in PV● persistent volume claim (PVC): mechanism for a user to

request a PV● Access types for volumes:

○ RWO - read write once (single node)○ RWX - read write many (multiple nodes)○ ROX - read only many (multiple nodes)

Persistent Storage in OpenShift

Dynamic Provisioning (OCP 3.4)

Details on Containerization

● gluster running as pods● aggregating host's disk devices into volumes● gluster storage nodes tied to openshift nodes (disk devices...)● gluster nodes on some, not necessarily on all openshift worker nodes● application pods can be on the openshift nodes (need extra memory!)

gluster cluster in openshift

● containerized systemd○ (running multiple processes: glusterd, brick servers, ...)

● privileged container (systemd, access host /dev, ...)● startup script● host /dev bind-mounted● bind-mount config

○ /etc/glusterfs, /var/lib/glsuterd, /var/log/glusterfs● use host network:

○ gluster ips need to be constant○ gluster config tied to the node○ performance

about the gluster containers

● high-level service interface for managing the lifecycle of gluster volumes● RESTful API and cli ("heketi-cli")● manages one or several gluster clusters● can create, expand, delete volumes (more coming)● hides nitty gritty details of volume creation from caller● just takes size and desired durability type

○ (currently only replicate is supported in CNS)● automatically finds cluster and disks to satisfy the request● stores its state in a database (currently Bolt)● https://github.com/heketi/heketi

about heketi

Wrap Up & Q/A