Post on 26-Jul-2018
OpenShift supports Persistent Storage
AUTOMATED CONFIGURATION
SINGLE CONTROL PANEL
CHOICE OF PERSISTENT STORAGE
Ceph RBD
Amazon EBS
Azure Disk
GCE Disk
iSCSI
NFS
GlusterFS
GlusterFS Support in OpenShift
CONTAINER NODE 1
CONTAINER NODE 2
CONTAINER NODE 3
CONTAINER NODE 4
NGINX Container
NGINX Container
NGINX Container
NGINX Container
Scalable Highly-Available Automated Elastic
● Red Hat Gluster Storage runs inside OpenShift in a container (in kubernetes pods)
● Application and storage containers can be co-located
● There’s exactly one RHGS container per host
● Uses host networking, not overlay
● Use OpenShift templates to deploy
● Use a dynamic volume allocator (heketi) to automate volume creation
Launch Container-Native Storage July 2016
VOLUMEA namespace
presented as a POSIX mount point and is
comprised of bricks.
BRICK
The basic unit of storage, represented by an export directory on a
server
SERVER/NODES
Contain the bricks
GLUSTER STORAGE CONCEPTS
GAJAH ANNUAL REPORT 2015 |
BRICKS
/export1
/export2
/export3
/export4
/export5
/export6
/export7
/export8
/export9
/export10
/export11
/export12
/export13
/export14
/export15
STORAGE NODE STORAGE NODE STORAGE NODE
A Brick is the combination of a node and file system (hostname:/dir)
Each brick inherits limits of underlying file system (XFS)
Red Hat Gluster Storage operates at the brick level, not the node level
Ideally, each brick in a volume should be the same size
GAJAH ANNUAL REPORT 2015 |
ELASTIC HASH ALGORITHM
#
BRICK BRICK
0 1 1 0 1 0 1 0
No Central Metadata Server• Suitable for unstructured data storage• No single point of failure
Elastic Hashing• Files assigned to virtual volumes• Virtual volumes assigned to multiple bricks• Volumes easily reassigned on-the-flyLocation Hashed on Filename• No performance bottleneck• Eliminates risk scenarios
GAJAH ANNUAL REPORT 2015 |
VOLUMES
Bricks taken from multiple hostsbecome one addressable unit
High availability as needed
Load balanced data
Managed by Gluster
CLIENTS
SERVER SERVER
BRICK BRICK BRICK BRICK
VOLUMES
● CNS: providing dynamic persistent storage for openshift with GlusterFS in a hyper-converged fashion
● Heketi: the high-level service interface to gluster to manage the lifecycle of volumes in multiple Gluster clusters.
Openshift ⇐⇒ Heketi ⇐⇒ Gluster
What is Container native Storage … and what is Heketi?
● openshift/kubernetes○ dynamic glusterfs provisioner○ glusterfs plugin
● Heketi○ high-level service interface for gluster volume lifecycle
management● glusterfs:
○ one or more glusterfs clusters○ running hyper-converged in openshift
● cns-deploy:○ tool to deploy gluster and heketi into an existing openshift
cluster
Components of CNS
● pod: group of one or more containers that form an entity● persistent volume (PV): to be mounted by application pod● provisioner: to provide PVs upon request● plugin: mechanism to mount the PV, referenced in PV● persistent volume claim (PVC): mechanism for a user to
request a PV● Access types for volumes:
○ RWO - read write once (single node)○ RWX - read write many (multiple nodes)○ ROX - read only many (multiple nodes)
Persistent Storage in OpenShift
● gluster running as pods● aggregating host's disk devices into volumes● gluster storage nodes tied to openshift nodes (disk devices...)● gluster nodes on some, not necessarily on all openshift worker nodes● application pods can be on the openshift nodes (need extra memory!)
gluster cluster in openshift
● containerized systemd○ (running multiple processes: glusterd, brick servers, ...)
● privileged container (systemd, access host /dev, ...)● startup script● host /dev bind-mounted● bind-mount config
○ /etc/glusterfs, /var/lib/glsuterd, /var/log/glusterfs● use host network:
○ gluster ips need to be constant○ gluster config tied to the node○ performance
about the gluster containers
● high-level service interface for managing the lifecycle of gluster volumes● RESTful API and cli ("heketi-cli")● manages one or several gluster clusters● can create, expand, delete volumes (more coming)● hides nitty gritty details of volume creation from caller● just takes size and desired durability type
○ (currently only replicate is supported in CNS)● automatically finds cluster and disks to satisfy the request● stores its state in a database (currently Bolt)● https://github.com/heketi/heketi
about heketi