THE POWER OF RED HAT CEPH STORAGE - OpenStack · THE POWER OF RED HAT CEPH STORAGE Jean-Charles...

THE POWER OF RED HAT CEPH STORAGE

Jean-Charles Lopez S. Technical Instructor, Global Storage Consulting Practice Red Hat, Inc.

[email protected]

May 2017 – OpenStack Summit, Boston

And how it’s essential to your OpenStack environment

STORAGE CONCEPTS

3

DIFFERENT KINDS OF STORAGE

FILE STORAGE File systems allow users to organize data stored in blocks using hierarchical folders and files.

OBJECT STORAGE Object stores distribute data algorithmically throughout a cluster of media, without a rigid structure.

BLOCK STORAGE Physical storage media appears to computers as a series of sequential blocks of a uniform size.

4

REPLICATION VS ERASURE CODING

OBJECT OBJECT

COPY COPY COPY

REPLICATED POOL

CEPH STORAGE CLUSTER CEPH STORAGE CLUSTER

1 2 3 4

ERASURE CODED POOL

X Y

FULL COPIES OF STORED OBJECTS • Very high durability • Quicker recovery • Performance optimized

ONE COPY PLUS PARITY • Cost-effective durability • Expensive recovery • Capacity optimized

RED HAT CEPH STORAGE ARCHITECTURAL OVERVIEW

6

RED HAT CEPH STORAGE ARCHITECTURAL COMPONENTS

RBD A reliable, fully distributed block device

with cloud platform integration

RGW A web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

LIBRADOS A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOS A software-based reliable, autonomous, distributed object store comprised of self-

healing, self-managing, intelligent storage nodes and lightweight monitors

CEPHFS* A distributed file system with POSIX

semantics & scale-out metadata

CLIENT

* CephFS is Tech Preview in RHCS2

7






APP HOST/VM


Reliable Autonomous Distributed Object Store Software-based, comprised of self-healing, self-managing, intelligent storage nodes and

lightweight monitors



CLIENT


RADOS CLUSTER

RADOS CLUSTER

9

OBJECT STORAGE DAEMONS (OSDs)

OSD

FS

DISK

OSD

FS

DISK

OSD

FS

DISK

OSD

FS

DISK

OSDs •  10s to 10000s in a cluster •  One per disk (SSD, SAS, SATA, …) •  Serve stored objects to clients •  Minimum 3 per cluster

10

MONITORS (MONs)

Monitors •  Maintain cluster membership and state •  Track health of the cluster •  Provide consensus for distributed decision-making •  Small, odd number •  These do not serve stored objects to clients •  Minimum 3 per cluster

WHERE DO OBJECTS LIVE?

APPLICATION

OBJECTS

RADOS CLUSTER

12

INTRODUCTION TO CEPH DATA PLACEMENT : POOLS & PLACEMENT GROUPS

10

10

10 10

10 01

01

01 01

10

01 01

01 11

01 11

POOL A

POOL B

POOL C

POOL D

10 11 10 01

10 11 10 01

10 11 10 01

10 11 10 01

10 11 10 01

01 10 11 10

10 01 01 01

01 10 11 10

CONTROLLED REPLICATION UNDER SCALABLE HASHING

OBJECTS

CLUSTER

10

01

11

PLACEMENT GROUPS

10

01

10

10

10 10

10 01

01

01 01

10

01 01

01 11

01 11

10

01

01

10

10

10 10

10 01

01

01 01

10

01 01

01 11

01 11

14

CRUSH IS A QUICK CALCULATION

OBJECTS

CLUSTER

CRUSH - DYNAMIC DATA PLACEMENT

CRUSH

• Pseudo-random placement algorithm • Fast calculation, no lookup • Repeatable, deterministic

• Statistically uniform distribution • Stable mapping

• Limited data migration on change • Rule-based configuration

• Infrastructure topology aware • Adjustable replication • Weighting

16






APP HOST/VM






CLIENT


ACCESSING A RADOS CLUSTER

LIBRADOS

APPLICATION

OBJECTS

Socket

RADOS CLUSTER

18






APP HOST/VM






CLIENT


THE RADOS GATEWAY (RGW)

RADOSGW RADOSGW

LIBRADOS

REST

Socket

LIBRADOS

RADOS CLUSTER

20






APP HOST/VM






CLIENT


21

STORING VIRTUAL DISKS VM

HYPERVISOR

LIBRBD

RADOS CLUSTER

22

SEPARATE COMPUTE FROM STORAGE

VM HYPERVISOR

LIBRBD

HYPERVISOR

LIBRBD

RADOS CLUSTER

23

KERNEL MODULE FOR MAX FLEXIBILITY

LINUX HOST KRBD

RADOS CLUSTER

24






APP HOST/VM






CLIENT


25

CEPHFS* - SEPARATE METADATA SERVER

KERNEL MODULE

LINUX HOST

DATA METADATA

RADOS CLUSTER


RED HAT CEPH STORAGE OPENSTACK USE CASE

RED HAT CEPH STORAGE AND OPENSTACK

KEYSTONE SWIFT CINDER NOVA GLANCE

RADOSGW LIBRADOS LIBRBD

OPENSTACK

HYPERVISOR

RADOS CLUSTER

MANILA

CEPHFS*


RED HAT CEPH STORAGE DOING IT!

RED HAT CEPH STORAGE RBD & GLANCE On ceph admin node, run: ceph osd pool create {pool_name} 2x

ceph auth get-or-create {user_name} ... –o {keyring_file}scp {keyring_file} {unix_user}@{glance_node}:{path} <- Provide read permission for Glancescp /etc/ceph/ceph.conf {unix_user}@{glance_node}:{path} <- Provide read permission for Glance

Add the following to /etc/ceph/ceph.conf on Glance node [{user_name}]keyring = {path}

Edit /etc/glance/glance-api.conf on Glance node ... [glance_store] stores = rbd default_store = rbd show_image_direct_url = true rbd_store_user = {user_id} <- If user name is client.{id}, use {id} rbd_store_pool = {pool_name} rbd_store_ceph_conf = {Ceph configuration file path} rbd_store_chunk_size = {integer} <- Uses 8 by default for 8MB object RBDsflavor = keystone

Restart Glance services

RED HAT CEPH STORAGE RBD & CINDER On ceph admin node, run: ceph osd pool create {pool_name} 2x

ceph auth get-or-create {user_name} ... –o {keyring_file}scp {keyring_file} {unix_user}@{cinder_node}:{path} <- Provide read permission for Cinderscp /etc/ceph/ceph.conf {unix_user}@{cinder_node}:{path} <- Provide read permission for Cinder

Add the following to /etc/ceph/ceph.conf on Cinder node [{user_name}]keyring = {path}

Edit /etc/cinder/cinder.conf on Cinder node. Note that you can create multiple storage backends ... [cinder_backend_name]volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf = {Ceph configuration file path} rbd_pool = {pool_name} rbd_secret_uuid = {UUID} rbd_user = {ceph_userid}

Restart Cinder services

RED HAT CEPH STORAGE RBD & LIBVIRT Create a file with on compute node <secret ephemeral=”no” private=”no”> <uuid>{UUID}</uuid> <usage type=”ceph”> <name>{username} secret</name> </usage> </secret>

Run command virsh secret-define --file ceph.xmlvirsh secret-set-value --secret {UUID} -base64 $(cat {ceph_user_name}.key)*

Synchronize libvirt secrets across compute nodes

RED HAT CEPH STORAGE RBD & NOVA Edit /etc/nova/nova.conf on Nova nodes [libvirt] libvirt_images_type = rbd libvirt_images_rbd_pool = {pool_name} libvirt_images_rbd_ceph_conf = {Ceph configuration file path} libvirt_disk_cachemodes = "network=writeback" rbd_secret_uuid = {UUID} rbd_user = {ceph_userid}

Restart Nova services

RED HAT CEPH STORAGE VM TROUBLESHOOTING

On each compute node, make sure your /etc/ceph/ceph.conf file has [client.{user_name}]admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok log file = /var/log/qemu/qemu-guest-$pid.log

VMs need restart for changes to take effect ceph –admin-daemon /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok {command}

Useful commands help <- List available commandsperf dump <- Dump performance countersconfig show <- View all run time parametersconfig get {parameter} <- View the specific run time parameterconfig set {parameter} {value} <- Modify the specific run time parameter

RED HAT CEPH STORAGE SWIFT On OpenStack controler node, create a Swift service and endpoint openstack service create --name=swift --description="Swift Service" object-store openstack endpoint create --region RegionOne --publicurl "http://{rgw}:8080/swift/v1" \

--adminurl "http://{rgw}:8080/swift/v1” \ --internalurl "http://{rgw}:8080/swift/v1" swift

On your Keystone server, create an NSS database mkdir {certificate_directory} openssl x509 -in /etc/keystone/ssl/cert/ca.pem -pubkey | certutil -d {certificate_directory} \

-A -n ca -t "TCu,Cu,Tuw" openssl x509 -in /etc/keystone/ssl/cert/signing_cert.pem -pubkey |certutil -d {certificate_directory} \

-A -n signing_cert -t “TCu,Cu,Tuw”

On your Keystone server, copy the NSS database to the RADOS Gateway nodes scp –R {certificate_directory} {ceph_linux_user}@{rgw_node}:{certificate_directory}

RED HAT CEPH STORAGE SWIFT On your RADOS Gateway servers [{username}] rgw_keystone_url = http://a.b.c.d:{port} rgw_keystone_admin_user = {admin-user} rgw_keystone_admin_password = {admin-password} rgw_keystone_admin_tenant = {admin-tenant} rgw_keystone_accepted_roles = admin member swiftoperator rgw_keystone_token_cache_size = 200 rgw_keystone_revocation_interval = 300 nss_db_path = {certificate_directory}

Restart your RADOS Gateways

CLOSING TIME

WHERE TO GO TO ENGAGE

!  Red Hat Subscriptions: https://access.redhat.com/subscription-value

!  Evaluation, Pre-production, and Production contracts available through Red Hat sales

!  Red Hat Consulting: http://www.redhat.com/en/services/consulting/storage !  Red Hat Storage training: https://www.redhat.com/en/services/training

!  Red Hat Storage test-drive: http://red.ht/cephtestdrive

!  To engage a Territory Service Manager in your area, ask for a local Red Hat Storage sales professional at:

NORTH AMERICA: 1 (888) REDHAT-1; LATIN AMERICA: 54 (11) 4329-7300; EMEA: 00800 7334 2835

APJ: 65 6490 4200; Brazil: 55 (11) 3529-6000,; Australia: 1800 733 428; New Zealand: 0800 733 428

•  Web: www.redhat.com/storage

•  Blog: http://redhatstorage.redhat.com/

•  Twitter: www.twitter.com/redhatstorage

•  Facebook: www.facebook.com/RedHatStorage/

•  YouTube: www.youtube.com/user/redhatstorage

•  Slideshare: www.slideshare.net/Red_Hat_Storage

WHERE TO GO ON SOCIAL MEDIA

SERVICES AVAILABLE FROM RED HAT for Red Hat Ceph Storage & Red Hat OpenStack Platform

•  Subscription Benefits – Streamlined product and resource access, lifecycle management, Knowledgebase and deployment resources, security and accountability

•  Jumpstart - 3 day consulting for pre-production cluster implementation at customer site

•  Health Check - 3-5 day evaluation of total solution best practice engagement

•  Red Hat Storage Consulting Services - recommended for initial storage design and set-up

•  Global Services and Support - production subscriptions for smooth lifecycle deployments

•  Red Hat Global Learning Services – 2 and 5 day for storage or cloud admins

THANK YOU

THE POWER OF RED HAT CEPH STORAGE - OpenStack · THE POWER OF RED HAT CEPH STORAGE Jean-Charles...

Documents

Transcript of THE POWER OF RED HAT CEPH STORAGE - OpenStack · THE POWER OF RED HAT CEPH STORAGE Jean-Charles...