600-0027NA - Key Design Considerations for Scalable OpenStack Block Storage (white paper)

SnapshotsDashboard Workloads Replication Usage Hardware

Key Design Considerations for Scalable OpenStack Block StorageBuilding for Scale with Coho DataStream

WHITE PAPER

For DataStream OS 2.7.0+ Document # 600-0027NAVersion 1.16 • April 2016

www.cohodata.com

http://www.cohodata.com

White PaperKey Design Considerations for Scalable OpenStack Block Storage

Contents

Abstract ..................................................................................................................................................3

Openstack Design Philosophy ..............................................................................................................3

The Common Challenges with Block Storage .....................................................................................5

Scalability ......................................................................................................................................5

Simplicity .......................................................................................................................................5

Performance .................................................................................................................................5

Coho DataStream Architecture ............................................................................................................6

Shared-Nothing Architecture ..........................................................................................................6

SDN-assisted Workload Distribution ..............................................................................................7

Automated Data Rebalance ............................................................................................................7

Elastic Scale-out ...............................................................................................................................7

How the Coho Data Cinder Driver Works ...........................................................................................8

Why a Coho Data Cinder Solution for OpenStack ..............................................................................9

Scalability - Enables Business Agility .............................................................................................9

Simplicity - Enables Operational Efficiency ....................................................................................9

Performance - Maximize Utilization of Storage Resources .........................................................9

Use Cases ............................................................................................................................................ 10

Conclusion ........................................................................................................................................... 11

Resources ............................................................................................................................................ 12

About the Author ................................................................................................................................ 12

600-0027NA • Key Design Considerations for Scalable OpenStack Block Storage • 02.7.1.16



AbstractOpenStack is a rapidly growing open source cloud platform supported by many vendor communities and individual contributors. According to the latest OpenStack October 2015 user survey, OpenStack is demonstrating its increasing maturity - with 60% of deployments in production; nearly doubled from their survey of less than two years ago.

Some of the top business drivers to adopt OpenStack are:

• Accelerate organization’s ability to innovate

• Enables business agility

• Increases operational efficiency to drive down expenses

OpenStack delivers the right cloud management framework or tools to enable your organization’s business objectives.

This paper provides key design considerations to build Scalable Block Storage for OpenStack cloud infrastructure to enable the organization’s business objectives. Additionally, how Coho Data addresses those requirements by way of an in depth description of the Coho DataStream architecture, with suitable storage driven OpenStack use cases.

Openstack Design PhilosophyScalability and elasticity are primary design goals for OpenStack. Key design concepts are also defined as:

• All required components must be horizontally scalable

• Shared nothing architecture or sharding

• Distribute everything

OpenStack is clearly different from the infrastructure built on virtualization technologies (e.g. VMware ESXi hypervisor and vSphere) in that virtualization supports consolidation on a small number of larger servers. The solution works well for the application hosted on a single virtual machine and grows by scaling up resources on a single physical server running the ESXi hypervisor. OpenStack or cloud computing technology is designed to be used with a different class of applications architected to scale horizontally and are resilient against virtual machine instance shutdowns. Although, recent OpenStack releases enhanced VM live migration to support some of the use cases. Resources can be expanded by adding more application instances and re-balancing workloads across those instances.

OpenStack components such as Compute, Networking, Storage are tied together with OpenStack shared service and managed through common API to allow the infrastructure to scale those component resources quickly in response to the application demands.

https://wiki.openstack.org/wiki/BasicDesignTenets


Figure 1: Logical OpenStack Architecture (Reference: openstack.org).


The following diagram shows logical OpenStack architecture with how the components are tied together.

In the example of OpenStack Compute (Nova), it is designed to run OpenStack instances in distributed fashion and easily scale by adding more Compute nodes running on hypervisors such as KVM, Xen, and VMware ESXi.

At high level:

1. When users request to provision a VM instance via the Horizon UI self-service portal or CLI, the request first goes to nova-api, which is RESTful API service that interacts with Nova.

2. Then nova-scheduler determines on which compute host to run the instance, based on various filters such as availability zones, host affinity, and host resources availability.

3. Finally, nova-compute, the worker daemon running on the selected Compute node, will create the instance.

Communications between the nova component services and other OpenStack components are going through a message queue. This enables OpenStack as component-based architecture for high availability and scalability at the component basis. When you need more compute resource, it allows you to add compute nodes on demand. The nova-compute daemon, running on new compute node, will seamlessly join the compute cluster and update Nova database.

Also, OpenStack has a flexibility allowing additional compute hosts with a different core count and RAM size. nova-scheduler will take care of differences in sizing, which enables new hardware to become part of the same infrastructure. It allows you to take advantage of Moore's Law on both a price and performance perspective, in addition to a simplified hardware life cycle process.



The Common Challenges with Block StorageCinder is a Block Storage Service for OpenStack, providing abstraction and automation on top of various back-end storage platform via a common OpenStack API. End users may provision storage resources, from self-service portal, without any knowledge of back-end storage platform.

It is important to understand how back-end storage works once cinder driver send the request. It depends on each vendor's storage architecture on how storage is laid out, presented to the instances.

Ceph RBD is often chosen as Cinder storage due to open source and it also comes with various OpenStack distributions. Many traditional storage vendors offer cinder drivers that work with their storage platforms and it is rational to leverage the storage platform you have, in your existing environment, when you’re experimenting OpenStack technology. However, if you’re building a production OpenStack cloud and would like block storage to work as defined as its philosophy, scalability, simplicity, and performance need to be considered.

These following storage design points are important, regardless of building an OpenStack cloud your own, working with system integrators, or larger vendors of OpenStack solution.

Scalability• Ability to seamlessly scale, in response to both capacity and performance demand,

as nova compute scales. The capability to scale performance is crucial because it is a common mistake to use scale-up storage in an infrastructure designed to support scalable distributed applications because storage will become a bottleneck at scale.

• Intelligently and dynamically distribute and load-balance workloads as it scales.

Simplicity• The process involved to configure and to grow, including physical cabling to logical

configuration and how the process affects an existing OpenStack infrastructure.

• Refreshing the hardware life cycle. Data migration with monolithic storage system is a

common challenge with traditional infrastructure.

Performance• Ability to scale performance linearly. In today's multi-tenant environments, it is a common

challenge to achieve linear performance on distributed storage system at large scale.

• Network optimization to avoid storage networking bottlenecks, at scale.


Figure 2: Coho DataStream Architecture.

• Workload Distribution (scheduler)

• Dynamic Load-balancing

• Data Placement Decision

• Single Namespace

• Single NFS IP Address

• Single API Endpoint


Coho DataStream ArchitectureCoho DataStream is a software-defined scalable distributed storage system consisting of following two primary components:

• X86 based commodity storage servers - It provides a balanced ratio of CPUs, 10GbE Networking, NVMe flash cards and high-density SSDs or HDDs. Certified 2U SuperMicro server chassis comes with two independent storage nodes called MicroArray works as single fault domain.

• OpenFlow-enabled SDN ToR switches - Low-latency Arista Network switches works as protocol virtualization layer allows transparently scale out across multiple real storage server nodes while maintaining single namespace consistency with single TCP endpoint. It provides dynamic and centralized control over data placement, load-balancing and workload distribution based on resource availability by leveraging OpenFlow controller.

Client hosts can be directly connected to the SDN ToR switches or through uplink to non-Coho Data managed spine switches. Currently certified Arista Network switch has four 40Gbs QSFP+ ports available for uplinks on each switch.

Shared-Nothing ArchitectureMicroArray node is independent, and self-sufficient storage server; each has own CPU’s, RAM and storage accessible from a pair of 10GbE Network Interface uplinked to redundant OpenFlow-enabled SDN switches. Those MicroArray nodes are clustered together with low-latency shared persistent memory and presented as globally accessible single namespace through the protocol virtualization layer in SDN switches.



SDN-assisted Workload DistributionThe protocol virtualization layer is tightly integrated with OpenFlow-enabled Ethernet switching fabric, allowing clients connecting to a single IP address while transparently and dynamically balancing traffic across 10GbE network interfaces on all of the MicroArray. This dynamic client connection migration or load-balancing is triggered by a software controller running on switch in response to application level heartbeat failure from MicroArray, or to rebalance based on client connection level throughput. The OpenFlow controller interact with the switch by pushing flow-specific rules onto the switch’s TCAM or L2/L3 forwarding tables for the associated connection migration action.

Automated Data RebalanceFile or volume is broken into multiple stripe units (objects) and stored across multiple storage devices and MicroArrays in fully balanced and distributed manner. Each stripe unit (object) is synchronously replicated to a different MicroArray, with a replication factor of two. The system periodically performs an automated object rebalance process in the background, based on capacity utilization and to improve load distribution across all MicroArrays. It helps to eliminate hot spots and improve overall performance to utilize all available storage resources. In the event of a failure, regardless of single SSD device or entire MicroArray node, it triggers system-wide reconstruction of lost objects. It accomplishes this by creating from its replicas on different MicroArrays in the system using the available storage space.

Elastic Scale-outCoho DataStream allows dynamically scaling performance and capacity to be as simple as plug-and-play. A new MicroArray will first boot off from the controller software on SDN switch when connected and automatically joins to the cluster. The system responds to the introduction of new MicroArray hardware, in the same way as a SDN-assisted client connection rebalance and the system-wide data reconstruction behaves in response to failure. Individual stripes or objects are moved to new hardware based on new data placement layout. That is determined by the software controller on SDN switches, in a controlled manner, to limit the impact of background rebalance I/O on active client workloads. Coho DataStream provides underlying hardware flexibility allow to have different generation of MicroArray hardware within same cluster.


Figure 3: OpenStack Cinder Workflow with Coho Data driver.


How the Coho Data Cinder Driver WorksCoho Data offers a NFS-based cinder driver, which enables the organization to leverage unique architecture benefits which include: workload distribution, network optimization, and elastic scale-out, into your OpenStack Block Storage environment.

Cinder API workflow with Coho Data driver is as below:

1. cinder-api provides REST API to respond Block storage related request.

2. cinder-scheduler choose cinder-volume with an appropriate volume driver to be used based on the requested storage type. Volume drivers are management interface of corresponding backend storage.

3. Volume information is queried and updated on cinder database during the volume creation process. All API communications between cinder internal services go through message queues.

4. When coho cinder driver is used it mounts an NFS exported directory defined in a cinder configuration file and create disk images files.

5. Nova compute nodes use the NFS mount to access the image files to present to a VM instance through the hypervisor.



A single IP address endpoint, provided by the Coho Data protocol virtualization layer, on SDN switches is used as NFS target. This endpoint is persistent through hardware refresh cycles or even when the back-end storage servers scales.

NFS connections from nova compute hosts are intelligently distributed and load-balanced based on connection level throughput which leverages OpenFlow at this protocol virtualization layer to optimize storage network traffic, as well as to maximize back-end storage resource utilization.

Volume clone and snapshot operations through the Coho Data driver are instantly completed and spaced efficiently by utilizing Coho Data API.

Coho DataStream provides a unique VM instance level real-time performance, capacity reporting via REST API, or Coho Data management UI to supplement the OpenStack common interface. It helps OpenStack administrators to know how applications or instances are performing from a storage perspective.

Why a Coho Data Cinder Solution for OpenStackAs described in earlier sections, the Coho DataStream architecture shares similar design concepts as OpenStack cloud to solve common cloud infrastructure scalability and agility challenges.

Scalability - Enables Business Agility • Coho Data Cinder Block storage solution is the ability to scale seamlessly and horizontally in

response to both capacity and performance demands; for example, Nova compute can add more compute resources as required.

• It can avoid storage bottlenecks by distributing client connections at the SDN storage network layer to optimize storage traffic as well as rebalancing data to eliminate hotspots.

Simplicity - Enables Operational Efficiency• Deploying the Coho Data cinder storage into OpenStack is simply as modifying few configuration

files and point to a single endpoint and no vendor specific agent need to run on either controller or compute nodes. Growing storage is independent of the infrastructure and does not require any changes, including physical cabling as it goes to the Coho Data SDN ToR switches.

• MicroArray storage nodes building block approach provides predictable capacity/performance increase when scaling, which simplifies storage upgrade planning.

• No forklift upgrade is required when blending different generations of MicroArray within the same cluster. Hardware refresh is much like how Nova compute designed - add new hardware and remove hardware.

• The shared-nothing architecture, with fully redundant distributed objects, allows running a non-disruptive rolling upgrade.

Performance - Maximize Utilization of Storage Resources • The Coho Data solution is optimized for mixed workloads, at large scale. It performs well

because of intelligent decisions made at the SDN protocol virtualization layer that distribute workloads to optimize storage network throughput. Data is also automatically rebalanced to achieve maximum utilization of storage resources.



Use CasesIt is important to design cinder storage infrastructure to scale as easily as compute resources scales to achieve maximum utilization of storage resources and avoid storage bottleneck, thus, supporting your business objectives.

Some of the following storage driven OpenStack use cases are designed to parallelize process, at the application layer, by distributing the workloads to multiple VM-instances, running on different hypervisor hosts. The Coho DataStream enables the same level of workload distribution and parallelization at Block storage layer as well.

Infrastructure services IaaS environment, including service provider and internal application development, requires quick response to storage resource demand.

High Throughput Computing HTC cloud is designed to scale to a large number of compute resources to parallelize the computation process. It’s a key to maximizing throughput from available storage resource at any scale.

Web service and e-commerce OpenStack cloud is used as a backend of Web services and e-commerces at scale.

Database as a Service Project Trove provides scalable and reliable cloud database as a service provisioning to support various database engines, including both SQL and NoSQL.

Big Data Analytics Project Sahara allows users to provision data processing frameworks such as Hadoop and Spark, on top of OpenStack cloud. Clustered instances to perform analytic tasks are provisioned to parallelize data processing.

Container Optimized Project Magnum provides OpenStack API to integrate with container orchestration engines like Kubernetes and Docker swarm, to manage application containers running on Nova instances.

Table 1: OpenStack Cinder Use Cases.



ConclusionDesigning Block Storage infrastructure to scale is essential for the organizations to meet their business objectives using OpenStack cloud at any scale.

The architecture combines commodity storage servers and SDN OpenFlow-enabled networking to provide horizontal scalability, network optimized workload distribution, parallelization, and maximum storage resource utilization to meet your cloud application needs.

With Coho DataStream, organizations can build OpenStack infrastructure that allows you to dynamically add storage capacity and performance easily and on demand, as compute resources scales, which enables business agility and increased operational efficiency.

To schedule a demo and detailed technical discussion, contact us at [email protected]. For more information, visit www.cohodata.com.

mailto:sales%40cohodata.com?subject=Request%20for%20more%20information.

http://www.cohodata.com



ResourcesCoho Data github repository (driver for Kilo and Liberty):

https://github.com/coho-data/coho-cinder-driver

OpenStack foundation:

http://www.openstack.org

OpenStack logical architecture diagram:

http://docs.openstack.org/admin-guide/common/get_started_logical_architecture.html

About the AuthorMasa Kawano is a Principal Sales Engineer with Coho Data. Masa has over 15 years of Storage Industry experience in various roles spanning Pre-sales, Solutions Architect and Tech support supporting large scale infrastructure. Masa has been co-organizing OpenStack NYC meetup to support a local OpenStack community in NYC.

Masa has several industry certifications including Mirantis MCA200, Cloudera CCAH, VMware VCP-DCV, etc. You can follow Masa on Twitter @masa_kawano

4151 Middlefield Road, Suite 100, Palo Alto, CA 94303Phone: 408.419.1907 | Email: learnmore@cohodata @cohodata

Publication Date: April 25, 2016 1:27 PM

©2016 Coho Data. All rights reserved. The Coho Data logo, DataStream, and MicroArray are trademarks of Coho Data in Canada, United States and other jurisdictions. All other trademarks, service marks, and trade names referenced in this document are those of their respective owners. No part of this document or other Coho Data document assets may be reproduced without the express written consent of Coho Data.

Every effort has been made to ensure that the information in this document is accurate; however, errors and/or omissions in content are possible. In no event shall Coho Data be liable for incidental or consequential damages arising from use of this document or the software and hardware described in this document. Content is subject to change without prior notice.

https://github.com/coho-data/coho-cinder-driver

http://www.openstack.org

http://docs.openstack.org/admin-guide/common/get_started_logical_architecture.html

https://twitter.com/masa_kawano

mailto:learnmore%40cohodata?subject=I%20would%20like%20to%20learn%20more%20about%20Coho%20Data.

https://twitter.com/cohodata

https://twitter.com/cohodata

600-0027NA - Key Design Considerations for Scalable OpenStack Block Storage (white paper)

Documents

Transcript of 600-0027NA - Key Design Considerations for Scalable OpenStack Block Storage (white paper)