Dockerizing OpenStack for High Availability

© 2014 IBM Corporation

Dockerizing OpenStack High Availability A Practical Approach Manuel Silveyra - Senior Cloud Solutions Architect @manuel_silveyra Daniel Krook - Senior Certified IT Specialist @DanielKrook Shaun Murakami - Senior Cloud Solution Architect @stmuraka Kalonji Bankole - Cloud Architect @k_bankole


OpenStack Summit Atlanta May 2014

A Practical Approach to Deploying a Highly Available OpenStack


OpenStack high availability challenges

•  There were a lot of possible configuration options •  Active/Active •  Active/Standby

•  Installing and configuring is complicated •  Keep track of configurations, ports, services, etc.

•  Scaling increases complexity •  Distributing load has different requirements than availability


Our OpenStack HA architecture


That architecture leaves room for improvement

•  Existing challenges •  Many configuration options •  Installation is complex •  Scaling increases complexity

•  Automation and visibility •  Deployment •  Patching •  Monitoring


Can Docker help?

•  A technology that allows applications (and all related dependencies) to be packaged in individual containers.

•  Containers run as isolated userspace processes on the host OS. •  Containers share the Linux kernel.

Benefits include •  Service isolation •  Security •  Version control •  Portability •  Repeatable •  Rapid deployment •  Very lightweight (close to bare metal)

Bare metal Container Virtual machine


Advantages of OpenStack on Docker

Faster scaling •  New Docker

containers start up in seconds

Higher density •  Lower overhead

means more available resources on the host

Greater flexibility •  Docker standardizes

the packaging, configuration, and deployment of services.

Which all add up to faster response to changing business requirements for our

OpenStack deployments


Before and after

Bare Metal Docker Deployment Method Chef Cookbooks Custom Scripts

Deployment Preparation Days Hours

Deployment Time 15 Mins 5 Mins Scale Time 7 Mins Seconds Scaling Unit Bare Metal Node Service Containers


Our newly Dockerized OpenStack


Docker is a technology that...

Leverages Linux containers

• Process isolation •  libcontainer (abstraction) •  cgroups (resource control) • namespaces (isolation)

• Host kernel reuse • eliminates redundancy

Simulates a VM without overhead

• Faster lifecycle operations • minimal operating system •  copy, start, stop, delete

• Better resource utilization •  smaller footprint for both

containers and images

Provides additional benefits over VMs

• Versioning and layering • promotes rapid

collaboration and reuse

• No hypervisor dependency • highly portable • high performance


Understanding Docker concepts

Containers

•  create, delete, start, stop, restart, pause, resume, save

•  inspect – view metadata about a container

•  logs – view stdout and stderr from a container

Images

•  create, delete, export, import

• history – show commands used to make an image

• along with Dockerfiles, the key persistent unit of Docker

Registries

• pull, push, tag, search

•  central location for sharing images

•  contains community or trusted images


Host

Docker Client

Base OS/Kernel

Docker Daemon Isolation

Con

tain

er

Con

tain

er

Con

tain

er

Docker Registry

libcontainer / LXC

Requires kernel compatible

images

Expose select ports on Host

App Client

Understanding Docker management


Docker managed container features •  Expose from the container •  Proxy through the host mapping Network ports

•  Pass in to set runtime configuration values Environment variables

•  Set DNS servers and search domains •  Set modes: bridged, none, container, host Network configuration

•  Limit memory •  Limit CPU Resource constraints

•  Mount from host •  Share volumes between containers Storage volumes

•  Set to: on failure, never, always Restart policy

•  Escalate container access to host resources Container privileges


Bringing it all together: A simple workflow with Docker •  Create and start a new container with docker run

docker run –ti ubuntu bash You're now in a new Ubuntu container running bash – experiment or iterate to develop and test apps and configuration.

•  Create new container using a Dockerfile: FROM ubuntu

RUN apt-get update && apt-get install -y openssh-server

EXPOSE 22

CMD ["/usr/sbin/sshd", "-D"]

docker build –t simple:sshd .

docker run -p 2222:22 simple:sshd Now the SSH server is running in a container and ready to be used on port 2222

Start with Ubuntu base image

Each RUN action creates a new filesystem layer

Only port 22 is available from outside container

Command to run when container starts

Map port 22 on container to 2222 on host

Start Ubuntu and run the bash shell


Running highly available OpenStack services in Docker


Running OpenStack services in Docker

1.  Build an image

2.  Start a container instance

3.  Update load balancer(s)

(repeat for all services)


OpenStack Dockerfile example (nova-api) # Create the base operating system layer FROM ubuntu:trusty MAINTAINER Shaun Murakami [email protected] # Update base image RUN apt-get -y update RUN apt-get -y upgrade # Install OpenStack components RUN apt-get -y install python-software-properties python-mysqldb nova-api # Prepare filesystem for OpenStack components RUN chown -R nova:nova /etc/nova \ && chown -R root:root /etc/nova/root* \ && rm /var/lib/nova/nova.sqlite \ && cp /etc/nova/api-paste.ini /etc/nova/api-paste.ini.orig \ && echo "admin_token = oWKwDPaUWBNzif92" >> /etc/nova/api-paste.ini \ && cp /etc/nova/nova.conf /etc/nova/nova.conf.orig # Import nova.conf from the host ADD ./nova.conf /etc/nova/ # Customize container runtime EXPOSE 8774 8775 CMD /usr/bin/python /usr/bin/nova-api --config-file /etc/nova/nova.conf --logfile /var/log/nova/api-`hostname`.log


Create the Docker image docker build –t nova:api .

Step 0 : FROM ubuntu:trusty ---> 6b4e8a7373fe Step 1 : MAINTAINER Shaun Murakami <[email protected]> ---> Using cache ---> 96345089d832 Step 2 : RUN apt-get -y update ---> Running in fc22a3c8812b

Step 6 : ADD ./nova.conf /etc/nova/ ---> ba53dd03fcf0 Removing intermediate container 910c4ff92b18 Step 7 : EXPOSE 8774 8775 ---> Running in 5cc44c54c15d ---> a8840d052474 Removing intermediate container 5cc44c54c15d Step 8 : CMD /usr/bin/python /usr/bin/nova-api --config-file /etc/nova/nova.conf --logfile /var/log/nova/api-`hostname`.log ---> Running in e876b1085db9 ---> a35112f528b0 Removing intermediate container e876b1085db9 Successfully built a35112f528b0

...


OpenStack services running in containers

docker run -d -P nova:api


Sharing images using a shared private registry

1.  docker tag nova:api 9.30.211.23:5000/nova:api

2.  docker push 9.30.211.23:5000/nova

3.  docker pull 9.30.211.23:5000/nova


Scaling OpenStack services with Docker

1.  Share images in Docker registry

2.  Start a container instance

3.  Update load balancer(s)


Lessons learned

3. Layer limitations •  Combine commands in Dockerfile

1. Docker random port generation makes service management difficult •  Fixed ports & script automation

4. Debugging isn’t easy (Docker ver. <1.3) •  Consolidated logging

2. Services that require multiple processes •  Supervisord to manage and run multiple processes


Docker processes with consolidated logging

•  Run command: /usr/bin/python /usr/bin/nova-api \ --config-file /etc/nova/nova.conf \ --logfile /var/log/nova/api-`hostname`.log

•  Export volume when starting: -v /root/openstack_logs/nova:/var/log/nova


OpenStack Docker container management options


Shipyard

•  Written in Python

•  Manages multiple Docker hosts

•  Provides a customizable UI (Django)

•  Utilizes Docker API to retrieve information

•  Active community


Summary

•  Docker improves our highly available architecture in several areas without a major redesign •  Faster scaling •  Higher density •  Greater flexibility

•  OpenStack services can be encapsulated very easily within Docker containers •  Easy to test iteratively •  Easy to declare in a Dockerfile •  Easy to run and scale

•  Orchestration of a Docker based OpenStack cluster needs improvement •  Many fast moving options are available •  Customization of Shipyard worked best for us


IBM technical sessions at the Paris Summit IBM Sessions on Monday, November 3rd 15:20

R.251 When Disaster Strikes the Cloud: Who, What, When, Where and How to recover Ronen Kat, Michael Factor, and Red Hat

11:40 A.Blue IPv6 Features in OpenStack Juno Xu Han Peng, Comcast, and Cisco

15:20 R252 Why is my Volume in 'ERROR' State!?! An Introduction to Troubleshooting Your Cinder Configuration Jay Bryant

16:20 A.Blue Group Based Policy Extension for Networking Mohammad Banikazemi, Cisco, Midokura, and One Convergence

IBM Sessions on Tuesday. November 4th 11:15

R252 The perfect match: Apache Spark meets Swift Gil Vernik, Michael Factor, and Databricks

15:40 R242 Docker Meets Swift: A Broadcaster's Experience Eran Rom, and RAI

16:40 Maillot User Group Panel: India, Japan, China Ying Chun Guo, Guang Ya Liu, Qiang Guo Tong

14:50 Passy A Practical Approach to Dockerizing OpenStack High Availability Manuel Silveyra, Shaun Murakami, Kalonji Bankole, Daniel Krook

IBM Sessions on Wednesday, November 5th 09:00

R241 Monasca DeepDive: Monitoring at scale Tong Li , Rob Basham, HP and Rackspace

09:00 R242 Beyond 86: Managing multi-platform environments with OpenStack Shaun Murakami, Philip Estes

09:50 R253

Troubleshooting Problems in Heat Deployments Fabio Oliveira, Ton Ngo, Priya Nagpurkar, Winnie Tsang

11:50 R251 Keystone to Keystone Federation Enhancements for Hybrid Cloud Enablement Steve Martinelli, Brad Topol, CERN, and Rackspace

17:50 R253 Practical advice on deployment and management of enterprise workloads Jarek Miszczyk, Venkata Jagana


Learn more at these IBM sponsored sessions on Wednesday: 9:50 Room 243 Step on the Gas: See how Open Technologies are driving the future of the enterprise 11:50 Room 212/213 IBM and OpenStack: Collaborations beyond the code 1:50 Room 212/213 A Use Case Driven view of IBM’s OpenStack based Offerings 2:40 Room 212/213 IBM OpenStack Offerings in Action

Stop by the IBM Booth (B4) Demos, games and FREE tee

shirt.


Legal Disclaimer

•  © IBM Corporation 2011. All Rights Reserved. •  The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any

kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

•  References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

•  If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete: Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

•  If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete: All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

•  Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

•  If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete: Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

•  If you reference Java™ in the text, please mark the first use and include the following; otherwise delete: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

•  If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete: Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

•  If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete: Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

•  If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete: UNIX is a registered trademark of The Open Group in the United States and other countries.

•  If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete: Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

•  If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.

Dockerizing OpenStack for High Availability

Technology

Transcript of Dockerizing OpenStack for High Availability