1
April 26, 2013
Deploying OpenStack with Chef and Operational Tooling
#chefconf 2013
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Who Are We?
Justin Shepherd
Principal Architect
Rackspace Private Cloud
github.com/galstrom21
Joseph Breu
Deployment Engineer
Rackspace Private Cloud
github.com/rackerjoe
@rackerjoe
3RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Only consume Open Source projects
• All changes go upstream
• CHEF-3889 - tunables for postgres
• CHEF-3849 - redirect login bug
• Everything must be automated through the deployment and management stacks.
• Everything we build is Open Souce
• https:/github.com/rcbops
• github.com/rcbops-cookbooks
Rackspace Private Cloud Philosophy
4RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Jenkins is used for testing and gating of our cookbooks
• Syntax Verification (pep8, foodcritic)
• Unit Tests for OpenCenter
• Full deployment of OpenStack utilizing the cookbooks with the proposed changeset applied
• Functional Tests of OpenStack
• OpenStack API Testing
• OpenStack CLI Testing
• Upstream opscode cookbooks are tested before inclusion
Testing
5RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How Did We Get Here?
5
6RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Installing Bexar and Cactus by hand
• Problems
• Deployed from trunk
• No packages available for distributions
• Not easily repeatable
• Not easily supported
In The BeginningEarly 2011
7RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Defining Our Mission
7
8RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to model “Real World Deployments”
The MissionEarly 2011
9RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Automating OpenStack Deployment
9
10
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Cactus and Diablo
• Puppet
• Inherited from OpenStack Team
• djeep based
• Very simplistic, not flexible
• Represented the developer case - not real world deployments
10
Automate!Late 2011
11
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Mission Status
We need to make it easy to install FAIL
We need to make the installation repeatable PASS
We need to model “Real World Deployments” FAIL
12
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Crowbar
• Dell platform for deployment
• Chef based
• Very heavyweight
• ISO installation
• Boot shim for system discovery
• Used for a few deployments and contributed code upstream
• Upgrades were painful or not possible at all
12
Automate!Late 2011 - Diablo
13
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Mission Status
We need to make it easy to install Almost Passed
We need to make the installation repeatable PASS
We need to model “Real World Deployments” Almost Passed
14
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Feature Creep
14
15
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to mode “Real World Deployments”
• We need to handle updates
• We need to quickly consume new OpenStack services
• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)
15
We now need to…March 2012
16
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did Puppet do?
We need to make it easy to install FAIL
We need to make the installation repeatable PASS
We need to model “Real World Deployments” FAIL
We need to handle updates FAIL
We need to quickly consume new OpenStack Services FAIL
We need to plug into existing infrastructure FAIL
17
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did Crowbar do?
We need to make it easy to install Almost Passed
We need to make the installation repeatable PASS
We need to model “Real World Deployments” Almost Passed
We need to handle updates FAIL
We need to quickly consume new OpenStack Services FAIL
We need to plug into existing infrastructure FAIL
18
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Next Steps in Automation
18
19
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Essex
• Started to build our own cookbooks
• Heavy utilization of Chef search
• Modeled real world deployments
• Support for multiple Operating Systems
Next Steps in AutomationMarch 2012
20
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Folsom
• Creating LWRPs for REST based Services
• Rewrites of many cookbooks to support multiple Operating Systems
• Debian and RedHat based
• Search wrapper functions for ease of use
Next Steps in AutomationLate 2012
21
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install PASSED
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure Almost PASSED
22
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
More Feature Creep
22
23
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to mode “Real World Deployments”
• We need to handle updates
• We need to quickly consume new OpenStack services
• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)
• We need provide Highly Available OpenStack Services
• We need to provide support for Vendor Integration for Block Storage
• We need to support Software Defined Networks
23
We now need to…December 2012
24
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• keepalived for failover IP
• haproxy for load balancing REST based services
• Multi-master MySQL replication and keepalived
• Active/Passive RabbitmQ and keepalived
• This is sketchy in Folsom - better in Grizzly
Creating Highly Available ServicesEarly 2013
25
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Support for multiple Cinder Backends
• Currently NetApp and EMC
• Support for SDN
• Currently only OVS
• LDAP Support
• Greenfield LDAP deployments only
• No AD support... yet...
Other ServicesEarly 2013
26
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install FAILED!
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure PASSED
We need to provide Highly Available OpenStack services
PASSED
We need to provide support for Vendor Integration for Block Storage
PASSED
We need to Support Software Defined Networks PASSED
27
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
The Tipping Point
27
28
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We had a system that can do real interesting things
• You now need to have staff with strong Chef knowledge to use it
• And strong OpenStack knowledge...
• OpenSource Chef is great for some things – but not everything
• We need to orchestrate changes across a cluster
• We need to do it in a defined order
The Tipping PointEarly 2013
29
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
OpenCenter
29
30
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Simplifies, speeds up deployment
• Lowers Operator overhead
• Minimizes the internal OpenStack knowledge
• Provides resiliency in the Cloud
• API for command line access
What is OpenCenter?
31
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is OpenCenter?
32
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is OpenCenter
33
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Lowers Operator Overhead
34
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resiliency for Private Cloud
35
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Solver – Automates Operator Tasks
36
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Extensible
Write your own Python/Bash scripts to create workflows that extend OpenCenter
37
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install PASSED
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure PASSED
We need to provide Highly Available OpenStack services
PASSED
We need to provide support for Vendor Integration for Block Storage
PASSED
We need to Support Software Defined Networks PASSED
38
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Errata
38
39
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Partial Searches and Templates
• Remove dependencies on monitoring and logging
• Make our cookbooks more easily consumable
• Upstream our cookbooks to the OpsCode OpenStack repos
• Utilize OpenStack CI testing and gating
Looking forward
4040
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
Top Related