Chef and OpenStack Workshop from ChefConf 2013
-
Upload
matt-ray -
Category
Technology
-
view
12 -
download
3
description
Transcript of Chef and OpenStack Workshop from ChefConf 2013
Chef and OpenStackMatt RayApril 24, 2013
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
Introductions
• Matt Ray• [email protected]• mattray IRC, GitHub• @mattray
http://upload.wikimedia.org/wikipedia/commons/2/27/Hi_How_Are_You_Austin_2005.jpg
Presenter notes: Austin, San Antonio, Santa Clara, Boston, San Francisco, San Diego, PortlandBexar and onMercado Libre, Dell, Rackspace, HP, DreamHost
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
"To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to
implement and massively scalable."
Mission Statement
Why OpenStack?
‣ Control. Open source, no vendor lock in. Apache 2 license.
‣ Flexibility. Modular design integrates legacy and third party technologies.
‣ Emerging Industry Standard. More than 180 technology industry leaders backing it and major public clouds built on OpenStack.
‣ Proven. Originally built for scale and redundancy at NASA and Rackspace. More than 200 large-scale deployments worldwide.
‣ Compatible and Connected. Enables portability.
Control. Open source means you’re never locked to a proprietary vendor. You always have visibility and the ability to directly influence the roadmap through the open design process.
Flexibility. Modular design can integrate with legacy systems and third-‐party technologies, so you don't have to rip-‐and-‐replace your exisAng infrastructure.
Emerging Industry Standard. More than 170 leading technology companies across the globe are developing and building tools for OpenStack, including AT&T, Cisco, Dell, HP, Intel, IBM, MicrosoP and Red Hat, and new OpenStack clouds are coming online daily.
Proven and Scalable. OpenStack was built for scale and redundancy. You can run the same soPware that today powers some of the world's largest public and private clouds.
Compa<ble and Connected. CompaAbility with public OpenStack clouds means enterprises are prepared for the future—making it easy to migrate data and applicaAons to public clouds when condiAons are right.
OpenStack Controls Compute, Storage & Networking
Architecture
Architecture: Nova
Compute: Nova‣ Virtual Machines
‣ Provision and manage large pools of on-demand computing resources (hypervisors & instances)
‣ KVM
‣ Xen
‣ LXC
‣ Hyper-V
‣ VMware
‣ Bare-metal
Compute (codenamed "Nova") provides virtual servers upon demand. Rackspace and HP provide commercial compute services built on Nova and it is used internally at companies like Mercado Libre, AT&T and NASA (where it originated).
Architecture: Cinder
Block Storage: Cinder‣ Virtual Block Storage Devices
‣ Volumes on commodity storage gear
‣ Drivers for more advanced systems like NetApp, Solidfire, Ceph and Nexenta
‣ Released in Folsom Fall 2012
Block Storage (codenamed "Cinder") provides persistent block storage to guest VMs. This project was born from code originally in Nova (the nova-volume service described below). Please note that this is block storage (or volumes) not filesystems like NFS or CIFS share.Cinder is new for the Folsom release.
Architecture: Quantum
Networking as a Service: Quantum‣ Virtualized Networking
‣ Software Defined Networking (SDN)
‣ Automation of hardware & software
‣ Define network connectivity & addressing used by devices from other services
‣ Drivers for Ryu, Floodlight, Nicira, Midokura, Cisco and many more
‣ Released in Folsom Fall 2012
Network (codenamed "Quantum") provides "network connectivity as a service" between interface devices managed by other OpenStack services (most likely Nova). The service works by allowing users to create their own networks and then attach interfaces to them. Quantum has a pluggable architecture to support many popular networking vendors and technologies. Quantum is new in the Folsom release.
Architecture: Glance
Image Registry: Glance‣ Multi-format virtual disk image registry & catalog‣ Delivery of images to Nova Compute‣ Allows uploads of private and public images in a wide variety of formats
‣ Machine (kernel/ramdisk outside of image, a.k.a. AMI)
‣ qcow2 (Qemu/KVM)
‣ VMDK (VMWare)
‣ OVF (VMWare, others)
‣ And more
Image (codenamed "Glance") provides a catalog and repository for virtual disk images. These disk images are mostly commonly used in OpenStack Compute. While this service is technically optional, any cloud of size will require it.
Architecture: Keystone
Identity: Keystone‣ Unifies all core projects with common authentication system‣ Provides authorization for multiple log-in credentials
‣ Username/password
‣ Token-based
‣ AWS-style logins
‣ Integrate with existing systems
Identity (codenamed "Keystone") provides authentication and authorization for all the OpenStack services. It also provides a service catalog of services within a particular OpenStack cloud.
Architecture: Swift
Object Storage: Swift‣ Redundant, resilient, horizontally scalable object storage‣ Petabytes of reliable storage on standard gear‣ Examples include virtual machine images, photo storage, email storage
and backup archiving‣ Rackspace Cloud Files
Object Store (codenamed "Swift") allows you to store or retrieve files (but not mount directories like a fileserver). Several companies provide commercial storage services based on Swift. These include KT, Rackspace (from which Swift originated) and Internap. Swift is also used internally at many large companies to store their data.
Architecture: Horizon
Web Dashboard: Horizon‣ Self-service, role-based web interface for users and administrators‣ Provision cloud-based resources through a self-service portal‣ Create and manage projects and users, defining resources available to
them‣ Extensible design makes it easy to plug in and expose third party
products and services‣ Django application that consumes APIs
Dashboard (codenamed "Horizon") provides a modular web-based user interface for all the OpenStack services. With this web GUI, you can perform most operations on your cloud like launching an instance, assigning IP addresses and setting access controls.
OpenStack Community
‣ OpenStack Summits (Spring & Fall)‣ IRC (irc.freenode.net)
‣ #openstack, #openstack-meetings, #openstack-chef, many more
‣ Mailing Lists (lists.openstack.org)‣ OpenStack.org
‣ Blog, Docs, Wiki
‣ Twitter @OpenStack
OpenStack Code
‣ GitHub‣ https://github.com/openstack/
‣ Gerrit‣ https://review.openstack.org
‣ Jenkins‣ https://jenkins.openstack.org/
‣ DevStack.org
OpenStack Foundation
Led by Executive Director, Jonathan Bryce, the Foundation is hiring 10-12 employees who, under the strategic direction of the Board, will help carry out the OpenStack mission. Specific responsibilities include coordinating the project's infrastructure, such as systems for testing the software at scale, community building activities, and managing the OpenStack trademark, which was transferred from Rackspace following the first board meeting.
OpenStack Distributions
‣ Linux Distributions‣ Debian
‣ Fedora
‣ Red Hat
‣ SUSE
‣ Ubuntu
‣ Commercial Offerings‣ CloudScaling
‣ Mirantis
‣ Nebula
‣ Piston
‣ Rackspace
‣ ...many more
Grizzly Release‣ 550 Developers‣ 2500 conference attendees‣ Red Hat, IBM, Rackspace‣ Incubating:
‣ Commons: Oslo
‣ Metering: Ceilometer
‣ Orchestration: Heat
• Ceilometer is a metering project. The project offers metering information and the ability to code more ways to know what has happened on an OpenStack cloud. While it provides metering, it is not a billing project. A full billing solution requires metering, rating, and billing. Metering lets you know what actions have taken place, rating enables pricing and line items, and billing gathers the line items to create a bill to send to the consumer and collect payment. Ceilometer is available as a preview.
• Heat provides a REST API to orchestrate multiple cloud applications implementing standards such as AWS CloudFormation.
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
Why Chef?
I'm here to tell you
Chef is Infrastructure as Code
http://www.flickr.com/photos/louisb/4555295187/
• Programmatically provision and configure
• Treat like any other code base
• Reconstruct business from code repository, data backup, and bare metal resources.
When dealing with Chef, need to literally “think outside the box”, by shifting your thinking about configuration away from a single system, to that of an Application Infrastructure. The concept of an Infrastructure is an abstract one with a specific technical meaning. When we talk about Infrastructure, we mean..
Declarative Interface to Resources
• Define policy
• Say what, not how
• Pull not Push
http://www.flickr.com/photos/bixentro/2591838509/
Chef gives you declarative interfaces into the Resources on those Nodes.Being declarative means that you say what you want to do, instead of how to do it.For example, you declare that package foobar-1.2.3 should be installed, or that the directory /var/log/foobar should exist.Chef pulls down policy from the chef-server, ensuring that a node down for maintenance will receive its policy update when it comes back online.
Ruby!
extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend
Because we use a 3GL for the recipe config files, we can use features of ruby like case statements and iterative loops.Sysadmins don’t need to be afraid of Ruby, they’ve been dealing with sub-standard programming languages like configuration files for years. They’re also not limited to what the language tells them they can do.
Recipes and Cookbooks
• Recipes are collections of Resources
• Cookbooks contain recipes, templates, files, custom resources, etc
• Code re-use and modularity
• Hundreds already on Community.opscode.com
http://www.flickr.com/photos/shutterhacks/4474421855/
The Chef Community
• Apache License, Version 2.0
• 1300+ Individual contributors
• 200+ Corporate contributors
• Dell, DreamHost, HP, Rackspace, VMware, SUSE and many more
• 900+ cookbooks
• http://community.opscode.com
Chef is hackable! Permissive Apache2 license, vibrant community of awesome folks.Community is very important to us.That's why we're here.
http://www.flickr.com/photos/16339684@N00/2681435235/
Chef and OpenStack?
That's great and all, but tell me about
OpenStack!
And a basic fact about Infrastructure -- it EVOLVES.
Deploying OpenStack
• Chef ties it all together automatically
• Scaling changes how we deploy
• Interchangeable components
• Configurations shared, supported & documented
• Licensing makes it available to everyone
We've learned a lot of things.
Overview & Status of Chef for OpenStack
Chef for OpenStack: Who
• Arista
• AT&T
• Baremetal Cloud
• Calxeda
• Dell
• DreamHost
• HP
• HubSpot
• IBM
• Intel
• Internap
• Mercado Libre
• Mirantis
• NTT
• Nebula
• Nicira
• Piston Cloud
• Rackspace
• SUSE
• TryStack.org
• Voxel
• ...and more
These companies are currently involved to some extent. Some are paying customers that we've done engagements with.
Chef for OpenStack: Why
• Community for the automated deployment and management of OpenStack
• Reduce fragmentation and encourage collaboration
• Deploying OpenStack is not "secret sauce"
• Project not a product
• Apache 2 license
Chef for OpenStack: What
• Chef Repository for Deploying OpenStack
• Documentation for Chef for OpenStack
• Cookbooks• Keystone
• Glance
• Nova
• Horizon
• Swift
• Quantum
• Cinder
• knife-openstack
Chef for OpenStack: Where
• #openstack-chef on irc.freenode.net
• github.com/opscode/openstack-chef-repo
• github.com/opscode-cookbooks/
• keystone, glance, nova, horizon, swift,quantum,cinder
• github.com/opscode/knife-openstack
• github.com/mattray/openstack-chef-docs
• groups.google.com/group/opscode-chef-openstack
• @chefopenstack
• Chef repo for Essex/Grizzly
• Operating Systems (Ubuntu 12.04)
• Hypervisors (KVM, LXC)
• Databases (MySQL)
• Nova network FlatDHCP HA & VLAN
• Quantum Nicira plugin available
• Test Kitchen integration
Chef for OpenStack: When (Today)
Chef for OpenStack: When (Tomorrow)
• Grizzly sprint scheduled in 2 weeks
• Merging AT&T, DreamHost, HubSpot and Rackspace code
• Documentation (docs.opscode.com)
• Build packages from source
• Continuous integration
• Hypervisors (Hyper-V, bare metal)
• Databases (PostgreSQL)
• Cinder (Ceph)
• Quantum (Midokura)
• Operating Systems (RHEL, Debian, SUSE)
• Documentation (docs.opscode.com)
• HA Configurations
Chef for OpenStack: When (Roadmap)
ResourcesChef for OpenStackEcosystem
Mercado LibreHPRackspaceDellTrystackVoxelOpenStack
AT&T
• github.com/att-cloud/
• primary Folsom merge source
• openstack-common
• cinder (lvm, netapp, rbd)
• lots of support cookbooks
Crowbar
• Hardware provisioning and application management platform
• crowbar.github.com
• Dell, SUSE, others
• likely Swift cookbook source combined with Rackspace's
DreamHost
• github.com/dreamhost
• ceph
• ceilometer
• quantum
• Nicira NVP cookbook
• Open vSwitch cookbook
• Development in progress by Opscode
• github.com/gmiranda23/nvp-cookbook
Nicira
Rackspace Private Cloud
• www.rackspace.com/cloud/private/
• github.com/rcbops/chef-cookbooks
• primary Essex merge source
• likely Red Hat source
• Cookbooks reusable outside of OpenStack
• Test Kitchen
• knife-rackspace/hp
• Crowbar, pxe_dust & Razor
• Arista EOS cookbook
• Berkshelf & Librarian
• Spiceweasel & Sputnik
Chef for OpenStack "Halo Effect"
Why the Cloud?Why OpenStack?
The solution to this perceived impediment to resources
• Instant infrastructure
• Unlimited capacity
• Autoscaling
• No commitment
• Immediate replacement
Why the Cloud?
Enforces good architecture No long term commitment Cloud benefits – instant infrastructure Immediate replacement, no sparing etc. Unlimited storage, snapshots (1 TB volume limit) Provisioning APIs, autoscaling, EU storage, geodistPublic, private, hybrid
• Real Open Source
• Anyone can play
• Choice of features
• Features achieving parity/accelerating ahead
Why OpenStack?
Know our escape plan for every infrastructure
provider
"Drink the cloud Kool-aid, but only drink our Kool-aid"If there are problems that you have with your cloud provider...
Not just the cloud
Chef for Infrastructure Portability
• knife ec2
• knife rackspace
• knife hp
• knife google
• knife azure
• knife cloudstack
• knife openstack
• knife vcloud
• ... and many others
From EC2 to Rackspace, HP or any other OpenStack provider
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
• AWS
• Rackspace
• HP
• Azure
• many others
Desktop, Virtualization, Private & Public Clouds
Chef is hackable! Permissive Apache2 license, vibrant community of awesome folks.More than 360 individual contributors, over 70 corporate contributors. Community is very important to us.That's why we're here.
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
Desktop, Virtualization, Private & Public Clouds
• AWS
• Rackspace
• HP
• Azure
• many others
Chef is hackable! Permissive Apache2 license, vibrant community of awesome folks.More than 360 individual contributors, over 70 corporate contributors. Community is very important to us.That's why we're here.
Chef for OpenStack TL;DL
• Project, not a product
• Lots of contributors with real deployments in a vibrant ecosystem
• Essex works, Grizzly soon
• Features driven by demand
• Documentation with examples
• Do real work with OpenStack
From EC2 to Rackspace, HP or any other OpenStack provider
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
knife openstack
Knife is our command line tool, literally a swiss army knife of cloud APIsIt talks to the Chef server to manage your infrastructure, but it also talks to APIs like the OpenStack one
So even if you're not managing your OpenStack layer, you have Chef to manage the components on top of it.
knife openstack
$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)
** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack group list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)
This is a supported knife plugin for Chef, so we have ticket tracking and everything for it.It has the basics, server creation, deletion and listing available images and servers
knife openstack flavor list
$ knife openstack flavor listID Name Virtual CPUs RAM Disk 1 m1.tiny 1 512 MB 0 GB 2 m1.small 1 2048 MB 10 GB3 m1.medium 2 4096 MB 10 GB4 m1.large 4 8192 MB 10 GB5 m1.xlarge 8 16384 MB 10 GB
update your knife.rb with
### Note: If you are not proxying HTTPS to the OpenStack EC2 API port, the scheme should be HTTP, and the PORT is 8773.
you can get these from "knife node show blahblah -a nova"
knife openstack group list
$ knife openstack group listName Protocol From To CIDR Descriptiondefault tcp 22 22 0.0.0.0/0 default default icmp -1 -1 0.0.0.0/0 default haproxy tcp 22002 22002 0.0.0.0/0 22022
knife openstack image list
$ knife openstack image listID Name 4a197431-503d-4b85-b61e-84af21ca8654 cirros-image f8ebb842-c0c0-4be3-8c4c-f72f48edec50 precise-image
update your knife.rb with
### Note: If you are not proxying HTTPS to the OpenStack EC2 API port, the scheme should be HTTP, and the PORT is 8773.
you can get these from "knife node show blahblah -a nova"
knife openstack server create -h
knife openstack server create (options) --bootstrap-version VERSION The version of Chef to install -N, --node-name NAME The Chef node name for your new node -s, --server-url URL Chef Server URL -k, --key KEY API Client Key --[no-]color Use colored output, defaults to enabled -c, --config CONFIG The configuration file to use --defaults Accept default values for all questions --disable-editing Do not open EDITOR, just accept the data as is -d, --distro DISTRO Bootstrap a distro using a template; default is 'chef-full' -e, --editor EDITOR Set the editor to use for interactive commands -E, --environment ENVIRONMENT Set the Chef environment -f, --flavor FLAVOR_ID The flavor ID of server (m1.small, m1.medium, etc) -a, --floating-ip [IP] Request to associate a floating IP address to the new OpenStack node. Assumes IPs have been allocated to the project. Specific IP is optional. -F, --format FORMAT Which format to use for output --[no-]host-key-verify Verify host key, enabled by default -i IDENTITY_FILE, The SSH identity file used for authentication --identity-file -I, --image IMAGE_ID The image ID for the server -u, --user USER API Client Username --openstack-api-endpoint ENDPOINT Your OpenStack API endpoint --insecure Ignore SSL certificate on the Auth URL -K, --openstack-password SECRET Your OpenStack Password -T, --openstack-tenant NAME Your OpenStack Tenant NAME -A, --openstack-username KEY Your OpenStack Username --prerelease Install the pre-release chef gems --print-after Show the data after a destructive operation --private-network Use the private IP for bootstrapping rather than the public IP -r, --run-list RUN_LIST Comma separated list of roles/recipes to apply -G, --groups X,Y,Z The security groups for this server -S, --ssh-key KEY The OpenStack SSH keypair id -P, --ssh-password PASSWORD The ssh password -x, --ssh-user USERNAME The ssh username --template-file TEMPLATE Full path to location of template to use -V, --verbose More verbose output. Use twice for max verbosity -v, --version Show chef version -y, --yes Say yes to all prompts for confirmation -h, --help Show this message
update your knife.rb with
### Note: If you are not proxying HTTPS to the OpenStack EC2 API port, the scheme should be HTTP, and the PORT is 8773.
you can get these from "knife node show blahblah -a nova"
$ knife openstack server listInstance ID Name Public IP Private IP Flavor Image Keypair State08f2d9f7-eeb0-45e7-8562-63aed8f096cc os-45539345723309377 50.56.12.229 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active43c6bbf5-b397-4986-8aec-392d955ce5b1 os-9924426691020416 50.56.12.232 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo activec1b9e3df-e566-4378-8a52-ed998b516608 os-553425714287088 50.56.12.230 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo activef3edc5da-ef99-4acb-a141-d957e09809e3 os-07459550287500682 50.56.12.231 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active
knife openstack server list
How did we get to the point where we can build a multi-tiered, monitored infrastructure?
knife openstack server create -a -f 2 -I 737969f8-6091-4896-ba9c-f3cf63bd25c5 -S rs-demo -i ~/.ssh/rs-demo.pem -x ubuntu -r "role[base]"
knife openstack server create
How did we get to the point where we can build a multi-tiered, monitored infrastructure?
knife openstack server create
Instance Name: os-45539345723309377Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc
Waiting for server.........Flavor: 2Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5SSH Identity File: /Users/mray/.ssh/rs-demo.pemSSH Keypair: rs-demoPublic IP Address: 10.241.0.12Floating IP Address: 50.56.12.229
Waiting for sshd.....doneBootstrapping Chef on 50.56.12.229
Instance Name: os-45539345723309377Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096ccFlavor: 2Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5SSH Keypair: rs-demoPublic IP Address: 50.56.12.229Environment: _defaultRun List: role[base]
update your knife.rb with
### Note: If you are not proxying HTTPS to the OpenStack EC2 API port, the scheme should be HTTP, and the PORT is 8773.
you can get these from "knife node show blahblah -a nova"
How did we get to the point where we can build a multi-tiered, monitored infrastructure?
How did we get to the point where we can build a multi-tiered, monitored infrastructure?
knife-openstack compatibility
• Uses the OpenStack API
• Diablo, Essex, Folsom, Grizzly
• Cloudscaling
• Crowbar
• DreamHost
• Nebula
• Piston
• Rackspace Private Cloud
knife-openstack Roadmap
• github.com/opscode/knife-openstack
• docs.opscode.com/plugin_knife_openstack.html
• tickets.opscode.com/browse/KNIFE/component/
• windows support
• floating IP address management
• quantum network management
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
Who was there
• AT&T
• Dell
• DreamHost
• HubSpot
• KT
• Midokura
• Opscode
• Rackspace
• SUSE
There had been several days of conversations, these companies were all represented in the meeting. Missing: eNovance, HP, IBM, KIO, Mirantis
Where
• #openstack-chef on irc.freenode.net
• groups.google.com/group/opscode-chef-openstack
• github.com/mattray/openstack-chef-docs
• @chefopenstack
The resources we're using
Licensing
• Apache 2
• Opscode CLA/CCLA required
• http://wiki.opscode.com/display/chef/How+to+Contribute
• http://wiki.opscode.com/display/chef/Approved+Contributors
Attendees were all covered already
Where on GitHub
• http://github.com/osops
• chef-repo/
• berkshelf, not git submodules
• cookbooks all end in "-cookbook"
• ie. "nova-cookbook"
• "operations" cookbooks outside scope
• ie. logging, monitoring, provisioning
Move to community GitHub repo, not Opscode's. Opscode will upstream from this repo.
Cookbooks
• cinder
• glance
• horizon
• keystone
• nova
• quantum
• swift
• ceilometer & heat eventually
The core OpenStack services
Goal of incorporating into OpenStack
• Get on StackForge
• will provide CI
• which everyone will probably slave
• Gerrit for code reviews
• we'll sort out reviewers once we start
• Testing with TestKitchen initially
• Grenade? Kong? SmokeStack?
We want to go into "mainline" OpenStack
• support alternative package sources
• source-built coming ("VanillaStack")
• packaging recipes before configuration
• ie. "nova/recipes/nova-compute-packages"
Packages
Use distro packages were applicable, but not everyone wants to use them. Build from source will come in eventually.
Chef Style Guide
• Chef 11 target release
• partial search
• partial templates
• Full-stack Chef-client compatible
• Ruby 1.9.x
• Upstream community cookbooks
• Foodcritic as much as possible
Table stakes
Chef Style Guide
• openstack-common instead of osops-utils
• Attribute injection
• attributes may short-circuit search
• few, if any, attributes in roles
• environment-driven attributes
• Chef Solo not actively supported
• platform logic in attributes files
Already using these patterns
• May release "2013.1.0"
• Chef repo for Grizzly
• Operating Systems (Ubuntu 12.04)
• Databases (MySQL)
• Hypervisors (KVM, LXC)
• Nova network FlatDHCP HA & VLAN
Initial osops release
Opscode employee Matt Ray and Chris McClimans are getting together after ChefConf to work on cleaning up Grizzly. Sources will be AT&T, Dell, HubSpot and Rackspace primarily.
• Operating Systems (RHEL, SUSE)
• Databases (Postgres)
• Hypervisors (Xen, bare metal)
• Cinder (Ceph, LVM, NetApp)
• Quantum (Bridge, Midokura, Nicira, OVS)
• Folsom backport
• HA Configurations may be stretch goal because of differing implementations
Grizzly Roadmap
SUSE: SLES, OpenSUSE, PostgresKT: XenHubSpot: bare metalDreamHost: CephAT&T: LVMRackspace: Bridge, NetApp, OVS, RHELOpscode: NiciraMidokura: Folsom, MidoNetAT&T, SUSE, Rackspace different HA setups
knife-openstack v0.7.0
$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)
** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack group list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)
Currently supported features.
knife-openstack compatibility
• Uses the OpenStack API
• Diablo, Essex, Folsom, Grizzly
• Cloudscaling
• Crowbar
• DreamHost
• Nebula
• Piston
• Rackspace Private Cloud
Continue to test for compatibility, will build out CI testing for Opscode-supported knife plugins.
knife-openstack Roadmap
• github.com/opscode/knife-openstack
• docs.opscode.com/plugin_knife_openstack.html
• tickets.opscode.com/browse/KNIFE/component/
• Continues to be managed by Opscode
• Test against multiple OpenStack deployments for compatibility
• next major release v0.8.0 (May)
• floating IP address management
• network assignment on server creates
More features will undoubtedly show up
• Submit talk "Chef for OpenStack Fall 2013 Overview & Status"
• Review this deck
• Report progress
• Submit for developer track session as well
• See you in Hong Kong!
Fall 2013 OpenStack Summit
We'll see what actually happens in November.
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
Berkshelf
• http://berkshelf.com/• tool for managing your Chef Cookbooks and their dependencies
• Community site, Git, local development• Berksfile managed in version control
Spiceweasel
• https://github.com/mattray/spiceweasel• manages your Chef repositories and creating reproducible infrastructure
• nodes, cookbooks, roles, data bags & environments with a version controlled manifest
• validates dependencies • allows extraction and creation of infrastructure• lightweight orchestration and cluster management
• Sputnik Cloud Launcher• fills gap between the documentation and deployment of your Chef repository & infrastructure
Rackspace Private Cloud
• http://www.rackspace.com/cloud/private/• https://github.com/rcbops-cookbooks/• Session: Deploying OpenStack with Chef and Operational Tooling
Test Kitchen
• kitchen-openstack• https://github.com/RoboticCheese/kitchen-openstack• Session: Test Kitchen: Multi-Platform Integration Testing for the Masses
pxe_dust
• Provisioning solution for hardware• Initially developed by Matt Ray• https://github.com/opscode-cookbooks/pxe_dust• pxe_dust::bootstrap_template• pxe_dust::installers• pxe_dust::server
Crowbar
• Provisioning solution for hardware• Initially developed by Dell• crowbar.github.com• Barclamps!• 1.0 vs. 2.0
Razor
• Provisioning solution for hardware• Initially developed by EMC and Puppet• Open sourced as a Puppet Labs project• Install using Puppet, Chef, or manual• Auto-Discovered Real-Time Inventory Data• Dynamic Image Selection• Model-Based Provisioning• APIs and Plug-in Architecture• Metal-to-Cloud Application Lifecycle Management• Session: Harnessing the Power of Bare Metal with Razor and Chef Server
OpenStack Baremetal
• https://wiki.openstack.org/wiki/Baremetal• driver to allow OpenStack Compute to manage hardware directly (Grizzly)• provisioned via PXE and managed via IPMI• OpenStack Compute manages them via the Dashboard, CLI and API• OpenStack on OpenStack (aka "Triple-O")• authentication, authorization, quotas, a dashboard and an API provided by OpenStack• roadmap has device discovery, network management and additional hardware features
Agenda
• OpenStack Overview• Chef for OpenStack Overview• Knife OpenStack• Lunch 12:15• Chef for OpenStack Grizzly Roadmap• Related Technologies• Code Walkthroughs
currently Folsom release (v3.0.1)
open source Chef 11 server embedded
http://www.rackspace.com/cloud/private/
https://github.com/rcbops
https://github.com/rcbops-cookbooks/
Rackspace Private Cloud
Rackspace Private Cloud
github.com/rcbops/chef-cookbooks/tree/v3.0.1
roles/
environments/
cookbooks/
Rackspace Private Cloud
cookbooks/apache2/apt/aws/build-essential/cinder/collectd/collectd-graphite/collectd-plugins/corosync/database/dsh/erlang/exerstack/glance/graphite/haproxy/hardware/horizon/keepalived/keystone/kong/memcached/modules/
monit/monitoring/mysql/mysql-openstack/nova/nova-network/ntp/openssh/openssl/openstack-ha/osops-utils/postgresql/rabbitmq/rabbitmq-openstack/rsyslog/runit/selinux/sosreport/swift/sysctl/xfs/yum/
environments/example.json{ "name": "example_environment", "description": "Chef-server example environment", "cookbook_versions": { }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { }, "override_attributes": { "developer_mode": false, "monitoring" : { "metric_provider" : "collectd", "procmon_provider" : "monit" }, "glance": { "image_upload": true, "images": ["cirros", "precise"] },
environments/example.json "nova": { "ratelimit" : { "api" : { "enabled" : true }, "volume" : { "enabled" : true } }, "libvirt": { "virt_type": "qemu" }, "networks": [ { "label": "public", "ipv4_cidr": "10.10.100.0/24", "num_networks": "1", "network_size": "255", "bridge": "br100", "bridge_dev": "eth0.100", "dns1": "8.8.8.8", "dns2": "8.8.4.4" }, { "label": "private", "ipv4_cidr": "172.16.101.0/24", "num_networks": "1", "network_size": "255", "bridge": "br101", "bridge_dev": "eth0.101", "dns1": "8.8.8.8", "dns2": "8.8.4.4" } ] },
environments/example.json
"mysql": { "allow_remote_root": true, "root_network_acl": "%" }, "osops_networks": { "nova": "192.168.1.0/24", "public": "192.168.1.0/24", "management": "192.168.1.0/24" }, "package_component": "folsom" }}
roles/allinone.rbbase.rbcinder-all.rbcinder-api.rbcinder-scheduler.rbcinder-setup.rbcinder-volume.rbcollectd-client.rbcollectd-server.rbglance-api.rbglance-registry.rbglance-setup.rbglance.rbgraphite.rbha-controller1.rbha-controller2.rbhaproxy.rbhorizon-server.rbjenkins-allinone.rbjenkins-compute.rbjenkins-controller.rbjenkins-glance.rbjenkins-mysql-master.rbjenkins-nova-api.rbkeystone-api.rbkeystone.rbmysql-master.rb
nova-api-ec2.rbnova-api-os-compute.rbnova-api.rbnova-cert.rbnova-controller.rbnova-misc-services.rbnova-network-compute.rbnova-network-controller.rbnova-network.rbnova-scheduler.rbnova-setup.rbnova-vncproxy.rbnova-volume.rbopenstack-ha.rbquantum-network-manager.rbrabbitmq-server.rbrsyslog-client.rbrsyslog-server.rbsingle-compute.rbsingle-controller.rbswift-account-server.rbswift-all-in-one.rbswift-container-server.rbswift-management-server.rbswift-object-server.rbswift-proxy-server.rbswift-setup.rb
roles/allinone.rbname "allinone"description "This will create an all-in-one Openstack cluster"run_list( "role[single-controller]", "role[single-compute]")
expanded run list
osops-utils::packages, openssh, ntp, sosreport, rsyslog::default, hardware, osops-utils::default, mysql-openstack::server, erlang::default, rabbitmq-openstack::server, keystone::server, keystone::keystone-api, glance::setup, glance::registry, glance::api, nova::nova-setup, nova-network::nova-controller, nova::scheduler, nova::api-ec2, nova::api-os-compute, nova::volume, nova::nova-cert, nova::vncproxy, mysql::client, mysql::ruby, horizon::server, nova-network::nova-compute, nova::compute
roles/single-controllername "single-controller"description "Nova Controller (non-HA)"run_list( "role[base]", "role[mysql-master]", "role[rabbitmq-server]", "role[keystone]", "role[glance-setup]", "role[glance-registry]", "role[glance-api]", "role[nova-setup]", "role[nova-network-controller]", "role[nova-scheduler]", "role[nova-api-ec2]", "role[nova-api-os-compute]", "role[nova-volume]", "role[nova-cert]", "role[nova-vncproxy]", "role[horizon-server]")
roles/basename "base"description "Base role for a server"run_list( "recipe[osops-utils::packages]", "recipe[openssh]", "recipe[ntp]", "recipe[sosreport]", "recipe[rsyslog::default]", "recipe[hardware]", "recipe[osops-utils::default]")default_attributes( "ntp" => { "servers" => ["0.pool.ntp.org", "1.pool.ntp.org", "2.pool.ntp.org"] })
roles/mysql-mastername "mysql-master"description "Installs mysql and sets up replication (if 2 nodes with role)"run_list( "role[base]", "recipe[mysql-openstack::server]")
roles/rabbitmq-servername "rabbitmq-server"description "RabbitMQ Server (non-ha)"run_list( "role[base]", "recipe[erlang::default]", "recipe[rabbitmq-openstack::server]")
roles/keystonename "keystone"description "Keystone server"run_list( "role[base]", "recipe[keystone::server]", "role[keystone-api]")
roles/glance-setupname "glance-setup"description "sets up glance registry db and passwords"run_list( "role[base]", "recipe[glance::setup]")
roles/glance-registryname "glance-registry"description "Glance Registry server"run_list( "role[base]", "recipe[glance::registry]")
roles/glance-apiname "glance-api"description "Glance API server"run_list( "role[base]", "recipe[glance::api]")
roles/nova-setupname "nova-setup"description "Where the setup operations for nova get run"run_list( "recipe[nova::nova-setup]")
/nova-network-controller
name "nova-network-controller"description "Setup nova-networking for controller node"run_list( "recipe[nova-network::nova-controller]")
roles/nova-schedulername "nova-scheduler"description "Nova scheduler"run_list( "role[base]", "recipe[nova::scheduler]")
roles/nova-api-ec2name "nova-api-ec2"description "Nova API EC2"run_list( "role[base]", "recipe[nova::api-ec2]")
/nova-api-os-computename "nova-api-os-compute"description "Nova API for Compute"run_list( "role[base]", "recipe[nova::api-os-compute]")
roles/nova-volumename "nova-volume"description "Nova Volume Service"run_list( "role[base]", "recipe[nova::volume]")
roles/nova-certname "nova-cert"description "Nova Certificate Service"run_list( "role[base]", "recipe[nova::nova-cert]")
roles/nova-vncproxyname "nova-vncproxy"description "Nova VNC Proxy"run_list( "role[base]", "recipe[nova::vncproxy]")
roles/horizon-servername "horizon-server"description "Horizon server"run_list( "role[base]", "recipe[mysql::client]", "recipe[mysql::ruby]", "recipe[horizon::server]")
roles/single-computename "single-compute"description "Nova compute (with non-HA Controller)"run_list( "role[base]", "role[nova-network-compute]", "recipe[nova::compute]")