Ultimate SharePoint 2013 Infrastructure Best Practices Session - SPKSLO 2012
Infrastructure as Code - ABUG Session
-
Upload
patrick-debois -
Category
Technology
-
view
116 -
download
5
description
Transcript of Infrastructure as Code - ABUG Session
Infrastructure as
Patrick DeboisABUG-Session
Blog: http://jedi.be/blog - Twitter : @patrickdebois - #devops
Technical ReviewerMonitoring Chapter Co-Author Veewee / Sahara / Mccloud
Organizer Since 2009 Europe Organizer 2010
Engineer
First Europe Training 2010
Libvirt - Fog
Speaker
Freelance consultant http://github.com/jedi4ever
CONFIGURATION MANAGEMENT SYSTEMS
http://ansible.cc/
http://opscode.com/chef/
http://cfengine.com/http://palletops.com/http://puppetlabs.com/
http://www.nerdnirvana.org/wp-content/uploads/2010/12/apology-form.jpg
Chef & Puppet focusedbased on my experience
SERVICE
FILE2
FILE1PACKAGE
DECLARATIVE LANGUAGE
“KNOWN STATE”
STATE X STATE YCODE RUN
FILE1SERVICE
FILE2
FILE1PACKAGE
convergence
“IDEMPOTENCE”
STATE Y STATE YCODE RUN
SERVICE
FILE2
FILE1PACKAGE
SERVICE
FILE2
FILE1PACKAGE
GIVE ME THE CODE:“LANGUAGE”
Puppet Chef
class myapache {
package “apache2”
service “apache2”: ensure => “running”, require => Package[“apache2”]
}
package “apache2”
service “apache2 do action [:enable , :start]end
EXAMPLE SYNTAX
TERMINOLOGY
Java Puppet Chef
*.javamanifests
*.pprecipes
*.rb
package module cookbook
TERMINOLOGY (2)
Java Puppet Chef
singletonparametrized
classesrecipes with attributes
classdefines, classes
definitions, libraries
SYNTAXPuppet Chef
Limited DSL(by design) DSL + full ruby
Extended by Definitions
Light Weight Resource Providers /
DefinitionsCustom Resource providers
Providers, Light Weight Resource
ProvidersERB & Custom Libraries Libraries
LOOPING CONSTRUCTS
Puppet Chef
pass array of elementserb template for loops
create_resource (:type, hash)
do while , repeat etc. available through
native ruby
VARIABLES
Puppet Chef
special syntax$bla = “aa”
string interpolation$bla_string = “${bla}”
native rubybla = “aa”
string interpolationbla_string = #{bla}
SINGLE CLASSINHERITANCE
Puppet Chef
override via ‘module path’
override via ‘cookbook’ path
+> operator “cookbook” attribute
Puppet Module Chef Cookbook
files/lib/
README.rdocmanifests/templates/
attributes/definitions/files/libraries/metadata.rbproviders/README.rdocrecipes/resources/templates/
PROJECT STRUCTURE
EXECUTION MODEL
Puppet Chef
Compile catalog(directed graph
model)Then Execute
Top Down execution
PLUGINSPuppet Chef
FACES KNIFE PLUGINS
REPORTS REPORT HANDLERS
http://puppetlabs.com/blog/puppet-faces-what-the-heck-are-faces/http://wiki.opscode.com/display/chef/Knife+Plugins
CODE EDITOR
https://github.com/rodjek/puppet-pygments-lexerhttp://redmine.djagios.org/projects/puppet-lexer
LEXERS
VIM SYNTAX HIGHLIGHTIING
https://github.com/rodjek/vim-puppethttps://github.com/t9md/vim-chef
https://github.com/vim-scripts/Cfengine-version-3-syntax
http://cloudsmith.github.com/geppetto/https://github.com/cloudsmith/geppetto
GEPPETTO - ECLIPSE IDE
JETBRAINS RUBYMINE (PUPPET)http://youtrack.jetbrains.com/issue/RUBY-10832
REACTOR8http://beta.r8network.com/
COMING SOON
https://github.com/janschumann/intellij-lang-puppet
REVERSE ENGINEERING
https://github.com/devstructure/blueprint
Existing Server -> Manifest/Recipes
CODING STYLE
Puppet Chef
http://puppet-lint.com/ http://acrmp.github.com/foodcritic/
• Must use two-space soft tabs.• Must not use literal tab characters.• Must not contain trailing white space.• Should not exceed an 80 character line width
An exception has been made for source => 'puppet://...' lines as splitting these over multiple lines decreases the readability of the manifests.
• Should align arrows (=>) within blocks of attributes.• ....
•
• FC001: Use strings in preference to symbols to access node attributes• FC002: Avoid string interpolation where not required• FC003: Check whether you are running with chef server before using server-
specific features• FC004: Use a service resource to start and stop services• FC005: Avoid repetition of resource declarations• FC006: Mode should be quoted or fully specified when setting file
permissions• FC007: Ensure recipe dependencies are reflected in cookbook metadata• FC008: Generated cookbook metadata needs updating• FC009: Resource attribute not recognised• FC010: Invalid search syntax• FC011: Missing README in markdown format• FC012: Use Markdown for README rather than RDoc• FC013: Use file_cache_path rather than hard-coding tmp paths•
http://docs.puppetlabs.com/guides/style_guide.htmlhttps://github.com/ampledata/cookbook-style-guide
http://www.slideshare.net/PuppetLabs/modern-module-development-ken-barber-2012-edinburgh-puppet-camp
CODE VS CONFIG(THINK “PROFILES”)
http://wiki.opscode.com/display/chef/Data+Bags
http://projects.puppetlabs.com/projects/hierahttps://github.com/crayfishx/hiera-gpg
https://github.com/jedi4ever/stop-the-forkhttp://realityforge.org/code/2012/11/12/reusable-cookbooks-revisited.htmlhttp://devopsanywhere.blogspot.it/2012/11/how-to-write-reusable-chef-cookbooks.html
“STOP THE FORK(ING)”
PRE-COMMIT CHECKS
DEPENDENCY MGMT
GitSubmodules
Subtree à la Pom File
UPSTREAM REPOS
http://forge.puppetlabs.com/http://community.opscode.com/
Puppet Chef
https://github.com/rodjek/librarian-puppet
https://github.com/applicationsonline/librarian
http://berkshelf.com/
site "http://community.opscode.com/api/v1"
cookbook "ntp"cookbook "timezone", "0.0.1"
cookbook "rvm", :git => "https://github.com/fnichol/chef-‐rvm", :ref => "v0.7.1"
cookbook "cloudera", :path => "vendor/cookbooks/cloudera-‐cookbook"
CHEF METADATA
http://bitfieldconsulting.com/puppet-dependency-graphs
Dependency Graphs
https://github.com/miketheman/knife-role-spaghetti
DEBUGGING
LOGGING WITH DIFFERENT LEVELS
Info only, no exceptionChef::Log.info('Some useful info')
Fatal, raising exceptionChef::Log.fatal('Something bad')
REPL - STYLE(READ EVAL PRINT)
http://wiki.opscode.com/display/chef/Shefhttps://github.com/pry/pry
DEV ENVIRONMENT
Info & Credits
• Mitchell Hashimoto & John Bender
• http://vagrantup.com
• Based on Virtualbox , Ruby
• OS-Installers (not gem based)
• Runs on Mac OS, Linux, Windows
Vagrant 101
Step 1 : installation
The power of CLI
$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init$ vagrant up
Step 2 : base box
$ gem install vagrant
$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant up
Downloads predefined VM - f.i. Ubuntu 10.04 (Lucid)and registers it as name ‘base’
‘BASE’LUCID
Step 3 : configuration
$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init$ vagrant up
Creates a standard configuration file
Vagrantfile
Vagrant::Config.run do |config| # Every Vagrant virtual environment requires a box to build off of. config.vm.box = "base"
end
‘BASE’LUCID
Vagrantfile = Project fileLinks our project with the ‘Base’ box
Step 4 : boot
$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init
$ vagrant up
Extracts the ‘basebox’Creates a new VM in Virtualbox based on ‘basebox’
Boots the new VM
Ready to rock SSH
KA-CHING
I ♥ Vagrant
Reduces Setup Time
1 2 3 4
No-VMSSH
Ready
♥
Automated
Simplicity♥
$ gem install vagrant$ vagrant box add base http://files.vagrantup.com/lucid32.box$ vagrant init$ vagrant up
VBoxManage guestcontrol exec[ute] <vmname>|<uuid> <path to program> --username <name> --password <password> [--arguments "<arguments>"] [--environment "<NAME>=<VALUE> [<NAME>=<VALUE>]"] [--flags <flags>] [--timeout <msec>] [--verbose] [--wait-for exit,stdout,stderr||]
copyto|cp <vmname>|<uuid> <source on host> <destination on guest> --username <name> --password <password> [--dryrun] [--follow] [--recursive] [--verbose]
createdir[ectory]|mkdir|md <vmname>|<uuid> <directory to create on guest> --username <name> --password <password> [--parents] [--mode <mode>] [--verbose]
updateadditions <vmname>|<uuid> [--source <guest additions .ISO>] [--verbose]
Self-Servicing♥
Basebox
“PULL”
Consistency
♥
Basebox
DEVELOPER1 DEVELOPER2
Repeatability
♥
Basebox
UP DESTROY UP DESTROY
t1 t2 t3 t4
Contained
♥
Project 1 Project 2 Project 3
Laptop
Vagrant 201
FileSharing
/home/patrick/data
/data
config.vm.share_folder "datastore", "/data", "./data"
Port Forwarding
80
8080
# Forward a port from the guest to the host, which allows for outside # computers to access the VM, whereas host only networking does not. config.vm.forward_port "http", 80, 9000 config.vm.forward_port "redis", 6379, 6379 config.vm.forward_port "mysql", 3306, 3306
Non-Intrusive
♥
Project 1 Project 2 Project 3
Laptop
My Editor My IDE My Browser
Multi VM Stack
♥
DB DB DB
Project 1 Project 2 Project 3
APP APP APP
WEB WEB WEBconfig.vm.define :web do |web_config| web_config.vm.box = "ubuntu"
...endconfig.vm.define :app do |app_config| app_config.vm.box = "ubuntu"
...endconfig.vm.define :db do |db_config| db_config.vm.box = "ubuntu"
...end
Vagrant 301
Basebox
Provision
Just EnoughOperating System
Provisionerchef/puppet/script
+
Cookbooks/Manifests
VM
Package
Basebox
Just EnoughOperating System
Provisionerchef/puppet/script
+
Cookbooks/Manifests
VM Basebox
Provision Configconfig.vm.provision :chef_solo do |chef| chef.cookbooks_path = ["chefrepo/cookbooks"]
chef.add_recipe("app_rails")chef.json.merge!({
:apache => { :listen_ports => [ "8080"]})
end ChefSolo
ChefServer
Puppet standalone
Puppet Server
config.vm.provision :puppet do |puppet| puppet.pp_path = "/tmp/vagrant-puppet" puppet.manifests_path = "./manifests" puppet.module_path = "./modules" puppet.manifest_file = "newbox.pp" end
Veewee
https://github.com/jedi4ever/veewee
Basebox
Just EnoughOperating System
Provisionerchef/puppet/script
ISO
Kickstart/Preseed/Autoyast/
AutoUnattendUbuntu, Debian, Centos,
Fedora, Gentoo, Solaris, Suse, Archlinux,
Windows
JEOS CREATION
Just Enough Operating System+ Config Management Tool
Your Laptopvirtualization
Virtualbox , FusionKVM, Parallels
Linux & Windows
https://github.com/jedi4ever/veewee/
Centos, Ubuntu, Debian, Solaris, Gentoo, Suse, Oracle Linux, ....
And windows now too!
SERVICE
FILE2
FILE1PACKAGE
Virtual Machine (1)
Just Enough Operating System+ Config Management Tool
Your Laptopvirtualization
http://vagrantup.com/
Virtualbox + Fusion + EC2
Linux & Windows
Self-Executable Boxes
• java -jar mygreatbox.jar
http://www.jedi.be/blog/2011/03/31/installable-vagrant-boxes/https://github.com/maestrodev/wanton
“proof of concept”
https://github.com/opscode/bento
uses #veewee
CREATING BASEBOX ‘EN MASS’
https://github.com/atkm/seisan-linehttps://github.com/atkm/origami
http://www.vagrantbox.es/
http://www.morethanseven.net/2011/05/08/Vagrant-plugin-for-interacting-with-vagrantboxes/
http://vagrantfil.es/
http://www.nodescription.net/ideation-into-the-void-vagrant-plugin-for-vag
SHARE YOUR SETUP
CLI Vagrantbox.es
>> vagrant vagrantboxes search centos
3 centos 5.5 http://dl.dropbox.com/u/15307300/vagrant-0.7-centos-64-base.box6 opscode centos 5 http://opscode-vagrant-boxes.s3.amazonaws.com/centos5-gems.box7 opscode ubuntu 10.04 http://opscode-vagrant-boxes.s3.amazonaws.com/ubuntu10.04-gems.box9 puppet centos 5.5 64 http://puppetlabs.s3.amazonaws.com/pub/centos5_64.box10 puppet centos 4 64 http://puppetlabs.s3.amazonaws.com/pub/centos4_64.box21 centos 5.6 32 http://yum.mnxsolutions.com/vagrant/centos_56_32.box
https://github.com/garethr/ruby-vagrantboxes
Version ControlledBasebox
Just EnoughOperating System
Provisionerchef/puppet/script
ISO
Kickstart/Preseed/Autoyast/
AutoUnattend
VM
♥
Git Repo Git Repo
Cookbooks/Manifests
Vagrantfile
Git Repo
Complex Setups
VM
♥
Cookbooks/Manifests
Rails Nodejs FlumeHadoop Esper
Mysql HDFSPostgres Java
Sharing on Github
Kickstart/Preseed/Autoyast/
AutoUnattend
♥
Git Repo Git Repo
Cookbooks/Manifests
Vagrantfile
Git Repo
Veewee Chef/Puppet Vagrant
Kickstart/Preseed/Autoyast/
AutoUnattend
Git Repo Git Repo
Cookbooks/Manifests
Vagrantfile
Git Repo
Git Repo
Application Code
Shared “codebase”♥
Kickstart/Preseed/Autoyast/
AutoUnattend
Git Repo
Git Repo
Cookbooks/Manifests
Vagrantfile
Git Repo
Git Repo
Application Code
Co-Ownership♥
DEV & OPSpairing
More Ops into Dev
• can we speed up this provisioning?
• we need a migration script from vX -> vY
• we need start/stop scripts
• we need consistent data restore
♥
Synchronized Release Cycle
♥
Application vX
System vX
Application vX+1
System vX+1
Tools/Config Consistency
♥
Cookbooks/Manifests
Cookbooks/Manifests
DEV PROD
Provisionerchef/puppet/script
Provisionerchef/puppet/script
Kickstart/Preseed/Autoyast/
AutoUnattend
Kickstart/Preseed/Autoyast/
AutoUnattend
Disposable Test Environments♥
Cookbooks/Manifests
Cookbooks/Manifests
DEV PROD
Provisionerchef/puppet/script
Provisionerchef/puppet/script
Kickstart/Preseed/Autoyast/
AutoUnattend
Kickstart/Preseed/Autoyast/
AutoUnattend
TEST/QA
http://paperairoplane.net/?p=240
Cookbooks/Manifests
Provisionerchef/puppet/script
Kickstart/Preseed/Autoyast/
AutoUnattend
Reuse Workflow
Vagrant
https://github.com/geemus/fog
♥
https://github.com/BIAINC/vagrant-windows
https://github.com/dotless-de/vagrant-vbguesthttps://github.com/folken-laeneck/vagrant-bindfs
https://github.com/BerlinVagrant/vagrant-dnshttps://github.com/dwt/vagrant-hosts
USEFULVAGRANT PLUGINS
http://marketplace.eclipse.org/node/519961#.UKISB-Oe-d4
VAGRANT ECLIPSE INTEGRATION
http://www.jetbrains.com/ruby/webhelp/configuring-remote-interpreters-via-virtual-boxes.html
VAGRANT - CLOJURE
https://github.com/tbatchelli/vmfest
VAGRANT - JAVA
https://github.com/guigarage/vagrant-binding
UNIT TESTING
https://github.com/nistude/cucumber-puppet
https://github.com/rodjek/rspec-puppet
https://github.com/calavera/rspec-chefhttps://github.com/acrmp/chefspec
https://github.com/sstephenson/bats
DON’T TEST YOUR CONFIG MANAGEMENT
TEST YOUR LOGIC
NOT
Was the package installed
YES
If (OS=’X’ ) and (file exists) ...
did it give an error?
FAKE THE FACTS
https://github.com/customink/fauxhai
AUTO-TRIGGER TESTS RUN
https://github.com/guard/guard-chefhttps://github.com/alister/guard-puppet-linthttps://github.com/johnbintz/guard-puppet
https://github.com/guard/guard-rspec
CODE EDIT
AUTO RUNFEEDBACK
CI INTEGRATION
https://github.com/nicksieger/ci_reporter
Junit processor for Ruby
SYNTAX CHECKSCOMPILE CHECKS
UNIT TESTS
SMOKE TESTS
NO VM NEEDED
REAL VM
SMOKE TESTING(ON REAL MACHINE)
Code Applied
CLEAN VM
STATE A STATE BCode Run Run TestsCreate VM
SNAPSHOT
http://auxesis.github.com/cucumber-nagios/
RE-USABLE CUCUMBER STEPS
https://github.com/hedgehog/cukenhttps://github.com/cucumber/aruba
http://www.cucumber-chef.org/
https://github.com/calavera/minitest-chef-handlerhttps://github.com/jedi4ever/puppet-assert
Validate “inside” the provision-run cycle
VM CREATION IS EXPENSIVE
https://github.com/jedi4ever/saharahttps://github.com/t9md/vagrant-snap
Code Applied
CLEAN VM
STATE A STATE BCode Run Run TestsCreate VM
SNAPSHOT
ROLLBACK
https://github.com/opscode/test-kitchen
USEFULVAGRANT PLUGINS
https://github.com/riotgames/vagrant-berkshelf
https://github.com/schisamo/vagrant-omnibus
https://github.com/rtyler/vagrant-plugin
TARGETED TESTINGIMPACT OF CHANGE
https://github.com/jedi4ever/puppet-cichttps://github.com/jonlives/knife-preflight
ROLE1
ROLE2
ROLEN
MODULEA
MODULEB
MODULEC
MODULE...
Real VM testingis expensive
Only test what’s needed
PERFORMANCE/METRICS
https://github.com/rodjek/puppet-profiler
https://github.com/joemiller/puppet-graphite_eventhttps://github.com/imeyer/chef-handler-graphite
https://github.com/krux/puppet-module-graphite-report
execution timenumber of classes
“It works on my machine”
♥
For Devs(Test Code)
For Ops(Test Manifests)
For QA(Explore)
For Sales(To show off)
Bridgingfour worlds
without changing!
♥
http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/
http://foodfightshow.org/
https://twitter.com/hangops
QUESTIONS?
OLDER OR RELATED STUFF
https://github.com/tknerr/bills-kitchen
Windows
LOCAL VM TESTING
http://www.cucumber-chef.org/
https://github.com/exceedhl/toft
Faster testing via LXC
spinning up a vm is expensiveunless in container
EC2+LXC
Vagrant+LXC
http://wiki.opscode.com/display/chef/Whyrun+Testing
https://github.com/lak/puppet-memcached
https://github.com/lak/puppet-static-compiler
https://github.com/rtyler/blimpy
https://github.com/jedi4ever/mccloud/
Vagrant ‘clones’
EC2, KVM, FUSION,HOSTS, Vagrant
EC2
https://github.com/neerolyte/vagueanthttps://github.com/lstoll/tenderloin FUSION
LXC
http://puppetlabs.com/blog/what-is-user-experience-in-puppet/
AUDITING
http://www.normation.com/en/solutions/rudderhttp://www.fusioninventory.org/
http://comodit.github.com/synapse-agent/
https://github.com/tobami/littlechef
https://github.com/dwt/vagrant-hosts
https://github.com/opscode/omnibus
https://github.com/puppetlabs/puppet-module-tool
http://wiki.opscode.com/display/chef/Recipes#Recipes-ExceptionsandLogging
http://projects.puppetlabs.com/projects/puppet/wiki/Anchor_Pattern
VENDORING
https://github.com/railsmachine/rump
http://www.example42.com/?q=Puppi_A_Puppet_module_for_Deployment_Automation
http://www.sonian.com/cloud-monitoring-sensu/
http://docs.puppetlabs.com/guides/parameterized_classes.html
AUTHENTICATION
DASHBOARDS
http://puppetlabs.com/puppet/related-projects/dashboard/
http://blog.mornati.net/2012/01/29/kermit-a-webui-for-mcollective/
http://www.opscode.com/hosted-chef/
https://github.com/chapmanb/cloudbiolinux
https://github.com/aetherical/nimblestratus
https://github.com/fiddyspence/puppet-mconotify
https://github.com/ankurcha/stemcell