Infrastructure Automation
Ohad Levy – Infineon [email protected]
Page 2
Automated
”…a process which may once have been performed manually but has been altered in some way which allows a machine or computer to either wholly or partially manipulate the process to save time”
Page 3
Infrastructure
“Infrastructure is generally a set of interconnected structural elements that provide the framework supporting an entire structure”
Page 4
Automated Infrastructure
“Having the basic services necessary for your infrastructure to operate largely without the aid of a keeper.”
Page 5
Grow – from This:
http://i68.photobucket.com/albums/i2/RealConnections/googlecomputers.jpg
Page 6
Large Scale Environment
Page 7
Where does it fit in?
Page 8
Typical System Lifecycle
Pre/ Installation
DNS/DHCP…
Initial Configuration
KickstartJumpstart
Custom Scripts
FixesUpdatesAudits
Page 9
System configuration possibilities Manual System Configuration
„log in and „just do it!““¬ + OK for small number of servers¬ + Quick and easy
¬ - Very difficult to ensure similar configurations¬ - No auditing capability¬ - No change history¬ - No release documentation¬ - No way to deploy multiple servers quickly.¬ - No way to rebuild critical servers quickly.
Page 10
System configuration possibilities Install Time Auto-Configuration
„Deploy using Kickstart, Jumpstart etc“¬ + Ensures that new systems are brought into proper state as part
of installation process ¬ + Possible to deploy multiple servers quickly
¬ - No Validation that settings remain correct¬ - No easy way to deploy changes¬ - Auto-Configuration process is different between the sites leads
to different configurations ¬ - No Auditing¬ - No change history¬ - No release documentation
Page 11
Typical System Lifecycle
Pre/ Installation
DNS/DHCP…
Initial Configuration
KickstartJumpstart
Custom Scripts
FixesUpdatesAudits
Page 12
Puppet What is Puppet ?
A product designed to deploy system configurations. It is – ¬ Open source based on Ruby¬ Policy based¬ Runs every 30 minutes¬ An abstraction layer between the system administrator and the
system¬ Capable to run on any UNIX operating system
Page 13
Our Challenges
Keep our systems "harmonized“ Audit / Know what's going on each system Replace a server if it dies or to be able to add another server
that is exactly like it Similar Applications, different OS's Push out changes to all the servers that need a particular change Stop duplicating effort Go home early
How does Puppet works?
Page 15
Puppet Types
A Type is a particular element that Puppet knows how to configure
Files (content, permissions, ownership) Packages (installations, updates..) Services (enabled/disabled, running/stopped) Exec (run commands) Full List: cron, exec, file, filebucket, group, host, interface,
k5login, mailalias, maillist, mount, nagios*, package, service, sshkey, tidy, user, yumrepo, zone
Page 16
Example: Managing sudoers file
file { /etc/sudoers: owner => root, group => root, mode => 600, source => puppet://server/files/sudoer
}
Page 17
Dependencies
"require" and "before" / "after" settings ensures that types are applied in the correct orderfile { "/etc/sudoers": ... require => Package[sudo]}package { "sudo": ensure => present, before => File["/etc/sudoers"]}
Page 18
Dependencies - continued "notify" and "subscribe" settings can trigger cascaded updates
Particularly useful in services, exec
file { "/etc/ssh/sshd_conf": ... notify => Service["sshd"]
}service { "sshd": subscribe => File["/etc/ssh/sshd_conf“]}
Page 19
What is Facter?
Facter gathers information about the client, which can be used as variables within puppet.
You can add custom facts as needed.package {"sshd":
ensure => installed,name => $operatingsystem ? {
solaris => "IFKLssh",default => “openssh-server”
}}
Page 20
Example Facts$ sudo facterarchitecture => amd64domain => sin.infineon.comfacterversion => 1.3.8fqdn => sinn1636.sin.infineon.comhardwaremodel => x86_64hostname => sinn1636ipaddress => 172.20.89.113kernel => Linuxkernelrelease => 2.6.24-21-genericlsbdistcodename => hardylsbdistdescription => Ubuntu 8.04.1lsbdistid => Ubuntulsbdistrelease => 8.04macaddress => 00:1c:25:14:26:abmanufacturer => LENOVOmax speed => 2000 MHzmaximum memory module size => 4096 MB
maximum total memory size => 8192 MBmemoryfree => 988.30 MBmemorysize => 1.94 GBoperatingsystem => Debianoperatingsystemrelease => 2.6.24-21-genericprocessor0 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessor1 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessorcount => 2productname => 6458B43ps => ps -efpuppetversion => 0.24.5rubysitedir => /usr/local/lib/site_ruby/1.8rubyversion => 1.8.6serialnumber => L3A8632swapfree => 912.23 MBswapsize => 956.99 MBtype => Notebookuniqueid => 007f0101vendor => LENOVO
Page 21
What is a Class? A named collection of type objects Can include or inherit from other classesclass sudo_class { include foo_class file { "/etc/sudoers": ... } package{ "sudo": ... }}
Page 22
Class inheritance
class afile { file { "/tmp/foo": ensure => file source => "/src/versionA" }}class another_file inherits afile { File["/tmp/foo"] { source => "/src/versionB" }}
Page 23
What is a Node ? A configuration block matching a client Can contain types, classes "default" node matches any client without a node block
node "ohad.myself" { include sudo_class include other_class}
Page 24
External Nodes Node definitions can be defined outside of puppet – e.g. LDAP,
external script… Ideal when you have too many nodes..
Page 25
Classes and definitions Classes are groups of resources. Definitions are similar to classes, but they can be instantiated multiple
times with different arguments on the same node.
class apache2 {define simple-vhost ( $admin = "webmaster", $aliases, $docroot) {
file { "/etc/apache2/sites-available/$name":mode => "644",require => [ Package["apache2"], Service["apache2"] ],content => template("apache/vhost.conf"),
} }
}
Page 26
Classes and definitions - Continued
node mywebserver {include apache2
apache2::simple-vhost { "test.example.com": docroot => "/var/www/test“
}
apache2::simple-vhost { "debian.example.com": docroot =>"/var/www/debian“,alias => [“debiantest.example.com”,”debian”],admin => “[email protected]”
}}
Page 27
vhost.conf template Puppet uses Ruby's ERB template system:<VirtualHost *> ServerAdmin <%= admin %> DocumentRoot <%= docroot %> ServerName <%= name %><% aliases.each do |al| -%> ServerAlias <%= al %><% end -%> ErrorLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/error-%d" CustomLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/access %d" sane</VirtualHost>
Page 28
Template output
more /etc/apache2/sites-available/debian.example.com<VirtualHost *> ServerAdmin [email protected] DocumentRoot /var/www/debian ServerName debian.example.com ServerAlias debiantest.example.com ServerAlias debian ErrorLog |/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/error-%d“ CustomLog "|/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/access-%d“ sane
</VirtualHost>
Page 29
OS API - It also works theother way around:
$ ralsh user levyouser { 'levyo': groups => ['tidc'], ensure => 'present', password => 'x', shell => '/bin/bash', uid => '49960', comment => 'Ohad Levy, +65 xxxx xxxx', gid => '11100', home => '/home/levyo'}
Page 30
Puppet Modules Modules allow you to group both the logic and the files for an
application together. Puppet automatically searches its module path to find modules. Modules can contain four types of files, each of which must be
stored in a separate subdirectory: Manifests – must be stored in manifests/, and if you create
manifests/init.pp then that file will be loaded if you import that moudle name directly (e.g. import “mymodule”)
Templates – must be stored under templates/, put in your manifest some where template(“mymodule/mytemplate”)
Files – stored in files/, can be accessed as: source => puppet://”mymodule”/myfile
Plugins – Additional types, provides or facts
Page 31
File server and File Bucket Puppet also includes a file server which you can use for
transferring files from the server to the client. If you configure it, puppet can also save a backup of each file
that is changed on the client to the server. The backups go in a filebucket and can be retrieved later.
Page 32
Getting Started Install puppet (yum,pkg-get,…) or manually install ruby, facter
and puppet Setup a puppet server (puppetmaster / puppet-server). Write a manifest for your node (classes etc) Start puppet master on the server Start puppetd on the client
Page 33
Some more info Puppet uses SSL for all communications, therefor it includes a
CA, you can automatically sign CSR or use puppetca tool to mange them.
Storeconfig, option to save machine states(facts, configuration runs) and special facts (e.g. SSH keys) in a database.
Reporting, puppet has a few reporting options, most common are emails with changes, RRD files, yaml files and puppetshow web interface.
Puppet HA, load balancing etc
Page 34
What are the benefits ? Costs
Based on open source tools – No license cost Reduced Labor Reduced Cost
Productivity Reproducibility, Accuracy and homogenous environment (Reduce needed knowledge to operate our environment) Allow easy ramp up of new sites and services
Change Management Testing Processes, Auditing Global control over servers Harmonization Consolidation
…
Page 35
Conclusions We're all stuck on the hamster wheel Makes easy stuff easier, hard stuff possible Harmonize! (not just today, also in the future) Enable the business for consolidation Similar projects
cfengine bcfg2
Additional Resources http://reductivelabs.com/trac/puppet http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
#puppet on irc.freenode.org
Backup SlidesAdditional information
Page 37
Configuration DB - Terminology Module – A bundle of settings related to a single applicationAt Infineon we use 3 types of modules:
Host Type - e.g. Login Servers, Compute nodes… Service - module which is needed by other modules, e.g. ssh, x11 etc. Sites Modules - sites customizations
Versioning – The ability to refer (tag) to a certain state/PiT Host Type and Service modules are associated with a version, e.g.
Login Server v1.0 Site Modules are not tagged in order to reduce complexity
* Every change in a module, requires a check in action, which is identified by a user, and a comment (change log)
* A user access restriction should be applied to each module and to the tagging mechanism
Page 38
Configuration DB - Terminology Environments – The ability to select a group of modules in a
certain versionEach site is associated with at least two environments – Production
and Testing: Testing – an environment which refer to versioned modules
under a test tag. Production – A selection of modules in a certain version that
has already been approved for Production usage. Development environment – The state where the modules are in
the latest check-in version, but are not tagged (e.g. not a testing candidate yet).
Development Testing Production (Per Host Type)
Page 39
Configuration DB - Terminology Global modules / owners
Each owner is responsible for his module (or modules) A special attentions is required for Service module as many
other modules might depend on them
Site modules As Site modules are not tagged (versioned) any change in the
site specific settings, will be rolled out immediately.
Page 40
Risks & Challenges If the right processes are not in place, it would be possible to
implement the wrong changes globally Learning Curve
How our platform works How to use version control How to work in the new processes/workflows
Fear (Am I obsolete?, will I continue to learn?)
Page 41
Why Did we choose Puppet Open source More flexible than other open source alternatives Really works (Declarative language, Abstracts configuration as
resources, Allows relationships, Transactional, Modular) Provides an API to the OS, making it very simple to support other
OS’s in the future Puppet is now being adopted in many projects For more info see
http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppethttp://reductivelabs.com/trac/puppet/wiki/WhosUsingPuppet
Page 42
Datacenters Challenges Harmonization Simplify Servers management Implement and maintain Global standards Accurate Configuration Database Rapid recovery and deployment of services globally Eliminates duplicate / unnecessary effort Same look and feel everywhere (enables consolidation / global
operation) Audits / Compliance
Page 43
Infineon Puppet Infrastructure
Configuration Database(Puppeteer, SVN, Netapp)
configuration Server
Site A
deployment &monitoring
Managed Servers
Failover
Site B
deployment &monitoring
Managed Servers
configuration Server
• Puppeteer (Puppet Master of site PM’s) • Puppet Master per site (or based on size/load)• SQL Database for Web portal, Inventory• Configuration database (Version control & Ticketing system)
Page 44
Gini’s Features AD/LDAP Authentication DHCP/DNS Connectors (API, + MS(yacks) DHCP, MS(yacks) DNS and infoblox) Puppet Host type management Disk Layout Management Host Hardware management (needed for Solaris DHCP jumpstart, and general host info) Multi OS management (e.g. RH, CentOs, Solaris.... in their various versions) User role management (who is allowed to install, in which site, which host type etc) Subnet and IP Management (auto assignment of IP address, DHCP reservation DNS records etc) Multi Domain / site management Hands-free auto OS installation (currently Kickstart and Jumpstart (sparc) using DHCP) Management of Puppet Environments, choose an env per host (e.g. we have different production states for
different sites, or for different host types). Management of puppet certificates (also support chained certificates). Import of Machine Facts (we cant use store config due to the large setup of our env) Nice Graphs (with Gruff) Puppet modules selection per host(for those who needs more than the default host type) SP (Service Processor, iLOM etc) management TFTP management Status indications for machine state (pre installation, during puppet first run, etc) ... and probably a lot of other small stuff that I cant think of now
Page 45
Our Solution
Web Portal
Puppet
Configuration DBInventory
DHCP/DNS..
Page 46
Where does GINI live?
GINI
DBDNS
TFTP
CERTIFICATES
CMDB
DHCP
LOGS LSF
Future work
Top Related