Post on 12-Jul-2015
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Managing Fileswith Puppet
An introduction to configuration management
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Who am I?
• Walter Heck, Software engineer turned DBA, turned Sysadmin, turned entrepreneur
• Founder of OlinData (http://www.olindata.com) o Puppet Labs training partner for all of Asia and part of
Europeo Node.JS, OpenStack, Linux Foundation trainingo MySQL consulting
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Overview
• What is puppet (for those not aware)?
• Which modules to use for file management?
• Built-in puppet resources for working with files
• How to deal with cross-resource type conflicts
• Exported resources and the concat module
• Questions
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
What is Puppet and why do we care?
• Configuration management software
• Scales well (1-200k+ nodes)
• Multi-platform (windows, *nix, Mac OS, BSD)
• Commercially supported Open Source
• Infrastructure as code
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
1. source and content attributes of the file resource type
2. the concat module
3. the file_line resource type from the stdlib module
4. the inifile module
5. the augeas tool
What options do we have?
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The file resource type• used to manage whole files (and directories and symlinks)
• content of the file is managed in two mutually exclusive ways:o source => ‘puppet:///modules/apache/httpd.conf’o content => template(apache/httpd.conf.erb’)
• in heavy or large environments, split off file serving to a separate puppetmaster
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The file resource type - source
• Source attribute can use o local file on the agent
• syntax: source => ‘/usr/local/myfile.txt’
o remote file on the master• syntax: source => ‘puppet://<puppetmaster>/modules/apache/httpd.conf’
• source attribute copies content from source non-modifiableo great for static files eg. init.d scripts
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
$master: cat mymodule/manifests/bar.pp
class mymodule::bar {file { '/tmp/file01':ensure => file,owner => 'root',group => 'root',mode => '0644',source => 'puppet:///modules/mymodule/file01',
}file { '/tmp/file02':ensure => file,owner => 'root',group => 'root',mode => '0644',source => '/opt/local/file02',
}}
$master: cat mymodule/files/file01
This is a static file$agent: cat /opt/local/file02
This is a locally sourced static file
The file resource (source example)
$agent: puppet apply -e ‘include mymodule::bar’
$agent: cat /tmp/file01
This is a static file$agent: cat /tmp/file02
This is a locally sourced static file
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The file resource type - content
• Content attribute assigns a static string to the content of the file
o use the template() function to parse an erb template and assign the result.
• eg. content => template(‘apache/httpd.conf’)
• erb templates: standard ruby embedded templates• https://docs.puppetlabs.com/guides/templating.html
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
$ cat mymodule/manifests/foo.pp class mymodule::foo {
$say_hello_to = 'dude'$myname = 'file03'
file { "/tmp/$myname":ensure => file,content => template('mymodule/polite-file.erb'),
}}
$ cat mymodule/templates/polite-file.erb <% if @say_hello_to -%>Hello <%= @say_hello_to %>,<% end -%>I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.
$ puppet apply -e ‘include mymodule::foo’
$ cat /tmp/file03Hello dude,I'm file03, on a Ubuntu system, nice to meet you.
Note: the @operatingsystem variable (replaced here by “Ubuntu”) value is provided by Facter
The file resource (content example)
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The concat module
• puppet module developed by R.I.Pienaar originally in 2010o adopted by PuppetLabs:
https://github.com/puppetlabs/puppetlabs-concat• Allows to concatenate separate sections of files
o eg. puppet.conf on your puppet master• Uses a local nifty trick with a shell script
o check out: files/concatfragments.sh• Bonus: use exported resources on fragments to collect for
instance haproxy loadbalancer backends
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The concat module (example)class motd {$motd = '/etc/motd'
concat { $motd:owner => 'root',group => 'root',mode => '0644'
}
concat::fragment{ 'motd_header':target => $motd,content => "\nPuppet modules on this server:\n\n",order => '01'
}
# local users on the machine can append to motd by just creating# /etc/motd.localconcat::fragment{ 'motd_local':target => $motd,source => '/etc/motd.local',order => '15'
}}
#Note: concat::fragment resources for a single target file can appear anywhere in your repository, across modules and classes
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The stdlib module - file_line
• used for managing single lines in a file you otherwise don’t care about
o supports regex matchingo caution: managing the same file with file_line and a file
resource can cause flip-flopping file content• part of the puppetlabs/stdlib module
o https://github.com/puppetlabs/puppetlabs-stdlib• implemented as custom resource type
o lib/puppet/provider/file_line/ruby.rb
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
class foo::bar {
file_line { 'sudo_rule':path => '/etc/sudoers',line => '%sudo ALL=(ALL) ALL',
}
file_line { 'change_mount':path => '/etc/fstab',line => '10.0.0.1:/vol/data /opt/data nfs defaults 0 0',match => '^172.16.17.2:/vol/old',
}
}
file_line (example)
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
The inifile module
• used for managing inifile style fileso created by Chris Price, maintained by PuppetLabso https://github.com/puppetlabs/puppetlabs-inifile
• most important resource type is ini_settingo Auto-creates ini section headers
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
class foo::baz {ini_setting { "sample setting":
ensure => present,path => '/tmp/foo.ini',section => 'foo',setting => 'foosetting',value => 'FOO!',
}}
$ puppet apply -e ‘include foo::baz’
$ cat /tmp/foo.ini[foo]foosetting = FOO!
The inifile module - example
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Augeas
• Advanced parsing of config fileso http://augeas.net/
• Uses so-called lenses that define the grammar of known config files. Many available for well known software (http://augeas.net/stock_lenses.html)
o Creating your own lense possible, but tough!• After parsing, a configuration file will be available in tree
format, with read/write capabilities to each node in the tree• Works regardless of order of lines in the file
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
walter@web02:~$ cat -n /etc/hosts1 # HEADER: This file was autogenerated at Sun May 12 07:31:47 +0200 20132 # HEADER: by puppet. While it can still be managed manually, it3 # HEADER: is definitely not recommended.4 ### OlinData installimage5 # nameserver config6 # IPv47 127.0.0.1 localhost8 1.2.3.4 web02.olindata.com9 1.2.3.5 db01.olindata.com10 1.2.3.6 test01.olindata.com11 192.168.100.100 db0112 192.168.100.101 web0213 1.2.3.7 mail01.olindata.com puppet[..SNIP..]
Augeas (example)walter@web02:~$ augtoolaugtool> ls /files/etc/hosts/7ipaddr = 1.2.3.7canonical = mail01.olindata.comalias = puppetaugtool> ls /files/etc/hosts/07augtool> ls /files/etc/hosts/7ipaddr = 1.2.3.7canonical = mail01.olindata.comalias = puppetaugtool> match /files/etc/hosts/*/ipaddr 1.2.3.4/files/etc/hosts/2/ipaddraugtool> print /files/etc/hosts/*/ipaddr/files/etc/hosts/1/ipaddr = "127.0.0.1"/files/etc/hosts/2/ipaddr = "1.2.3.4"/files/etc/hosts/3/ipaddr = "1.2.3.5"/files/etc/hosts/4/ipaddr = "1.2.3.6"/files/etc/hosts/5/ipaddr = "192.168.100.100"/files/etc/hosts/6/ipaddr = "192.168.100.101"/files/etc/hosts/7/ipaddr = "1.2.3.7"
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
root@web02 /home/walter # cat test.ppclass test {
file { '/tmp/motd':ensure => present,content => "foo \nbar \n"
}
file_line { 'test line':path => '/tmp/motd',line => 'baz',match => '^ba.',
}}
include testroot@web02 /home/walter # puppet apply test.ppNotice: Compiled catalog for web02.olindata.com in environment dev in 0.06 secondsNotice: /Stage[main]/Test/File[/tmp/motd]/ensure: createdNotice: /Stage[main]/Test/File_line[test line]/ensure: createdNotice: Finished catalog run in 0.71 seconds
Trivia - what happens?
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
root@web02 /home/walter # cat test.ppclass test {
file { '/tmp/motd':ensure => present,content => "foo \nbar \n"
}
file_line { 'test line':path => '/tmp/motd',line => 'baz',match => '^ba.',
}}
include testroot@web02 /home/walter # puppet apply test.ppNotice: Compiled catalog for web02.olindata.com in environment dev in 0.06 secondsNotice: /Stage[main]/Test/File[/tmp/motd]/ensure: createdNotice: /Stage[main]/Test/File_line[test line]/ensure: createdNotice: Finished catalog run in 0.71 seconds
Trivia - what happens?
root@web02 /home/walter # cat /tmp/motdfoobaz
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resources
PuppetDB
Puppet
Master
DB01WEB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
5. Retrieve from PuppetDB3. Store in PuppetDB
2. Export to PM
4. Collect on DB01
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Exported resourcesnode /^web\d{3}.olindata.vm$/ {
include role::haproxy::backend
}
node 'proxy01.olindata.vm' {
include role::haproxy
}
class role::haproxy {
concat {‘/etc/haproxy.cfg’:
ensure => present,
}
Concat::Fragment <<| |>>
}
class role::haproxy::backend {
@@concat::fragment { $::fqdn:
ensure => 'present',
target => '/etc/haproxy.cfg'
}
}
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Upcoming training
• Puppet Fundamentals Training, Vienna
Monday, November 17, 2014
• Puppet Fundamentals Training, Barcelona
Monday, November 24, 2014
• Puppet Fundamentals Training, Hyderabad
Monday, November 24, 2014
• Puppet Fundamentals Training, Pune
Monday, December 1, 2014
• Puppet Architect Training, Singapore
Wednesday, December 17, 2014
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
We’re hiring!EU and Asia based trainers
jobs@olindata.com
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files
Questions?
@walterheck / @olindata
http://www.olindata.com
walterheck@olindata.com
http://github.com/olindata