...

AugeasAugeasSwiss-knife resources for your puppet treeSwiss-knife resources for your puppet tree

Julien Pivotto

Belgian Puppet User GroupHoliday is over Meetup!!! - November 12th, 2014

..

whoamiwhoamiJulien PivottoJulien Pivotto

• Open-Source consultant at inuits.eu• FOSS defender since 2004• DevOps believer and evangelist• Puppet User since 2011• @roidelapluie on twitter/github

..

..

ınuits.eu

..

..

Sysadmin 101Sysadmin 101CC BY-SA 2.0 https://www.flickr.com/photos/arthur-caranta/2926332140

..

Setting up a serviceSetting up a service

• Install the package• Change the configuration• Start the daemon

..

3 steps.What can go wrong?

..

PackagingPackaging

• Where is the package?• Which version do we need?• Does it conflict with something else?

..

..

Dependencies HellDependencies Hell

CC BY-SA 2.0 https://www.flickr.com/photos/coconinonationalforest/4587053982

..

ConfigurationConfiguration

• Where is the file?• How many files?• Configuration is in the database?• The file is *huge*

..

Starting the serviceStarting the service

• Does not start▶ Bad config file▶ Stale lock file▶ Data corruption

• High Availability• Replication

..

Let's talk about Puppet and filesLet's talk about Puppet and files

• Classical approach: File[] resource• Advanced approach: Concat[] define• Broken approach: Exec[sed] resource• Surgical approach: Augeas[] resource

..

Hidden ways to manage filesHidden ways to manage files

• Ssh_authorized_key[]• Nagios_*• To purge or not to purge

..

..

The File[] resourceThe File[] resourceCC BY 2.0 https://www.flickr.com/photos/80497449@N04/10567875696/

..

FileFile

• Built-in puppet resource• Most used• Works with a lot of usecases• Text files, binary files

..

.

.

file{"${::icinga::confdir_server}/cgi.cfg":ensure => present,content => template('icinga/redhat/cgi.cfg.erb'),owner => $::icinga::server_user,group => $::icinga::server_group,require => Class['icinga::config'],notify => [

Service[$::icinga::service_client],Service[$::icinga::service_server],Exec['fix_collected_permissions']

],}

..

Content of a fileContent of a file

• content => String, template(), file()• source => puppet:///, /local/file

..

File[] behaviourFile[] behaviour

• Array as "source": Puppet will pick the firstavailable one

• Multiple arguments to template(): Puppetwill concatenate them all

• File[/foo/bar] will autorequire File[/foo]

..

Downside of File[]Downside of File[]

• You can only have at one "content"• That resource describe the whole file• Works in almost every situation

..

..

concatPublic Domain http://commons.wikimedia.org/wiki/File:Adhesive_tapes_clear.JPG

..

ConcatConcat

• A "reference" puppet module:puppetlabs/concat

• https://github.com/puppetlabs/puppetlabs-concat

• Provides definitions to manage file• Alternative modules:

▶ onyxpoint/pupmod-concat▶ theforeman/puppet-concat (fork of onyxpoint)

..

Concat?Concat?

• Concat takes a bunch of snippets• Assemble them info a file• Each snippet is a define• The final file is a define

..

.

.

concat { '/tmp/file':ensure => present,

}

concat::fragment { 'tmpfile':target => '/tmp/file',content => 'test contents',order => '01'

}

..

Base and fragmentsBase and fragments

• Concat[] defines owner, ensure, mode of thefile

• Concat::Fragment[] defines the contents ofthe file

• One Concat[] has multipleConcat::Fragment[]

..

Advantages of concatAdvantages of concat

• More flexibility▶ if▶ virtual resource▶ exported resources▶ create_resources

• Mix templates and files

..

Disadvantages of concatDisadvantages of concat

• External Puppet module• Concat[] is the whole file• Performances

..

..

Exec{sed: onlyif => grep}

CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Ca%C3%AFn_par_Henri_Vidal.jpg

..

..https://github.com/search?o=desc&q=exec+sed+onlyif+grep+language%3APuppet

..

exec[sed] is br0kenexec[sed] is br0ken

• Which options to pass to sed and grep?• You should use as few Exec[] as possible• grep ....• Escape, regexes…

..

Another alternative: conf.dAnother alternative: conf.d

• Some services support conf.d directories• But it is hard to change existing parameters• In which order are the files read?• Don't forget to purge

..

..

Augeas

CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Students_assisting_surgery.JPG

..

AugeasAugeas

• Configuration editing tool• First release in 2007• API coded in C• Command-line tools• bindings for different languages

..

Configuration editing toolConfiguration editing tool

• Parsing the configuration files• Turning them into a tree• Edit the tree & save the configuration

..

.

.

$ cat /etc/nsswitch.conf# /etc/nsswitch.conf## Example configuration#

passwd: db filesgroup: db filesinitgroups: db [SUCCESS=continue] filesshadow: db filesgshadow: files

..

.

.

augtool> ls /files/etc/nsswitch.conf/#comment[1] = /etc/nsswitch.conf#comment[2] = Example configurationdatabase[1]/ = passwddatabase[2]/ = groupdatabase[3]/ = initgroupsdatabase[4]/ = shadowdatabase[5]/ = gshadow

..

.

.

augtool> ls /files/etc/nsswitch.conf/database[1]/service[1] = dbservice[2] = files

..

Native format -> treeNative format -> tree

• Augeas understand comments• Augeas does not care about empty lines• The cli tool (augtool) has autocomplete• It recognize a lot of formats

..

.

.

augtool> set /files/etc/nsswitch.conf/database[1]/service[last()+1] ldapaugtool> saveSaved 1 file(s)

..

.

.

$ cat /etc/nsswitch.conf# /etc/nsswitch.conf## Example configuration#

passwd: db files ldapgroup: db filesinitgroups: db [SUCCESS=continue] filesshadow: db filesgshadow: files

..

.

.

augtool> match /files/etc/nsswitch.conf/*/* ldap/files/etc/nsswitch.conf/database[1]/service[3]augtool> print /files/etc/nsswitch.conf/database[1]/files/etc/nsswitch.conf/database[1] = "passwd"/files/etc/nsswitch.conf/database[1]/service[1] = "db"/files/etc/nsswitch.conf/database[1]/service[2] = "files"/files/etc/nsswitch.conf/database[1]/service[3] = "ldap"

..

.

.

augtool> rm /files/etc/nsswitch.conf/database[1]/service[3]rm : /files/etc/nsswitch.conf/database[1]/service[3] 1augtool> print /files/etc/nsswitch.conf/database[1]/files/etc/nsswitch.conf/database[1] = "passwd"/files/etc/nsswitch.conf/database[1]/service[1] = "db"/files/etc/nsswitch.conf/database[1]/service[2] = "files"augtool> saveSaved 1 file(s)

..

One API to edit them allOne API to edit them all

• Can talk XML, ini, named, nginx, …• Only change what is needed• Ensure the syntax is right

..

Augeas LensesAugeas Lenses

• Lenses are files that explain how to edit files• It contains paths and syntax• There are a lot of them available• You can write your own lenses

..

”This brings the total number of lenses to178. […] It’s depressing to think that

Linux/Unix systems have managed to growthis many special snowflake formats.”

David Lutterkort, main developerabout Augeas 1.3.0

..

178 lenses178 lensesactivemq_conf activemq_xml aliases aptconf

apt_update_manager backuppchosts bbhosts bootconf buildcarbon cgrules channels cobblermodules cobblersettings collectd

crypttab cyrus_imapd darkice debctrl desktop device_map dhcpddnsmasq dovecot dpkg dput ethers exports fai_diskconfig fonts

fuse gdm grub gshadow hostname inetd inputrc interfaces iproute2iptables jaas jmxaccess keepalived known_hosts koji krb5 ldif limits

login_defs logrotate mcollective memcached mke2fsmongodbserver mysql nagioscfg nagiosobjects netmasks nginx ntpntpd odbc openshift_config openshift_http openvpn pam passwd

pbuilder postfix_main postfix_transport postfix_virtualpuppet_auth qpid rabbitmq resolv rmt securetty sep services shells

shellvars_list sip_conf slapd smbusers squid sshd stunnelsubversion sudoers sysconfig systemd thttpd up2date vfstab

..

A short lenseA short lense

.

.

module Hostname =autoload xfm

(* View: lns *)let lns = [ label "hostname" . store Rx.word . Util.eol ]

(* View: filter *)let filter = incl "/etc/hostname". incl "/etc/mailname"

let xfm = transform lns filter

..

Puppet <3 augeasPuppet <3 augeas

• Native "augeas" resource• Support for pluginsync• Helpers available

..

Puppet examplePuppet example

.

.

augeas { $name:context => "/files${fstab::variables::fstab_file}",changes => [

"rm ${fstab_match_line}",],onlyif => "match ${fstab_match_line} size > 0"

}

..

Real usecasesReal usecases

• Change grub options• Modify /etc/hosts• Modify XML's (puppetlabs-tomcat)• Configure Jenkins

..

PluginsyncPluginsync

• Puppet has pluginsync support for Augeas• Drop your lenses in your modules• lib/augeas/lenses• Use the "lens" parameter of the augeasresource

..

Puppet examplePuppet example

.

.

augeas{"jboss_conf":context => "/files/etc/jbossas",changes => [

"set jbossas.conf/JBOSS_IP $ipaddress","set jbossas.conf/JAVA_HOME /usr",

],lens => "Jboss.aug",

}

..

Augeas commandsAugeas commands

set rm mv clear insert …

..

Augeas comparators (onlyif)Augeas comparators (onlyif)

match get

..

AugeasprovidersAugeasproviders

• Helpers around augeas• Puppet modules• No augeas knowledge needed

..

apacheapache

.

.

apache_setenv { "SPECIAL_PATH":ensure => present,value => "/foo/bin",

}

..

kernel_parameterkernel_parameter

.

.

kernel_parameter { "quiet":ensure => present,bootmode => "normal",

}

..

Conclusion

..

DisadvantagesDisadvantages

• Learning required• Library to install• Writing lenses is hard

..

AdvantagesAdvantages

• Augeas is a mature tool• Preserves comments in files• It fails (if needed)• Only changes what is needed• A lot of lenses available• Puppet integration• Helpers available

..

Final noteFinal note

Most of the time, File[] resources are the wayto go. Augeas can help when you need tochange files generated by an application orthat you can not manage entirely.

..

ReadingsReadings

• http://augeas.net/• http://augeasproviders.com/• https://docs.puppetlabs.com/

..

Thank youThank you

Any question?Thanks to @raphink

..

ContactContact

Julien Pivottojulien@inuits.eu@roidelapluie

INUITS bvbaBelgium+32 473 441 636https://inuits.eu