Linux Clusters Institute: Configuration...
Transcript of Linux Clusters Institute: Configuration...
![Page 1: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/1.jpg)
Linux Clusters Institute:Configuration Management
Garrett McGrath, Princeton Neuroscience Institute
![Page 2: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/2.jpg)
Goals
● Detail what configuration management is and why it is useful● Review the current landscape of available tools● Be able to convey what configuration management is and why /
where it should be used to coworkers and managers
This will not be a deep dive into tools or a full course implementation of a solution. We will use puppet for some examples but many of the concepts transfer relatively broadly.
![Page 3: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/3.jpg)
Defining Configuration Management
At its broadest level, the management and maintenance of operating system and service configuration via code instead of manual intervention.
More formally:
● Declaring the system state in a repeatable and auditable fashion and using tools to impose state and prevent systems from deviating
![Page 4: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/4.jpg)
State
All system have a ‘state’ comprised of:
● Files on Disk● Running services
State can be supplied by:
● Installation / provisioning systems● Golden Images● Manual steps including direct configuration changes and setup
scripts
![Page 5: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/5.jpg)
Modern Configuration Management Features
● Idempotency○ Declaration and management of files and services to reach a ‘desired
state’● Revision Control
○ Systems are managed with an ‘Infrastructure as code’ model● Composable and flexible
![Page 6: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/6.jpg)
Why bother?
● Configuration management can operate as a ‘work multiplier’● Standard configuration steps can be made into components you
mix to define a machine● Tools can be used to verify a machines current state vs. the official
central manifest for the machine● Revision history allows you to track what changes were made to a
system, by who, and why
![Page 7: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/7.jpg)
Benefits of configuration version control
•Built-in documentation (change logs, summaries, etc.)
•Peer review (issue tracking, merge requests, email alerts)
•Reverts (at least partially)
![Page 8: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/8.jpg)
Benefits of configuration management
•Centralized catalog of all system configuration
•Automated enforcement of system state from an authoritative source
•Ensured consistency between systems
•Rapid system provisioning from easily-composed components
•Preflight tests to ensure deployments generate expected results
•Collection of system ‘ground truths’ for better decision making
![Page 9: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/9.jpg)
Modern configuration-management systems
• Puppet• Ruby based DSL
• Chef Infra
• Ruby based DSL
• CFEngine
• C based DSL
• Salt
• Python based DSL
• Ansible
• Python based DSL
![Page 10: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/10.jpg)
Getting started (puppet example)
● Pick a simple, small part of your primary system checklist○ sudoers○ resolv○ nsswitch
● Implement and Test (start with “no-op”)
![Page 11: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/11.jpg)
Puppet Directory Structure
• production/• hiera/• modules/
• ntp
• manifests/
• Init.pp
• files/
• ntp.conf
• manifests/• site.pp
![Page 12: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/12.jpg)
# modules/ntp/manifests/init.pp
class ntp {
package { 'ntp':
ensure => installed,
}
file { '/etc/ntp.conf':
source => 'puppet:///modules/ntp/ntp.conf',
owner => 'root',
group => 'root',
mode => '0644',
require => Package['ntp'],
}
service { 'ntp':
ensure => running,
enable => true,
require => File['/etc/ntp.conf'],
}
}
![Page 13: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/13.jpg)
# manifests/site.pp
node 'node1' { include ntp}
![Page 14: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/14.jpg)
Testing the prototype
# puppet apply --noop \ --modules modules manifests/site.pp
![Page 15: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/15.jpg)
Next steps
•Top-level node roles
•Add features you need now (don't try to do everything at once)
•Convince, teach, and assist your team
•Continue until you have no more questions about your environment
• Find more modules on https://forge.puppet.com/
![Page 16: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/16.jpg)
Puppet Workflow in PNI
● Primarily a Component Object Model○ Secondary configuration with Profiles
● Environment Folders○ production/dev/User workspaces
● Hiera for almost everything● Mercurial based version control with dev and production branches
○ Production env folder follows production branch○ Dev env folder follows dev branch
![Page 17: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/17.jpg)
What does this workflow look like?
● Pull from dev to make sure you’ve got the latest dev code being shared by others
● Make changes in your env folder
● Test changes against test nodes with --environment <name> and --noop as appropriate
● commit
● Merge changes from dev to production if complete and ready to use everywhere
● Push from local env folder to ../dev and ../production as appropriate
● Issue ‘update’ commands in those folders so they are running the latest revs of the code.
![Page 18: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/18.jpg)
Component Object Model
● Machines are self classifying○ Facilitated by ‘custom facts’; rules that specify things like ‘member of
cluster X, access to network Y’○ Host groups for smaller sets of configurations based simply on
hostnames○ Additional behaviors added based on hardware facts
■ Raid cards
■ Nvidia gpus
■ Virtual machine vs. physical
![Page 19: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/19.jpg)
Puppet workflow in HCC
•Roles and profiles
•Hiera
•R10K
•Git
Puppet
R10KGit
![Page 20: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/20.jpg)
Puppet
R10KGit
clone, commit, pushpull, merge….etc.
Puppetfile
environment = productionenvironment = test
You can add more:Gerrit,
Jenkins….
![Page 21: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/21.jpg)
What does this workflow look like?
• git clone git@git-server:puppet
• git checkout –b mybranch
•… make some changes…• git add/commit/push
•On you test node: puppet agent –t –environment=mybranch
•Merge it to production!
![Page 22: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/22.jpg)
Roles and Profiles
![Page 23: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/23.jpg)
Advocating to colleagues
• Work is front-loaded, so early work seems much more costly
• System might undo work done by others
• Add comments at the top of managed config files
• Offer to help colleagues port
• Work with at least one other person
• Be as transparent as possible
• Commit emails
• Slack updates either automated or manually when pushing out changes to production
• Document how to port an existing host
![Page 24: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/24.jpg)
Advocating to management
• Work more efficiently (get more done)• Properly tooled configuration management operates as a ‘work multiplier’ doing something one
time expands to doing the right thing on all systems every time
• Not an all-or-nothing proposition: start with a few systems and go slow
• Document and report success stories
• Deployment speed improvements
• Patch deployment improvements
• Peer review anecdotes
• Corrections made
![Page 25: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/25.jpg)
Things to watch out for
•Also easy to make a mistake on several hosts at once• Test in isolation first, and with a no-op mode• Changes can have wide sweeping effects if not deployed carefully, resulting
in a denial of service failure if not carefully tested before deployment to production.
• It's easy to get lazy and allow systems to fall out-of-sync• This is avoidable with systems using active clients, but scaling this approach is
difficult
• It's easy to let perfectionism take over
![Page 26: Linux Clusters Institute: Configuration Managementlinuxclustersinstitute.org/workshops/archive/intro19/pdfs/8-config.pdf · Getting started (puppet example) ... Hiera for almost everything](https://reader035.fdocuments.us/reader035/viewer/2022071012/5fca591665544404843b3c97/html5/thumbnails/26.jpg)
Reference
•Puppet: https://puppet.com/
•Puppet forge: https://forge.puppet.com/
•R10k: https://github.com/puppetlabs/r10k
•Roles and profiles: http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/