Integrating Puppet with Cloud Infrastructures-Remco Overdijk
-
Upload
maxserv -
Category
Technology
-
view
588 -
download
0
Transcript of Integrating Puppet with Cloud Infrastructures-Remco Overdijk
Remco OverdijkLEAD OPERATIONS ENGINEER
Automating the Cloud
Integrating Puppet with Cloud Infrastructures
@MaxServ @RemzJay
AUTOMATING THE CLOUD
Single Server Infrastructure Issues
PROBLEM
Limited headroom
Service issues affect other services
One outage means downtime
Maintenance during deployment
4
Image credit: https://commons.wikimedia.org/wiki/File:Grumpy-Cat.jpg - Rjommabolli (CC 4.0)
AUTOMATING THE CLOUD
Single Server Infrastructure Issues6
Scalability
Service Isolation
Redundancy
Zero-Downtime Deployments
SOLUTIONPROBLEM
Limited headroom
Affected services
One outage means downtime
Maintenance during deployment
• AWS specific, but applies to most (if not any) Cloud platforms.
AUTOMATING THE CLOUD
Scope7
• LAMP stack, but works for most stacks.
• Mix & match for best results.
• The method that works best depends on your own setup.
• Based on Puppet Open Source.
• Things may be different in Puppet Enterprise (Orchestrator).
• Ready-to-go AMI’s may be faster to launch, but harder to maintain.
AUTOMATING THE CLOUD
AWS Infrastructure Principles
Read more
https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf
8
• Infrastructure is failure-prone; Service robustness is achieved through redundancy.
• EC2 instances should be considered volatile.
• Scaling should be both vertical and horizontal.
• Legacy applications aren’t magically cloud-ready.
• Loose coupling improves scalability.
• Isolation increases security and decreases dependencies.
• IAM Server Certificates
• IAM Instance Profiles
• IAM Roles
• IAM Policies
• IAM Role Policies
• CodeDeploy Apps
• CodeDeploy
Deployment Groups
• EC2 Instances
• EC2 Elastic IP’s
• CloudWatch Metrics
• CloudWatch Alarms
• EIP Attachments
• ElastiCache Subnet
Groups
Provisioning Infrastructure
AWS Resources
• ElastiCache Clusters
• ElastiCache Parameter
Groups
• EC2 Elastic Load Balancers
• ELB Health Checks
• ELB App Cookie Stickiness
Policies
• Key Pairs
• RDS Subnet Groups
• RDS Parameter Groups
• RDS DB Instances
• Route53 Zones
• Route53 Records
• S3 Buckets
• S3 Policies
• S3 CORS Configuration
• Security Groups
• SNS Topics
• SNS Topic
Subscriptions
• VPC’s
• VPC Subnets
• VPC Internet Gateways
• VPC (S3) Endpoints
• VPC Route Tables
• VPC Customer
Gateways
• VPC VPN Gateways
• VPC DHCP Option Sets
• VPC VPN Routes
11
Provisioning Infrastructure
That’s a lot of clicking13
• Manual changes are extremely error-prone.
• Manual changes result in an inconsistent platform.
• Collaboration is difficult; People are scared to break things.
• Changes are scattered throughout the AWS console.
• Uses the same DSL as your ”regular” Puppet code.
Provisioning Infrastructure
Puppetlabs-AWS15
• Most benefits from ”regular” Puppet;like relationships, defined types and the graph.
• At the moment this module only supports a few of the resources in the AWS API.
• Does NOT run using agents / puppet masters, but manually fromyour workstation using `puppet apply`.
• Not as idempotent as you’d like at times.
Read more
https://github.com/puppetlabs/puppetlabs-aws
https://forge.puppetlabs.com/puppetlabs/aws
`AWS_PROFILE=my-aws-account AWS_REGION=eu-west-1 puppet apply aws-stack.pp --templatedir ./templates/`
Provisioning Infrastructure
Hashicorp Terraform
• Uses HCL, which looks a lot like Puppet DSL.
• Maintains a dependency graph, just like Puppet.
• Runs from your workstation.
• Is not AWS API feature complete, but covers most services.
• Allows flexible scaling and destruction.
17
Read more
https://github.com/hashicorp/terraform
https://www.terraform.io/
Provisioning Infrastructure
Puppet?20
• Automated Infrastructure is awesome.
• We like Puppet too.
• Empty EC2 instances don’t serve websites.
• How do we connect Puppet to the infrastructure we’ve just provisioned?
SLIDE CREDIT: Tim Bell, CERN – PuppetConf 2012
https://speakerdeck.com/puppetlabs/cern-accelerating-science-with-puppet
https://youtu.be/-Ykb2j2ojYU?t=19m33s
Bootstrapping Puppet
So, Cattle.
• Configuration should be at type level. Not node level.
• Nodes should be replaceable.
• Data on the nodes should be considered volatile.
• Only versioned and cached content should be present.
25
• Provisioning should be automated; No manual intervention should be required.
Bootstrapping Puppet
Puppet: Autosigning
• Automates indoctrination for new nodes.
• Multiple mechanisms available:
Naïve, Basic and Policy-based Autosigning.
26
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/puppet.conf
Read more
https://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html
Be very careful with naïve autosigning.
Don’t do it in production without strict firewalls.
Bootstrapping Puppet
Node Manifests using Regex28
Read more
https://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html#regular-expression-names
Bootstrapping Puppet
Hiera
• Configuration at `type` level instead of `clientcert`.
• Use node-level overrides when required.
• Combine with host `%H` mount points for master/slave.
29
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Puppet/hieradata
Read more
https://docs.puppetlabs.com/hiera/latest/
https://docs.puppetlabs.com/guides/file_serving.html#file-server-configuration
Bootstrapping Puppet
ENC’s in Foreman31
Read more
https://docs.puppetlabs.com/guides/external_nodes.html
http://theforeman.org/manuals/1.10/index.html#2.Quickstart
Bootstrapping Puppet
Foreman: Default Host Groups
• The same idea as Autosigning in Puppet.
• Uses a Foreman Plugin.
• Use facts for assigning host groups.
32
Read more
https://github.com/theforeman/foreman_default_hostgroup
Bootstrapping Puppet
Assigning Configuration: Facter34
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/templates/userdata.tpl
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/Vagrantfile
Read more
https://docs.puppetlabs.com/facter/2.4/custom_facts.html#structured-data-facts
Bootstrapping Puppet
AWS: User data
• User data allows you to pass configuration data to an instance during launch.
• User data can be used to provide cloud-init configuration.
35
Bootstrapping Puppet
Cloud-init37
• Handles early initialization of cloud instances.
• Supports EC2, CloudStack, OpenStack, OnApp, OpenNebula, RHEVm, vSphere & more.
• Can install packages, do basic config and more.
• Is able to bootstrap Puppet.
Read more
http://cloudinit.readthedocs.org/en/latest/index.html
http://cloudinit.readthedocs.org/en/latest/topics/examples.html#setup-and-run-puppet
Bootstrapping Puppet
Cloud-init with Puppetlabs-Aws38
Read more
https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
Bootstrapping Puppet
Cloud-init with Terraform39
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/ec2.tf
Read more
https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
Bootstrapping Puppet
Don’t forget to clean up!
• Revoke Puppet-CA certificates for decommissioned nodes.
• Clean up Salt keys.
• Remove old reports, exported resources and catalogs from PuppetDB.
• Clean connected resources like Load Balancers.
40
Sloppiness will catch up with you when it hurts the most.
Demo
Terraform & Puppet
THIS DEMONSTRATION INCLUDES:
VIEW & TRY THE FULL SOURCE OF THIS DEMO
https://github.com/MaxServ/Terraform-Puppet-Demo
A
B
C
Docker container running a Puppetmaster.
AWS Stack with 4 webservers using Terraform.
Webserver provisioning using Puppet based on type.
42
0416 - 30 10 00
Remco OverdijkLEAD OPERATIONS ENGINEER
Questions?
@MaxServ @RemzJay