Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
-
Upload
oholiab -
Category
Technology
-
view
597 -
download
3
Transcript of Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
Keeping AWS in check with Puppetor
“Managing cloud networking with Puppet with a healthy dose of trolling yourself on the side”
Matt Carroll
HELLO.
What’s all this about then?
● We use AWS extensively, especially EC2● We needed a way to organise our AWS networking (VPCs)● It needs to be:
○ Centrally managed○ Reproducible○ Declarative (idempotent)○ Ideally not another solution on top of a stack of solutions
We (Tom) decided to use Puppet
We wrote a set of types and providers to use with the AWS API
Spoiler
What I’m going to cover
1. Intro2. AWS networking basics3. Why we chose Puppet4. How it actually works5. Some of the “interesting” things we learned6. Summary7. Questions
YOU ARE HERE
What I hope you’ll get from this
● A sense of Puppet as an extensible framework that can manage dependencies in external APIs
● Nodes are just units of computing capacity - an operation does not have to be a subset of a node
● Some insight into the types and providers system● Some of the strange things we learned along the way
… and as always...
To serve as a lesson to others.
SOME AWS BASICS.
Some of this
isn’t exactly
retina resolution
The (simplified) Hierarchy
Account
Region
VPC
Routetable
Subnet
The (needlessly complicated simplified) Hierarchy
Account:• Region
• dopt: DHCP Options• vpc: Virtual Private Cloud
• igw: Internet Gateway• cgw: Customer Gateway• vgw: Virtual Gateway
• vpn: VPN• routetable
• subnet
How do we make all these objects?
Just make them in the console!
How do we make all these objects?
Just make them in the console!
IF YOU HATE YOURSELF
How do we actually make all of these objects?
• AWS::SDK!
• We’ll have to ensure that the resources get created (and
purged)...
• Idempotently…
• With all their dependencies…
• Remind you of something?
USE PUPPET!
Create Puppet types for VPC objects!● Resources for each of the objects
○ All API calls made on this level○ Will contain all code for reading and creating individual
resources○ No dependencies other than autorequires in the types -
those in the hierarchy earlier● Business logic in manifests
○ For your site-specific dependencies and network structure○ e.g. we have a separate VPC for each environment
● Data in hiera
(Actually though) Why Puppet?
• With the ability to query and create and modify objects through the API we can state them declaratively
• We can thus create resources which can be included or purged idempotently
• Rather than specify order, we can state dependencies and allow Puppet to figure order out
Why Puppet?Using this model you can even collect dependencies and resources:• Make all resources of a type with a parameter evaluate after
another
• Control purge and no-op of all aws resources
Why Puppet?Admittedly it seems a little asymmetrical:• Puppet runs on nodes which creates
resources in AWS• AWS networking is not a subset of a node• Nodes just act as executors for creating AWS
resource
But actually this plays to our advantage (more on this later)
HOW IT WORKS.
• The bit that you specify in the manifest
• Really just a DDL for the resource’s metadata
• The “front end” for the pluggable “back end” (the provider)
• Interface to specify all properties and parameters
Puppet Types
• The business end• This is where the API
code lives• Quite a lot longer• Handles all application
and querying of the resource including prefetch
Puppet Providers
• Some resources are expensive to read, so you read them all once when you first come across one
• This is done before the catalog is completely compiled
What’s a “prefetch”?
Manifest examples
Associated Infrastructure
• How do you handle multiple accounts?• Dedicated AWS admin box within each
account to apply the resources on• IAM roles to handle credentials• Logging resource changes separately• Why not do it in the code...?
LESSONS LEARNED.(the hard way)
Don’tdo
whatDonny
Don’tdoes
Why not put credentials in the code?
Why not put credentials in the code?• We (I) tried to make an
aws_credentials type• This requires access to the
catalog in the prefetch phase so other resources can query it
• You ALSO need to guarantee type evaluation order and access to credentials in the catalog so that you can use prefetch
Why not put credentials in the code?
• The instances method suddenly requires an argument
• In fact, so does anything that isn’t an instance method
• Actually ended up copying and pasting the resources resource to aws_resources and adding in a “credentials” parameter
On second thoughts...
Prefetch isn’t… Exceptional
• https://tickets.puppetlabs.com/browse/PUP-3656• This means that if anything goes awry in prefetch, puppet will
swallow it• In our case, we hit the API limit occasionally, meaning we got
duplicate resources
So we did a bad thing...
Asynchronous APIs tell lies• When you create an object you get
a 200 OK• This doesn’t mean “I’ve done it”,
this means “I’ll do it”• In a dependency chain, this can
mean that a resource is about to be created, but when it’s checked by another resource it’s not there yet
• In the Puppet paradigm it’s best just to run until convergence.
SO IN SUMMARY...
What I learned (the hard way)
• The “resources” resource• Is hard to understand unless
you read the code (type only) because Googling it is impossible.
• Could do with being able to apply other arbitrary properties and parameters?*
*this may be an awful idea
What I learned (the hard way)
• Instances vs. class methods paradigm gets very confusing because it is tied to applying catalogue vs. prefetch
• Prefetch is generally pretty confusing when you add in dependencies TO the prefetch
What really cool things I learned• You can reproduce AWS networking (and other objects
thanks to other contributors!)• A really cool insight into the types and providers system and
how it could grow in the future• Learning to treat servers not like pets OR cattle but as a
medium by which you do useful stuff
… and you could do this with a whole bunch of APIs
github.com/bobtfish/puppet-aws_apiMatt CarrollSRE at Yelp
yelp.com/careers (we’re hiring)
THANKS FOR LISTENING!