2013 AWS Worldwide Public Sector Summit Washington, D.C.

Infrastructure Automation with Python

Christian Arllen

Solution Architect

2013 AWS Worldwide Public Sector Summit

Abstract Outline

• During this presentation, we will

walk through built in tools in the

AWS SDK for Python and provide

examples and use cases where

Python enables automation. The

session will also demonstrate

methods of combining simple

Python tools into complex services.

• AWS Software Development Kit

(SDK) for Python (boto)

• Command-line tools bundled with

boto

• AWS Command Line Interface

• Reporting

• Manage and automate resource

growth

2013 AWS Worldwide Public Sector Summit

Getting Started

• AWS Software Development Kit (SDK) for Python (boto) – Description and References

• I want to follow along … Get me started now! – AWS Free Usage Tier

– Create an Amazon EC2 Key Pair

– Launch a pre-configured dev environment • AWS CloudFormation template deploying an instance with the ‘develop’ branch of the AWS SDK for Python

using an IAM Role for seamless API access.

• Command-line tools bundled with boto – fetch_file, s3put, lss3, kill_instance, list_instances, …

• AWS Command Line Interface

• Reporting – boto.utils.get_instance_metadata() queries the local instance metadata service

– User Data initiated launch notification with Amazon SNS

– ec2-resources.py <region>

• Manage and automate resource growth

2013 AWS Worldwide Public Sector Summit

http://aws.amazon.com/free/

2013 AWS Worldwide Public Sector Summit

http://aws.amazon.com/tools/

2013 AWS Worldwide Public Sector Summit

Boto bundled command-line tools

[ec2-user@ip-10-72-230-7 ~]$ list_instances --help Usage: list_instances [options] Options: -h, --help show this help message and exit -r REGION, --region=REGION Region (default us-east-1) -H ID,Zone,Groups,Hostname,State,T:Name, --headers=ID,Zone,Groups,Hostname,State,T:Name Set headers (use 'T:tagname' for including tags) -t, --tab Tab delimited, skip header - useful in shell scripts -f FILTER, --filter=FILTER Filter option sent to DescribeInstances API call, format is key1=value1,key2=value2,...

[ec2-user@ip-10-72-230-7 ~]$ list_instances -t --headers=ID,Zone,Hostname,State i-fcc17a81 us-east-1d ec2-54-234-249-183.compute-1.amazonaws.com running i-224a2449 us-east-1a terminated

2013 AWS Worldwide Public Sector Summit

UserData and an IAM Role

2013 AWS Worldwide Public Sector Summit

User Data initiated

launch notification

with Amazon SNS

Note this example used the default

“>” for the continuation prompt

username@instance-id ~> ec2_metadata_publish_sns.py \

> …. arn:aws:sns:us-east-1:593283415949:AWS_WWPS_Summit_2013

2013 AWS Worldwide Public Sector Summit

Amazon SNS

• Subject: New m1.small instance launch: i-fcc17a81 in us-east-1d

• Message contains

• The notification of a new instance launch, location, type, address, etc. …

• Human or Machine readable output (JSON)

• Build notification into a work flow

• AMI build process

• Mandatory USERDATA

• Project specific packages

2013 AWS Worldwide Public Sector Summit

Manage information dissemination

Given a growing system • API call rates increase

• Information dissemination among nodes

bottlenecks on Controller Node(s)

Evolve to Publish/Subscribe (PubSub) model • Utilize the local instance metadata service …

localize queries where feasible

• Utilize systems designed to scale with your work

loads

• Publish updates to AWS SNS topics or

Amazon DynamoDB

Control

Node Amazon EC2

API

DynamoDB

Amazon EC2

API

Amazon SNS

2013 AWS Worldwide Public Sector Summit

Cluster Instance Metadata Service

• Utilize simple tools to build complex system

• Given a 5 or 50+ node compute cluster

– Performance and efficiency for all nodes to query instance metadata for all other

instances

– Publish to an Amazon DynamoDB table

– Enables cluster manager selection and replacement using conditional batch writes to

manage cluster leadership election

Thank You