Post on 15-Jul-2015
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Deep Dive: Infrastructure as Code Steven Bryen – Solutions Architect, AWS
Raj Wilkhu – Principal Engineer, JUST EAT Bruce Jackson – CTO, Myriad Group AG
You are on-board …
needs to experiment, innovate, reduce risk Business
of services and applications Continuous
Delivery
culture, automation, measurement, sharing DevOps
infrastructure-as-code Cloud
AWS CloudFormation • Create templates of the infrastructure and
applications you want to run on AWS
• Have the CloudFormation service automatically provision the required AWS resources and their relationships from the templates
• Easily version control, replicate or update the infrastructure and applications using the templates
• Integrates with other development, CI/CD, and management tools.
Create template – Example web application
security group
Auto Scaling group
EC2 instance
Elastic Load Balancing
DB Service
Inventory Service
Recommendations Service
ElastiCache memcached cluster
Software pkgs, config, & data CloudWatch
alarms
Create template – Resources
security group
Auto Scaling group
EC2 instance
Elastic Load Balancing
ElastiCache memcached cluster
Software pkgs, config, & data CloudWatch
alarms
"Resources" : { "SecurityGroup" : {}, "WebServerGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "MinSize" : "1", "MaxSize" : "3", "LoadBalancerNames" : [ { "Ref" : "LoadBalancer" } ], ... } }, "LoadBalancer" : {}, "CacheCluster" : {}, "Alarm" : {} },
CloudFormation Template
Create template – Parameters
Auto Scaling group
EC2 instance
Recommendations Service Inventory
Service Customer DB
Service
Info to Customize Stack at Creation. Examples: Instance Type, App Pkg Version
"Parameters" : { "CustomerDBServiceEndPoint" : { "Description" : "URL of the Customer DB Service", "Type" : "String" }, "CustomerDBServiceKey" : { "Description" : "API key for the Customer DB Service", "Type" : "String", "NoEcho" : "true" }, "InstanceType" : { "Description" : "WebServer EC2 instance type", "Type" : "String", "Default" : "m3.medium", "AllowedValues" : ["m3.medium","m3.large","m3.xlarge"], "ConstraintDescription" : "Must be a valid instance type"
CloudFormation Template
Create template – Outputs
Elastic Load Balancing
"Resources" : { "LoadBalancer" : {}, ... }, "Outputs" : { "WebsiteDNSName" : { "Description" : "The DNS name of the website", "Value" : { "Fn::GetAtt" : [ "LoadBalancer", "DNSName" ] } } }
CloudFormation Template
Create template – Deploy and configure software
Auto Scaling group
EC2 instance
Software pkgs, config, & data
"AWS::CloudFormation::Init": { "webapp-‐config": { "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}
}, "chef-‐config" : {} }
CloudFormation Template ü Declarative ü Debug-able ü Updatable ü Highly Secure ü BIOT™ Bring In
Other Tools
Create template – Language features
Use a wide range of AWS services
ü Auto Scaling ü Amazon CloudFront ü AWS CloudTrail ü AWS CloudWatch ü Amazon DynamoDB ü Amazon EC2 ü Amazon ElastiCache ü AWS Elastic Beanstalk ü AWS Elastic Load Balancing ü Amazon Kinesis
ü AWS Identity and Access Mgmt
ü AWS OpsWorks ü Amazon RDS ü Amazon Redshift ü Amazon Route 53 ü Amazon S3 ü Amazon SimpleDB ü Amazon SNS ü Amazon SQS ü Amazon VPC
and more … As of April 2015
Nested CloudFormation Stacks
"myStackWithParams" : { "Type" : "AWS::CloudFormation::Stack", "Properties" : { "TemplateURL" : "https://s3.amazonaws.com/template-bucket/mystack.template", ”Parameters" : { "InstanceType" : "t1.micro",
"KeyName" : "mykey"
}
} }
Pass parameters to nested CloudFormation Stacks
Iterate on infrastructure
Update stack In-place Blue-Green
Faster
Cost-efficient
Simpler state and data migration
Working stack not touched
Extending AWS CloudFormation
Extend with Custom Resources
security group
Auto Scaling group
EC2 instance
Elastic Load Balancing
ElastiCache memcached cluster
Software pkgs, config, & data CloudWatch
alarms Web Analytics
Service AWS
CloudFormation
Provision AWS Resources
"Resources" : { "WebAnalyticsTrackingID" : { "Type" : "Custom::WebAnalyticsService::TrackingID", "Properties" : { "ServiceToken" : "arn:aws:sns:...", "Target" : {"Fn::GetAtt" : ["LoadBalancer", "DNSName"]}, "Plan" : "Gold" } }, ...
“Success” + Metadata
“Create, Update, Rollback, or Delete” + Metadata
Lambda-powered custom resources
security group
Auto Scaling group
EC2 instance
Elastic Load Balancing
ElastiCache memcached cluster
Software pkgs, config, & data CloudWatch
alarms
// Implement Custom Logic Here
Lookup an AMI ID
Lookup VPC ID and Subnet ID
Reverse an IP Address
Application Lifecycle
Infrastructure Lifecycle
EC2
SQS, SNS, Kinesis, etc.
Databases
VPC
IAM
Application Lifecycle
Download Packages, Install Software, Configure Apps, Bootstrap Apps,
Update Software, Restart Apps,
etc.
CloudFormation
• Templatize • Replicate • Automate
AWS::CloudFormation::Init
"AWS::CloudFormation::Init": {
"webapp-config": {
"packages" : {}, "sources" : {}, "files" : {},
"groups" : {}, "users" : {},
"commands" : {}, "services" : {}
Declarative
AWS::CloudFormation::Init Supports updates
"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}
AWS::CloudFormation::Init
"install_chef" : {},
"install_wordpress" : {
"commands" : {
"01_get_cookbook" : {}, ...,
"05_configure_node_run_list" : {
"command" : "knife node run_list add -z `knife node list -z` recipe[wordpress]",
"cwd" : "/var/chef/chef-repo",
"env" : { "HOME" : "/var/chef" }
Flexibility to bring in other tools such as AWS CodeDeploy and Chef
ow.ly/DiNkz
Use AWS::CloudFormation::Init "UserData": { "# Get the latest CloudFormation helper scripts package\n", "yum update -y aws-cfn-bootstrap\n", "# Trigger CloudFormation::Init configuration \n", "/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n", "# Signal completion\n", "/opt/aws/bin/cfn-signal –e $? --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n"
Use CloudWatch Logs for debugging
"install_logs": {
"packages" : { ... "awslogs" ... },
"services" : { ... "awslogs" ... }
"files": {
"/tmp/cwlogs/cfn-logs.conf": {}
file = /var/log/cfn-init.log log_stream_name = {instance_id}/cfn-init.log file = /var/log/cfn-hup.log log_stream_name = {instance_id}/cfn-hup.log
ow.ly/E0zO3
Use CloudWatch Logs for debugging
ow.ly/E0zO3
Wait Conditions & DependsOn
Example Wait for EC2 instance to signal success
"Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "UserData" : { "Fn::Base64" : {"Ref" : "myWaitHandle"}},
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]} } },
Example
"myWaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle",
}, "myWaitCondition" : {
"Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "Ec2Instance", "Properties" : { "Handle" : { "Ref" : "myWaitHandle" }, "Timeout" : "4500" }
}
Wait for EC2 instance to signal success
CI/CD Pipeline
CloudFormation in a CI/CD Pipeline
AWS CloudFormation Issue Tracker
App Developers
DevOps Engineers, Infrastructure Developers,
Systems Engineers
Dev Env Code Repo
App Pkgs, CloudFormation Templates, Etc.
CI Server
Test
Staging
Prod Code Review
"Infra-‐as-‐Code"
App Code & Templates
LONDON
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Continuous Integration and Deployment
Raj Wilkhu – Principal Engineer, JUST EAT
JUST EAT Platform
• High volume e-commerce platform • 45-60 minutes cycle • Real time message delivery and confirmation network • Hardware in 22,500 restaurants in the UK, 40,800 worldwide • Peak traffic is 10,000% of normal daytime traffic • 3 cities • Over 130 releases a month • Primarily Windows
Building reliable and high quality software
• Test-Driven Development, Continuous Deployment and Immutable Infrastructure
• Every part of the infrastructure replaced several times a day • Incentive to continuously speed up deployment process
Deploying components
• Platform decomposed into components • Each component defined by single AWS Cloudformation
template • Single Build Artifact • Instances boot and bootstrap themselves from
• pre-baked AMI • Content in S3 based on Cloudformation::Init metadata
Pre-baked AMI vs Dynamic config
• System dependencies (eg OS features, .NET, ruby, gems, etc)
• Static configurations
• Continuous Deployment (latest code) • Environment specific configuration • Automation to deploy feature
Increasing Boot time
• Unzip package • Install and configure dependencies • Execute deployment script(s) • Warm up feature • Tagged by deployment scripts
• Baked via CI pipeline • Tested and tagged
Pre-baked AMI CloudFormation::Init
Cloudformation template generation
• Single json from multiple yaml and erb files • Similar to HTML templating engines • Organized by component • Convention over configuration
"<%= @feature_name %>_pre_deploy": { "commands": { <% cmd_prefix = 250 services_to_disable = %w(aelookupsvc bits scdeviceenum trustedinstaller wuauserv) services_to_disable.each do |service_name| %> "<%= cmd_prefix %>_<%= @feature_name %>_stop_<%= service_name %>_service_because_unnecessary": { "waitAfterCompletion": "0", "command": "powershell -noninteractive -noprofile -command \"& { stop-service -Name <%= service_name %> -verbose }\"", "ignoreErrors": true }, "<%= cmd_prefix + 1 %>_<%= @feature_name %>_disable_<%= service_name %>_service_because_unnecessary": { "waitAfterCompletion": "0", "command": "powershell -noninteractive -noprofile -command \"& { set-service -Name <%= service_name %> -StartupType Disabled -verbose }\"", "ignoreErrors": true }, <% cmd_prefix += 2 end %>
--- :feature_config: :ami_id: ami-c99544ef :elb: true :use_logging_client: true :elb_dns: true :parameters: :ElbTarget: "HTTP:80/status" :AsgDesiredSizeDuringPeakTime: 10 :InstanceType: c4.xlarge
Example yaml: Example json.erb template:
{ Master
Immutable infrastructure - Cloudformation
Engineers
configs in yaml, template extensions in
json.erb + master
Tests in QA environment
Engineers
Component code
Continuous Integration Server (TeamCity Agent Cloud)
Build & Test Pipeline
Artifact
JustDeploy
Production/QA
EC2 Web/worker roles
Route 53
Rooms
Artifact
Git Git
Immutable infrastructure - Baking AMIs
Engineers
AMI Bakery
Continuous Integration Server (TeamCity Agenet Cloud)
EC2 base image
configurations and resources
Shared AMI
Latest Amazon OS image
Git
Automated Tests
config.yaml updated
Future
• Automatically select last tested AMI image • Use lambda to trigger AMI builds based on AMI release
notifications via SNS • Failover to secondary region • Publish more tools as OSS
– https://github.com/justeat
@justeat_tech + @rajwilkhu http://tech.just-eat.com
Yes, we’re recruiting too. http://tech.just-eat.com/jobs
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Deployment on the fly Bruce Jackson – CTO, Myriad Group AG
From IMPS to msngr
database
app server
web server web server web server
...x16
• Move from a mid-2000 carrier platform to web scale
• …with 38M users on 16 stacks
• …in 6 months • …with a team of 4
Basic foundations of our approach
Rules • OSGi used for all
application stacks • Make use of as much
managed infrastructure as possible
• Avoid any environment configuration
• Make deployment as familiar as coding
Rationale • Rapid creation of services
against interfaces • We don’t have any devops
so outsource management where possible
• No files/scripts that only one person understands
• The team is made up of developers
What did we do? • Created utility libraries
that use code annotations and AWS tags for configuration
• Use Eclipse features to define bundles to deploy on systems
What did we do (2) ?
• Created Eclipse tooling to manage deployment & code/feature checkout
A quick demonstration…
LONDON