Automating Management of Amazon EC2 Instances with Auto Scaling - March 2017 AWS Online Tech Talks
-
Upload
amazon-web-services -
Category
Technology
-
view
345 -
download
3
Transcript of Automating Management of Amazon EC2 Instances with Auto Scaling - March 2017 AWS Online Tech Talks
©2017,AmazonWebServices,Inc.oritsAffiliates.Allrightsreserved.
Michael Hanisch, AWS Solutions Architecture
3/23/2017
Automating Management of Amazon EC2 Instances with Auto Scaling
AutoScalinggroupAutoScalinggroup
AutoScaling
Dynamic Scaling
ELB
EC2Instances
ELB
CPUUtilization
EC2Instances
Fleet Management
Myth Fact
MyApplicationDoesn’tNeedScaling,SoIDon’tBenefitFromAutoScaling
It’sHardToUse
MyInstancesareStateful orUnique;Ican’tuseAutoScaling
ItWorksWellwithStateful Instances
YouCanGetStartedinMinutes
ItMonitorsandHealsInstances
Is Fleet Management For You?
“I’vegotinstancesservingabusiness-impactingapplication”
“Ifmyinstancesbecomeunhealthy,I’dlikethemreplacedautomatically”
“Iwouldlikemyinstancesdistributedtomaximizeresilience”
Auto Scaling – The Basics
instanceAvailability Zone a
instance instanceAvailability Zone bAvailability Zone a
instance instanceAvailability Zone bAvailability Zone a
Auto Scaling group
Minimum#=2 Maximum#=2
Desired#ofinstances=2
instance instanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
instance instanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
Auto Scaling group
Minimum#=2 Maximum#=2
Desired#ofinstances=2
Auto Scaling Groups
- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs
Launch Configurations
Determine what is going to be launched:- EC2 instance type & size- Amazon Machine Image (AMI)- Security groups, SSH keys, IAM instance profile- User data…
Bootstrapping
Installation & setup needs to be fully automated:- Use Amazon Machine Image (AMI) with all required
configuration & software (“golden image”)- Base AMI + install code & configuration as needed
- Via Userdata + scripts- Via Chef/Puppet/Ansible/…- Using AWS CodeDeploy- Using Amazon EC2 Systems Manager
…
Bootstrapping
#!/bin/bash# Install updatessudo yum update -y;# Install Amazon EC2 Systems Manager Agentcd /tmp;curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm -o amazon-ssm-agent.rpm && yum install -y amazon-ssm-agent.rpm;
Bootstrapping
#!/bin/bash# Install updatessudo yum update -y;# Install AWS CodeDeploy agentcd /home/ec2-user; curl https://aws-codedeploy-us-east-1.s3.amazonaws.com/latest/install \ -o install && chmod +x ./install && sudo ./install auto && sudo service codedeploy-agent start;
Monitoring
Auto Scaling gives you access to new metrics in Amazon CloudWatch:
- group-level metrics like number of running instances- aggregate metrics like average CPU utilization for all
instances in the group
Monitoring
Auto Scaling Concepts
LaunchConfiguration• AutoScalinggroupsusea
launchconfiguration tolaunchEC2instances.
• ProvidesinformationabouttheAMIandEC2instancetypes/size
ScalingPlan• AscalingplantellsAuto
Scalingwhenandhowtoscale.
• Createascalingplanbasedontheoccurrenceofspecifiedconditions(dynamicscaling)orcreateaplanbasedonaspecificschedule.
AutoScalingGroups• EC2instancesare
managedbyAutoScalinggroups.
• CreateAutoScalinggroupsbydefiningtheminimum,maximum,and,optionally,thedesirednumberofrunningEC2instances.
Termination Policies
Determine which instances are terminated first:- Longest running- Oldest launch configuration- Closest to full billing hour
But: rebalancing of capacity across AZs takes precedence!
Scaling Plans
Determine when the Auto Scaling group will scale in or out:
desired capacity > current capacity: launch instancesdesired capacity < current capacity: terminate instances
Scaling Plans
- Default: ensure current capacity of healthy instances remains within boundaries (never less than minimum)
- ‘Manual scaling’: modify desired capacity (via API, console, CLI) to trigger a scaling event
- Scheduled: scale in / out based on timed events- Dynamic scaling: scale based on CloudWatch metrics
Getting Started
What happens next?
Availability & Reliability
Auto Scaling Groups
- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs
Auto Scaling Groups
- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs- Replace unhealthy instances
Auto Scaling Groups
- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs- Replace unhealthy instances
Health Checks
- Performed periodically- Instances are marked as “Unhealthy” when checks fail- Unhealthy instances are terminated and replaced
(if new number of instances < minimum or < desired capacity)
Different Kinds of Health Checks
- EC2 instance status: Instance is unhealthy when instance state != ‘running’ or system health check == ‘impaired’
- ELB health checks:instance is unhealthy when ELB health check results in “OutOfService” (or EC2 health check failed)
- Manual: mark individual instances as ‘unhealthy’Instance unhealthy when marked as such or EC2 health check failed. Use to integrate with external monitoring systems.
instance instanceinstance instance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceinstanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
Unhealthy Instances Get Replaced…
instance instanceinstance instance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceinstanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
Unhealthy Instances Get Replaced…
instance instanceinstance instance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceinstanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
Unhealthy Instances Get Replaced…
instance instanceinstance instance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceinstanceAvailability Zone bAvailability Zone a
Elastic LoadBalancing
…In a Different AZ if Necessary
instanceinstance instanceinstance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceAvailability Zone bAvailability Zone a
instance
Elastic LoadBalancing
Rebalancing Capacity
instanceinstance instanceinstance
Auto Scaling group
Minimum=2 Maximum=6
Desired#ofinstances=6
instanceAvailability Zone bAvailability Zone a
instance
Elastic LoadBalancing
instance instance instance
What happens when an instance is terminated?
The Instance Lifecycle
Instance Lifecycle
Scale OutEvent
Instancelaunching:Pending
InService
TerminatingTerminated
Scale InEvent
Health checkfailed
Instance Lifecycle
AddanInstance
Instancelaunching:Pending
InService
TerminatingTerminated
RemoveanInstance
Health checkfailed
Instance Lifecycle
Instancelaunching:Pending
InService
TerminatingTerminated
Entering Standby…
Standby
How can we influence the instance lifecycle?
Why? – Common Use Cases
• Assign Elastic IP address or ENI on launch• Register new instances with DNS, external monitoring
systems, firewalls, load balancers, …• Load existing state from S3 or other system• Pull down log files before instance is terminated• Investigate issues with an instance before terminating it• Persist instance state to external system• …
Lifecycle Hooks & Notifications
Instance Lifecycle Notifications
AddanInstance
Instancelaunching:Pending
InService
TerminatingTerminated
RemoveanInstance
Health checkfailed
EC2InstanceTerminateSuccessful
EC2InstanceLaunchSuccessful
EC2InstanceLaunchUnsuccessful
Instance Lifecycle Notifications
• Notifications get sent after a state transition.• Rely on notifications to react to changes that happened.• Available via Amazon Simple Notification Service and
Amazon CloudWatch Events.• Prefer CloudWatch Events due to ease of use and
extended feature set!
Sample Notification
Service:AutoScalingTime:2017-03-23T21:53:43.989ZRequestId:52e21eba-718a-43a7-81a8-3b379054cba6LifecycleActionToken:979c0f97-80c5-44bd-a2b6-5a8aae339f35AccountId:XXXXXXXXXAutoScalingGroupName:demo-asgLifecycleHookName:do-something-on-launchEC2InstanceId:i-XXXXXLifecycleTransition:autoscaling:EC2_INSTANCE_LAUNCHINGNotificationMetadata:null
Lifecycle Hooks
Instance Lifecycle Hooks
• When Lifecycle Hooks are defined, instances enter special “WAIT” states during state transitions.
• Allows you to react to lifecycle events & impact the state• WAIT states bring their own notifications, too.
Instance Lifecycle Hooks
Instancelaunching:Pending
InService
TerminatingTerminated
Pending:WAIT
Terminating:WAIT
InvokeHook(s)
Lifecycle Hooks- Executed before taking a new
instance into service / terminating it
- Put instances into a WAIT state while work can happen
Auto Scaling Notifications- Notifications get sent after an
instance has entered “InService” or “Terminated” state, respectively
- Cannot influence or stop a transition
Let’s see this in action…
TODO:Tasks? Secondpartofdemo:showhowtosetup
CWEevent+ruletoinvokeEC2SystemsManagerRunCommand directlytosavewebserverlogfiles.(Pointoutthatthisisacommonexamplebutthatwe’drecommendtouseaproperloggingsolution,e.g.CloudWatch Logs,instead)
TODO:Tasks?
Addarchitecturediagramofthesolutionwejustsetup– ordothisbefore?
instance instanceAvailability Zone bAvailability Zone a
Auto Scaling group
AutoScaling
Lambdafunction
CloudWatch Events AmazonEC2SystemsManager
1. Eventfires,triggersRule
2.RuleinvokesLambda 3.AsksEC2SSM
ToRunCommand
4.InvokeRunCommandonterminatinginstance
5.CommanduploadslogstoS3
AutoScaling6.CompleteHook
How Do I Write a Lifecycle Hook?
1. Code the lifecycle hook’s action2. Create new Rule in CloudWatch Events3. Associate the lifecycle hook with the Auto Scaling group
Writing a Lifecycle Hook
1. Code the lifecycle hook’s action1. Extract instanceID, auto scaling group, other params.2. Do stuff…
• Beware of timeouts!• Send “heartbeats” if you need more time
3. Call CompleteLifecycleAction to signal that you’re done!
Writing a Lifecycle Hook
1. Code the lifecycle hook’s action• AWS Lambda function• Amazon EC2 Systems Manager RunCommand• Any Code that Consumes Kinesis Streams/ SQS/ SNS
Writing a Lifecycle Hook
{ "schemaVersion": "1.2",
"description": "Backup logs to S3", "parameters": {},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [ {
"id": "0.aws:runShellScript",
"runCommand": [ "",
"ASGNAME='demo-asg'",
"LIFECYCLEHOOKNAME='demo-asg-backup-hook'",
"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)",
"REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)",
"REGION=${REGION::-1}",
"HOOKRESULT='CONTINUE’”,
[…]
Writing a Lifecycle Hook
[…]
aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup",
" MESSAGE=$(cat /tmp/backup)",
"fi",
"",
"aws autoscaling complete-lifecycle-action
--lifecycle-hook-name ${LIFECYCLEHOOKNAME}
--auto-scaling-group-name ${ASGNAME}
--lifecycle-action-result ${HOOKRESULT}
--instance-id ${INSTANCEID} --region ${REGION}”
]
} }
}
}
Writing a Lifecycle Hook
2. Create new Rule in CloudWatch Events
Writing a Lifecycle Hook2. Create new Rule in CloudWatch Events
Writing a Lifecycle Hook
aws autoscaling put-lifecycle-hook \
--auto-scaling-group-name demo-asg \
--lifecycle-hook-name demo-hook-terminate \
--lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING
3. Associate the lifecycle hook with the Auto Scaling group
Dealing with StatefulApplications
Dealing With Stateful Applications
- While ”InService”:- Persist state to EBS volume on a regular basis- Tag with InstanceId, application name
- On “Instance-terminating Lifecycle Action”:- Detach EBS volume with state information- Remove InstanceId tag, keep application name tag
- On “Instance-launch Lifecycle Action” event:- Find & Attach EBS volume tagged w/ application name- Tag with InstanceId, Resume Application
Conclusion
Fact
ItWorksWellwithStateful Instances
YouCanGetStartedinMinutes
ItMonitorsandHealsInstances
• Direct Integration with CloudWatch• Instance Replacement• AZ Rebalancing
• Options for Easy Bootstrapping• Start off with Existing Instances
• Lots of Control via Lifecycle Hooks• Keep Track with Notifications
Questions?
Questions?
https://aws.amazon.com/blogs/compute/fleet-management-made-easy-with-auto-scaling/
http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html
https://aws.amazon.com/autoscaling/getting-started/
Thankyou!