NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
-
Upload
amazon-web-services -
Category
Technology
-
view
29 -
download
1
Transcript of NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
![Page 1: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dickson YueSolutions Architect
NEW LAUNCH! Advanced Task Scheduling with
Amazon ECS and Blox
January 19, 2017
![Page 2: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/2.jpg)
What is Amazon ECS?
Scalable Container Management
Flexible ContainerPlacement
AWS Platform Integration Extensible by Design
![Page 3: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/3.jpg)
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
Scheduler
Cluster State Service
Placement Engine
Event Stream
![Page 4: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/4.jpg)
Task Placement Engine
![Page 5: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/5.jpg)
Typical User Workflow
Create Task Definition Amazon ECS
Task Definition- Docker Image(s)- CPU, memory- Port mapping- CMD, ENV
> aws ecs register-task-definition --cli-input-json file://ecs-task.json
![Page 6: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/6.jpg)
Task
{ "family": "kata-demo-task", "containerDefinitions": [ { "name": "ecs-demo", "image": "dicksonyue/aws-voting-app:alpha", "cpu": 10, "memory": 500, "portMappings": [{ "containerPort": 8080 }], "essential": true, "command": [ "npm", "start"], "environment" : [ { "name" : "REDIS_HOST", "value" : "ecs-demo.cw7bo2.0001.usw2.cache.amazonaws.com" }, { "name" : "REDIS_PORT", "value" : "6379" } ] }
]}
Resources
Docker image
CMD, ENV
![Page 7: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/7.jpg)
New Placement Constraints & Attributes
Name Example
AMI ID attribute:ecs.ami-id == ami-eca289fb
Availability Zone attribute:ecs.availability-zone == us-east-1a
Instance Type attribute:ecs.instance-type == t2.small
Distinct Instances type=“distinctInstance”
Custom attribute:stack == prod
![Page 8: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/8.jpg)
Anatomy of Task Placement
Cluster Constraints
Custom Constraints
Placement Strategies
Apply Filter
Satisfy CPU, memory, and port requirements
Filter for location, instance-type, AMI, or custom attribute constraints
Identify instances that meet spread or binpack placement strategy
Select final container instances for placement
![Page 9: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/9.jpg)
Supported Placement Strategies
Binpacking Spread Affinity Distinct Instance
![Page 10: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/10.jpg)
Placement Strategy Chaining
Spread tasks across Zonesand Binpack within each Zone
![Page 11: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/11.jpg)
New Cluster Query Language
![Page 12: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/12.jpg)
Filtering: Match on Instance Family or Type
![Page 13: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/13.jpg)
Filtering: Match on Multiple Expressions
![Page 14: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/14.jpg)
Filtering: Match on Custom Attributes
![Page 15: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/15.jpg)
Task Placement Examples
![Page 16: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/16.jpg)
g2.2xlarge t2.small g2.2xlarge g2.2xlarge
Placement: Targeting Instance Type
![Page 17: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/17.jpg)
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge
t2.small
t2.small t2.medium
us-east-1aus-east-1d
Placement: Targeting Instance Type & Zone
![Page 18: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/18.jpg)
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Availability Zone Spread
![Page 19: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/19.jpg)
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Spread across Zone and Binpack
![Page 20: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/20.jpg)
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Affinity and Anti-Affinity
![Page 21: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/21.jpg)
Running a Service
![Page 22: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/22.jpg)
t2.medium t2.small t2.small
us-east-1aus-east-1d
t2.medium t2.micro t2.small
us-east-1c
Placement: Multiple Services on a Cluster
![Page 23: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/23.jpg)
t2.medium g2.2xlarge t2.micro t2.small
t2.small t2.small g2.2xlarge t2.small
t2.small t2.small
g2.2xlarge t2.small
Placement: Services – Distinct Instances
![Page 24: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/24.jpg)
Console: Getting Started with Placement
![Page 25: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/25.jpg)
Console: Placement Templates to Get Started
![Page 26: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/26.jpg)
Console: Customizing Placement Strategies
![Page 27: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/27.jpg)
Live Demo
![Page 28: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/28.jpg)
Event Stream
![Page 29: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/29.jpg)
Consuming Real-time Events
![Page 30: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/30.jpg)
Three Steps to Getting Started with Events
Step 1: Create CWE Rule Step 2: Create SNS Topic Step 3: Put Events to SNS
![Page 31: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/31.jpg)
Consuming Events for Service Discovery
app1-tst 10.1.0.11db1-tst 10.1.0.14app2 10.1.0.16db2 10.1.0.18my-app 10.1.0.20
websrv1 10.1.0.1websrv2 10.1.0.2websrv3 10.1.0.4app-dev1 10.1.0.9app-dev2 10.1.0.5app-dev3 10.1.0.8
db-dev 10.1.0.19
![Page 32: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/32.jpg)
Introducing Blox
![Page 33: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/33.jpg)
What is Blox?
Choice
Control
Developer Experience
![Page 34: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/34.jpg)
Building with Blox
scheduler cluster state service
![Page 35: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/35.jpg)
Set Up Blox LocallyStep 1: git clone https://github.com/blox/blox.git
Step 3: docker-compose up -d
Step 4: start using Blox locally
Step 2: deploy CloudFormation template to configure ECS event stream and SQS queue
![Page 36: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/36.jpg)
Swagger Spec: Cluster State Service
![Page 37: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/37.jpg)
Swagger Spec: Daemon Scheduler
![Page 38: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/38.jpg)
Daemon Scheduler
![Page 39: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/39.jpg)
t2.small t2.small t2.small
Blox: Daemon Scheduler
t2.small t2.small t2.small
scheduler cluster state service
t2.small t2.small t2.small
![Page 40: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/40.jpg)
Live Demo
![Page 41: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/41.jpg)
The Blox Vision
![Page 42: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/42.jpg)
Contributing to Blox
• Blox is licensed under Apache 2.0• Open an issue or pull request• Watch our roadmap on GitHub• Check out our Gitter channel
![Page 43: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/43.jpg)
Summary
![Page 44: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/44.jpg)
Summary
![Page 45: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/45.jpg)
Thank you!
![Page 46: NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a377b51a28abaa488b5701/html5/thumbnails/46.jpg)
Remember to complete your evaluations!