© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1

( )

EC2

2019-03-18

Amazon EC2

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2

•

••

•

•

••

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4https://www.itmedia.co.jp/enterprise/articles/1902/22/news006.html

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 5

AI AWS (1/4) - ITmedia

https://www.itmedia.co.jp/enterprise/articles/1902/22/news006.html

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 6

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 7

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 9

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 10

( )

( 5 )

2019-03-18

Amazon EC2

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 11

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 12

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 13

•

••

•

•

••

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 14

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 15

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 16

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 17

•

•

•

••

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 18

•

•

•

••

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 19

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20

AWS

CloudFormation

AWS

OpsWorks

Amazon

EMR

Amazon

ECS

AWS Data

Pipeline

AWS BatchAuto Scaling

AWS Thinkbox

Deadline

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 21

•

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 22

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 23

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 24

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 25

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 26

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 27

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 28

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 29

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 30

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 31

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 32

• 1

• ( )

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 33

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 34

AWS

CLI

(aws ec2 ...) (EC2)

run-instances

(request-spot-

instances)

one-time (sync) /

persistent (sync)

request-spot-

fleet/

vCPU( )

request (async) /

maintain (async)

EC2create-fleet /

vCPU( )

instant (sync) /

request (async) /

maintain (async)

Auto Scaling

(New!)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 35

AWS

CLI

(aws ec2 ...) (EC2)

run-instances

(request-spot-

instances)

one-time (sync) /

persistent (sync)

request-spot-

fleet/

vCPU( )

request (async) /

maintain (async)

EC2create-fleet /

vCPU( )

instant (sync) /

request (async) /

maintain (async)

Auto Scaling

(New!)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 36

AWS

CLI

(aws ec2 ...) (EC2)

run-instances

(request-spot-

instances)

one-time (sync) /

persistent (sync)

request-spot-

fleet/

vCPU( )

request (async) /

maintain (async)

EC2create-fleet /

vCPU( )

instant (sync) /

request (async) /

maintain (async)

Auto Scaling

(New!)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 37

AWS

CLI

(aws ec2 ...) (EC2)

run-instances

(request-spot-

instances)

one-time (sync) /

persistent (sync)

request-spot-

fleet/

vCPU( )

request (async) /

maintain (async)

EC2create-fleet /

vCPU( )

instant (sync) /

request (async) /

maintain (async)

Auto Scaling

(New!)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 38

m4.large Spot ASG Min: 1 Max: 10

m5.large Spot ASG Min: 1 Max: 10

c4.xlarge O-D ASG Min: 1 Max: 10

Availability

Zone 1

Availability

Zone 2

Availability

Zone 3

-

ASG

ASG

-

ASG

ASG: Auto Scaling Group

O-D: On Demand Instance

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 39

m4.large Spot Instances

m5.large Spot Instances

c4.xlarge On-Demand instances

Availability

Zone 1

Availability

Zone 2

Availability

Zone3

ASG

Single ASG

ASG: Auto Scaling Group

O-D: On Demand Instance

RI: Reserved Instance

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 40

•

Auto Scaling

Group

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 41

EC2 Auto Scaling Spot Fleet

AZ

[1]1. Total target capacity

2. "Optional On-demand

portion

3.

[2]1. Desired Capacity

2. "On-Demand base"

3. OD:Spot Ratio

ELB

Launch Template

$latest, $default

[1] https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot

[2] https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/AutoScalingGroup.html#asg-purchase-options

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 42

•

•

•

•

•

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 43

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 44

On Demand Auto-

Scaling group

Output S3

bucket

Availability Zone A

Availability Zone BSpot Fleet

Upload object

into input S3

bucket

Job SQS

Queue

Auto Scaling group and Spot fleet

scale up based on queue depth and

scale down based on CPU utilization

CW metrics

Workers will

check for jobs

in the queue

Workers will update Job status

(start time, SLA end time, etc) in

DynamoDB

Uploads to S3 will trigger

a Lamda function to put

jobs in SQS and

DynamoDB

EFSEC2 instance

worker fleet

Cost-effective Batch Processing with Amazon EC2 Spot |

AWS Compute Blog

https://aws.amazon.com/blogs/compute/cost-effective-

batch-processing-with-amazon-ec2-spot/

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 45

JobQueuesubmit-job

list-jobs

describe-jobs

cancel-jobs

terminate-jobs

Compute Environment

ECR/Docker

Registry

S3/EFS/NFS

Autoscaling

ECS

Docker Image

JobDefinitio

n

jobAWS Batch

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 46

ビッグデータ領域のリファレンスアーキテクチャ

Master

Node (OD

or RI)

S3

EMRFS

Master Fleet

Task

Node

(Spot)

Task

Node

(Spot)

Task

Node

(Spot)

Task Fleet

Instance

Store

EBS

Volume

Instance

Store

EBS

Volume

Instance

Store

EBS

Volume

Core Fleet

HDFS

Instance Fleet

Core Node

(OD or RI)

Core Node

(OD or RI)

Core Node

(OD or RI)

Amazon EMR

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 47

コンテナを活用したステートレスウェブアプリケーション

Stateless Web

Server (Spot)

Stateless Web

Server (Spot)

Session State Data

Stateless Web Server

(On-Demand)

Elastic Load

Balancing

Amazon EC2

Auto Scaling

ECS

container 1

Stateless Web Server

(On-Demand)

ECS

container 2

ECS

container 1ECS

container 2

https://github.com/awslabs/ec2-spot-labs/tree/master/workshops/ec2-spot-fleet-web-app

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 48

•

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 49

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 50

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 51

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 52

•

•

•

••

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 53

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 54

•

••

•

•

••

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 55

•

••

•

•

••

•

•

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 56