© 2019, Amazon Web Services, Inc. or its Affiliates. All rights ......2019/09/11 · © 2019,...

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

https://amzn.to/JPWebinar

https://amzn.to/JPArchive

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

❏

❏

❏

❖

❖

❖


•

•

①吹き出しをクリック②質問を入力③ Sendをクリック


•

•

•

•


❏❖

❖

❖

❏❖

❖


•

•

•

•

•

•


•

•

•

•


•

•

•

https://en.wikipedia.org/wiki/CRAY-1

https://en.wikipedia.org/wiki/EPFL

https://en.wikipedia.org/wiki/Lausanne


CPU

< >(queueing)

(dispatch)

dispatch

B A

A B


•

•

•

•

•


•

•

•

•

•

•


従来は処理のピークに合わせてリソースを予め用意していた。ピーク時に賄えないリソースは、ジョブスケジューラにより、処理するタイミングをずらして対応

クラウドの場合はその時に必要なリソースを立ち上げて、処理を実行する

Time

Time


従来は手持ちの限られたリソースで、逐次処理していたジョブもクラウドであれば必要な台数、インスタンスを起動して、一斉処理が可能

のインスタンス費用は原則「時間×台数」なのでどちらもほぼ同じコスト


オンプレミス環境

従来のクラスタ構成は固定

固定のリソースに、ジョブをスケジューリング


クラウド環境従来のクラスタ構成は固定



クラウド環境


必要に応じて必要な台数でクラスタを構成ジョブをスケジューリング




クラウド環境

クラウド環境



ジョブが終了したらインスタンスを終了計算ノードの課金が停止



クラウド環境


•

•

•

•

•


Media & Entertainment:

Healthcare & Life Sciences:

Internet Companies:

Scientific Research:

Engineering and Design:


上にコアを伸縮自在に追加するシステムを構築。万ジョブを処理し、従来週間かかるシミュレーションを日で完了

ドライブヘッド設計で、数百万パターンに及ぶパラメータスタディ実行。ピーク時以上のコアを同時利用、従来ヶ月かかる処理を時間で完了

万の化合物のスクリーニング実行。ピーク時約コアを同時利用し年分の計算処理を時間で完了


•

•

•

•

•

•


• がインスタンスの起動や停止を行うため、スケジューラや計算ノードなどの管理が不要

• ジョブはコンテナイメージを元に作成し、自動でスケールするコンピューティング環境で実行する

• コンピューティング環境ではインスタンスタイプや数、スポットインスタンス利用有無などを任意に指定可能

• 万クラスの大規模な計算にも対応で管理

処理を依頼

スケジュール実行


•

•

•

•

•

•

•


•

•

•

•

•

•

•

•

•


•

•

•

•


•

•

•

•

•

•


インスタンス


•

•

•

•

•

•

•

•

•


•

•

•

•


•

•

•

•

•

•

•

•

•

•


•

•


インスタンス


•

•

•

•

•

•


•

•

•

•

•

•

Submitted

Runnable

Starting

Running

Succeeded FAILED

Pending

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/batch_sns_tutorial.html

https://aws.amazon.com/jp/premiumsupport/knowledge-center/batch-job-stuck-runnable-status/


•

•

$AWS_BATCH_JOB_ARRAY_INDEX

•

•“Job-A”

“Job-B”

“Job-A”

Job-A:0

…

Job-A:1 Job-A:2

Job-A:3 Job-A:4 Job-A:5


https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/array_jobs.html

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/job_dependencies.html


“Job-A”

Job-A:0

…

Job-A:1

Job-A:99

Job-A:2

“Job-A”

Job-A:0

…

Job-A:1 Job-A:2



“Job-B”

“Job-B”

Job-B:0

…

Job-B:1 Job-B:2

Job-B:3 Job-B:4 Job-B:5

Job-B:97 Job-B:98 Job-B:99

“Job-A”


•

•

•

•

•

•

•

•

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/launch-templates.html

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/job_timeouts.html

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/job_retries.html


•

•

•

•

•

•

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/ExamplePolicies_BATCH.html#iam-example-job-def

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/job_definition_parameters.html#containerProperties

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/logging-using-cloudtrail.html


•

•

•

•

•

•

•

“Job-A”

https://aws.amazon.com/jp/blogs/compute/building-a-tightly-coupled-molecular-dynamics-workflow-with-multi-node-parallel-jobs-in-aws-batch/

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/multi-node-parallel-jobs.html

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/efa.html


•

•

•

•

•


•

•

•

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/batch-cwe-target.html#scheduled-batch-job


•

•

•

•

•

•

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/job_retries.html

https://www.slideshare.net/AmazonWebServicesJapan/20190306-aws-black-belt-online-seminar-amazon-ec2


インスタンス

•

•

•

https://docs.aws.amazon.com/ja_jp/batch/latest/userguide/launch-templates.html#lt-user-data

https://aws.amazon.com/jp/blogs/compute/scalable-deep-learning-training-using-multi-node-parallel-jobs-with-aws-batch-and-amazon-fsx-for-lustre/

https://www.slideshare.net/AmazonWebServicesJapan/20190319-aws-black-belt-online-seminar-amazon-fsx-for-lustre


https://docs.aws.amazon.com/ja_jp/codebuild/latest/userguide/sample-docker.html


•

•

•

https://aws.amazon.com/jp/blogs/compute/building-simpler-genomics-workflows-on-aws-step-functions/

https://www.slideshare.net/AmazonWebServicesJapan/20190522-aws-black-belt-online-seminar-aws-step-functions


•

•

•

•

•

•


•

•

•

https://aws.amazon.com/jp/about-aws/whats-new/2019/04/AWS-Batch-supports-GPU-Scheduling/

https://aws.amazon.com/jp/blogs/compute/deploy-an-8k-hevc-pipeline-using-amazon-ec2-p3-instances-with-aws-batch/


•

•

https://www.nextflow.io/

https://www.nextflow.io/blog/2017/scaling-with-aws-batch.html

https://aws.amazon.com/jp/blogs/compute/using-cromwell-with-aws-batch/

https://aws.amazon.com/jp/quickstart/architecture/illumina-dragen/


•

•

•



•

•

•

•

•



•

•

•

•


•

•

•

•

•

•

•

•

•

•

•

•

•

https://www.slideshare.net/AmazonWebServices/set-up-a-millioncore-cluster-to-accelerate-hpc-workloads-cmp404-aws-reinvent-2018

https://www.slideshare.net/AmazonWebServices/intro-to-aws-batch-how-aqr-capital-leverages-aws-to-identify-new-investment-signals-cmp372-aws-reinvent-2018


https://aws.amazon.com/jp/blogs/compute/building-simpler-genomics-workflows-on-aws-step-functions/

https://github.com/aws-samples/aws-batch-genomics

https://aws.amazon.com/jp/quickstart/architecture/illumina-dragen/

https://aws.amazon.com/jp/blogs/compute/accelerating-precision-medicine-at-scale/


•

•

•

•

•

•

https://aws.amazon.com/pt/blogs/compute/deep-learning-on-aws-batch/


https://aws.amazon.com/jp/blogs/compute/deploy-an-8k-hevc-pipeline-using-amazon-ec2-p3-instances-with-aws-batch/


•

•

•

•

•

•

•

•

•

•

•

•

https://www.slideshare.net/AmazonWebServicesJapan/20190306-aws-black-belt-online-seminar-amazon-ec2

https://www.slideshare.net/AmazonWebServicesJapan/20180214-aws-black-belt-online-seminar-amazon-container-services

https://www.slideshare.net/AmazonWebServicesJapan/20190731-black-belt-online-seminar-amazon-ecs-deep-dive-162160987

https://www.slideshare.net/AmazonWebServicesJapan/20190522-aws-black-belt-online-seminar-aws-step-functions

https://www.slideshare.net/AmazonWebServicesJapan/20190423-aws-black-belt-online-seminar-aws

https://www.slideshare.net/AmazonWebServicesJapan/20190319-aws-black-belt-online-seminar-amazon-fsx-for-lustre


https://aws.amazon.com/jp/blogs/news/


AWS Well-Architected 個別技術相談会

•

•

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights ......2019/09/11 · © 2019,...

Documents

Transcript of © 2019, Amazon Web Services, Inc. or its Affiliates. All rights ......2019/09/11 · © 2019,...