© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Usman Shakeel – Amazon Web Services

Cloud Rendering at scale

on AWS

Visual Effects and

Animation1

Who is using AWS for Rendering?

3 Theme Parks

5 Gaming

Marketing2

4 Manufacturing

6 Life Sciences

7 Engineering and Architecture

Visual Effects and

Animation1

VFX/Animation Rendering - workflow components

CompostingModeling Rendering

Asset Management

Collaboration and Task Management

The challenge of making a film

The challenge of making a film

On-premise capacity

The challenge of making a film

On-premise capacity

Rendering in the Cloud

The challenge of making a film

On-premise capacity

Rendering in the CloudCloud provides you the capability to

scale fast and get the outputs faster

Initial project on-boarding

artwork

Classifying Studios By Consumption

They all ask us the same thing …

The ability to spin up thousands of cores on-demand

… without any upfront investment

… and leverage the most up-to-date configurations

A project based “disposable” infrastructure

… with a flexible licensing / utility / by the hour

They all tell us the same thing …

=< $0.01per core/hour

Access to thousands of

cores whenever needed

No upfront investments in

infrastructure

Easier collaboration

Ecosystem of software

providers

Access to large memory

configs to do 6K/10K renders

Project based “disposable” infrastructure

… when rubber meets the road !

Share FS Everywhere Latency Large Datasets Lots of instances

{Data/Content}

Rendering in the Cloud

Rendering in the Cloud - State of the Union

Scale at a very cheap price

EC2 Spot

Leveraging Spot successfully today requires some

effort

Build stateless, distributed, scalable applications

Choose which instance types fit your workload the best

Ingest price feed data for AZs and regions

Make run time decisions on which Spot pools to launch in based on price and volatility

Manage interruptions

Monitor and manage market prices across AZs and instance types

Manage the capacity footprint in the fleet

And all of this while you don’t know where the capacity is

Serve your customers

Spot Fleet

Instead of writing all that code to manage Spot Instances,

simply specify:

• Target Capacity – The number of EC2 instances that you want

in your fleet.

• Maximum Bid Price – The maximum bid price that you are

willing to pay.

• Launch Specifications – # of and types of instances, AMI id,

VPC, subnets or AZs, etc.

• IAM Fleet Role – The name of an IAM role. It must allow EC2 to

terminate instances on your behalf.

Spot Fleet Example – Instance WeightingSay your workload needs at least 60 GB of memory

Want capacity to complete 20 units of work

Choices:• r3.2xlarge (61.0 GB, 8 vCPUs) = 1 unit of 20

• r3.4xlarge (122.0 GB, 16 vCPUs) = 2 units of 20

• r3.8xlarge (244.0 GB, 32 vCPUs) = 4 units of 20

An option to bid for all of these instance types:

AWS Cloud Scale is “Large”

•10s/100s/1000s/10000s cores on-demand in the Cloud

•A “Large” renderfarm (Actual example):

55,000 cores

•At re:Invent 2015:

~40,000 cores on

EC2 Spot Market

Rendering in the Cloud - State of the Union

Scale at a very cheap price

• BYOLVray, Solid Angle, Autodesk Maya 3DsMax

• SaaS

Thinkbox Deadline Cloud Wizard

• AWS Marketplace

• Elastic Licensing models

Thinkbox Deadline Usage Based Licensing

• Render nodes pull metered licenses from Cloud-based license server

• Usage is tracked per minute

• Bulk minutes will be available via Thinkbox’s online store

• Store will eventually host 3rd party licensing (Nuke, VRay, etc)

Pixar Renderman/Tractor (demo at SIGGRAPH)

Rendering in the Cloud - State of the UnionLicensing at Cloud Scale

Rendering in the Cloud - State of the UnionHydrating the Cloud Renderfarm

S3 as the source of truth for your content/data

• On AWS Marketplace/SaaS(Aspera, Signiant, File Catalyst, Expedat)

• S3 Multi-part Upload

• AWS import/export Snowball

Direct to Shared File Systems

• EFS throughput scales linearly to the storage

• Lustre can hydrate from an S3 bucket

• Avere can be fronted to S3 or an on-premise NAS

+ Direct Connect

EFSS3

Multipart

AWS Snowball

Rendering in the Cloud - State of the UnionShared FileSystem Everywhere (some ideas)

Shared Storage

On-prem StorageAWS Direct Connect

Storage Cache

Amazon S3

Luster on EC2

Avere on EC2

EFS

AWS Direct Connect

Hydrate workers

EC2 Spot

Shared Storage

FXT on-prem

Rendering in the Cloud - State of the UnionShared FS (Content/Data Share) Everywhere (some ideas)

Elastic File System (Amazon EFS)

• Designed to support Peta-Byte scale file systems

• Throughput scales linearly to storage

• Same latency spec across each AZ

• Thousands of concurrent NFS connections

• Works great for Large I/O sizes

• Pay for only what you use not what you provision

• Managed with multi-copy durability Amazon EFS

Rendering in the Cloud - State of the UnionMove the Graphic Artist to the Cloud …

Rendering is going Global• NVIDIA GPU based EC2 instances

• Nice DCV

• Teradici PCoIP

• Frame, Otoy

• Windows and Linux (VNC+VirtualGL)

3D Modeler

Modeling Dumb Client

Remote Application

running on a G2 instance

G2

Rendering in the Cloud - State of the UnionManaging your “disposable” infrastructure

Launch a CloudFormation stack

with all the infrastructure

resources for a specific project

Autoscale the stack as

appropriate

AMI

CloudFormation

TemplateCloudFormation

Terminate

Template

Rendering in the Cloud - State of the UnionThe Crown Jewels

• AWS alignment with the latest MPAA cloud based application

guidelines for content security – August 2015

• VPC private endpoint for S3 – enables a true private workflow

capability

• Encryption & key management capabilities

• Glacier Vault for high-value media/originals

Rendering in the Cloud - A Sample Architecture

(All in Cloud Pipeline)

Shared Storage

Renderfarm

On-Prem Storage

Pipeline and License Manager

3D Modeler

Remote

App Visualization

AWS Direct Connect

Modeling Dumb Client

Storage Cache

Amazon S3

Avere on EC2

Scalable Renderfarm on EC2

Appstream or Teradici running on a G2 instance

Pipeline Manager running on EC2

G2

EC2 SPOT

EFS

Hydrate workers

EC2 Spot

Render Farm

Rendering in the Cloud - A Sample Architecture

(A Hybrid Pipeline)

Shared Storage

Renderfarm

On-Prem Storage

AWS Direct Connect

Storage Cache

Amazon S3

Avere on EC2

Scalable Renderfarm on EC2

EFS

Hydrate workers

EC2 Spot

On-premise

RenderfarmCloud renderfarm as an

extension of on-prem renderfarm

FXT on-prem

Pipeline and License

Manager (also manage

cloud renderfarm)

A Customer Example

A large scale test:

• Deploying ~40K cores

• In 20 min

• Leveraging Spot Fleet

• Hybrid rendering environment

using Avere (on-premise Data)

• EFS

Faster than on-premise

environment

Disney Animation Renderfarm

RenderfarmAvere FXT

cluster

WDAS Data Center

Renderfarm

Avere FXT

cluster

Storage

Remote Data Center

RenderfarmAvere FXT

cluster

Remote Data Center

San Francisco

Los Angeles

Burbank

Artists

Redundant 10Gb

Disney Animation Renderfarm (on the Cloud)

RenderfarmAvere FXT

cluster

WDAS Data Center

Renderfarm

Avere FXT

cluster

Storage

Remote Data Center

RenderfarmAvere FXT

cluster

Remote Data Center

San Francisco

Los Angeles

Burbank

Artists

Redundant 10Gb

virtual private cloud

Avere vFXT

Oregon

Spot InstancesEFS

Spot Fleet Deployment

Another Customer Example

Cloud Rendering Benchmarks

Benchmarks: On Premise vs. The Cloud

0

20

40

60

80

100

120

streamtriad diskread diskwrite

OnPrem

r3.4xlarge

r3.8xlarge

m4.4xlarge

m4.10xlarge

cr1.8xlarge

Higher is better

Average Open Latency

Average Read Latency

0

100

200

300

400

500

600

700

100 500 800 1200 2400 4000

Tim

e (

µs)

Render Processes

Mid-TierA

Mid-TierB

Mid-TierC

Archive

EFS

Rendering in the Cloud vs. On-Premise

-

5,000

10,000

15,000

20,000

25,000

30,000

1 10 20 30 40 50 60 70 80 90

Re

nd

er

Tim

e (

s)

Frame #

EC2/EFS

OnPrem

Lower is better

Lessons Learned

• Use as many different instance types as you can. Especially older generations.

• Think about ways to modify your workload

• Use every availability-zone

• Check your limits, especially your EBS limit and VPC setup (address space)

• Resource-Oriented Bidding

• Diversified Allocation

• Benchmark your workload and set pricing accordingly

• Set ONLY realistic pricing that you will pay for

• Don’t be afraid to ask for help or pre-planning your run from AWS

Conclusion

Cloud Rendering on AWS

State of the Union is getting stronger …

Rendering Forecast

Partly cloudy with a chance of all in the cloud…

Technology Trends

•Containers for Rendering

•GPU based Rendering

•More vendors investing in Elastic Licensing models due to the push from their

customers

•All in-Cloud rendering with more robust application streaming solutions

Thank you!