Amazon SWF and Gordon

Amazon SWF Ruby User Group Berlin

Peritor GmbH

EC2 & Eucalyptus Cluster Management

§  Automatic Configuration

§  Self Healing Clusters

§  Auto Scaling Clusters

§  One Click Deployment

§  One Click Cluster Cloning

§  Monitoring & Alerting

§  Logs & Accounting

§  Firewalls & Backups

§  Access & Rights Management

2

Architecture

3

DB

Web��App

Architecture

4

DB

Web��App

EC2

S3

Euca

Architecture

5

DB

Web��App

EC2

S3

Euca

Agent

RabbitMQ

RabbitMQ

6

RabbitMQ

7 http://www.heavensend-design.com/happy/

Agent/Queue Requirements

§ Scalability We manage thousands of instances

§ Availability & Reliability We need to be available even if an AWS region fails

8

DJ?

9

Resque?

10

Hmmmmm….

11

SWF: Simple Workflow Service

From http://aws.amazon.com/swf/

§ Maintains application state

§ Tracks workflow executions and logs their progress

§ Holds and dispatches tasks

§ Controls which tasks each of your application hosts will be assigned to execute

13


“Workflows as a Service”

§ Hosted

§ Scalable § Consistent (uses DynamoDB for state)

14

SWF: Pricing

§ $0.0001 per workflow execution

§ Until a Workflow Execution completes: $0.000005 per workflow execution for every 24-hours

§ $0.000005 per retained workflow execution per day

+ traffic

15

SWF: Workflow

16

Start ��Instance

Observe��Start

Send Init JSON

Cleanup

Activity 1 Activity 2 Activity 3

Activity 4

SWF: Architecture

17 http://docs.amazonwebservices.com/amazonswf/latest/developerguide/swf-welcome.html?r=8856

SWF: Decider

§  Implements workflow logic

§ Gets history of events

§ Decides what to do next: §  Call next activity

§  Abort workflow

§  Re-schedule

§  …

18

SWF: Worker

§  Implements activity logic

§ Gets input for current activity

§ Processes activity: §  Complete

§  Fail

§  Timeout

§  Heartbeat

19

Separate Task Order/Graph From Task Implementation

20

Worker Decider

Gordon

Ruby framework to use SWF

§ Syntactic sugar for creation of deciders and activities

§ Daemons for processing tasks

§ Unicorn-style master/child handling of processes

22

Gordon: Decider

23

Gordon: Decider

24

Gordon: Activity

25

Gordon: Start Workflow

26

Gordon: Processing

27

Gordon: Processing

28

Gordon: config.decider.yml

29

DEMO

30

Available soon at http://github.com/scalarium/gordon

Under Apache 2 license

31


Many nice features

§ Retention period

§ Task lists

§ Timeouts §  Event history

§ Abort/cancel/re-schedule

§ Delay

§ Mark & count

§  ...

32

We’re Hiring!

DevOps

§  Scalarium operations

§  Write and maintain Chef cookbooks

§  Improve and scale backend

§  Security

§  Help us support more OSs

§  Vagrant integration

33

Ruby Developers

§  Rails frontend

§  Ruby backend / agents

§  Chef

§  Scaling evented agents

§  API

§  Command line tools

§  EC2, S3, SWF, SQS, …

[email protected]

Amazon SWF and Gordon

Technology

Transcript of Amazon SWF and Gordon