Orchestrating AWS Lambda with Step Functions · What is AWS Lambda Serverless As far as you are...
Transcript of Orchestrating AWS Lambda with Step Functions · What is AWS Lambda Serverless As far as you are...
Orchestrating Lambda with Step Functions
@technovangelist
@technovangelist
@technovangelist
@technovangelist
Who am I
Matt Williams
Evangelist at Datadog@[email protected]
Organizer of DevOps Days Boston 2017 & 2018
@technovangelist
Who is Datadog
SaaS-based Monitoring & Analytics Infrastructure, APM, LogsOpen Source AgentTrillions of data
points per day
We are hiring!!
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
I l ike to talk about Serverless…
@technovangelist
What is AWS Lambda
ServerlessAs far as you are concernedDon’t pay for idleScale with ease
FaaSNode 8.10 (java, go, python, c#)
Trigger-based
@technovangelist
Trigger Based?
S3DynamoDBKinesisSNSSESCognitoCloudFormationCloudwatch Logs & Events
CodeCommitConfigAlexaLexAPI GatewayIoTCloudFrontLambda
@technovangelist
AWS Lambda
@technovangelist
@technovangelist
@technovangelist
That’s pretty cool right? But not perfect
You still have to configure everything elseS3DynamoDBAPI GWetc
SAMCloudformationServerless
@technovangelist
AWS Lambda
@technovangelist
…but serverless !== ec2
@technovangelist
…but serverless !== applications
@technovangelist
…applications are complicated
@technovangelist
…serverless functions should respect the single responsibility principle…
@technovangelist
@technovangelist
@technovangelist
we can create many serverless functions
@technovangelist
we can create many serverless functions
but how do we orchestrate them?
@technovangelist
Event/Message/Database as Traffic Cop
Amazon Kinesis Amazon SQS Amazon DynamoDB
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://github.com/Nordstrom/hello-retail
@technovangelist
Enter AWS Step Functions
What about Azure?
@technovangelist
a step function is
@technovangelist
a step function is
a state machine
@technovangelist
but what about SWF?(Amazon Simple Workflow Service)
@technovangelist
AWS Step Function Concepts
State MachineStates
TaskChoiceSucceed/FailPassWaitParallel
TransitionsExecutions
You define the State Machine with the Amazon States Language
A diagram will be generated based on that structure
@technovangelist
State Example: Task
@technovangelist
State Example: Choice
@technovangelist
State Example: Succeed / Fail
@technovangelist
State Example: Pass
@technovangelist
State Example: Wait
@technovangelist
State Example: Parallel
@technovangelist
Many ways to build Step Functions
I used Serverless Framework
@technovangelist
Benefits of Serverless Framework
Define everything in a single file:IAM Role StatementsLambda Function HandlersLambda Function TriggersStep Function StatesCloudFormation for other things
@technovangelist
Amazon ECRAWS
Lambda
AmazonS3
AmazonDynamoDB
Amazon CloudFront
AWS CodeBuild
AWS CodeCommit
AWSCloudFormation
IAM
Amazon API Gateway*
AWS Step Functions
AmazonSNS
@technovangelist
@technovangelist
1.Use ML to figure out which slide2.Is this slide different than the last time?3.If so, pull slide data from Dynamo4.If slide different then create document
in s35.Build the site6.Wait for it
1.Is it built7.Done
@technovangelist
Type: TaskResource: Lambda Function
@technovangelist
@technovangelist
Type: TaskResource: Lambda Function
@technovangelist
@technovangelist
Type: TaskResource: Lambda Function
@technovangelist
Type: TaskResource: Lambda Function
@technovangelist
How about monitoring?
We monitor Lambda via Logs
MONITORING|unix_epoch_timestamp|value|count|my.metric.name|#tag1:value,tag2
console.log(monitoringstring);
@technovangelist
What are the Lambda Metrics
aws.lambda.duration, min, max, sumaws.lambda.errorsaws.lambda.invocationsaws.lambda.throttlesaws.lambda.iterator_age
@technovangelist
What are the Step Function Metrics
aws.states.execution_time (.maximum, .minimum)aws.states.executions_aborted, failed, started, succeeded, timed_outaws.states.lambda_functions_scheduled, started, succeeded, timed_outaws.states.execution_throttledaws.states.lambda_function_time (.maximum, .minimum)aws.states.lambda_function_run_time (.maximum, .minimum)aws.states.lambda_function_schedule_time (.maximum, .minimum)aws.states.lambda_functions_started
@technovangelist
@technovangelist
@technovangelist
@technovangelist
@technovangelist
@technovangelist
@technovangelist
Lambda and Step Function Challenges
• Logging and debugging is a bit…painful• Don’t trust the SF when it claims a step is complete• Cold start is a thing but it may not be as bad as you think. • Cold start is a thing and it could be worse than you think• Every Lambda should do one thing…except if you can’t
afford it
@technovangelist
@technovangelist
Summary
Step Functions make AWS Lambda orchestration easyLambda, Serverless, and Step Functions can be an interesting part of your infrastructureMonitor your Lambda functions to ensure peak performance
Source can be found at:https://github.com/DataDog/mattw-reinvent2017-demo
@technovangelist
Matt Williams
Evangelist at Datadog@[email protected]
Organizer of DevOps Days Boston 2017 & 2018