Orchestrating Data Workflows Using a Fully Serverless ...

Orchestrating Data Workflows Using a Fully Serverless Architecture

@tomer_levi

About Me

2

● Data Engineer at Fundbox

● Technology enthusiast - especially data, ML and serverless

● Tennis (go Roger!), Football (Chelsea)

● Beer

About FundboxOVERVIEW

Products Data-driven business credit solutions

Customers>150k US-based small businesses

Founded2013

Locations• San Francisco: headquarters• Tel Aviv: R&D• Dallas: credit and support• 225 employees and 12 dogs

Americas most promising AI companies

Investors

Tectonic shifts in data landscapeTHE OPPORTUNITY

Expensive: ~$3,500 per underwriteInaccurate: low approval rateOverly reliant on personal credit

Data-Native Approach

Tectonic shifts in data landscapeTHE OPPORTUNITY

Scalable: ~$0 cost per underwriteAccurate: high approval rateComprehensive view of business

Expensive: ~$3,500 per underwriteInaccurate: low approval rateOverly reliant on personal credit

Data-Native Approach

6

Data Scientist

Data Engineer

Data Analyst

Data

7

Data Scientist

Data Engineer

Data Analyst

Data

SQL ReportsETL

Training

Feature engineering

Data preparation

Motivation

8

Motivation

SSH

9

SSH

10

Our goal:

Let users think less about how their jobs are run on our infrastructure, and instead focus on their core inquiry.

11

Meet BIJO

Extensible Multiple runtime support

Runtime params Low level API

Simple Easy to use and maintain

MonitoredMetrics collection

Logs collection

12

Principles

Simple

Create a folder in Github

1

User creates a job definition folder in a dedicated Github repository

3-step job creationSimple

Create a folder in Github Add job files and config

1 2

User adds his python or sql file to the resource folder and created a

config.yaml for his job



Create a folder in Github Add job files and config Get job notifications

1 2 3

User adds his python or sql file to the resource folder and created a

config.yaml for his job

User get notified on job events such as job started and finished



17

Detailed documentation and Examples

18

Jenkins

19

Jenkins

ECR - Container Registry

20

Jenkins


21

CloudWatch Cron Schedule

Jenkins


22


Jenkins


23


Jenkins


CloudWatch Logs

Python Support

24

Python Support

25

Python Support

26

Monitored

Monitoring jobs BIJO uses AWS StepFunctions to manage job state and catch failures.

28

29


Jenkins


30


Jenkins


SNS Topic

31

Email Address

SMS

SNS - Notification Service

A message

SNS Topic

32

Email Address

SMS

Lambda Function

SNS - Notification Service

A message

Monitoring

StepFunctions SNS Topic

Slack Lambda #bijo_notifications

CloudWatch Logs

http post

Job A has finished


Pull docker image

33


Monitoring


Slack Lambda #bijo_notifications

CloudWatch Logs

http post

Job A has finished


Pull docker image

34



Sns-to-Slack Lambda #bijo_notifications

CloudWatch Logs

http post

36

Job dependency service

Job-Dependency Lambda



CloudWatch Logs

http post

37



DynamoDB table



CloudWatch Logs

http post

38



DynamoDB table

Run job

Dependency

39



CloudWatch Logs

http post


DynamoDB table

42


API Gateway

REST API

New Object creationS3

Run job



CloudWatch Logs

http post


DynamoDB table

43


API Gateway

http request

REST API


Run job



CloudWatch Logs

http post


DynamoDB table

44


API Gateway

http request

REST API


Run job



CloudWatch Logs

http post

Run job

48



API Gateway

http request

Dependency Service


DynamoDB table

Extensible

Principles

REST API Custom Runtime

REST API

Python Client

Custom runtime support using Docker containers

Custom runtime support through DockerSupport for any runtime environment and configuration using Docker

REST API

REST API

Python Client

BIJO APIsExtensible

Manger Service Notifications

Run jobs on demand

Get job execution history

Get job status

More…

Send Slack notifications easily

Plots and images support

Manager API - Job execution

55

Job Manager Service


Sns-to-Slack Lambda#bijo_notifications

CloudWatch Logs

http post


DynamoDB table

Job-Manager Lambda

API Gateway

Client

http request

Key Value Parameter Store 56


Job Manager Service


Sns-to-Slack Lambda#bijo_notifications

CloudWatch Logs

http post


DynamoDB table

Job-Manager Lambda

API Gateway

Client

http request

Key Value Parameter Store

Execute job

57


Notifications API

58

Simple v2

BIJO Slack Bot

60

BIJO Slack Bot

62


Sns-to-Slack Lambda

#bijo_notifications

CloudWatch Logs

http post


DynamoDB table

API Gateway

Client

http request

Bot Lambda


-Get history -Execute

64

Execute job


Job-Manager Lambda


Sns-to-Slack Lambda

#bijo_notifications

CloudWatch Logs

http post


DynamoDB table

API Gateway

Client

http request

Bot Lambda

Bot calls



65

Execute job


Job-Manager Lambda


Sns-to-Slack Lambda

#bijo_notifications

CloudWatch Logs

http post


DynamoDB table

API Gateway

Client

http request

Bot Lambda

Bot calls



66

Execute job


Job-Manager Lambda

67

Wait ….

S3 object creation == 1 job



New Object creation

S3




DynamoDB table

New Object creation

S3




DynamoDB table

New Object creation

S3

Number of ECS Fargate tasks is limited




DynamoDB table

New Object creation

S3




DynamoDB table

New Object creation

S3

Job-Manager Lambda




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Any available resources?




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Yes




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Run job




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Any available resources?




DynamoDB table

New Object creation

S3

Job-Manager Lambda

No :(




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Job queue




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Job queue

Job X done




DynamoDB table

New Object creation

S3

Job-Manager Lambda

Job queue

Run job

Extensible Multiple runtime support

Runtime params Low level API

Simple Easy to use and maintain

MonitoredMetrics collection

Logs collection

91

Principles

35

>4000$0.01

92

Impact

• Dynamic job resource allocation

• UI Dashboard

• Release as an open-source project

93

The Future

Thank you@tomer_levi

[email protected]

Rate today ’s session

95

Session page on conference website O’Reilly Events App

@tomer_levi

[email protected]

Orchestrating Data Workflows Using a Fully Serverless ...

Documents

Transcript of Orchestrating Data Workflows Using a Fully Serverless ...