CLOUD 39. Berner Architekten-Treffen

+41 79 422 11 87 | lukas.zaugg@spoud.io | CTO

+41 76 498 59 07 | patrick.boenzli@spoud.io | CEO

2017-03-23

ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT

WE DELIVER DATA SO YOU CAN FOCUS ON VALUE

+41 76 498 59 07 | patrick.boenzli@spoud.io | CEO

1SITUATION

The world’s most valuable resource is no longer oil, but data. - The Economist

Data = Resource

OR

Every IT Project

Creates more Data

Consumes existing Data

Explosion of Sources and Consumers

Isolated

DataIsolated

Consumers

Isolated

DataIsolated

Consumers

We observe a Gap in between

?

You don’t know what data assets your company has

You already know the Gap, if…

You have to go through an IT project to connect to data

You have no idea how good or bad data quality is

You don’t know who has access to sensitive data

Sensitive data ends up in test environments

Or if you create DB dumps to allow access to your data

You have nightmares about GDPR

We call the Gap…

DATA LOGISTICS

We deliver data so you can create value.

2SPOUD DATA MARKET

Connector

System

Syst

em

System

Compliance and Governance

Collaborative, Streaming Data-Market for your Company.

SPOUD Data Market

Your company data is easy to understand

and at your service with 1-click.

Imagine a world where…

INSURANCERETAIL SPACE SPORTS

Customers

CLOUD! WHAT’S THE QUESTION?

WE DELIVER DATA SO YOU CAN FOCUS ON VALUE

+41 79 422 11 87 | lukas.zaugg@spoud.io | CTO

From Zero to Cloud and Back

GET OUT! FAST!

#TIME-TO-MARKET

#MVP

#PYTHON

#AWS

#1-WEEK

#ETL

#SIMPLE-IOT-ANALYTICS

1 2 3 4

January 2016

65 7

Today

Simple IoT Analytics - Need

- DB - Server for APIs and website

Webserver

Simple IoT Analytics - What we got

WebserverWebserverWebserver

Load

balanc

er

- Auto Scaling - Health Checks - Monitor and Alerting - Managed Database (Backups, Updates, …) - Managed Server (Updates, Cleanup, …

From Zero to Cloud and Back

REINVENTING US! AGILE!

#EVENT-DRIVEN

#MORE-DATA

#ELASTICSEARCH

#AWS-SERVICES

#1-MONTH

#KAFKA

#MORE-ANALYTICS

#REALTIME

#FLINK

1 2 3 4

April 2016

65 7

Today

More Data, more Analytics

- Instead of vertical scaling, redesign architecture - Distributed Log, Message-Driven - Use of Realtime Data Processing with Flink - Use SaaS for everything with standard interfaces (e.g. MySQL, Redis, )

ETL

~ 10 - 30 Instances

From Zero to Cloud and Back

OVERLOAD! SCALE!

#HIGH-VVV

#GT-10-CUSTOMERS

#GT-2000-EVENTS-PER-SEC

#NOMAD

#ELASTIC-BEATS

#HA

#TERRAFORM

January 2017 Today

1 2 3 4 65 7

Handling Load

- 10 customers, 500 events/s on average, 2000 events/s peak - High latency and backpressure! Cause? CPU or I/O? Which

application? - CPU on Flink Node because bad partitioning (and bug) - Vertical scaling as quick fix - Horizontal scaling needs an even load distribution - Partitioning is key - Later, saturated network I/O on message broker

Management & Automation

- Using the existing AWS services for easy management no longer reasonable

- Welcome to the container orchestration level - Welcome to more complexity

~ 50 - 90 Instances

~ 50 - 90 Instances

From Zero to Cloud and Back

THE COSTS!

#PRICING-JUNGLE

#MONEY

#OPTIMIZE

#AUTOMATE

1

April 2017

65 7

Today

2 3 4

2017

4

April 2017 2018

$

2K 10K 12K 12K

0.5K 1K 1.5K 3K

43M 86M 129M 259M

2017

4

April 2017 2018

$

events/sec max

events/sec avg

events/day avg

~ 3000 bytes / event

Cost Factors

- Guarantees are not free (e.g. Message delivery, availability, order) - It’s too easy to use Cloud products :) - No cleanup - Flexibility has its price - Most expensive technologies are Kafka, Flink, ElasticSearch - Idle time, because no reasonable automatic downscaling

Cost Optimisation

- Clean Cloud - Everyone should be aware of the costs and budget - Know your cost indicator (e.g events/s and retention time) - Emerging patterns and products (e.g. serverless) - Automation - Auto Scaling

Cloud

From Zero to Cloud and Back

RESIST COZY LOCK-INS!

#FOCUS

#LOCK-IN

#ENABLER

#STANDARDS

1

August 2017

65 7

Today

2 3 4

Lock-Ins

- Do not depend on specific proprietary services for our base stack - CH and On-Premise is around the corner - Handle specific services as optional volatilities

From Zero to Cloud and Back

ENTERPRISES THINK DIFFERENT!

#EVALUATION

#CH-CLOUD

#ON-PREMISE

#ORGANIZATION

1

September 2017

65 7

Today

2 3 4

Evaluation Cloud Provider CH

- We are spoiled by GCP, AWS, AZ - Be aware of legacy Hosting Business Models with a Cloud paint - Focus on scalability, cost transparency and automation - Our customers are ready to pay more

On-Prem?

- Container to the rescue (LCD) - On-Prem is always highly specific - Be aware of the organisational structures

From Zero to Cloud and Back

EVERYTHING! CROSS-CLOUD!

#CROSS-SILO

#PRIVATE-CLOUD

#ON-PREMISE

#BRIDGE

#MULTI-CLOUD

#EDGE-COMPUTING

1 65 7

Today

2 3 4

Private Cloud, Multi Cloud, Everything

- We have to use different services to survive. Think AI. - Focus on Data in-motion and Event-Driven Architectures - Data Logistics is bridging the silos

Bridging through Data Logistics

Logistics Layer

Infrastructure 1 Infrastructure 2

Logistics Layer

Service AService B

CLOUD! What’s the question?

+41 79 422 11 87 | lukas.zaugg@spoud.io | CTO

+41 76 498 59 07 | patrick.boenzli@spoud.io | CEO

ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT

+41 79 422 11 87 | lukas.zaugg@spoud.io | CTO

+41 76 498 59 07 | patrick.boenzli@spoud.io | CEO