CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming...
Transcript of CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming...
CLOUD 39. Berner Architekten-Treffen
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO
2017-03-23
ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT
WE DELIVER DATA SO YOU CAN FOCUS ON VALUE
+41 76 498 59 07 | [email protected] | CEO
1SITUATION
The world’s most valuable resource is no longer oil, but data. - The Economist
Data = Resource
OR
Every IT Project
Creates more Data
Consumes existing Data
Explosion of Sources and Consumers
Isolated
DataIsolated
Consumers
Isolated
DataIsolated
Consumers
We observe a Gap in between
?
You don’t know what data assets your company has
You already know the Gap, if…
You have to go through an IT project to connect to data
You have no idea how good or bad data quality is
You don’t know who has access to sensitive data
Sensitive data ends up in test environments
Or if you create DB dumps to allow access to your data
You have nightmares about GDPR
We call the Gap…
DATA LOGISTICS
We deliver data so you can create value.
2SPOUD DATA MARKET
Connector
System
Syst
em
System
Compliance and Governance
Collaborative, Streaming Data-Market for your Company.
SPOUD Data Market
Your company data is easy to understand
and at your service with 1-click.
Imagine a world where…
INSURANCERETAIL SPACE SPORTS
Customers
CLOUD! WHAT’S THE QUESTION?
WE DELIVER DATA SO YOU CAN FOCUS ON VALUE
+41 79 422 11 87 | [email protected] | CTO
From Zero to Cloud and Back
GET OUT! FAST!
#TIME-TO-MARKET
#MVP
#PYTHON
#AWS
#1-WEEK
#ETL
#SIMPLE-IOT-ANALYTICS
1 2 3 4
January 2016
65 7
Today
Simple IoT Analytics - Need
- DB - Server for APIs and website
Webserver
Simple IoT Analytics - What we got
WebserverWebserverWebserver
Load
balanc
er
- Auto Scaling - Health Checks - Monitor and Alerting - Managed Database (Backups, Updates, …) - Managed Server (Updates, Cleanup, …
From Zero to Cloud and Back
REINVENTING US! AGILE!
#EVENT-DRIVEN
#MORE-DATA
#ELASTICSEARCH
#AWS-SERVICES
#1-MONTH
#KAFKA
#MORE-ANALYTICS
#REALTIME
#FLINK
1 2 3 4
April 2016
65 7
Today
More Data, more Analytics
- Instead of vertical scaling, redesign architecture - Distributed Log, Message-Driven - Use of Realtime Data Processing with Flink - Use SaaS for everything with standard interfaces (e.g. MySQL, Redis, )
ETL
~ 10 - 30 Instances
From Zero to Cloud and Back
OVERLOAD! SCALE!
#HIGH-VVV
#GT-10-CUSTOMERS
#GT-2000-EVENTS-PER-SEC
#NOMAD
#ELASTIC-BEATS
#HA
#TERRAFORM
January 2017 Today
1 2 3 4 65 7
Handling Load
- 10 customers, 500 events/s on average, 2000 events/s peak - High latency and backpressure! Cause? CPU or I/O? Which
application? - CPU on Flink Node because bad partitioning (and bug) - Vertical scaling as quick fix - Horizontal scaling needs an even load distribution - Partitioning is key - Later, saturated network I/O on message broker
Management & Automation
- Using the existing AWS services for easy management no longer reasonable
- Welcome to the container orchestration level - Welcome to more complexity
~ 50 - 90 Instances
~ 50 - 90 Instances
From Zero to Cloud and Back
THE COSTS!
#PRICING-JUNGLE
#MONEY
#OPTIMIZE
#AUTOMATE
1
April 2017
65 7
Today
2 3 4
2017
4
April 2017 2018
$
2K 10K 12K 12K
0.5K 1K 1.5K 3K
43M 86M 129M 259M
2017
4
April 2017 2018
$
events/sec max
events/sec avg
events/day avg
~ 3000 bytes / event
Cost Factors
- Guarantees are not free (e.g. Message delivery, availability, order) - It’s too easy to use Cloud products :) - No cleanup - Flexibility has its price - Most expensive technologies are Kafka, Flink, ElasticSearch - Idle time, because no reasonable automatic downscaling
Cost Optimisation
- Clean Cloud - Everyone should be aware of the costs and budget - Know your cost indicator (e.g events/s and retention time) - Emerging patterns and products (e.g. serverless) - Automation - Auto Scaling
Cloud
From Zero to Cloud and Back
RESIST COZY LOCK-INS!
#FOCUS
#LOCK-IN
#ENABLER
#STANDARDS
1
August 2017
65 7
Today
2 3 4
Lock-Ins
- Do not depend on specific proprietary services for our base stack - CH and On-Premise is around the corner - Handle specific services as optional volatilities
From Zero to Cloud and Back
ENTERPRISES THINK DIFFERENT!
#EVALUATION
#CH-CLOUD
#ON-PREMISE
#ORGANIZATION
1
September 2017
65 7
Today
2 3 4
Evaluation Cloud Provider CH
- We are spoiled by GCP, AWS, AZ - Be aware of legacy Hosting Business Models with a Cloud paint - Focus on scalability, cost transparency and automation - Our customers are ready to pay more
On-Prem?
- Container to the rescue (LCD) - On-Prem is always highly specific - Be aware of the organisational structures
From Zero to Cloud and Back
EVERYTHING! CROSS-CLOUD!
#CROSS-SILO
#PRIVATE-CLOUD
#ON-PREMISE
#BRIDGE
#MULTI-CLOUD
#EDGE-COMPUTING
1 65 7
Today
2 3 4
Private Cloud, Multi Cloud, Everything
- We have to use different services to survive. Think AI. - Focus on Data in-motion and Event-Driven Architectures - Data Logistics is bridging the silos
Bridging through Data Logistics
Logistics Layer
Infrastructure 1 Infrastructure 2
Logistics Layer
Service AService B
CLOUD! What’s the question?
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO
ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO