Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"
-
Upload
dataconomy-media -
Category
Technology
-
view
204 -
download
3
Transcript of Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"
Hadoop & Germany & 2016
uweseiler
/whoami &
/disclaimer
Hadoop & Germany & 2016
We finally stopped talking infrastructure!
Hadoop & Germany & 2016
We now talk architectures and use cases!
Hadoop & Germany & 2016
#1 The Big Data Lake is an illusion!
Hadoop & Germany & 2016
Da
ta S
ourc
esD
ata
Sys
tem
sA
pp
lica
tion
s
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
RDBMS EDW MPP …
Business Intelligence
BusinessApplications
Custom Applications
Operation
Manage &
Monitor
Dev Tools
Build &
Test
New Sources
Logs Mails Sensor …SocialMedia
EnterpriseHadoop Plattform
#1 The Vision of the Big Data Lake
Hadoop is not the one tool to rule them all
#1 Vision & Reality
Embrace heterogeneity! (and learn to deal with the complexity)
#1 After the reality shock…
#1 Real world architecture - Insurance
Da
ta S
ourc
esD
ata
Sys
tem
sA
pp
lica
tion
s
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
DWH
BusinessIntelligence
New Sources
Logs Sensor …SocialMedia
Enterprise Hadoop Plattform
SAS LASR Server
Apache Zeppelin
#2 Speed is the new king!
Hadoop & Germany & 2016
#2 The “classic“ Lambda Architecture
Batch Layer
Speed LayerData Ingestion
Data Processing
Data Storage
Data Storage Data Analysis
Visualization
Visualization
…
DataChannels
ms - s
min - h
#2 Lambda in Action - (e)Commerce
SMACK Spark Mesos Akka
Cassandra Kafka
#2 The lust for speed
Data Ingestion
Data Processing
Raw Data
#2 Cassandra & Hadoop - AdServing
Data Processing
User Journey
Aggregated Data
Web Frontend
Aggregated Data< 120 days
Data Science
#3 Data Science to the help!
Hadoop & Germany & 2016
Hadoop is about to become commodity
#3 Let’s face it..
Algorithms will be the new differentiator
#3 We need new challenges…
Batch Layer
Speed LayerData Ingestion
Stream Processing
ms - s
min - h
#3 Fraud detection - Financial services
DataImport
Data Preparation
Model Generation
Model Validation
Feature & Parameter Selection
Manual or automatic Iterations to tune
parameters
Use Model
Refresh Model from latest input data
Every major company is building teams of unicorns
#3 The solution?
#4 Hadoop for good!
Hadoop & Germany & 2016
Hadoop User Group Rhein-Mainhttp://www.meetup.com/de-DE/HUG-Rhein-Main/
Next Meetup: 23.06.2016, Talks welcome