Analytics in the cloud

Analytics in the CloudNatalino Busa - Head of Data Science

2 Natalino Busa - @natbusa

Distributed computing Machine Learning

Statistics Big/Fast Data Streaming Computing

Head of Applied Data Science at Teradata

On most networks:

@natbusa

https://twitter.com/natbusa


Let’s define Cloud Services



Analytics in the cloud: stacking layers

Bare Metal: Physical Machines





IAAS: Virtual Resources






CAAS: Containers,






CAAS: Containers,

dPAAS: Datastores, Data Engines iPAAS: Tools Integration, Flows & Processes





CAAS: Containers,


DAAAS: Data Analytics as a Service

Watson ServicesAzure ML

GoogleCloud MLBigML




Analytics in the cloud: today’s talk



CAAS: Containers,


DAAAS: Data Analytics as a Service



“we live in an age of open source datacenters, so we can stack all these things together and we have open source from the ground to ceiling.”

Sam Ramji, CEO of Cloud Foundry

https://www.youtube.com/watch?v=7oCSFcUW-Qk





Containers vs VMs



Techs based on Containers

YARN



Containers as a Service

https://aws.amazon.com/ecs/

For example: Amazon ECS





CaaS: 6 offerings

https://www.linux.com/news/5-container-service-tools-you-should-know-about

Project Magnum

Amazon ECSDocker DataCenterGoogle

Container Engine





Most new PaaS solutions are containerized



PaaS: Big Data SQL Queries

Batch OrientedLarge Aggregations

Interactive QueriesData Exploration

Interactive QueriesMachine Learning

Streaming:Micro-batching

Interactive QueriesMachine Learning

Streaming:Event-driven



Advanced Analytics: models and algorithms



PaaS: Advanced Analytics

Graph analytics:

- Cluster items- Extract similarities- Detect patterns




Text analytics:

- Sentiment Analysis- Language Detection- Summarization- Entity extraction




Machine Learning:

- Classification- Regression- Clustering- Forecasting- Anomaly detection




AI and Deep Learning- Unstructured Data- Object Detection- Natural Language Processing- Video Summarization- Speech Recognition




SQL + Graph + Text + Machine Learning + Voice/Image/Video



dPaaS: Machine (deep) Learning

… this are just a few examples ...



Analytics Everywhere

Public Cloud Managed Cloud Private Cloud Private Infra



iPaas: Components for Analytics in the Cloud

SQL : Big Data Data Warehousing

NoSQL

Machine LearningObjects Stores

Streaming Computing

SQL: RelationalTransactional DB



iPaas, dPaaS:

Objects Stores

HDFSGlusterFSCephFSNFSSwiftNovaCassandraRedis

S3 (AWS)Storage (GCP)...



iPaas, dPaaS:

NoSQLObjects Stores



CassandraRedisHBaseAccumulo

Neo4JElasticSearchMongoDBCouchbase

BigTable (GCP)DynamoDB



iPaas, dPaaS:

NoSQLObjects Stores




MySQLPostgreSQLMariaDB

Oracle (AWS MP)






iPaas, dPaaS:


NoSQLObjects Stores





Oracle (AWS MP)

HivePrestoSpark SQLImpala

Redshift (AWS)BigQuery (GCP)Big SQL (IBM)

Teradata (AWS MP)SAP Hana(AWS MP)Vertica (AWS MP)






iPaas, dPaaS:


NoSQL Machine Learning

Objects Stores





Oracle (AWS MP)







Spark MLH2OFlinkAreosolveTheanoTensorflowXGboost

Azure MLAWS MLGoogle MLIBM Watson



iPaas, dPaaS:


NoSQL Machine Learning

Objects Stores

Streaming Computing





Oracle (AWS MP)







Spark MLH2OFlinkAreosolveTheanoTensorflowXGboost

Azure MLAWS MLGoogle MLIBM Watson

Heron (Storm)NiFiSpark StreamingFlinkKafka StreamsLogstashStreamSQL

Google DataFlow (GCP)



iPaaS: Selecting your Analytical Stack

� Flexible. Powerful.- Combinations for this example:

8 * 3 * 4 * 8 * 7 * 7 = 37632

� Right tool for the right job- Fit for purpose- Multi-Genre Analytics

Hard to maintain and upgrade:- Extended Skills and Know-how- Components upgrades must be compatible

Hard to configure: - no matter if cloud or bare or vms- complex stacks with many tools and services



iPaaS: Deploy & Manage your own Analytics

How to simplify? Select a bundle!



iPaaS: bundled recipes & stacks

Select a recipe:- Hortonworks Data Platform- Cloudera Data Platform- Reactive Platform - Smack Stack- Pancake Stack- ELK Stack- Select your own



iPaaS: my favs analytical stacks

Objects Stores

NoSQL SQL : Big Data Data Warehousing

Machine Learning Streaming Computing

All Hadoop (5) HDFS Hbase Hive Spark Storm

Smack stack (2) Cassandra Cassandra Spark Spark Spark

Elastic (5) HDFS ElasticSearch Hive H2O Kafka

Data Science (8) HDFS ElasticSearch Hive, Presto Spark, H2O, Tensorflow Flink

Real Time (2) Cassandra Cassandra Flink Flink Flink



dPaaS: Managed Analytics

This is hard ! Can we access it as a service?



dPaaS: Managed Hadoop & Spark

HDInsight: Hadoop, Spark, and R as services

Managed Spark Clusters, BigInsight (Hadoop)

DataFlow and DataProc: Flink, Spark and Hadoop Clusters as a Service

EMR: Hadoop components a la carte



PaaS: Analytical clusters

Ephemeral

Create then Dispose

Clusters are Short-Lived

Data Exploration

Isolated, Personal

Simple Access Management

Interactive Analytics

Permanent

Clusters are Long Lived

Scheduled Operations

Production ETL

Co-Ordinated

Complex Access Management

Batch Analytics

vs



DAaaS: Microsoft’s Cortana and ML Studio



DAaaS: IBM Watson



DAaaS: Google ML and AI as a service

Cloud Computing forDeep Neural Networks > Train, Score, Data

AI and ML models for:

● Speech (audio)● Language (text)● Vision (images/video)



Summary

• Analytics in the Cloud:

The dawn of a new computing era

• IPaas, dPaas:

complexity vs flexibility, it’s a tradeoff

• Computing clusters:

Ephemeral and Persistent



Head of Applied Data Science at Teradata

Distributed computing Machine Learning

Statistics Big/Fast Data Streaming Computing

Linkedin and Twitter:

natbusa


Analytics in the cloud

Data & Analytics

Transcript of Analytics in the cloud