Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED...

31
Darwin Schweitzer Big Data Analytics

Transcript of Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED...

Page 1: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Darwin Schweitzer

Big Data Analytics

Page 2: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

C LO U D

D ATA A I

Organizations that harness Data, Cloud, and AI outperform

Page 3: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Security and performanceFlexibility of choiceReason over any data, anywhere

Data warehouses

Data Lakes

Operational databases

Hybrid

Data warehouses

Data Lakes

Operational databases

SocialLOB Graph IoTImageCRM

T H E M O D E R N D A T A E S T A T E

Page 4: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Classic AnalyticsTransaction-driven

Cloud-born AnalyticsEvent-driven

COSMOS DB Databricks

SQL DB/DW and Analysis Services

Page 5: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Big Data & Advanced Analytics in Azure

Page 6: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Model & ServePrep & Train

Databricks

HDInsight

Data Lake AnalyticsCustom

apps

Sensors

and devices

Store

Blobs

Data Lake

Ingest

Data Factory(Data movement, pipelines & orchestration)

Machine

Learning

Cosmos DB

SQL Data

Warehouse

Analysis Services

Event Hub

IoT Hub

SQL Database

Analytical dashboards

Predictive apps

Operational reports

Intelligence

B I G D ATA & A D VA N C E D A N A LY T I C S AT A G L A N C E

Business

apps

1001

SQLKafka

Page 7: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Azure DatabricksPowered by Apache Spark

Page 8: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 9: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure

Best of Databricks Best of Microsoft

Designed in collaboration with the founders of Apache Spark

One-click set up; streamlined workflows

Interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

Native integration with Azure services (Power BI, SQL DW, Cosmos DB, Blob Storage)

Enterprise-grade Azure security (Active Directory integration, compliance, enterprise -grade SLAs)

Page 10: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Get started quickly by launching

your new Spark environment with

one click.

Share your insights in powerful

ways through rich integration with

Power BI.

Improve collaboration amongst

your analytics team through a

unified workspace.

Innovate faster with native

integration with rest of Azure

platform

Simplify security and identity control

with built-in integration with Active

Directory.

Regulate access with fine-grained user

permissions to Azure Databricks’

notebooks, clusters, jobs and data.

Build with confidence on the trusted

cloud backed by unmatched support,

compliance and SLAs.

Operate at massive scale

without limits globally.

Accelerate data processing with

the fastest Spark engine.

ENHANCE PRODUCTIVITY BUILD ON THE MOST COMPLIANT CLOUD SCALE WITHOUT LIMITS

Page 11: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Optimized Databricks Runtime Engine

DATABRICKS I/O SERVERLESS

Collaborative Workspace

Cloud storage

Data warehouses

Hadoop storage

IoT / streaming data

Rest APIs

Machine learning models

BI tools

Data exports

Data warehouses

Azure Databricks

Enhance Productivity

Deploy Production Jobs & Workflows

APACHE SPARK

MULTI-STAGE PIPELINES

DATA ENGINEER

JOB SCHEDULER NOTIFICATION & LOGS

DATA SCIENTIST BUSINESS ANALYST

Build on secure & trusted cloud Scale without limits

Page 12: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 13: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Demo Azure Databricks

Page 14: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

https://github.com/Azure/data-ai-iot/tree/master/databricks

Page 15: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

https://gallery.azure.ai/Solution/Azure-Databricks-Spark-Streaming-4

Page 17: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Use Cases

Page 18: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Business / custom apps

(Structured)

Logs, files and media

(unstructured)

Azure storage

Polybase

Azure SQL Data Warehouse

Data factory

Data factory

Azure Databricks

(Spark)

Analytical dashboards

Model & ServePrep & TrainStoreIngest Intelligence

Page 19: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Web & mobile appsAzure Databricks

(Spark Mllib,

SparkR, SparklyR)

Azure Cosmos DB

Business / custom apps

(Structured)

Logs, files and media

(unstructured)

Azure storage

Polybase

Azure SQL Data Warehouse

Data factory

Data factory

Analytical dashboards

Model & ServePrep & TrainStoreIngest Intelligence

Page 20: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Unstructured data

Azure storage

Polybase

Azure SQL Data Warehouse

Azure HDInsight

(Kafka)

Azure Databricks

(Spark)

Analytical dashboards

Model & ServePrep & TrainStoreIngest Intelligence

Page 21: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Pricing @General Availability

Release Standard Premium

General availability Data analytics: $0.40/DBU/hr + VM

Data engineering: $0.20/DBU/hr + VM

Includes:

Compliance: SOC2, HIPAA, AAD Integration

Data connectors (Blob Storage, Data Lake, SQL DW,

Cosmos DB, Event Hub), GPU Instances

Data analytics: $0.55/DBU/hr + VM

Data engineering: $0.35/DBU/hr + VM

Includes:

Everything from standard +

Fine grained control for notebooks & clusters, structured

data controls

JDBC/ODBC endpoint

Governance logs

.NET Integration

Integrations with Azure apps like Power BI, etc.

Page 22: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Engage Microsoft experts for a workshop to help identify

high impact scenarios

Try a Quickstart or Tutorial at:

https://docs.microsoft.com/en-us/azure/azure-databricks/

https://gallery.azure.ai/Solution/Azure-Databricks-Spark-Streaming-4

https://github.com/Azure/data-ai-iot

Learn more about Azure Databricks www.azure.com/databricks

Page 23: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

https://github.com/Azure/data-ai-iot

Accelerate learning and using Try → Learn → Build:

Try• Demos (IDEA)

• Introduce

• Demo

• Explain

• Attend

Learn• GitHub Samples

• Solution Templates

• Data Science VM

Build

• Documentation &

Solution Architectures

Transform individuals to transform business

Transformation of individuals → teams → organizations

Page 24: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Darwin Schweitzer | WW INTELLIGENT CLOUD – Big Data / AI Advanced Workload LeadWorldwide Commercial Business (WCB) – Intelligent Cloud(425) 638-9068 | [email protected] | @DataSnowman | GitHub DataSnowmanPlease check out Data and AI and IoT resources at https://github.com/Azure/data-ai-iot

Page 25: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Thank you

Page 26: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache

Appendix

Page 27: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 28: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 29: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 30: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache
Page 31: Big Data Analytics€¦ · Predictive apps Operational reports Intelligence BIG DATA & ADVANCED ANALYTICS AT A GL ANCE Business apps 10 01 Kafka SQL. Azure Databricks Powered by Apache