Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, Anton Gorshkov
Introduction to MemSQL · 2020. 2. 5. · Scalable ANSI SQL Petabyte scale Real-time and historical...
Transcript of Introduction to MemSQL · 2020. 2. 5. · Scalable ANSI SQL Petabyte scale Real-time and historical...
Introduction to MemSQL
1
(1) Import image to presentation (2) Go to crop tool, click down arrow (3) Select circle shape (4) Adjust image size using handles and place over this text
Today’s Agenda
● Company Overview
● Common Use Cases
● Product Architecture
● Customer Examples
● Q&A
2
Mike Boyarski
Sr Director, Product Marketing
[email protected] questions in the Question Box
MemSQL at a Glance
3
Mission Deliver speed and scale for critical data infrastructure
Product Top-ranked database and data warehouse
Customers
4
Latency Holding Back the Enterprise
Lengthy Queries
Slow query responsesBillion to trillion row tables
No real-time response
Limited User Access
Single-threaded operationsData silos for mixed workloads
Single-node performance
Slow Data Loading
Batch processingHours to load
Sampled data views
5
The Enterprise Requires Performance
Fast Queries
Scalable ANSI SQLPetabyte scaleReal-time and historical insights
Scalable User Access
Scale-out for performanceConverged transactions and analyticsPowerful workload management
Live Loading
Stream and bulk dataOn-the-fly transformations
Multiple sources
6
What is MemSQL
7
MemSQL: The No-Limits Database
● Scale-out relational database
● Extreme performance at scale
● Lock-free ingest
● High concurrency
● Deploy anywhere
TM
10x faster and 3x more cost effective than legacy database providers
8
Common Use Cases
Modernize Legacy Workloads
● Enable real-time analytics for operational systems
● Accelerate performance of single- node RDBMS like Oracle RAC/Exadata, SQL Server, MySQL, Greenplum, Netezza, Vertica
9
Common Use Cases
Modernize Legacy Workloads Enable New Workloads
● Enable real-time analytics for operational systems
● Accelerate performance of single-node RDBMS like Oracle RAC/Exadata, SQL Server, MySQL, Greenplum, Netezza, Vertica
● Streaming and IoT analytics● Augment and accelerate Hadoop● Real-time machine learning● Replace and consolidate single-use
databases like MongoDB, Redis, and Elastic
10
High-Speed IngestFast bulk load or stream data
with built-in pipelines
Memory-Optimized TablesUltra-low latency for
transactions and analytics
Disk-Optimized TablesPetabyte scale analytics with
compression and performance
Unified Platform
Relational Data
11
Ecosystem Overview
Streaming Ingest
MemoryOptimized
Tables
Disk Optimized
Tables
Real-Time Data
Transforms
Data Lakes
ETL/ELT
Spark
DBMS
Hadoop Amazon S3
Bare Metal, Virtual Machines, Containers On-Premises, Cloud, As a Service
Modern Applications
TableauLooker Microstrategy
Kafka
CDC
Informatica, Talend, Alooma
Attunity
Data WarehouseBI Dashboards
12
MemSQL Architecture
13
MemSQL is a Distributed Relational Database
14
MemSQL uses Aggregator Nodes and Leaf Nodes
LeafLeafLeafLeaf
AggregatorAggregator
15
Leaf Nodes Store Data in Partitions
LeafLeafLeafLeaf
AggAggregator
MasterAggregator
LeafLeafLeafLeafPARTITIONS PARTITIONS PARTITIONS PARTITIONS
AggregatorAggregator
16
LeafLeafLeafLeaf
AggAggregator
MasterAggregator
LeafLeafLeafLeafPARTITIONS PARTITIONS PARTITIONS PARTITIONS
AggregatorMaster Agg
Every Cluster has at Least One Master Aggregator and One Leaf Node
17
LeafLeafLeafLeaf
AggAggregator
MasterAggregator
LeafLeafLeafLeafPARTITIONS PARTITIONS PARTITIONS PARTITIONS
AggregatorMaster Agg
Applications Connect to an Aggregator
Application / Database Client
18
Pipelines Ingest Across all Leaf Nodes
LeafLeafLeafLeaf
AggAggregator
MasterAggregator
LeafLeafLeafLeafPARTITIONS PARTITIONS PARTITIONS PARTITIONS
AggregatorMaster Agg
19
MemSQL Capabilities
High-Speed Ingestion
● Stream ingestion
● Fast parallel bulk loading
● Built-in Create Pipeline
● Transactional ACID Consistency
● Exactly-Once Semantics
● Native integrations with Kafka, AWS S3, Azure Blob, HDFS
20
Durable Distributed Storage
21
Highly Available
Online replication ensures data consistency and protects
against outages
Big Data Capacity
Petabyte scale with up to 10x compression and instant
query retrieval
Distributed and Durable
Store and process on clusters of machines for performance
and persistence
Ultra-Fast Insights
● Scalable ANSI SQL
● Support for JSON, Geospatial, and Full-Text Search
● Fast Query Vectorization and Compilation
● Extensibility with Stored Procedures, UDFs, UDAs
22
Built for Enterprise Demands
● Interoperable with existing tools & skills
● Deploy on-premises or cloud
● Run on bare metal, VMs, or containers
● Simplified management
● Comprehensive security
23
Enterprise-Grade Security
● Role-Based Access Control (RBAC)
● Encryption
● Authentication
● Audit logging
● Strict mode
24
Easy to Manage and Tune
● Monitor performance and capacity
● API-based deployment and config
● Full administration and scaling
● Query profiler
● Visual query plans
25
Customer Examples
26
27
How Fanatics Powered Their Wayto a Better Simplified Future
28
Company Profile
• The global leader in licensed sports merchandise
• $2 billion revenue
• Run online stores of all major North American sports leagues, more than 200 professional and collegiate teams
Place Image Here
29
Event-Driven ArchitectureFor responsiveness, reliability, and scalability
Place Image Here
30
Before MemSQL: Event-Driven Analytics Architecture
Place Image Here
31
With MemSQL: Consolidation, Simplification, and Performance
Place Image Here
With MemSQL: Consolidation, Simplification, and Performance
● Migrated from to
● Worked great in the beginning
● Unfortunately, ran into:
○ Scalability issues as the workload increased
○ Significant increase in overall cost
○ Support challenges
○ Outages on Tier 1 workloads impacted visibility of network health
33
+ Leading Wireless Provider in the US
34
Business Benefits:▪ Collect and process KPIs about wireless
network ▪ Reports on health of the wireless network▪ Significant cost savings over Oracle - 2.3x
times less
▪ No change to reporting front-end
Technical Benefits:▪ Consolidate ~100 TB, ~50K tables, ~55K reports,
~1K concurrent users from 4 Exadatas into 2 MemSQL Clusters
▪ Simpler and easier to manage compared to Oracle
▪ 80-100x faster than Exadata
▪ Run Oracle queries as-is
High Speed Operational Analytics - Call Quality Analysis
+ Leading Wireless Provider in the US
A top five financial institution uses MemSQL to power its real-time customer dashboard to show the most up-to-date (millisecond) portfolio summaries
3535
Financial Services
Real-Time Decision MakingRisk Management, Algorithmic Trading, and Fraud Detection
Using MemSQL, a top energy provider ingestsdrill bit data and reports back key indicators within minutes, allowing operators to respondto live conditions and preventing costly drill failures and outages
3636
Energy & Utilities
IOT AnalyticsPredictive Maintenance, Supply Chain Analytics, and Utility Grid Management
At Uber, MemSQL provides a live operational view into activity for millions of drivers and customers every day, delivering geospatial views for optimizing service delivery and managing day-to-day operations
3737
High Technology
Operational AnalyticsReal-time dashboards, geospatial analytics, and customer 360
Media &
38
Ad Analytics, Personalization, and Streaming Media Analytics
Comcast uses MemSQL to deliver analytics on data from millions of DVR boxes in real time,helping Comcast to provide superior customer experience with proactive service alerts and user insights for optimizing advertising programs
38
Communication
Place Image Here
MemSQLRecap
39
• Distributed, ANSI SQL, database
• Full ACID transactions
• Lock free, shared nothing
• Compiled queries
• Massively parallel
• Geospatial and JSON
• In-memory and on-disk
• MySQL wire protocol
• Comprehensive security
• Rowstore & columnstore tables
40
Thank You! Questions?
Learn more at memsql.com/product
Questions? Email us at [email protected]
Try at memsql.com/download● No time limit● Deploy to production● Full featured● Up to 128GB of RAM, unlimited disk● Get support at forum.memsql.com