Journey to the Cloud - Starburst Data · Ivan Black, Director, Big Data Platforms Journey to the...
14
Ivan Black, Director, Big Data Platforms Journey to the Cloud FINRA, Starburst, and Presto
Transcript of Journey to the Cloud - Starburst Data · Ivan Black, Director, Big Data Platforms Journey to the...
Ivan Black, Director, Big Data Platforms
Journey to the CloudFINRA, Starburst, and Presto
AGENDA
Confidential | Copyright 2019 FINRA
FINRA Mission01
Journey to the Cloud02On-Prem History03Cloud-based Future04
Starburst Partnership05Move to Presto06
1
Growth of Presto07Future Growth08
Confidential | Copyright 2019 FINRA
FINRA Mission
2
09 | FINRA @ Scale
Confidential | Copyright 2019 FINRA 3
Up to
135 Billion
Events Per DayInvestor
ProtectionMarket Integrity
Monitor
100% Equities &
45% Options
in the US
Run Hundreds of
surveillance patterns
plus a wide swath of
data science, ad-hoc &
interactive analytic
query tools
Reconstruct Trillions of
Market Nodes & Edges
Confidential | Copyright 2019 FINRA
Journey to the Cloud
4
03 | On-Prem History
Confidential | Copyright 2019 FINRA 5
Growing 20 to 30 percent YoY
Data growth
Costly to build for peak; constant EOL cycles. Spend more
on infrastructure or core mission?
Infrastructure cost
What do we have? Source? Versions? Retention?
Tracking 40M+ tables is not easy
Data governance questions
How do we manage data at scale?
How can we run analytics despite fragmentation?
Data management problems
04 | Cloud-based Future
Confidential | Copyright 2019 FINRA 6
One location, scalable, durable, performant, cost-effective, cross-region replicated
All data is in Amazon S3 - source of truth
Separation of storage from computeElastic compute, engine-agnostic, rapidly evolving ecosystem of open source software
Confidential | Copyright 2019 FINRA
Starburst Partnership
7
06 | Move to Presto - 2015
Confidential | Copyright 2019 FINRA 8
Better than classic Big Data engines, but behind dedicated
analytic appliances
PerformanceLacked LDAP integration and
internode encryption
Security
Only rudimentary resource management
Resource Management
07 | Growth of Presto - Present
Confidential | Copyright 2019 FINRA 9Confidential | Copyright 2019 FINRA 9
Performance of some queries has increased 20x
over past 4 years, resulting in improved price
performance and new use case development
PerformanceInternode encryption, extensible
authentication plugin architecture
Security
Sophisticated, group-based
resource management
Resource Management
Confidential | Copyright 2019 FINRA
Future Growth
10
New Use Cases
Confidential | Copyright 2019 FINRA 11
Improved performance and security allows for additional Presto use cases
Replacement of classic big-data engine to create DataMarts for thousands of end-users
Replacement of common key/value query system for trillionsof events
Consolidated Audit Trail
Confidential | Copyright 2019 FINRA 12
In excess of135 BillionEvents Per Day Investor
ProtectionMarket Integrity
MonitorAll Equities & Options Exchanges and Broker Dealers in the US
Run Hundreds of surveillance patterns, plus a wide swath of data science, ad-hoc & interactive analytic query tools
Reconstruct Trillions of Market Nodes & Edges
Confidential | Copyright 2019 FINRA
Q A
13
