One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker
-
Upload
kendall-jefferson -
Category
Documents
-
view
29 -
download
4
description
Transcript of One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker
© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 1
One Size Fits All: An Idea Whose Time has Come and Gone
Michael Stonebraker
www.streambase.com 2
Alternate TitleAlternate Title
The elephants are selling 30 year old “bloatware”
That is not good at anything
And you should send them to the “home for old software”
www.streambase.com 3
Three Financial Services MarketsThree Financial Services Markets
Stream processing (electronic trading)
Tick stores (data warehouses)
OLTP (transaction processing)
www.streambase.com 4
Stream ProcessingStream Processing(Electronic Trading)(Electronic Trading)
A feed comes out of the wall
Compute a “secret sauce” looking for events of interest
Trade based on the result
But only if you are more nimble than the next guy….
www.streambase.com 5
Traditional RDBMS ModelOutbound Processing
Store the data before processing!
LatencyWhat if the data is not important?
Too many processes! Optimized for business data
processing Where you don’t trust the app.
Queries
Memory
Disk
Updates
Processing
Too slow to be interesting!
www.streambase.com 6
Stream Processing Engine with StreamSQL
Database paradigm (SQL) a good one
But need a different architecture
Straight through processing
No task switches
Lightweight scheduling
Inbound Processing
Memory
Disk
StreamBase Application
Event Data
Queries
Alerts Actions
Alerts Actions
Streambase Application
www.streambase.com 7
• Example: Every minute for every stock I am trading: Calculate VWAP (vol. weighted avg. price) for my trades & all trades Alert whenever my personal trading execution is inferior to market
5 Streambase operators, 30 min to build Streams of “tuples” (time-series data) flow through query
Queries run continuously
StreamSQL Application Example
”
Market_Feeds
My_Buys
Alerts
www.streambase.com 8
StreamSQL Will Dominate Rule Engines
Essentially all applications entail a mix of stored and real-time dataStreamSQL covers both kinds of data in a single paradigmA rule engine must switch paradigms
StreamSQL amenable to compilationKnow what is the next event to processIn contrast, hard to figure this out in a rule engine
www.streambase.com 9
Performance Benchmark
Financial Services Application:
Construct a virtual feed of “first arrivers” on a low end Linux machine
Relational DB: 11,000 messages/secStreambase: 300,000 messages/secAnother StreamSQL vendor: 20,000 messages/sec
Result: Streambase was a factor of 27 faster
www.streambase.com 10
Tick Stores Tick Stores (and Other Warehouse Applications)(and Other Warehouse Applications)
Store all market data for the last 10 years
To back test “secret sauce” models
To answer ad-hoc queries – “how many times has X happened”
Typical size – 100 Tbytes
Append only
www.streambase.com 11
Terminology -- “Row Store”
Record 2
Record 4
Record 1
Record 3
E.g. DB2, Oracle, Sybase, SQLServer, …
www.streambase.com 12
Rotate Your Thinking 90 Degrees Rotate Your Thinking 90 Degrees
Column stores read only the columns required
Not all of them
Compression works better
By a factor of 2-3 against the elephants
No record headers
Which are big ticket items
No padding to byte or word boundaries
www.streambase.com 13
Benchmark SummaryBenchmark Summary
Vertica has been baked off about 30 times
Typically against the incumbent
Has yet to win by less than a factor of 30 against a
row store
Beats most other column stores by around 10X
KX is the only system to come within an order of
magnitude
www.streambase.com 14
Maybe Elephants are Good Maybe Elephants are Good at OLTP……at OLTP……
OLTP is a main memory market
Not a disk-based one
Transactions are short and have no I/O or user stalls
Run to completion (single threaded)
Disaster Recovery (and HA) a requirement
Build it into the bottom of the system
www.streambase.com 15
TPC-C Performance TPC-C Performance on a Low-end Machineon a Low-end Machine
Elephant
850 TPS (1/2 the land speed record per processor)
H-Store (so far – a university prototype)
70,416 TPS (41X the land speed record per processor)
Factor of 82!!!!!
www.streambase.com 16
Implications for the ElephantsImplications for the Elephants
They are selling “one size fits all”
Which is 30 year old legacy technology that is good at nothing
www.streambase.com 17
Pictorially:
OLTPData Warehouse
Streaming data
DBMS apps
www.streambase.com 18
The DBMS Landscape – Performance Needs
OLTPData Warehouse
Streaming data
low
high
high
high
www.streambase.com 19
One Size Does Not Fit All -- Pictorially
Open source
Vertica H-Store
successors
StreambaseElephants get only “the crevices”
© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 20
Thank You
Member
Corporate Headquarters181 Spring StreetLexington, Massachusetts 02421+1 866 STRMBAS+1 866 787 6227+1 781 761 0800
New York City Office220 West 42nd Street, 20th FloorNew York, New York 10036+1 866 STRMBAS+1 866 787 6227
Reston, Virginia Office11921 Freedom Drive, Suite 550 Reston, VA 20190+1 703 608 6958
London Office107-111 Fleet StreetLondon EC4A 2ABUnited Kingdom+44 (0)20 7936 9050