CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the...
Transcript of CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the...
![Page 1: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/1.jpg)
ABenchmarkingStory
CockroachDB:FromOLTPtoHTAP
presentedby ArjunNarayan
![Page 2: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/2.jpg)
@cockroachdb
WARNING!• Opinionatedspeculationahead
• Here’swhatI wantCockroachDBtolooklike
• Notnecessarilywhatwillhappen
![Page 3: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/3.jpg)
@cockroachdb
HTAP: What is it?Here’s To A Perspective
![Page 4: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/4.jpg)
@cockroachdb
Databasestoday(simplified)
OLTP Database
OLAP Warehouse
ETL
TPC-C
TPC-H
![Page 5: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/5.jpg)
@cockroachdb
Databasestomorrow(simplified)
HTAP Database
TPC-C
TPC-H
![Page 6: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/6.jpg)
@cockroachdb
Dataarchitecturestoday(reality)
OLTP
OLTP
OLTP
OLTP
KAFKA
Hadoop
Vertica
Storm
Samza
Custom denormalizationmicroservices
Ad-hocData Science
Reporting
Realtime feeds
Dashboards
Memcache
![Page 7: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/7.jpg)
@cockroachdb
Avisionfordataarchitecturestomorrow
Cockroach Zone: OLTP
Cockroach Zone:Incrementally Updated
Materialized Views
Ad-hocData Science
Reporting
Realtime feeds
Dashboards
Cache
![Page 8: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/8.jpg)
@cockroachdb
HTAP: How do we benchmark it?You are what you measure
![Page 9: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/9.jpg)
@cockroachdb
CockroachDB Architecture in 60 secondsScale out SQL
![Page 10: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/10.jpg)
@cockroachdb
ArchitectureOverview
Raft
Fully featured SQL API
Distributed Query Execution
Node 1
Node 2
Node 3
Replicas
![Page 11: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/11.jpg)
@cockroachdb
TheCockroachDBKVLayer
![Page 12: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/12.jpg)
@cockroachdb
UsingClocksforFastReads
Node 1
Lease
Node 2
Lease
Node 3
Lease
![Page 13: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/13.jpg)
@cockroachdb
DistributedQueryExecutioninCockroachDB
![Page 14: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/14.jpg)
@cockroachdb
Building CockroachDB
![Page 15: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/15.jpg)
@cockroachdb
BuildingCockroachDB:AnOrderofPriorities
Correctness •Jepsen testing
Stability •Chaos testing
Performance •Benchmarking
![Page 16: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/16.jpg)
@cockroachdb
BuildingCockroachDB:AnOrderofPriorities
Scale-outOLTP
• Where we are today
AcceptableOLAP
• Where we are keeping an eye on
HTAP • The distant future (the year 2000)
![Page 17: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/17.jpg)
@cockroachdb
CockroachDB: An HTAP vision
![Page 18: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/18.jpg)
@cockroachdb
HTAP:Astrawman
OLTP Database
OLAP Warehouse
ETL
TPC-C
TPC-H
HTAP Database
TPC-C
TPC-H
![Page 19: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/19.jpg)
@cockroachdb
Dataarchitecturestoday
OLTP
OLTP
OLTP
OLTP
KAFKA
Hadoop
Vertica
Storm
Samza
Custom denormalizationmicroservices
Ad-hocData Science
Reporting
Realtime feeds
Dashboards
Memcache
![Page 20: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/20.jpg)
@cockroachdb
HTAP:Desiderata
• Computeanalyticsqueriesandkeepresultsup-to-date
• ResourceisolationforOLTPwork
![Page 21: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/21.jpg)
@cockroachdb
Dataarchitecturestomorrow
KAFKA
Cockroach Zone: OLTP
Hadoop
Vertica
Storm
Samza
Ad-hocData Science
Reporting
Realtime feeds
Dashboards
Memcache
![Page 22: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/22.jpg)
@cockroachdb
IncrementallyUpdatedMaterializedViews
• Real-timeChangefeedsfromOLTPreplicas
• DifferentialDataflowforincrementalcomputationofallSQLqueries
• BuiltrightintotheRDBMS,noseparateETLstep
• CockroachtimestampsindataflowgivesserializabletransactionsthatspanOLTP+OLAP
• “ReadyourOLTPwritesinOLAPreads”transactionalguarantees
• SeeNaiad(SOSP2013),DifferentialDataflow(CIDR2013),FrankMcSherry’s Blog(GitHub)
![Page 23: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/23.jpg)
@cockroachdb
Dataarchitecturesthedayafter
Cockroach Zone: OLTP
Cockroach Zone:Incrementally Updated
Materialized Views
Ad-hocData Science
Reporting
Realtime feeds
Dashboards
Memcache
![Page 24: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/24.jpg)
@cockroachdb
BenchmarkingHTAP:Whatnottodo
• Don’tjustrunTPC-C+TPC-Honthesamesystem
• Why?MaterializedviewsbreakTPC-H
• It’snotjustaboutrunningad-hocTPC-Hqueries
• HTAPdoesn’tmatterifyoucan’tguaranteepredictableTPC-Cperformance
![Page 25: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/25.jpg)
@cockroachdb
Benchmarkingmaterializedviews
• Howlongdoesittaketocomputeincrementalreports?
• Howlongdoesittaketocomputead-hocqueries?
• Howlongdoesittaketobuildamaterializedviewfromscratch?
• HowlongdoesittaketopropagateOLTPchanges?
• HowmuchoverheaddoweimposeontheOLTPsystem?
![Page 26: CockroachDB: From OLTP to HTAP · 2017-10-27 · keeping an eye on HTAP •The distant future (the year 2000) @cockroachdb CockroachDB: An HTAP vision. @cockroachdb ... •Real-time](https://reader033.fdocuments.us/reader033/viewer/2022050220/5f662a7387c5ad2ce92ba168/html5/thumbnails/26.jpg)
@cockroachdb
Questions?