TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why?...
Transcript of TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why?...
![Page 1: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/1.jpg)
TiDB: Overview, New Features, Multi-Cloud
Shen Li, Kevin Xu PingCAP
![Page 2: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/2.jpg)
!2
What is Ti(tanium) DB?
• NewSQL (RDBMS at Scale)
• MySQL Compatible (5.7)
• Hybrid OLTP + OLAP Workloads
• Multi-Cloud
![Page 3: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/3.jpg)
!3
Product: the TiDB Platform
• TiDB Platform:• TiDB (stateless SQL layer)
• TiKV (persistent key-value store w/ RocksDB)
• TiSpark (Apache Spark plug-in to TiKV)
• Open Sourced from Day 1• GA 2.0: April 2018
![Page 4: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/4.jpg)
Community
!4
• Stars:
• TiDB: 15,600+
• TiKV: 3,800+
• Contributors:
• TiDB: 210+
• TiKV: 105+
![Page 5: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/5.jpg)
Recent News
!5
![Page 6: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/6.jpg)
TiDB Overview
![Page 7: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/7.jpg)
Platform Architecture
!7Monitoring & Observability: Prometheus + Grafana
![Page 8: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/8.jpg)
TiKV: the Storage Foundation
!8
![Page 9: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/9.jpg)
TiDB: the (My)SQL Layer
!9
![Page 10: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/10.jpg)
300+ Companies in Production
!10
![Page 11: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/11.jpg)
MySQL Compatible Tools and Migration
![Page 12: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/12.jpg)
Tools
!12
• TiDB-DM (Data Migration)
• TiDB-Lightning
• TiDB-Binlog
![Page 13: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/13.jpg)
TiDB-DM
!13
• Why? To support migrating data from MySQL/MariaDB to TiDB
• Full data migration + binlog replication
• Features:
• Data migration job management
• Monitoring system
• Sharded table merge migration
• Data transformation
![Page 14: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/14.jpg)
DM Architecture
!14
![Page 15: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/15.jpg)
Sharded Table Merge
!15
![Page 16: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/16.jpg)
Data Transformation
!16
![Page 17: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/17.jpg)
TiDB-Lightning
!17
• Why? Enables fast, full import of SQL dump -> TiDB cluster• Fast:
• Can import > 100 GB / hour on production grade hardware
• Full Import:
• Import the entire database once from scratch; not incremental import
• Suitable for migrating an existing database to a new TiDB cluster
• TiDB Cluster:
• Lightning interacts with TiDB’s internal structures
• Only works with TiDB cluster
![Page 18: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/18.jpg)
TiDB-Lightning Architecture
!18
![Page 19: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/19.jpg)
TiDB-Binlog
!19
• Why? Provides real-time data synchronization and backup
• Synchronizing downstream data platform in real-time to:
• Hot backup
• MySQL slave
• OLAP data warehouses
• Writing incremental backup files
• Event subscription & trigger
![Page 20: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/20.jpg)
TiDB-Binlog Architecture
!20
![Page 21: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/21.jpg)
Migration: in and out of TiDB
!21
![Page 22: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/22.jpg)
Sneak Peek of TiDB 2.1
![Page 23: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/23.jpg)
TiDB 2.1 - SQL LayerSmarter SQL optimizer • More efficient plan • Dynamic stats update based on
query feedback Faster aggregation operator • New aggregation framework • Multi-thread hash aggregation
operator 50% faster OLTP point query
!23
Faster online DDL • Run DDL concurrently • Faster “Add Index” operation Improved debugging & diagnosis • More readable EXPLAIN result • “Explain Analyze” statement • Expose cluster information through
HTTP API
![Page 24: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/24.jpg)
TiDB 2.1 - Storage Engine (aka TiKV)
Raft learner • Improve cross-DC data availability Hotspot scheduling • Automatically distribute hotspots to more idle instances Batch region-splits • Avoid large region formation under heavy write workloads 2X read performance improvement
!24
![Page 25: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/25.jpg)
1.0 vs 2.0 on TPC-H Scale 50
!25
![Page 26: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/26.jpg)
2.0 vs 2.1 on TPC-H Scale 50
!26
![Page 27: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/27.jpg)
2.1 vs 2.0 on Sysbench Point Select
!27
![Page 28: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/28.jpg)
TiDB Cloud
![Page 29: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/29.jpg)
TiDB As A Service
!29
• Fully Managed by PingCAP
• Upgrades, Patching, Online DDL with No Downtime
• Backup and Restore
• Multi-Cloud Deployment
https://www.pingcap.com/tidb-cloud/
![Page 30: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/30.jpg)
TiDB Operator
!30
![Page 31: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/31.jpg)
GCP Marketplace
!31
![Page 32: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/32.jpg)
See it to believe it
!32
• PingCAP Booth #6
• Demo by Greg Weber TODAY at 1:20PM
![Page 33: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/33.jpg)
Oh…one more thing, TiDB Academy
!33https://www.pingcap.com/tidb-academy/
![Page 34: TiDB: Overview, New Features, Multi-Cloud - Percona · 2018-11-22 · TiDB-Lightning!17 • Why? Enables fast, full import of SQL dump -> TiDB cluster • Fast: • Can import > 100](https://reader033.fdocuments.us/reader033/viewer/2022042319/5f0827b87e708231d4209ccc/html5/thumbnails/34.jpg)
Thank you!
@PingCAP, @shenli3514, @kevinsxu