ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the...
Transcript of ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the...
![Page 1: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/1.jpg)
Self-Driving Database Management Systems
@Andy_Pavlo // 15-721 // Spring 2019
ADVANCEDDATABASE SYSTEMS
Le
ctu
re #
25
![Page 2: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/2.jpg)
CMU 15-721 (Spring 2019)
Autonomous DBMS History
Self-Driving DBMSs
Learned Components
2
![Page 3: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/3.jpg)
CMU 15-721 (Spring 2019)
MOTIVATION
Personnel is ~50% of the TOC of a DBMS.
Average DBA Salary (2017): $89,050
The scale and complexity of DBMS installations have surpassed humans.
3
Source: https://www.highbeam.com/doc/1P3-1149052351.html
Source: https://www.bls.gov/oes/current/oes151141.htm
![Page 4: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/4.jpg)
CMU 15-721 (Spring 2019)
SELF-ADAPTIVE DATABASES (1970s -1990s)
Index Selection
Partitioning / Sharding Keys
Data Placement
4
![Page 5: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/5.jpg)
CMU 15-721 (Spring 2019)
SELF-ADAPTIVE DATABASES (1970s -1990s)
4
SELECT * FROM A JOIN BON A.ID = B.ID
WHERE A.VAL > 123AND B.NAME LIKE 'XY%'
A.IDA.VALB.ID
B.NAME
+100 +200 +50
TuningAlgorithm
Admin
![Page 6: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/6.jpg)
CMU 15-721 (Spring 2019)
SELF-ADAPTIVE DATABASES (1970s -1990s)
4
SELECT * FROM A JOIN BON A.ID = B.ID
WHERE A.VAL > 123AND B.NAME LIKE 'XY%'
A.IDA.VALB.ID
B.NAME
+100 +200 +50
TuningAlgorithm
Admin
![Page 7: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/7.jpg)
CMU 15-721 (Spring 2019)
SELF-ADAPTIVE DATABASES (1970s -1990s)
4
SELECT * FROM A JOIN BON A.ID = B.ID
WHERE A.VAL > 123AND B.NAME LIKE 'XY%'
A.IDA.VALB.ID
B.NAME
+100 +200 +50
TuningAlgorithm
Admin
![Page 8: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/8.jpg)
CMU 15-721 (Spring 2019)
SELF-TUNING DATABASES (1990s -2000s)
5
SELECT * FROM A JOIN BON A.ID = B.ID
WHERE A.VAL > 123AND B.NAME LIKE 'XY%'
A.IDA.VALB.ID
B.NAMETuning
Algorithm
Admin
AutoAdmin
OptimizerCost Model
![Page 9: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/9.jpg)
CMU 15-721 (Spring 2019)
SELF-TUNING DATABASES (1990s -2000s)
5
SELECT * FROM A JOIN BON A.ID = B.ID
WHERE A.VAL > 123AND B.NAME LIKE 'XY%'
A.IDA.VALB.ID
B.NAMETuning
Algorithm
Admin
AutoAdmin
OptimizerCost Model
![Page 10: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/10.jpg)
CMU 15-721 (Spring 2019)
SELF-TUNING DATABASES (1990s -2000s)
6
75
541
53
291
0
200
400
600
2000 2004 2008 2012 2016
Nu
mbe
r of
Kn
obs
Source: Dana Van Aken
Number of Configuration Knobs Per Release
![Page 11: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/11.jpg)
CMU 15-721 (Spring 2019)
CLOUD-MANAGED DATABASES (2010S)
Initial Placement
Tenant Migration
7
![Page 12: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/12.jpg)
CMU 15-721 (Spring 2019)
CLOUD-MANAGED DATABASES (2010S)
7
![Page 13: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/13.jpg)
CMU 15-721 (Spring 2019)
CLOUD-MANAGED DATABASES (2010S)
7
![Page 14: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/14.jpg)
CMU 15-721 (Spring 2019)
OBSERVATION
People have been working on autonomous database systems for 45 years.
Why is this previous work insufficient?
8
![Page 15: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/15.jpg)
CMU 15-721 (Spring 2019)
PREVIOUS WORK
Problem #1: Human Judgements→ User has to make final decision on whether to apply
recommendations.
Problem #2: Reactionary Measures→ Can only solve previous problems. Cannot anticipate
upcoming usage trends / issues.
Problem #3: No Transfer Learning→ Tunes each DBMS instance in isolation. Cannot apply
knowledge learned about one DBMS to another.
9
![Page 16: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/16.jpg)
CMU 15-721 (Spring 2019)
OBSERVATION
Just like there are different levels of autonomy in cars, there are different levels for databases.→ SAE (J3016) Automation Levels
We need to reason about the autonomous systems to understand their capabilities and limitations.→ This will help us reason about how much a human needs
to be involved in its administration.
10
![Page 17: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/17.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
11
System only does what humans tell it to do.
![Page 18: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/18.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
Level #1: Assistant
11
Recommendation tools that suggest improvements.
Human makes final decisions.
![Page 19: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/19.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
Level #1: Assistant
Level #2: Mixed Management
11
DBMS and humans work together to mange the system.
Human guides the process.
![Page 20: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/20.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
Level #1: Assistant
Level #2: Mixed Management
Level #3: Local Optimizations
11
Subsystems can adapt without human guidance.
No higher-level coordination.
![Page 21: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/21.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
Level #1: Assistant
Level #2: Mixed Management
Level #3: Local Optimizations
Level #4: Direct Optimizations
11
Human only provides high-level direction + hints.
System can identify when it needs to ask humans for help.
![Page 22: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/22.jpg)
CMU 15-721 (Spring 2019)
AUTONOMOUS DBMS TAXONOMY
Level #0: Manual
Level #1: Assistant
Level #2: Mixed Management
Level #3: Local Optimizations
Level #4: Direct Optimizations
Level #5: Self-Driving
11
![Page 23: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/23.jpg)
CMU 15-721 (Spring 2019)
SELF-DRIVING DATABASE
A DBMS that can deploy, configure, and tune itself automatically without any human intervention.→ Select actions to improve some objective function (e.g.,
throughput, latency, cost).→ Choose when to apply an action.→ Learn from these actions and refine future decision
making processes.
12
SELF-DRIVING DATABASE MANAGEMENT SYSTEMSCIDR 2017
![Page 24: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/24.jpg)
CMU 15-721 (Spring 2019)
ARCHITECTURE OVERVIEW
13
Workload ForecastsSQL Statements
Internal Metrics Component Models
Deploy &Observe
Search & Planning
Modeling
Where to Deploy?How to Deploy?
When to Deploy?Why?
Actions
![Page 25: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/25.jpg)
CMU 15-721 (Spring 2019)
SELF-DRIVING ENGINEERING
Environment Observations→ How the DBMS collects training data.
Action Meta-Data→ How the DBMS implements and exposes methods for
controlling and modifying the system's configuration.
Action Engineering→ How the DBMS deploys actions either for training or
optimization.
14
MAKE YOUR DATABASE DREAM OF ELECTRIC SHEEP: ENGINEERING FOR SELF-DRIVING OPERATIONUNPUBLISHED MANUSCRIPT 2019
![Page 26: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/26.jpg)
CMU 15-721 (Spring 2019)
ENVIRONMENT OBSERVATIONS
Logical Workload History→ SQL queries with their execution context.→ Need to compress to reduce storage size.
Runtime Metrics→ Internal measurements about the DBMS's runtime
behavior.
Database Contents→ Succinct representation/encoding of the database tables.
15
![Page 27: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/27.jpg)
CMU 15-721 (Spring 2019)
SUB-COMPONENT METRICS
If the DBMS has sub-components that are tunable, then it must expose separate metrics for those components.
Bad Example:
16
![Page 28: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/28.jpg)
CMU 15-721 (Spring 2019)
SUB-COMPONENT METRICS
17
RocksDB Column Family Knobs
![Page 29: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/29.jpg)
CMU 15-721 (Spring 2019)
SUB-COMPONENT METRICS
17
RocksDB Column Family Knobs
Column Family Metrics
Missing:Reads
Writes
![Page 30: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/30.jpg)
CMU 15-721 (Spring 2019)
SUB-COMPONENT METRICS
17
RocksDB Column Family Knobs
Global Metrics
AggregatedMetrics
![Page 31: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/31.jpg)
CMU 15-721 (Spring 2019)
ACTION META-DATA
Configuration Knobs→ Untunable flags→ Value ranges
Dependencies→ No hidden dependencies→ Dynamic actions (i.e., an action creates new actions).
18
![Page 32: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/32.jpg)
CMU 15-721 (Spring 2019)
UNTUNABLE KNOBS
Anything that requires a human value judgement should be marked as off-limits to autonomous components.→ File Paths→ Network Addresses→ Durability / Isolation Levels
19
![Page 33: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/33.jpg)
CMU 15-721 (Spring 2019)
KNOB HINTS
The autonomous components need hints about how to change a knob.→ Min/max ranges.→ Separate knobs to enable/disable a feature.→ Non-uniform deltas.
20
![Page 34: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/34.jpg)
CMU 15-721 (Spring 2019)
KNOB HINTS
The autonomous components need hints about how to change a knob.→ Min/max ranges.→ Separate knobs to enable/disable a feature.→ Non-uniform deltas.
20
1 KB 1 MB 1 GB 1 TB+10 KB +10 MB +10 GB
![Page 35: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/35.jpg)
CMU 15-721 (Spring 2019)
ACTION ENGINEERING
No Downtime
Notifications
Replicated Training
21
![Page 36: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/36.jpg)
CMU 15-721 (Spring 2019)
NO DOWNTIME
The DBMS must be able to deploy any action without incurring downtime.→ Restart vs. Unavailability
Without this, the system has to include the downtime in its cost model estimations.→ Bad Example: MySQL Log File Size
22
![Page 37: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/37.jpg)
CMU 15-721 (Spring 2019)
NOTIFICATIONS
Provide a notification to indicate when an action starts and when it completes.→ Need to know whether degradation is due to deployment
or bad decision.
Harder for changes that can be used before the action completes.
23
![Page 38: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/38.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
ML models need lots of training data. But getting this data is expensive in a DBMS.→ We don't want to slow down a production DBMS.→ Building a simulator for the DBMS is too hard.
Ongoing Research: How to use the DBMS's replicas to explore configurations and train its models.
24
![Page 39: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/39.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
Master Replica
Replica
Actions
Actions
Self-Driving Manager
![Page 40: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/40.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
Master Replica
Replica
Actions
Actions
Actions
Actions
Self-Driving Manager
![Page 41: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/41.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
Master Replica
Replica
Actions
Actions
??? Revert Actions
Wait Time
Self-Driving Manager
![Page 42: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/42.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
MasterReadsWrites
Writes
Writes
Replica
Replica
App Server
Self-Driving Manager
![Page 43: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/43.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
MasterReadsWrites
Physical Log
SQL Statements
Replica
ReplicaPhysical Log
SQL Statements
App Server
???
???
Self-Driving Manager
![Page 44: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/44.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
MasterReadsWrites
Physical Log
SQL Statements
Replica
ReplicaPhysical Log
SQL Statements
App Server
???
???
Self-Driving Manager
Component Models
Component Models
![Page 45: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/45.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
MasterReadsWrites
Physical Log
SQL StatementsReads
Reads
Replica
ReplicaPhysical Log
SQL Statements
App Server
Self-Driving Manager
Component Models
Component Models
![Page 46: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/46.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
Master Replica
Replica
App Server
Actions
Self-Driving Manager
Component Models
Component Models
![Page 47: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/47.jpg)
CMU 15-721 (Spring 2019)
REPLICATED TRAINING
25
Master Replica
Replica
App Server
???
???
Actions
Self-Driving Manager
Component Models
Component Models
![Page 48: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/48.jpg)
CMU 15-721 (Spring 2019)
ORACLE SELF-DRIVING DBMS
26
September 2017 January 2017
![Page 49: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/49.jpg)
CMU 15-721 (Spring 2019)
ORACLE SELF-DRIVING DBMS
Automatic Patching
Automatic Indexing
Automatic Recovery
Automatic Scaling
Automatic Query Tuning
26
September 2017
![Page 50: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/50.jpg)
CMU 15-721 (Spring 2019)
ORACLE SELF-DRIVING DBMS
Automatic Patching
Automatic Indexing
Automatic Recovery
Automatic Scaling
Automatic Query Tuning
26
September 2017
ReactionaryMeasures
No TransferLearning
![Page 51: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/51.jpg)
CMU 15-721 (Spring 2019)
ORACLE SELF-DRIVING DBMS
Automatic Patching
Automatic Indexing
Automatic Recovery
Automatic Scaling
Automatic Query Tuning
26
September 2017
ReactionaryMeasures
No TransferLearning
![Page 52: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/52.jpg)
CMU 15-721 (Spring 2019)
OBSERVATION
There are many places in the DBMS that use human-engineered components to make decisions about the behavior of the system.→ Optimizer Cost Models→ Compression Algorithms→ Data Structures→ Scheduling Policies
What if the DBMS could "learn" these policies based on the data.
27
![Page 53: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/53.jpg)
CMU 15-721 (Spring 2019)
LEARNED COMPONENTS
Replace DBMS components with ML models trained at runtime.
28
![Page 54: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/54.jpg)
CMU 15-721 (Spring 2019)
PARTING THOUGHTS
True autonomous DBMSs are achievable in the next decade.
You should think about how each new feature can be controlled by a machine.
29
![Page 55: ADVANCED 5 DATABASE SYSTEMS - CMU 15-721CMU 15-721 (Spring 2019) MOTIVATION Personnel is ~50% of the TOC of a DBMS. Average DBA Salary (2017): $89,050 The scale and complexity of DBMS](https://reader035.fdocuments.us/reader035/viewer/2022070817/5f135640c035167d7c2302eb/html5/thumbnails/55.jpg)
CMU 15-721 (Spring 2019)
NEXT CL ASS
SAP HANA Guest Lecture
30