Download - MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Transcript
Page 1: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

MariaDB for Advanced DBAsMariaDB Training

Page 2: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Introduction

Page 3: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Introducing MariaDB AbFounders from MySQL — the

Company and the Community Funded by Founders, Employees,

and Venture Capital Several former MySQL Employees

and Community Members Located in over 14 Countries

3

Page 4: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Personal IntroductionsInstructor

Name and Background

Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics

4

Page 5: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site

Location of Rest Rooms Smoking Areas Snacks and Drinks

5

Page 6: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Course OutlineHigh Availability

MariaDB Replication & GTID

Complex Scenarios & Semi-Sync

MariaDB Replication Manager

MariaDB Enterprise Cluster

Client Proxies

Optimizing Optimizing Overview

Schema Tuning

InnoDB In-Depth

Collecting Information

Benchmarking

Query Tuning The Query Process

Query Tuning

Advanced Index Usage

Diagnosing Systems Common Bottlenecks

Troubleshooting

Hacking with GNU Debugger

6

Page 7: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

High Availability Overview

Page 8: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

MariaDB Replication

Page 9: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Purpose of MariaDB Replication

9

Load Balancing — High-Traffic Reads on Slaves High Availability

Fail Over — Promote a Slave to Master Back-Ups — Take a Slave Off-Line

Minimize Downtime for Upgrades or Schema Changes Apply Changes to a Slave Promote Slave to Master

Page 10: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Replication Terrain

10

mysqld Data Storage

Slave 1

Relay Log

Data Storage

Slave 2/Master 2

Relay Log

Binary Log

mysqldSlave 2A

Slave 2B

Slave 2C

INSERT UPDATE DELETE

CREATE ALTER DROP

mysqld Data Storage

Master

Binary Log Dump Thread

Client Threads

SQL Thread

IO Thread

IO Thread

IO Thread

Page 11: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB Replication Factors

11

One Master, Multiple Slaves No True Multi-Master Solution, but Circular

Replication

Close to Real Time (Asynchronous) Semi-Synchronous Replication Mode Crash-Safe Slaves with InnoDB

Slaves may also be Masters (log_slave_updates) Replication Filtering Rules Storage Engine Changes Optional on Intermediate

Slaves

Documentation on log_slave_updates: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/#log_slave_updates

Page 12: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Replication Threads

12

Master binlog Dump Thread Pushes Binary Log Events to Slave

Slave IO Thread

Gets binlog Events from Master Writes to Local Relay Log

Slave SQL Thread

Reads Relay Log & Executes Checks Result Codes Match Master

Slave Multiple Execution Threads

Separates Events Based on Database Updates Applied in Parallel

SHOW SLAVE STATUS \G

... Slave_IO_Running: Yes Slave_SQL_Running: Yes ...

SHOW PROCESSLIST \G ***** 1. row ***** Id: 615 User: mariadb_replicator

Host: 12.34.56.01:42422 db: NULL Command: Binlog Dump Time: 798022

State: Master has sent all binlog to slave; waiting for binlog to be updated

Executed on Master

Executed on Slave

Page 13: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Parallel Replication

13

Documentation on Parallel Replication: https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

Replication Process on Slaves Events from Master through IO Thread and Queued in

Relay Log Each Relay Log Entry is Retrieved by SQL Thread Each Transaction is Applied to Slave

Non-Parallel Systems Apply Transactions Sequentially through SQL Thread

Parallel Systems Apply Transactions in Pool of Separate Replication Worker Threads

Smaller Transactions are Required

Page 14: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Troubleshooting Problems with Replication

14

Check Slave Error Log Look for Disconnects from Network Binary or Relay Log Event Corruption will cause Slave

SQL Thread to Stop Different Query Error Codes on Slave indicates Not

Synchronized

Percona Toolkit can Help with Troubleshooting May Need to Rebuild Slave with New Snapshot (Back-up

of Master or Another Slave)

Page 15: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Slave Filtering Rules — Database Level

15

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Exclude Specific Databases (e.g., mysql)

Include Specific Databases

Excluding can Cause Problems with Joins

SET GLOBAL replicate_ignore_db = ''; SET GLOBAL replicate_do_db = 'sales,inventory';

SET GLOBAL replicate_ignore_db = 'mysql';

Page 16: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Slave Filtering Rules — Table Level

16

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Ignore Specific Tables

SET GLOBAL replicate_ignore_db = (employees), replicate_do_table = (employees.names, employees.contacts);

SET GLOBAL replicate_ignore_table = (employees.salary);

SET GLOBAL

replicate_ignore_db = (sales), replicate_do_table = (sales.europe_%), replicate_ignore_table = (sales.europe_uk_%);

Include Specific Tables

Wildcards for Multiple Tables

Page 17: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Global Transaction Identification

Page 18: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

GTID (Global Transaction Identification)

Unique Identification for Binary Log Transactions

Used to Eliminate Confusion amongst Replication Servers

18

Documentation on GTID: https://mariadb.com/kb/en/mariadb/gtid/

Page 19: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Confusion Without GTIDReplication Process

Changes on Master are Recorded in Binary Log as Events Slaves get Events from Master and Execute Locally Slaves Use Master's Binary Log File Name and Position

Numbers for Clarity

Potential Source of Confusion Slaves may be Masters to Other Slaves (a.k.a., Intermediate

Masters) Intermediate Masters have Different Log Positions Promoting an Intermediate Master to Master is

Cumbersome

19

Page 20: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Clarity from GTIDUnique Identifier for Each binlog Event Group

Transactions are Collections of Events to Apply Together Can Include Non-Transactional Statements

Promoting a Slave to Master is Easy State of Slave is Recorded in Crash-Safe Method

20

Page 21: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

GTID FormatThree Numbers Separated by Dashes (e.g., 0-1-8000)

Domain ID (32-bit Unsigned Integer) Server ID (32-bit Unsigned Integer) Sequence Number (64-bit Unsigned Integer)

Domain ID Indicates Individual Write Stream Server ID is Set with server_id Variable — Must be Unique

(1 to 2 32) Sequence is Incremented with Each Transaction

Stored in gtid_seq_no

21

Page 22: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Effect of the GTID DomainBinlog Consists of Different Streams —

Not a Single Ordered Stream Each Identified by its Domain ID GTIDs always in Same Order

Different Streams can be Interleaved Slaves use Last GTID Applied for

Domain Slaves Start from a Different Point in

binlog for Each Domain when there's a New Master

22

Page 23: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Enabling GTID on MasterSet gtid_domain_id on Master & Restart

Execute SHOW MASTER STATUS to get Current binlog File Name and Position

Use BINLOG_GTID_POS() to Get the binlog GTID Position Equivalent

23

Article on Enabling GTID for Replication: https://mariadb.com/resources/blog/enabling-gtids-server-replication-mariadb-100

[mysqld] gtid-domain-id=1 SHOW MASTER STATUS\G

****** 1. row ****** File: mariadb-bin.000001 Position: 510 ...

SELECT BINLOG_GTID_POS('mariadb-bin.000001', 510) AS binlog_gtid_pos;

+-----------------+ | binlog_gtid_pos | +-----------------+ | 1-101-1 |

+-----------------+

Tell the slave this value.

Excerpt from master's configuration file.

Executed on master.

Page 24: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Informing Slave of GTIDSet gtid_slave_pos on Slave to

binlog GTID Position from Master

Use CHANGE MASTER to set master_use_gtid to slave_pos

Execute SHOW SLAVE STATUS to check if GTID is in Use

24

STOP SLAVE;

SET GLOBAL gtid_slave_pos = '1-101-1';

CHANGE MASTER

TO master_use_gtid=slave_pos;

START SLAVE;

SHOW SLAVE STATUS \G ... Using_Gtid: Slave_Pos

Gtid_IO_Pos: 1-101-1 ...

Executed on slave.

Executed on slave.

Page 25: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

GTID VariablesSeveral Positions are Stored Use SHOW VARIABLES to Find Values

25

SHOW VARIABLES

WHERE Variable_Name LIKE 'gtid%' OR Variable_Name LIKE 'last_gtid';

+------------------------+-----------+ | Variable_name | Value |

+------------------------+-----------+ | gtid_binlog_pos | 0-1-96217 | | gtid_binlog_state | 0-1-96217 | | gtid_current_pos | 0-1-96217 |

| gtid_domain_id | 0 | | gtid_ignore_duplicates | OFF | | gtid_seq_no | 0 | | gtid_slave_pos | | | gtid_strict_mode | OFF |

| last_gtid | | +------------------------+-----------+Documentation on GTID Variables: https://mariadb.com/kb/en/mariadb/gtid/#system-variables-for-global-transaction-id

Page 26: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Showing EventsUse the SHOW BINLOG EVENTS

Statement to see a list of Binary Log Events

Several Event Types Format_desc Gtid_list Binlog_checkpoint Gtid Query

26

SHOW BINLOG EVENTS IN 'ip-12-34-56-01-bin.000056' LIMIT 1 \G

*********** 1. row *********** Log_name: ip-12-34-56-01-bin.000005

Pos: 4 Event_type: Format_desc Server_id: 1 End_log_pos: 249 Info: Server ver: 10.1.23-MariaDB,

Binlog ver: 4

Documentation on SHOW BINLOG EVENTS: https://mariadb.com/kb/en/mariadb/show-binlog-events/

Page 27: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

GTID Strict ModeHelps keep binlogs Identical across

Multiple Servers Slave Stops when Encountering a Problem

Won't Allow Slave to use Lower Sequence for a Domain — SQL Thread Stops

Rejects Attempts to Set SESSION.gtid_seq_no to Lower Sequence

Stops Slave from Connecting with a GTID missing in Master's binlog

Controlled by Setting Strict Mode on Slave

27

Documentation on gtid_strict_mode: https://mariadb.com/kb/en/mariadb/gtid/#gtid_strict_mode

SET gtid_strict_mode = 'ON';

Page 28: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

GTID Ignores DuplicatesMulti-Source Masters may Process

Event Groups with same GTID Only One Applied — Others are Ignored

Sequence Number of Domain Used to Determine if Duplicate

Need to Ensure GTID Sequence are Increased Strictly

28

Documentation on gtid_ignore_duplicates: https://mariadb.com/kb/en/mariadb/gtid/#gtid_ignore_duplicates

SET gtid_ignore_duplicates = ON;

Page 29: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Switch Slave to GTIDSlaves can Use GTID or Binary Log File

Name with Position ID Use CHANGE MASTER Statement to Switch

to GTID — Give Value of NO to Disable

29

CHANGE MASTER TO master_use_gtid=slave_pos

Documentation on CHANGE MASTER Statement: https://mariadb.com/kb/en/mariadb/change-master-to/

Page 30: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Switching PlacesInitial Scenario

Server 1 is Master; Server 2 is Slave Server 1 goes Down Promote Server 2 to Master

New Scenario Server 1 is Revived and Made a Slave

Doesn't have Prior GTIDs since it was Never a Slave The gtid_slave_pos is Empty

Use Instead current_pos with CHANGE MASTER

30

mysqldData

Storage

Server 1

Binary Log Dump Thread

Data Storage

Server 2

Relay Log

Binary Log

mysqld

CHANGE MASTER

TO master_use_gtid=current_pos

Executed on Server 1

Page 31: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Complex Scenarios

Page 32: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Non-Typical Replication MethodsMariaDB Replication can use Non-

Traditional Topology Circular Replication Multi-Source Replication

32

Page 33: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Circular ReplicationClients can Write to any Server — All are

Masters and Accept Writes

Servers can Replicate Concurrent Changes to Each Other (e.g., Two Changes to Same Row)

No Conflict Resolution

33

server-id100

server-id200

server-id300

Page 34: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Multi-Source ReplicationSlaves are Used Typically for Read Load

Balancing Writes can also be Load Balanced with

Multi-Source Replication Don't Confuse with Multi-Master

Replication (i.e., Circular or Point-to-Point Replication)

34

Documentation on Multi-Source Replication: https://mariadb.com/kb/en/mariadb/multi-source-replication/

server-id101

server-id103

server-id102Two Masters

One Slave

t1 t2

The default_master_connection can equal t1 or t2.

Page 35: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Multi-Source PreparationUse an Alias with the CHANGE MASTER

Statement to Distinguish between Sources

Different Replication Filters may be Used for Differen Master Sources

35

CHANGE MASTER 'master1' TO

MASTER_HOST='35.161.145.71', MASTER_PORT=3306, MASTER_USER='maria_replicator', MASTER_PASSWORD='rover123';

START SLAVE 'master1';

Executed on slave.

Documentation on CHANGE MASTER with Alias: https://mariadb.com/kb/en/mariadb/change-master-to/#connection_name

Page 36: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Multi-Source MonitoringUse SHOW SLAVE STATUS with Alias to

Monitor Slave in Relation to a Specific Master

Use ALL SLAVES to Show Slave Status for All Sources

36

SHOW SLAVE 'master1' STATUS;

SHOW ALL SLAVES STATUS;

Documentation on SHOW MASTER STATUS: https://mariadb.com/kb/en/show-master-status/ Documentation on SHOW SLAVE STATUS: https://mariadb.com/kb/en/mariadb/show-slave-status/

Executed on slave.

Page 37: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Stopping Multi-Source ReplicationUse STOP SLAVE with Alias to Stop

Replicating a given Master Use ALL SLAVES Option to Stop

Replication Completely

37

STOP SLAVE 'master1';

STOP ALL SLAVES;

Executed on slave.

Page 38: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB Replication — AsynchronousMaster Doesn’t Wait for Slaves IO Thread may be Slow to Receive

binlog Packets Network Congestion or Disconnects

SQL Thread may be Slow in Processing Relay Log Events

Load on Slave or Network Problems

38

Page 39: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Semi-Sync Plugin

Page 40: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Semi-Synchronous

40

Documentation on Semi-Synchronous Replication: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

Master waits for a Slave to Acknowledge Receipt of Transaction

Waits for Slave to Write to Relay Log, Not to Execute — Slave SQL Thread may still Lag

One Slave Response needed for Master to Continue (i.e., Semi-Synchronous, Not Synchronous)

Can Significantly Affect Performance of Master

Page 41: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Enable Semi-SynchronousMaster Switches to Asynchronous

if Not Acknowledged Semi-Synchronous is Resumed

When One Slave Synchronizes

41

SET GLOBAL rpl_semi_sync_master_enabled = ON;

SET GLOBAL rpl_semi_sync_slave_enabled = ON;

SET GLOBAL

rpl_semi_sync_master_timeout = 10000;

INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

Documentation on Semi-Sync Status Variables: https://mariadb.com/kb/en/mariadb/semisynchronous-replication-plugin-status-variables/

Page 42: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Binary Log FormatStatement Based (SBR) — Original Queries are Replicated

Least Data Sent over Wire and Tested for Years Non-Deterministic Statements Executed on Slave — Slave Load is Increased

Row Based (RBR) — Table Rows are Replicated

Only Non-Deterministic Statements Executed on Master — Slave Load is Reduced vs. SBR

More Data sent over Wire — Not Supported by All Engines Mixed (default) - Smart Switching between SBR and RBR

Checksum (--binlog-checksum) in Binary and Relay Logs to detect Errors

Includes Errors in Memory, Disk, Network and Database Can be Implemented for each Slave

42

Documentation on Binary Log Format: https://mariadb.com/kb/en/mariadb/binary-log-formats/

[mysqld]

binlog-format=MIXED

Page 43: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Annotate RowsBinary Logging can Annotate Row

Events — Useful for Florensic Information Add the --binlog-annotate-row-events

Option — There's No Variable to use with SET

43

Documentation on https://mariadb.com/kb/en/mariadb/annotate_rows_log_event/

[mysqld] binlog-annotate-row-events

Page 44: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Register New Semi-Synchronous SlaveSlave will Continue as Asynchronous

Unless Restarted Properly Restart Slave IO Thread to Register as

Semi-Synchronous

44

STOP SLAVE IO_THREAD;

START SLAVE IO_THREAD;

Executed on slave.

Page 45: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Timeout for Semi-Synchronous ReplicationMaster Reverts to Asynchronous

when Time Exceeded for Acknowledgement

Default Time is 10,000 milliseconds

Status Variable rpl_semi_sync_master_status Set Automatically to OFF

45

SET rpl_semi_sync_master_timeout = ON;

Page 46: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Debugging Semi-Sync ReplicationFour Tracing Levels for Semi-

Synchronous 1 — General Level (e.g., Time Function

Failures) 16 — Detailed, Verbose Level 32 — Net Wait Level (e.g., Information

about Network Waits) 64 — Function Level (e.g., Information

about Function Entries and Exits) Default Value is 32

46

SET rpl_semi_sync_master_trace_level = 32;

Documentation on trace levels: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/#rpl_semi_sync_master_trace_level

Page 47: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Waiting Related to Slave CountMaster Reverts to Asynchronous when No Slaves

(i.e., rpl_semi_sync_master_clients) Enable rpl_semi_sync_master_wait_no_slave to

Wait for Timeout

47

SET rpl_semi_sync_master_wait_no_slave = ON;

Page 48: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Waiting for SynchronizationSet Parameters for Master to Wait for Acknowledgement

AFTER_SYNC — Transaction Written to Slave's binlog All Clients see Same Data at Same Time Failover is Lossless if Master Crashes

AFTER_COMMIT — After a Slave Acknowledges and Master Commits to Storage Engine

Other Clients may see Committed Transaction Clients may see Data Loss if Master Crashes

48

SET rpl_semi_sync_master_wait_point = AFTER_SYNC;

Documentation on rpl_semi_sync_master_wait_point: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/#rpl_semi_sync_master_wait_point

Page 49: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

MariaDB Replication Manager

Page 50: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB Replication ManagerAble to Handle Failovers of MariaDB

Replication with GTID Failover on a Dead Master Promote a Slave to Master

Arbitrator and Proxy Peacemaker Resource Use a Client Proxy to Route Queries to

New Hosts

50

Documentation on MariaDB Replication Manager: https://github.com/mariadb-corporation/replication-manager/blob/0.7/README.md

Page 51: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InstallationDownload a Binary Package from

Github Use yum or another Utility to

Install Binary

51

Download MariaDB Replication Manager: https://github.com/tanji/replication-manager/releases Article on Installing MariaDB Replication Manager: https://mariadb.com/resources/blog/mariadb-automatic-failover-maxscale-and-mariadb-replication-manager

wget https://github.com/tanji/replication-manager/.../xx.xx.version.rpm yum install xx.xx.version.rpm

Page 52: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Switch Over ManuallyUse switchover Option to Switch

Servers Selects Most Suitable Node in --

hosts to be Leader — Decision based on which is Most Up-to-Date

Include --interactive for Interactive Mode to Confirm New Topology

52

replication-manager switchover \ —hosts=node1,node2,node3 \ —user=root:pass —rpluser=replicator:pass \

--interactive

Node A Node CNode B

MariaDBReplication Manager

Page 53: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Procedural Command-Line ExampleNon-Interactive Failover Mode

Use root for Management Use repl for Replication Switchover Give Failover Script Set Maximum Slave Delay before Switchover

53

replication-manager failover \ --hosts=db1:3306,db2:3306,db2:3306 \

--user=root:pass --rpluser=repl:pass \ --pre-failover-script="/usr/local/bin/vipdown.sh" \ --post-failover-script="/usr/local/bin/vipup.sh" \ --verbose --maxdelay=15

Page 54: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MonitoringUse a Client Proxy to Monitor the Servers Add a Script Entry to Start Replication

Manager User Executing Script must have ALL

Privileges repluser must have at least REPLICATION

SLAVE Privilege

54

Download MariaDB Replication Manager: https://github.com/mariadb-corporation/replication-manager Donwload Go Programming Language: https://golang.org/dl/

[MySQL Monitor] type=monitor

module=mysqlmon servers=%%ENV:SERVERS_LIST%% user=root passwd=%%ENV:MYROOTPWD%% monitor_interval=1000

detect_stale_master=true

script=/usr/local/bin/replication-manager --user root:admin

--rpluser repluser:replpass --hosts $INITIATOR,$NODELIST --failover=force --interactive=false events=master_down

Page 55: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Monitoring with Console Mode

55

replication-manager monitor / --hosts=db1:3306,db2:3306,db2:3306 / --user=root:pass --rpluser=repl:pas

Page 56: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

HTTP ServerStart in Background to Monitor a

Cluster with HTTP Server Controlling Daemon

Accessible through Web Browser on http://localhost:10001 by Default

56

replication-manager monitor \ --hosts=db1:3306,db2:3306,db2:3306 \

--user=root:pass --rpluser=repl:pass \ --daemon --http-server

Page 57: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

MariaDB Enterprise Cluster

Page 58: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Advantages of MariaDB Enterprise ClusterParallel Slave Applying Practically No Slave Lag Instant Trivial Failover Automatic Node Provisioning Works Well in WAN

58

Page 59: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

NuancesUses Only InnoDB Primary Keys are Necessary Commit Latency Transaction Size Limited to 2GB —

Unlimited in Future Versions

DEADLOCK on COMMIT

AUTO_INCREMENT Handled Differently

No Cluster-Wide Read Locks

59

Page 60: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB Replication — Server Centric StyleAsynchronous Replication No Conflict Detection Multi-Master Replication

If Node C Crashes, Does Cluster Survive?

If Node B Crashes and Clients Switch to C, How does Node B Rejoin?

Which Node has Data X? How do you Back-Up Cluster?

60

MasterServer

SlaveServer

Node A

Node C

Node B

Page 61: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Galera Approach — Data Centric StyleData Doesn’t Belong to a Node Nodes Belong to Data Data is Synchronized among Two or

More Servers

61

Server 1 Server 2 Server 3 Server N

DataSet

Page 62: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Galera ApproachGalera Nodes are Anonymous — All are

Equal Galera Cluster is One Large Distributed

Master A DataSet Needs an Identifier DataSet Identier is a Cluster Identifier

00295a79-9c48-11e2-bdf0-9a916cbb9294

62

DataSet Cluster

Page 63: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)DataSet plus Sequence of Atomic

Changes equals GTID

63

DataSet00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

64201

Page 64: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)Initial DataSet

64

00295a79-9c48-11e2-bdf0-9a916cbb9294:0

00295a79-9c48-11e2-bdf0-9a916cbb9294:1

00000000-0000-0000-0000-000000000000:-1

First Change and Transaction

Undefined GTID

Page 65: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)MySQL 5.6 GTID

65

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

8182213e-7c1e-11e2-a6e2-080027635ef5:12345

Galera GTID

0-10-12345

MariaDB 10 GTID(server identifier : transaction processed by server)

(data & cluster Identifier : data change in cluster)

(domain - server identifier - data change in asynchronous cluster)

Page 66: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)Visible in MySQL 5.6

66

8182213e-7c1e-11e2-a6e2-080027635ef5:12345 8182213e-7c1e-11e2-a6e2-080027635ef5:12346 8182213e-7c1e-11e2-a6e2-080027635ef5:12347

New Master Promoted

f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:1 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:2 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:3

Page 67: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)Visible in MariaDB 10

67

0-10-12345 0-10-12346 0-10-12347

New Master Promoted

0-20-12348 0-20-12349 0-20-12350

Page 68: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Global Transaction Identifier (GTID)Visible in Galera

68

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201 00295a79-9c48-11e2-bdf0-9a916cbb9294:64202 00295a79-9c48-11e2-bdf0-9a916cbb9294:64203

New Master Promoted

00295a79-9c48-11e2-bdf0-9a916cbb9294:64204 00295a79-9c48-11e2-bdf0-9a916cbb9294:64205 00295a79-9c48-11e2-bdf0-9a916cbb9294:64206

Page 69: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Master or SlaveNot a Node Role or Function A Relation Between a Node and a Client

69

master1slave2

slave1slave2

slave1master2

slave1slave2

slave1slave2

Cluster

master for client1 master for client1

client2client1

Page 70: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Cluster Addresswsrep_cluster_address

70

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

10.0.0.5

Documentation on Galera Cluster Addresses: https://mariadb.com/kb/en/mariadb/galera-cluster-address/

Page 71: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Cluster Addresswsrep_cluster_address

71

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

gcomm://10.0.0.6

10.0.0.5handshake

Page 72: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Cluster Address

72

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

10.0.0.5

Page 73: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Cluster Address

73

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3

10.0.0.4

10.0.0.5

Page 74: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Cluster Addresswsrep_cluster_address = gcomm://node1,node2

Try to Connect to Members (node1, node2)

Can only Join a Running Cluster

74

10.0.0.5

Page 75: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Node Synchronization (State Transfer)

75

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3

10.0.0.4

10.0.0.5

Page 76: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Node Synchronization (State Transfer)

76

joined

syncedsynced

desync

synced

undefined

Page 77: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Node Synchronization (State Transfer)

77

catch-up

UNDEFINED

JOINER

JOINED

SYNCHED

catch-up

SYNCHED

DONOR

JOINED

SYNCHED

New Node Cluster Old Node

Verify Node Synched

Donor Found — Wait Be a Donor

Transfer Missing Data (Private Channel)

Page 78: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Avoiding Split BrainDistinguishing Server Crash from

Network Failure in Shared Nothing Architecture

Decision Algorithm Used to Avoid Spilt Brain

Absolute Majority Needed in Galera Uneven Number of Nodes Safer

78

Page 79: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Primary Component

79

Primary

Page 80: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Primary Component

80

Primary

Non-Primary

Continues Working

Page 81: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Primary Component

81

Primary

Non-Primary

Tries to ReconnectContinues Working

Page 82: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Primary Component

82

Primary

Page 83: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Split Brain

83

Non-Primary Non-Primary

Split Brain Possible with Even Number of Nodes

Page 84: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Synchronous PenaltiesGalera Copies Data Buffer to All Cluster

Members on COMMIT from Client (~1 RTT added latency)

Connection throughput equals 1/RTT trx/sec

Total throughput equals 1/RTT trx/sec ✕ #connections

A Given Row Can't be Modified More Than 1/RTT times a second

84

Round Trip Time (RTT) is Length of Time for a Signal to be sent and Receipt of Acknowledgement

Page 85: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Galera Synchronous Penalty in WAN (EC2)

85

Page 86: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Galera Synchronous Penalty in WAN (EC2)

86

Page 87: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Slave Lag in Galera

87

Client Master Node

START TRANSACTION

COMMIT Replicate Write Set

OK

Slave Node

COMMIT

PROCESS

APPLY

COMMIT

Acknowledge

SELECT (stale data)

Certify

Page 88: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Client Proxies

Page 89: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Client Proxy Objectives & FeaturesObjectives High Availability Minimal Latency Read Scalability Highly Scalable

Features Transparent to Applications Extendible Lightweight with Small

Footprint Authentication Required

89

Page 90: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Popular Client ProxiesHAProxy — TCP/HTTP Load Balancer

Distributes Traffic across Multiple Servers to Improved Performance and Resource Usage

MaxScale Database Proxy to Forward SQL Statements to Multiple

Database Servers Provides Transparently to Applications, Load Balancing

and High Availability Functionality Scalable and Flexible Architecture that Supports different

Protocols and Routing

Hardware Load Balancer

90

Tutorial on haproxy: https://mariadb.com/resources/blog/setup-mariadb-enterprise-cluster-part-3-setup-ha-proxy-load-balancer-read-and-write-pools Documentation on MaxScale: https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-21/

Page 91: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Proxy as an IntermediaryApplication-to-Database

Insulates Applications from Complexities of Backend Database Cluster

91

Proxy

MariaDB Master

Application

MariaDBSlave

MariaDBSlave

MariaDBSlave

Page 92: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Load Balancing — Read ScalingMariaDB Master-Slave Replication GTID Unique across Independent

Replication Streams Multi-Source Replication Optimistic Parallel Replication Slave Execution of Triggers

92

Proxy

MariaDB Master

MariaDBSlave

MariaDBSlave

Client Client Client

Page 93: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Load Balancing — Write ScalingMariaDB Enterprise Cluster Multi-Master Replication for Write

Scalability

93

Proxy

MariaDB

MariaDB

Client Client Client

MariaDB

Galera Cluster

Page 94: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Monitoring & Health CheckLoad Balancer should Monitor Often Hardware Load Balancer can Health Check through inetd Script

Limited CPU Available in Appliances

MaxScale Parameter monitor_interval — Default 500 milliseconds

94

Page 95: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Automatic FailoverAutomatic Detection of Master

Failure MaxScale Monitor Launches Script

upon Master Failure Promotes a Slave as New Master Instructs Other Slave of New

Master

95

MaxScale

Master

Slave Slave Slave

Monitor Detects Event: master_down

Execute Failover Script Promote a Slave to Master CHANGE MASTER on Slaves

Master Fails

binlog Cache

Page 96: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Optimizing

Page 97: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Optimizing Overview

Page 98: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Value in Performance TuningEfficient Use of Hardware Resources Tune Only when Needed Anticipate Needs

Increase in Traffic leads to Exponential Load Growth

User Data Grows Over Time

98

Page 99: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Tuning RoutineStart Early in Development Life Cycle Regularly Review Production Systems

Schema Changes, etc.

Monitor System Resources Emergency Response Tuning

99

Page 100: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

HardwareOne Service per Server is Ideal to Prevent Contention More CPU Cores is generally Good More Disk is usually Better

Large Datasets, Fast Disks are Ideal

RAM is usually Best — Traffic Dependent

More of Dataset in Memory, Fewer Slow Disk Operations

100

Page 101: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Schema Tuning

Page 102: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Train DevelopersConsistent Naming Conventions Appropriate & Accurate Data Type (e.g., INT for Numbers)

Variable Length Fields are Often Padded

Use NOT NULL where Practical A NULL field uses slightly More Disk and Memory (Depends on

Storage Engine)

Use PROCEDURE ANALYSE( )

102

Page 103: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Tuning Tables OverallMinimize Table Size on Disk and in Memory Archive Table Data if Possible and Appropriate Remove Duplicate or Unused Indexes Use Appropriate Data Types — Smaller is Better

Consider Sharding Large Tables across Multiple Servers

103

Page 104: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Number Data TypesUse UNSIGNED when Appropriate

INT(n) Specifies Display Precision, Not Storage Precision

Size and Precision is Storage Engine Dependent Define Handling of Out-of-Range Values with sql_mode

Default Mode: Values are Truncated Silently Strict Mode: Errors are Generated

104

Documentation on Integer Data Types: https://mariadb.com/kb/en/mariadb/data-types/

TINYINT

SMALLINT

MEDIUMINT

INTEGER,INT

BIGINT

Page 105: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

String Data TypesAll String Data Types have a Character Set CHAR(n) — number of characters, not bytes, wide VARCHAR(n)— Changes to CHAR in Implicit Temporary Tables and mysqld internal buffers

TEXT —Not Supported by the MEMORY Storage Engine; Implicit Temporary Tables may Convert to MyISAM

105

Documentation on String Data Types: https://mariadb.com/kb/en/mariadb/string-data-types/

CHAR

VARCHAR

TINYTEXT

TEXT

MEDIUMTEXT

LONGTEXT

Page 106: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Character Set & CollationCharacter Set may be Global or for Schema, Table

or Column Multi-Byte Character Sets Increase Disk Storage

and Working Memory Requirements (e.g, UTF-8 Requires 3 or 4 bytes per Character)

Collations affect String Comparison (Character Order) Collations can be Changed for Query

106

Documentation on Character Sets and Collation: https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

SELECT * FROM table1 ORDER BY col1 COLLATE latin1_german2_ci;

Page 107: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Binary Data TypesBINARY and VARBINARY are Case-Sensitive Versions of CHAR and VARCHAR

No Character Set and Collation for Binary Types - ordered by bytes

Blobs are Used often to Store Files in a Database Files on Disk are often Faster Blobs are Included in Transactions, Replication, and

Backups Blobs Inflate mysqld memory usage

107

Documentation on BINARY Data Type: https://mariadb.com/kb/en/mariadb/binary/ Documentation on VARBINARY Data Type: https://mariadb.com/kb/en/varbinary/ Documentation on BLOB Data Type: https://mariadb.com/kb/en/blob/

BINARY

VARBINARY

TINYBLOB

BLOB

MEDIUMBLOB

LONGBLOB

Page 108: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Date & Time Data TypesDATE — from 1000-01-01 to 9999-12-31 (YYYY-MM-DD)

TIME — from -838:59:59 to 838:59:59

DATETIME — Same Range with Time (YYYY-MM-DD HH:mm:ss)

TIMESTAMP — Unix timestamp, in seconds from 1970-01-01

Many Apps Store UNIX_TIMESTAMP() values in unsigned integer field YEAR — Accepts YY or YYYY

108

DATE

TIME

DATETIME

TIMESTAMP

YEAR

Documentation on Date and Time Data Types: https://mariadb.com/kb/en/mariadb/date-and-time-data-types/Documentation on Microseconds in MariaDB: https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

SELECT CURTIME(4); +---------------+ | CURTIME(4) |

+---------------+ | 05:33:09.1061 | +---------------+

Page 109: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Listing Acceptable ValuesENUM is an Enumerated List of

String Values — uses a 2-byte integer index

109

Documentation on ENUM: https://mariadb.com/kb/en/mariadb/enum/ Documentation on SET: https://mariadb.com/kb/en/mariadb/set-data-type/

CREATE TABLE colors (primary_colors ENUM('red', 'yellow', 'blue'));

CREATE TABLE colors

(primary_colors SET('red','yellow','blue'));

INSERT INTO colors VALUES('red'), ('red,blue');

Simple Example for an ENUM Column

Simple Example for a SET Column

SET is a Specified List of String Values — Can Hold Multiple Specified Values

Not Indexable

Page 110: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Keys and IndexesMake PRIMARY KEY as Small as Practical

InnoDB Stores Copy of PRIMARY KEY with each Index Entry Secondary Indexes Grow with PRIMARY KEY

Use Surrogate PRIMARY KEY, Not Natural Ones AUTO_INCREMENT is like a Sequence AUTO_INCREMENT UNSIGNED NOT NULL Careful of AUTO_INCREMENT Contention for Pre-Plugin InnoDB

Consider Partial (Prefix) Index for String Indexes With Good Index Selectivity, Won't Affect Performance Much Without Good Selectivity is a Different Problem

110

Page 111: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

File StorageStoring Files in BLOB Useful

Included in Back-Up and Replication Guaranteed to be Consistent with Other Data

Storing Files in BLOB has Problems Database gets Too Large MariaDB Can't do Partial Field Read — MyISAM Reads Entire File InnoDB can Skip if Field Not Referenced in Query

Minimize by putting Files in Separate Table (1:1 relationship)

Store Files on File System — Put File Name and Path in Table Database is Smaller and More Efficient Organize Separate Back-Up, possibly with External Locking for

Consistency

111

Page 112: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Normalization of TablesProcess of Optimizing and Factoring a Schema

1:1, 1:n and n:n 1NF / 2NF / 3NF

Makes a Data Set Smaller by Eliminating Redundant or Duplicated Data

Problem if Data is Smaller than Key

Ensures Data Set Consistency and Integrity Often Improves Concurrency by Reducing Locking

Overhead — Not Always

Increases Number of Tables and Associated Maintenance

112

Page 113: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Denormalization of TablesComplete Normalization can Slow Queries Complex JOINs are Drains and Maintainence Problem

Add Redundant Data Back to Simplify with JOIN Queries Combine 1:n Relationships Add Pre-Computed Columns Use Fake Materialized Views

Denormalization Adds Redundant Data to Schema Write Queries become More Complex or Numerous as

Multiple Locations must be Maintained

Easier to Normalize First, Then Denormalize when Appropriate

113

Page 114: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Dynamic ColumnsStores Key/Value Pairs within a

BLOB Column with COLUMN_CREATE()

Used to Store many Attributes which are Unknown in Advance

Manipulated with Special Dynamic Column Functions:

114

SELECT item_name, COLUMN_GET(item_attributes, 'color' AS CHAR) AS color FROM clothes;

+-----------------+-------+ | item_name | color | +-----------------+-------+

| MariaDB T-shirt | blue | +-----------------+-------+

COLUMN_GET()

COLUMN_ADD()

COLUMN_LIST()

COLUMN_DELETE()

Documentation on Dynamic Columns: https://mariadb.com/kb/en/dynamic-columns/ Tutorial on Dynamic Columns: http://radar.oreilly.com/2015/04/dynamic-columns-in-mariadb.html

CREATE TABLE clothes

(item_name VARCHAR(32) PRIMARY KEY,

item_attributes BLOB);

INSERT INTO clothes

VALUES ('MariaDB T-shirt',

COLUMN_CREATE('color', 'blue', 'size', 'XL'));

Page 115: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Virtual ColumnsTwo Virtual Column Types:

PERSISTENT (stored) VIRTUAL (generated only)

All Data Types Supported Use PERSISTENT for Indexes - Cannot

be Primary Key

Used with InnoDB, Aria, MyISAM, CONNECT

115

Documentation on Virtual Columns: https://mariadb.com/kb/en/virtual-columns/

CREATE TABLE sales

(sales_id INT AUTO_INCREMENT KEY, qty INT, price DECIMAL(6,2), total DECIMAL(6,2) AS (qty * price) VIRTUAL);

Page 116: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

InnoDB In-Depth

Page 117: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Advantages & Disadvantages of InnoDB

117

Read Uncommitted Read Committed Repeatable Read (no fantom rows) Serializable

Fully Transactional,

ACID Compliant

Data & Indexes Cached by mysqld

in Buffer Pool

Row-Level Locking for High-

Concurrency

Supports Four Isolation Levels

Supports Foreign Keys and Multi-

Version Concurrency

Control

Reliable Crash Recovery

Slower than MyISAM, but

Higher Concurrency

Advantages Disadvantages

Isolation Levels

Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

117

Page 118: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Isolation LevelsInnoDB Supports All Four Transaction Isolation Levels*

READ UNCOMMITTED READ COMMITTED REPEATABLE READ SERIALIZABLE

* Listed in Order of Increasing Performance Potential

Increasing Isolation Protection means More Locks Needed

Reduces Query Concurrency Increases Memory Usage

Set Isolation Level on Per-Session Basis for Transactions Requiring More Protection

Leave Bulk of Traffic to Operate at Lower Isolation Level

118

Page 119: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Architecture

119

Additional Memory Pool

Buffer Pool *

Data Files * **

Redo Log (File 1/2)

Redo Log (File 2/2)

Memory

Disk

* Cached Data & Indexes, Stored in Pages ** Transaction Data Writes *** Data, Index, and Undo Log Files

INSERT, UPDATE, DELETE

1

COMMIT

CHECKPOINT

Log Buffer * *

2

3

119

Page 120: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Buffer PoolLarger is Generally Better for innodb_buffer_pool_size

For Servers with Only InnoDB Tables, 80% of Available Memory

MyISAM is Used Internally and Needs Resources along with OS

Dirty Buffer Pool Pages are Flushed in Background Allows Changes to be Merged and Written Sequentially to Disk Applications with Plenty of Writes, or Write Spikes, may need

Adjusting Use innodb_max_dirty_pages_pct to Control Frequency of

Flushing Dirty Pages

InnoDB Plugin can Disable the Adaptive Hash Index Mostly Improves Performance, but Not Always

120

Page 121: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Clustered Index

121

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER PRIMARY KEY ROW DATA

leaf

121

Page 122: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Secondary Index

122

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY

leaf

122

Page 123: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

IndexesInnoDB Indexes are Clustered

Row Data is Stored with Primary Key

Use Short Surrogate Primary Keys (INT UNSIGNED AUTO_INCREMENT)

Secondary Index Entries hold a Copy of PK — Large PK Increases All

Table Index Statistics are Vague for InnoDB Gathered by a Series of Random Index Dives It's Possible for Stats to be Inaccurate and Cause

Optimizer to Choose Bad Query Plan Run Periodically ANALYZE TABLE to Regenerate Stats

123

Page 124: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Lock MonitoringINNODB_LOCKS

INNODB_TRX

INNODB_BUFFER_POOL_STATS

INNODB_TABLE_STATS

INNODB_INDEX_STATS

SHOW TABLES LIKE 'INNODB%';

SELECT r.trx_id AS waiting_trx_id, r.trx_mysql_thread_id AS waiting_thread,

r.trx_query AS waiting_query, b.trx_id AS blocking_trx_id, b.trx_mysql_thread_id AS blocking_thread, b.trx_query AS blocking_query FROM information_schema.innodb_lock_waits w

INNER JOIN information_schema.innodb_trx b ON b.trx_id = w.blocking_trx_id INNER JOIN information_schema.innodb_trx AS r ON r.trx_id = w.requesting_trx_id;

Locked & Locking Queries

124

Page 125: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB MutexesProcess of Locking and

Latching Resources for InnoDB

InnoDB Keeps Track of All Mutexes

125

SHOW ENGINE INNODB MUTEX;

+--------+------------------------------+---------------+

| Type | Name | Status |

+--------+------------------------------+---------------+

| InnoDB | &rseg->mutex | os_waits=1 |

| InnoDB | &dict_sys->mutex | os_waits=1 |

| InnoDB | &log_sys->mutex | os_waits=116 |

| InnoDB | &buf_pool->flush_state_mutex | os_waits=3913 |

| InnoDB | &buf_pool->LRU_list_mutex | os_waits=16 |

| InnoDB | combined &block->mutex | os_waits=7 |

| InnoDB | &dict_operation_lock | os_waits=167 |

| InnoDB | &log_sys->checkpoint_lock | os_waits=2239 |

| InnoDB | &btr_search_latch_arr[i] | os_waits=6 |

| InnoDB | combined &block->lock | os_waits=47 |

+--------+------------------------------+---------------+

Documentation on SHOW ENGINE INNODB MUTEX: https://mariadb.com/kb/en/mariadb/show-engine/#show-engine-innodb-mutex

Page 126: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Transaction LogConverts Transactions from Random I/O to Sequential

Transaction Data is Written to Transaction Log for Durability and Speed - Not Directly to Data Files

Log Files Rotate as a Circular Buffer with Data Merged to Disk in Background

126

fsync() FrequencyVariable Setting Write and Flush Action

innodb_flush_log_at_trx_commit = 0 Write at Commit, but No Flush

innodb_flush_log_at_trx_commit = 1 Flush at Commit (Default)

innodb_flush_log_at_trx_commit = 2 Write at Commit, Flushed Every Second

Page 127: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Transaction LogLarge innodb_log_file_size (Default 50 MB) Results

Usually in Less I/O and Better Performance Crash Recovery Delay Increases with Log File Size

Increasing innodb_log_buffer_size (Default 1MB) May Reduce Log File I/O for Large Transactions

127

Page 128: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

The InnoDB CoreVariable innodb_flush_method Controls How Data is

Written to Disk Default is a Normal fsync( ) Operation Leads to "Double Buffering" of Data (Buffer Pool and OS Disk

Cache)

O_DIRECT Allows InnoDB to Use Direct I/O Bypasses any OS Cache

Changing Flush Method should be Tested Thoroughly Some Combinations of Device, Filesystem, OS may Not

Benefit

128

Page 129: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Collecting Information

Page 130: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Tools & StatisticsIdentify Accurately and Carefully Trouble

Spots Gather Performance Stats with MariaDB

and OS Tools SHOW Statements PERFORMANCE_SCHEMA CPU, Disk, Network, Memory, & Swap Stats

Retain Snapshots of Multiple Stats Data from a Single Point Shows Little

Automate Collection of Stats into Logs Useful for Emergency Tuning

130

Page 131: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

SHOW PROCESSLISTSnapshot of mysqld Activity

mysqld is Multi-Threaded, One Thread per Client Connection — a "process" is a "thread"

Accumulate SHOW PROCESSLIST Snapshots for History of Thread Activities

131

Time: Length of Time Thread has been in Current State State: Indicates Thread Activity

SHOW PROCESSLIST \G

****** 1. row ******** Id: 9546 User: monyog Host: localhost:40600 db: NULL

Command: Sleep Time: 192 State: Info: NULL Progress: 0.000

Page 132: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Thread StatesAfter Create Analyzing Checking Permissions Checking Table Cleaning Up Closing Tables Converting HEAP to MyISAM Copy to tmp Table Copying to Group Table Copying to tmp Table Copying to tmp table on Disk Creating Index

Creating Sort Index Creating Table Creating tmp Table Deleting from Main Table Deleting from Reference

Tables discard_or_import_tablespace End Executing Execution of init_command Flushing Tables Freeing Items FULLTEXT initialization

Init Killed Locked Logging Slow Query Login Manage Keys NULL Opening Tables, Opening

Table Optimizing Preparing Purging Old Relay Logs Query End

132

Documentation on General Thread States: https://mariadb.com/kb/en/mariadb/general-thread-states/

Page 133: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Thread StatesReading from Net

Removing Duplicates Removing tmp Table

Rename Rename Result Table

Reopen Tables Repair by Sorting

Repair Done

Repair with Keycache Rolling Back

Saving State Searching Rows for Update

Setup

Sorting for Group Sorting for Order

Sorting Index Sorting Result

Statistics System Lock

Table Lock

Updating Updating Main Table

Updating Reference Tables User Lock

User Sleep

Waiting for (tables|tables|table flush) Waiting for All Running Commits to

Finish

Waiting for Commit Lock Waiting for Global Read Lock

Waiting for lock_type Lock Waiting for Release of Read Lock

Waiting on Cond

Waiting to Get Read Lock Writing to Net

133

Page 134: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Check System StatusUse SHOW STATUS for List of Internal Counters

GLOBAL for System Status Since Start

SESSION for Client Connection

FLUSH STATUS Resets Local Counters

Monitor Changes to Counters to Identify Hot Spots Collect Periodically Status Snapshots to Profile Traffic

134

Documentation on SHOW STATUS: https://mariadb.com/kb/en/mariadb/show-status/

Page 135: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Connection & Thread StatusClients & Connections Aborted_clients

Aborted_connects

Connections

Max_used_connections

Ssl_*

Threads Delayed_insert_threads

Threads_cached

Threads_connected

Threads_created

Threads_running

Bytes & Writes Bytes_received

Bytes_sent

Compression

Delayed_errors

Delayed_writes

Uptime*

135

Docmentation on Server Status Variables: https://mariadb.com/kb/en/mariadb/server-status-variables/

Page 136: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Status CountersTable Status Created_tmp_disk_tables

Created_tmp_files

Created_tmp_tables

Opened_table_definitions

Opened_tables

Table_locks_immediate

Table_locks_waited

Queries Last_query_cost

Not_flushed_delayed_rows

Prepared_stmt_count

Qcache_*

Queries

Questions

Select_*

Slow_launch_threads

Slow_queries

Sort_*

Transaction Coordinator Tc_log_max_pages_used

Tc_log_page_size

Tc_log_page_waits

Other Status Counters Flush_commands

Handler_*

Innodb_*

Key_*

Open_*

Opened_files

136

Docmentation on Server Status Variables: https://mariadb.com/kb/en/mariadb/server-status-variables/ Documentaton on InnoDB Status Variables: https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-status-variables/

Page 137: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Binary Log Status VariablesFile Information Binlog_snapshot_file

Binlog_snapshot_position

Binlog_bytes_written

Caches Binlog_cache_disk_use

Binlog_cache_use

Binlog_stmt_cache_disk_use

Binlog_stmt_cache_us

Commits Binlog_commits

Binlog_group_commit_trigger_count

Binlog_group_commit_trigger_lock_wait

Binlog_group_commit_trigger_timeout

Binlog_group_commits

137

Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/

Page 138: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Replication Status CountersMaster Status Master_gtid_wait_count

Master_gtid_wait_time

Master_gtid_wait_timeouts

Rpl_status (deprecating)

Slave Status Slave_connections

Slave_heartbeat_period

Slave_open_temp_tables

Slave_received_heartbeats

Slave_retried_transactions

Slave_running

Slave_skipped_errors

Slaves_connected

Slaves_running

Slave Changes Com_change_master

Com_slave_start

Com_slave_stop

Com_start_all_slaves

Com_start_slave

Com_stop_all_slaves

Com_stop_slave

138

Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/

Page 139: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

PERFORMANCE_SCHEMAMonitors MariaDB Server Events

Function Calls, Operating System Waits, Internal Mutexes, I/O Calls

Detailed Query Execution Stages (Parsing, Statistics, Sorting)

Some Features Storage Engine Specific

Monitoring Lightweight and Requires No Dedicated Thread

Not a Simple, Quick Fix

Interpreting Output Requires Knowledge of Internals Designed to be Used Iteratively with Successive

Refinement

139

Documentation on performance_schema: https://mariadb.com/kb/en/mariadb/performance-schema/

Page 140: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

PERFORMANCE_SCHEMA Tablescond_instances

events_waits_current

events_waits_history

events_waits_history_long

events_waits_summary_by_instance

events_waits_summary_by_thread_by_event_name

events_waits_summary_global_by_event_name

file_instances

file_summary_by_event_name

file_summary_by_instance

mutex_instances

performance_timers

rwlock_instances

setup_consumers

setup_instruments

setup_timers

threads

140

Page 141: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

ProfilingUse Profiling to Measure Differences in

Execution between Queries Enable by Setting PROFILING to ON

141

SHOW PROFILE FOR QUERY 1;

+----------------------+----------+

| Status | Duration |

+----------------------+----------+

| starting | 0.000047 |

| checking permissions | 0.000005 |

| Opening tables | 0.000015 |

| After opening tables | 0.000004 |

| System lock | 0.000004 |

| Table lock | 0.000011 |

| init | 0.000017 |

| optimizing | 0.000007 |

| statistics | 0.000012 |

| preparing | 0.000014 |

| executing | 0.000003 |

| Sending data | 0.008304 |

| end | 0.000006 |

| query end | 0.000003 |

| closing tables | 0.000002 |

| Unlocking tables | 0.000007 |

| freeing items | 0.000004 |

| updating status | 0.000015 |

| cleaning up | 0.000014 |

+----------------------+----------+

SET PROFILING=ON;

SELECT * FROM company.employees;

SHOW PROFILES \G

***** 1. row ***** Query_ID: 1

Duration: 0.00849380 Query: SELECT * FROM company.employees

Documentation for SHOW PROFILE: https://mariadb.com/kb/en/mariadb/show-profile/

Page 142: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Slow Query LogContains Queries that took Longer

than long_query_time Seconds to Execute — Off by Default

Activate when First Using an Application to Monitor Slow Queries

Default Location is in Data Directory Use mysqldumpslow to Process Log

Use also pt-query-digest as an Alternative to Process Log

142

[mysqld] log-slow-queries = ON long-query-time = 2

log-queries-not-using-indexes log-slow-admin-statements log-slow-vervosity = query_plan,explain

Documentation on Slow Query Log: https://mariadb.com/kb/en/mariadb/slow-query-log-overview/ Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/ Documentation on pt-query-digest: https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html

Excerpt from /etc/my.cnf.d/server.cnf

Page 143: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

External ToolsApacheBench htop httperf innotop iostat / vmstat / dstat/top Percona Toolkit MyBench

MariaDB Benchmark Suite (sql-bench) mysqladmin (extended-status) mysqlslap mytop statpack Super Smack SysBench

143

ApacheBench: http://httpd.apache.org/docs/2.4/programs/ab.html InnoTop: https://github.com/innotop/innotop MyBench: http://jeremy.zawodny.com/mysql/mysql-query-benchmarking/img13.html mysqlslap: https://mariadb.com/kb/en/mariadb/mysqlslap/ Percona Toolkit: https://www.percona.com/software/database-tools/percona-toolkit

Page 144: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Benchmarking

Page 145: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Tests to RunNumber of Queries Expected per Unit of

Time Number of Concurrent Client

Connections Expected Average Number of Clients Peak Number of Concurrent Clients

Length of a Client Connection Expected — In Milliseconds

Size of Dataset — Fit in Memory When all Requirements will Double

145

Page 146: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

SysbenchOpen-Source Benchmarking Tool Contains oltp and mutex Tests

146

Documentation on sysbench and oltp: https://www.percona.com/docs/wiki/benchmark_sysbench_oltp.html

mysql -e "CREATE DATABASE sbtest"

sysbench --mysql-user=root --test=oltp prepare

sysbench --mysql-user=root --test=oltp run

Page 147: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Establish a Base LineUse Current Configuration Get Stable Results Use Same Hardware and Configuration

if Possible Use Same Size and Distribution Dataset

147

Page 148: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Construct TestsDetermine System Resource Limits and Design

Test Should Run Normally Should Exceed Server Resource Limits Should Simulate Nightly Back-up Load without

Users Noticing

Examine Impact of Configuration Changes Remove or Restrict Access to CPU core or Hard

Drives Start another Process via cron that Consumes

Suddenly Memory Observe What Happens When a Caching Layer

goes Offline

Use Tests to Identify Easy Solutions

148

Page 149: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Examine Test ResultsTest System Identical to Production

System Same Operating System (Version, Libraries,

Patches)

Hardware (Number of Cores, Physical Memory, Disk Speed)

Other Resources (Memory Available to MariaDB, Competing Services)

Test System Simulating Real Data and Traffic

Number of Clients (Concurrently, per Hour per Day, Peak Traffic)

Any Caching that might Skew Results

149

Page 150: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Query Tuning

Page 151: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

The Query Process

Page 152: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB Query Process

Connection Handling

Archive

InnoDB

Aria

Memory

Storage Engines

Query Cache

SQL Parser

Optimizer

Clients

152

Page 153: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Thread Cache

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Thread is Assigned to Each Connection

Threads may be Reused from the Thread Cache or Created

153

Page 154: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Thread Pool

Thread Pool

Query Cache

SQL Parser

Optimizer

Storage Engines

Main Thread Listens for Client Connections User Authentication based on Host, User, and

Password Client Buffers for Session Variables and Network

Communications

154

thread_handling = pool-of-threads thread_pool_size = 3

Excerpt from /etc/my.cnf.d/server.cnf

Documentation on Thread Pool: https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/

Page 155: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Query Cache

Thread Pool

Query Cache

SQL Parser

Optimizer

Storage Engines

Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/

SHOW VARIABLES LIKE 'query_cache_type';

Possible to Stipulate in a Query

Stores SELECT Query Result Sets

Useful for High Read, Low Write Servers

Cache Purged when Related Data Changed

SELECT SQL_CACHE * FROM ...

SELECT SQL_NO_CACHE * FROM ...

155

No Longer Recommended

Page 156: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

SQL Parser

Thread Pool

Query Cache

SQL Parser

Optimizer

Storage Engines

Converts SQL Text to Binary Format Parses SQL into Tokens (i.e., keyword, table,

field, value) Applies Grammar Rules to check

Validity (Lexical Scanner, Grammar Rules Module)

Construct a Parse Tree to be passed to Optimizer

156

Page 157: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Optimizer

Thread Pool

Query Cache

SQL Parser

Optimizer

Storage Engines

157

The Optimizer Reads the Parse Tree and Determines an Execution Plan

Locate any Related Indexes Compare Efficiency of Index Access to

Table Scan Determine JOIN order of Table Eliminate Unnecessary Tables and WHERE

clauses Find Indexes that can handle GROUP BY

and ORDER BY

Page 158: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Query Tuning

Page 159: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Query Tuning OverviewDon't Query Tune on Production Server

Use Test Server with same Hardware and OS

Use Copy of Production Data Set

Query Frequency and Query Speed are Both Important

Frequent Slow Queries are often Worse than Rarely Run Very Slow Query

159

Page 160: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

IndexingIndexes Improve Read Performance

Without Index, MariaDB Must Read Every Row

With Index, MariaDB Jumps to Requested Rows

Indexes Improve Writes since they Require Reads

Index for Speed but Avoid Indexing Excessively or Arbitrarily

Remove Unused or Redundant Indexes

160

Article on Getting Started with Indexes: https://mariadb.com/kb/en/mariadb/getting-started-with-indexes/

Page 161: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Index SizeKeep Indexes as Small as Practical

Faster if Fits in Memory Rebuilds Faster after Writes

Use Partial Prefix-Index for String Columns

Reduces Index Size — May Slow Searches Use Index Cardinality (Uniqueness Measure)

if Necessary — Re-evaluate as Data Grows Low Cardinality Indicates many

Duplicates High Cardinality is More Useful

161

Documentation on Index Statistics: https://mariadb.com/kb/en/mariadb/index-statistics/

Page 162: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Query AnalysisUse EXPLAIN to see how

Troublesome Query Executed Shows if Indexes are Used

Use EXPLAIN EXTENDED and SHOW WARNINGS to see how Query Rearranged before Execution

162

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/

EXPLAIN SELECT * FROM company.clients WHERE name_first = 'John' AND name_last = 'Smith' \G

id: 1 select_type: SIMPLE table: clients type: ref

possible_keys: names key: names key_len: 15 ref: const,const rows: 1

Extra: Using where

EXPLAIN EXTENDED SELECT ...; SHOW WARNINGS;

Page 163: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

ANALYZE vs. EXPLAINEXPLAIN is Expected Execution Plan

ANALYZE Executes and Shows Effect Writes are Executed

163

ANALYZE SELECT * FROM company.clients

WHERE name_first = 'John'

AND name_last = 'Smith' \G

***** 1. row *****

id: 1

select_type: SIMPLE

table: clients

type: ref

possible_keys: names

key: names

key_len: 15

ref: const,const

rows: 1

r_rows: 1.00

filtered: 100.00

r_filtered: 100.00

Extra: Using where

See Upcoming Slides for Lists of Select Types and Query Types

Page 164: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Fields from EXPLAINField Descriptionid Query identifierselect_type Type of SELECT performed

table The table in use (each table in a JOIN will use one row)

type The table access strategy

possible_keys Any available indexes that could resolve the query

key Index MariaDB chose from possible_keyskey_len The portion of the chosen index to be used (width in bytes)ref Columns from the index, or a constant, usedrows Number of rows the query is expected to touchExtra Extra information relating to the access strategy

164

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended

See Next Slide for List of Types

Page 165: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Scan or Index Method Types

Field Description

const A constant value can be read once and cached

eq_ref One index access per outer query row

ref Multiple index accesses per outer query row

index_merge Multiple indexes used, merging into a single result set

range Multiple index accesses to return all rows within a range

index Full index scan (every index entry is read sequentially)

all Full table scan (every record is read sequentially)

Method Used to Access Table

165

Page 166: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Select TypesType of SELECT being Performed

166

Type of SELECT Description

SIMPLE Normal SELECT Statement

PRIMARY First (Outermost) SELECT in JOIN

UNION Subsequent Query in JOIN

DEPENDENT UNION Subsequent Query Dependent on an Outer Query Result

UNION RESULT Query is the Results Set from a UNION

SUBQUERY First SELECT in a Sub-Query

DEPENDENT SUBQUERY A Sub-Query that must be Re-Executed for Each Combination of Values in Outer Query

DERIVED Sub-Query in a FROM Clause

UNCACHEABLE SUBQUERY Sub-Query that must be Re-Executed for Every Iteration of an Outer Query

UNCACHEABLE UNION Subsequent Query in a UNION belonging to an Uncacheable Subquery

Page 167: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Extra Explanationsconst row not found

Distinct

Full scan on NULL key

Impossible HAVING

Impossible WHERE noticed after reading const tables

Impossible WHERE

No matching min/max row

No matching row in const table

No tables used

Not exists

Open_frm_only

Open_full_table

Open_trigger_only

Range checked for each record (index map: N)

Scanned N databases

Select tables optimized away

Skip_open_table

Unique row not found

Using filesort

Using index

Using index condition

Using index condition(BKA)

Using index for group-by

Using intersect(…)

Using sort_union(…)

Using temporary

Using where

Using WHERE with pushed condition

Using buffer

167

Documentation on Extra Field of EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/#extra-column

Page 168: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Selecting All ColumnsSELECT with * Performs Badly

Increases Size of Results Set — Memory and Network Overhead on Client and Server

Increases Disk Activity for Storage Engines that can Read Partial Records (not MyISAM, but InnoDB)

Causes Development and Usage Problems

Ambiguity with JOIN tables having same Column Names

Problems when Schema has been Changed

168

Page 169: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

MariaDB OptimizerOptimizer calculates Best Plan

Can't Take too Long or Defeats Purpose

Many Possible Plans for Large Joins Greedy Searches used to Reduce

Possibilities Possible to Adjust Decision Parameters

Limit Number of Tables Considered (optimizer_search_depth)

Skip Plans Based on Row Count Estimates (optimizer_prune_level)

169

Documentation on optimizer_search_depth: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depth Documentation on optimizer_prune_level: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_level Article on Setting Optimizer Search Depth: https://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql

Page 170: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Optimizer HintsGiving Hints can Affect the Optimizer’s

Choices Index Choices USE INDEX IGNORE INDEX FORCE INDEX

Buffering Methods for GROUP BY SQL_BUFFER_RESULT SQL_SMALL_RESULT SQL_BIG_RESULT

170

SELECT * FROM people FORCE INDEX names WHERE name_first = 'Bob';

Documentation on Forcing Indexes: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/ Documentation on Indexing with GROUP BY: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by

Page 171: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Joining TablesJOIN is a Cartesian Product of Table

Subsets with Variations The Optimizer can Help

Orders Tables for Minimum Cost — Fewest Rows Examined

Converts OUTER to INNER when Possible

Ensure Columns in ON and USING Clauses are Indexed

Restrict GROUP BY and ORDER BY to Columns in same Table

Consider JOIN Decomposition

171

Page 172: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Sub-QueriesMariaDB uses Semi-Join Optimizations

— Queries within IN( ) Subquery Cache used to Optimize

Evaluation of Correlated Subqueries Views & Sub-Queries are Essentially the

Same Potential Performance Drain

172

Documentation on Subquery Optimization: https://mariadb.com/kb/en/mariadb/subquery-optimizations/ Documentation on Semi-Join Optimization: https://mariadb.com/kb/en/mariadb/semi-join-subquery-optimizations/ Documentation on Subquery Cache: https://mariadb.com/kb/en/mariadb/subquery-cache/

Page 173: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Limiting RowsLIMIT is Evaluated at End of Query Plan

ORDER BY with LIMIT needs to Sort All Rows

LIMIT(o, n) will still Parse o + n Rows Scan in Index Order and Apply LIMIT

vs. Sort and LIMIT

173

Documentation on LIMIT Clause: https://mariadb.com/kb/en/mariadb/select/#limit Documentation on LIMIT ROWS EXAMINED: https://mariadb.com/kb/en/mariadb/limit-rows-examined/

Page 174: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Ordering and AggregatingGROUP BY Triggers an Implicit ORDER BY (filesort)

ORDER BY NULL Avoids It

174

EXPLAIN SELECT Continent, COUNT(*) FROM country GROUP BY Continent \G

... Extra: Using temporary; Using filesort

EXPLAIN SELECT Continent, COUNT(*) FROM country

GROUP BY Continent ORDER BY NULL\G ... Extra: Using temporary

Page 175: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Advanced Index Usage

Page 176: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Faster ResultsUnderstanding of Indexes will Help to

Get More from Them Clustered Index InnoDB Secondary Indexes

Indexes can't Resolve Flawed Logic in SQL Statements

176

Page 177: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

B-Tree IndexesBalanced Binary Search Trees holding

Pointers to Rows Precision Depends on Storage Engine Some Store Pointers to Rows (Offsets) InnoDB Primary Key Stores Row — Other

Indexes Store Copy of Key

Good for Many Operations Equality (= <>) Range Access (> < >= <= BETWEEN)

Partial String (i.e., LIKE) — String Constants without Leading %

Sorting (ORDER BY) — Indexes are Pre-Sorted

177

Documentation on Indexes Used by Storage Engines: https://mariadb.com/kb/en/mariadb/storage-engine-index-types/

Page 178: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Secondary Index

178

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY

leaf

178

Article Comparing InnoDB Primary and Secondary Indexes: https://mariadb.com/resources/blog/innodb-primary-key-versus-secondary-index-interesting-lesson-explain

Page 179: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Composite IndexesSpans Multiple Columns Increases Search Speed Best when All Columns in Index

Specified in Query Not Used with Latter Columns of

Index Only One Index Used per Table

179

CREATE TABLE people

(name_first CHAR(20), name_last CHAR(20), city CHAR(20), KEY names(name_first,name_last));

SELECT * FROM people WHERE name_first = 'Bob' AND name_last = 'Smith';

SELECT * FROM people WHERE name_first = 'Bob';

Which Use an Index

SELECT * FROM people WHERE name_last = 'Smith';

Page 180: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Ordering with IndexesUsing an ORDER BY Clause can

Cause an Index Not to be Used

180

CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20),

INDEX names(name_first, name_last), INDEX city (city));

SELECT * FROM table1

WHERE city = 'Boston' ORDER BY name_first;

SELECT * FROM table1 WHERE name_first = 'Bob'

ORDER BY name_last;

Which Use an Index

Page 181: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Aggregating with IndexesUsing a GROUP BY Clause can

Cause an Index Not to be Used

181

CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20),

INDEX names(name_last, name_first), INDEX city (city));

SELECT name_first, COUNT(*)

FROM students WHERE name_last = 'Smith' GROUP BY name_first;

SELECT city, COUNT(*) FROM students WHERE name_first = 'Bob' GROUP BY city;

Which Use an Index

Page 182: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Straddling Columns with IndexesUsing AND in WHERE is Fine when

All Columns in Index can Cause an Index Not to be

Used

182

CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20), country CHAR(2)

INDEX name_city (name_first, name_last, city));

SELECT * FROM table1

WHERE name_first = 'Bob' AND name_last = 'Smith';

SELECT * FROM table1 WHERE name_first = 'Bob'

AND city = 'Boston';

Which Use an Index

SELECT * FROM table1 WHERE name_first = 'Bob' AND country = 'us';

Page 183: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Pagination by SkippingIndexes can Improve Query

Performance Performance can be Lost with

some Clauses (e.g., LIMIT) for Returning Pages of Results

Check Status of rows_read to see clearly Index Usage

183

FLUSH STATUS;

SELECT name_last FROM people WHERE name_first = 'Vito' LIMIT 100, 5;

SHOW STATUS LIKE 'rows_read' \G ... Value: 218907

Add an Index

Without Index

CREATE INDEX name ON people (name_first);

FLUSH STATUS;

SELECT ... LIMIT 100,5; SHOW STATUS LIKE 'rows_read' \G ... Value: 105

Only 100 More than Needed

Page 184: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Pagination by IndexIndexes can Improve Query

Performance Performance can be Lost with

some Clauses (e.g., LIMIT) for Returning Pages of Results

Check Status of rows_read to see clearly Index Usage

184

SELECT people_id , name_last FROM people WHERE name_first = 'Vito'

ORDER BY people_id LIMIT 5;

[results]

SELECT name_last FROM people WHERE name_first = 'Vito' AND people_id > [last_id] ORDER BY people_id LIMIT 5;

Page 185: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Handling Duplicate Row ValuesIndexes can Improve Query

Performance Performance can be Lost with

some Clauses (e.g., LIMIT) for Returning Pages of Results

Check Status of rows_read to see clearly Index Usage

185

SELECT people_id , salary, name_last FROM people WHERE name_first = 'Vito'

ORDER BY salary DESC LIMIT 5;

[results - last 3 same salary]

SELECT people_id , salary, name_last FROM people WHERE name_first = 'Vito' AND salary <= [last_salary] AND people_id NOT IN([id_same_salary])

ORDER BY salary DESC LIMIT 5;

Page 186: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Diagnosing Systems

Page 187: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Common Bottlenecks

Page 188: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Lack of CachingCaching can Occur at Many Levels

188

Applications User Session Data Pre-Compiled, Static Content

Database System Query Cache Storage Engine

Hardware File System Cache Proxy Server Content Distribution Network memcached

Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/ Documentation on InnoDB Buffer Pool: https://mariadb.com/kb/en/mariadb/xtradbinnodb-buffer-pool/

Page 189: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Buffer Pool SizeInnoDB Buffer Pool is a Key

Component for Optimizing MariaDB

Stores Data and Indexes Keeping in Memory Reduces Disk

IO — a Main Bottleneck

Buffer Pool keeps Frequently Used Blocks in the Buffer — Recently Used (63%) and Older Information (37%)

189

Documentation on InnoDB Buffer Pool: https://mariadb.com/kb/en/mariadb/xtradbinnodb-buffer-pool/

SHOW STATUS LIKE 'innodb_buffer_pool_read%';

+---------------------------------------+----------+

| Variable_name | Value |

+---------------------------------------+----------+

| Innodb_buffer_pool_read_ahead | 8848 |

| Innodb_buffer_pool_read_ahead_evicted | 185 |

| Innodb_buffer_pool_read_ahead_rnd | 0 |

| Innodb_buffer_pool_read_requests | 24549645 |

| Innodb_buffer_pool_reads | 177895 |

+---------------------------------------+----------+

Page 190: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Slow StartSlow at Start could be from Cold Index

& Data Caches — Try Preloading InnoDB Clustered Index InnoDB without PRIMARY Indexes

For Speed, Try Preloading PK Ranges in Parallel

190

SELECT COUNT(*) FROM table1 WHERE idx_col LIKE '%0%';

SELECT COUNT(*) FROM table1 WHERE non_idx_col=0;

Page 191: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Temporary Tables on DiskMEMORY Converts to MyISAM on

Disk if Temporary Table Not Possible

Check if Temporary Disk Tables is Incrementing using SHOW STATUS

Execute SHOW PROCESSLIST to find Threads Creating tmp table on disk

191

SHOW STATUS LIKE 'Created_tmp_%';

+-------------------------+-------+ | Variable_name | Value |

+-------------------------+-------+ | Created_tmp_disk_tables | 0 | | Created_tmp_files | 45 | | Created_tmp_tables | 1 | +-------------------------+-------+

Page 192: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Resolve Disk Temporary TablesMove TEXT and BLOB Columns to

another Table Make Large VARCHAR Columns as Small

as Possible Increase tmp_table_size and max_heap_table_size

192

SHOW VARIABLES

WHERE Variable_Name LIKE 'tmp_table_size' OR Variable_Name LIKE 'max_heap_table_size';

+---------------------+----------+ | Variable_name | Value | +---------------------+----------+ | max_heap_table_size | 16777216 | | tmp_table_size | 16777216 |

+---------------------+----------+

Page 193: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Maximum Connections ExeededCreating Connections has Little Impact

if Threads are Cached Use Thread Pool for Short-Lived

Workloads Session Buffers Allocated Only when

Needed

193

Documentation on Handling Excessive Connections: https://mariadb.com/kb/en/mariadb/handling-too-many-connections/

SHOW STATUS WHERE Variable_Name

LIKE 'Threads_connected' OR Variable_Name LIKE 'max_used_connections';

+----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 10 | | Threads_connected | 2 |

+----------------------+-------+

Page 194: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Domain Name ServiceDNS and Reverse DNS Used for IP and

Hostname with Client Connections Slow DNS Server can Slow MariaDB

Client Connection Local DNS Server Improves Lookup

Speed and Security Disable DNS Lookup

Use Only IP Addresses in Permissions Tables

194

[mysqld] skip-name-resolve

Disable DNS Lookup

Page 195: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

InnoDB Adaptive Hash IndexInnoDB performs like In-Memory

Database with AHI Leads to High CPU Usage

Mutex for Adaptive Hash Index

195

SHOW ENGINE INNODB MUTEX;

+--------+------------------------------+---------------+

| Type | Name | Status |

+--------+------------------------------+---------------+

| InnoDB | &rseg->mutex | os_waits=1 |

| InnoDB | &dict_sys->mutex | os_waits=1 |

| InnoDB | &log_sys->mutex | os_waits=210 |

| InnoDB | &buf_pool->flush_state_mutex | os_waits=5095 |

| InnoDB | &buf_pool->LRU_list_mutex | os_waits=41 |

| InnoDB | combined &block->mutex | os_waits=11 |

| InnoDB | &new_index->lock | os_waits=2 |

| InnoDB | &new_index->lock | os_waits=16 |

| InnoDB | &new_index->lock | os_waits=1 |

| InnoDB | &dict_operation_lock | os_waits=221 |

| InnoDB | &log_sys->checkpoint_lock | os_waits=2920 |

| InnoDB | &btr_search_latch_arr[i] | os_waits=6 |

| InnoDB | combined &block->lock | os_waits=62 |

+--------+------------------------------+---------------+

[mysqld] innodb_adaptive_hash_index=OFF

Page 196: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Sort BufferMemory Allocated for Sorting

Results Check the sort_merge_passes

Status to get Number of Merge Passes Performed

Set sort_buffer_size to Higher Value if Merge Passes is High

196

SHOW STATUS

LIKE 'sort_merge_passes';

+-------------------+-------+

| Variable_name | Value |

+-------------------+-------+

| Sort_merge_passes | 0 |

+-------------------+-------+

SHOW VARIABLES LIKE 'sort_buffer_size';

+------------------+---------+

| Variable_name | Value |

+------------------+---------+

| sort_buffer_size | 2097152 |

+------------------+---------+

Page 197: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Troubleshooting

Page 198: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Error LogContain Startup, Shutdown, Error

Messages Unix uses stderr, Sent to host_name.err in datadir

Some Unix systems Redirect to System Log

Windows uses host_name.err in datadir, or System Event Log

198

Set log-warnings to 2 for Verbose

Documentation on Error Log: https://mariadb.com/kb/en/mariadb/error-log/

Error Log

General Query Log

SQL Query Log

Slow Query Log

Binary Log

mysqld

[mysqld]

log-error = /path/file_name log-warnings = 1

Excerpt from my.cnf or my.ini configuration file

Page 199: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

General Query LogAll Queries Received from All Clients Order Received — Not Execution Potential Problems with Log

Possible Disk I/O bottleneck on High-Traffic Servers

Log can Quickly become Huge Contains Queries in Plain Text — a

Security Risk

199

Documentation on General Query Log: https://mariadb.com/kb/en/mariadb/general-query-log/

Error Log

General Query Log

SQL Error Log

Slow Query Log

Binary Log

mysqld

Excerpt from my.cnf or my.ini configuration file

[mysqld] general-log

general_log_file='/path/host.log'

Page 200: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

SQL Error LogLogs SQL Errors

Error Messages SQL Statement

Part of MariaDB Audit Plugin Used for Detecting SQL Injections

Logs User, Host, & Time

200

Documentation on SQL Query Log: https://mariadb.com/kb/en/mariadb/sql_error_log-plugin/

Error Log

Slow Query Log

Binary Log

mysqld SQL Error Log

General Query Log

INSTALL PLUGIN sql_error_log SONAME 'sql_errlog';

Requires INSERT Privilege on mysql.plugin Table

Page 201: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Binary LogWrites in the Order Executed Useful for Recovering Lost Data Used for Replication Move from to Separate Disk for

Security Set Expiration or Purge Occasionally

201

Documentation on Binary Log: https://mariadb.com/kb/en/binary-log/

Error Log

General Query Log

Binary Log

Slow Query Log

SQL Error Logmysqld

[mysqld] log-bin

expire_logs_days = 7

Excerpt from my.cnf or my.ini configuration file

Page 202: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Out of Memory (OOM Killer)

202

Page 203: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Galera Self Leaves

203

Page 204: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Stack Traces

204

Page 205: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Find Bug by BacktraceFind Identifying Part Search Issue Tracker (https://jira.mariadb.org)

Confirm Results and Watch Issue Check Comments for Work-Around

205

Page 206: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Conclusion

Page 207: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

On-Line ResourcesDocumentation Knowledge Base Open-Source Community

Post Questions on Forums Ask Questions on IRC — FreeNode

207

MariaDB Documentation and Knowledge Base: http://mariadb.com/kb

Page 208: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

[email protected]

Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

Advanced MariaDB for DBAs

Other Training CoursesPublic In-Person Courses Private In-Person Courses Live Virtual Training Courses

208

Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions

Page 209: MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch Replication Manager Client Proxies Optimizing Overview Schema Tuning

Advanced MariaDB

Introduction

MariaDB Replication

GTIDComplex Scenarios

Semi-Synch

Replication Manager

Client ProxiesOptimizing Overview

Schema Tuning

InnoDB In-Depth Collecting Information

Benchmarking

The Query Process

Query Tuning

Advanced Index UsageCommon Bottlenecks

Troubleshooting

Hacking with GNU Debugger

Conclusion

Thanks for Participating