Percona Toolkit for Effective MySQL Administration

44
Percona Toolkit for Effective MySQL Administration By MyDBOPS Remote DBA Team

Transcript of Percona Toolkit for Effective MySQL Administration

Page 1: Percona Toolkit for Effective MySQL Administration

Percona Toolkit for Effective MySQL Administration

By

MyDBOPS Remote DBA Team

Page 2: Percona Toolkit for Effective MySQL Administration

● MyDBOPS is on MySQL Consulting with core specialization on MySQL Administration and Support.

● MyDBOPS is formed by a group of MySQL and open source enthusiasts from TOP tech companies who have scaled TB’s of data in MySQL.

● MyDBOPS was created with a motto of developing a Devops model for MySQL administration. We help organisations to scale in MySQL and implement the advance technologies in MySQL.

About Us

Page 3: Percona Toolkit for Effective MySQL Administration

Agenda

● Introduction

● Installation of Percona tool kit ( Linux )

● Tools

➢ pt-config-diff➢ pt-query-digest➢ pt-table-sync➢ pt-summary➢ pt-duplicate-key-checker➢ pt-kill➢ pt-mysql-summary➢ pt-table-checksum➢ pt-online-schema-change

Page 4: Percona Toolkit for Effective MySQL Administration

INTRODUCTION

The Percona toolkit is a free, open source tool which used to monitor the mysql server and make the complex MySQL operations easier than normal way

Some of the operations :

● It verify master and slave data consistency.

● Find duplicate indexes.

● Summarize MySQL servers and linux server.

● Analyze queries from logs.

● It verify the config file differences.

Page 5: Percona Toolkit for Effective MySQL Administration

Installing Percona Toolkit

Percona package repos for most of Linux Flavour.

$ sudo yum install percona-toolkit ( RPM Based server )

$ sudo dpkg install percona-toolkit ( Debian Based )

http://www.percona.com/software/repositories

Other download options available for RPM, DEB, tarball, or individual tools.

http://www.percona.com/doc/percona-toolkit/ installation.html

Page 6: Percona Toolkit for Effective MySQL Administration

Tools

pt-config-diff pt-query-digest pt-table-sync

pt-summary pt-duplicate-key-checker pt-kill

pt-mysql-summary pt-table-checksum pt-online-schema-change

Page 7: Percona Toolkit for Effective MySQL Administration

pt-config-diff

Page 8: Percona Toolkit for Effective MySQL Administration

Find The Differences

Page 9: Percona Toolkit for Effective MySQL Administration

pt-config-diff

● It gives the differences of MySQL configuration files and server variables. CONFIG can be a filename or a DSN. At least two CONFIG sources must be given.

● Like standard Unix diff, there is no output if there are no differences.

Diff host1 config from SHOW VARIABLES against host2:

pt-config-diff “h=host1” “h=host2”

Diff config from [mysqld] section in my.cnf against host1 config:

pt-config-diff /etc/my.cnf h=host1

Diff the [mysqld] section of two option files:

pt-config-diff /etc/my-new.cnf /etc/my-old.cnf

Page 10: Percona Toolkit for Effective MySQL Administration

Difference between two config files :

$ pt-config-diff /etc/my-old.cnf /etc/my-new.cnf3 config differencesVariable /etc/my-old.cnf /etc/my-new.cnf========================= =============== ===============basedir /usr/local/mysql1 /usr/local/mysql2datadir /var/lib/mysql1 /var/lib/mysql2port 3306 3307

pt-config-diff

Page 11: Percona Toolkit for Effective MySQL Administration

Difference between two hosts :

$ pt-config-diff “10.20.30.40,u=root,p=root” “10.20.30.41,u=root,p=root”15 config differencesVariable Cent_Node1 khosla========================= ========================= =========================auto_increment_increment 3 1back_log 750 150binlog_cache_size 65536 32768binlog_format ROW STATEMENTcharacter_sets_dir /usr/share/percona-xtr... /usr/share/mysql/chars...datadir /data/mysql/ /var/lib/mysql/event_scheduler OFF ONexpire_logs_days 14 0explicit_defaults_for_... OFF ONgeneral_log_file /data/mysql/Cent_Node1... /var/lib/mysql/khosla.loghost_cache_size 778 628hostname Cent_Node1 khoslainnodb_autoinc_lock_mode 2 1innodb_buffer_pool_dum... ON OFFinnodb_buffer_pool_loa... ON OFF

pt-config-diff

Page 12: Percona Toolkit for Effective MySQL Administration

pt-summary

Page 13: Percona Toolkit for Effective MySQL Administration

pt-summary

● It summarizes the status and configuration of a server.

● It gives top running process and hardware informations.

● This tool works well on many types of Unix systems.

$ pt-summary # Percona Toolkit System Summary Report ###################### Date | 2016-01-08 23:22:42 UTC (local TZ: IST +0530) Hostname | [email protected] Uptime | 7:24, 4 users, load average: 0.48, 0.64, 0.58 System | Gigabyte Technology Co., Ltd.; To be filled by O.E.M.; vTo be filled by O.E.M. (Desktop) Service Tag | To be filled by O.E.M. Platform | Linux Release | Fedora release 21 (Twenty One) Kernel | 4.1.13-100.fc21.x86_64Architecture | CPU = 64-bit, OS = 64-bit Threading | NPTL 2.20 Compiler | GNU CC version 4.9.2 20150212 (Red Hat 4.9.2-6). SELinux | Enforcing Virtualized | No virtualization detected

Page 14: Percona Toolkit for Effective MySQL Administration

# Processor ################################################## Processors | physical = 1, cores = 2, virtual = 2, hyperthreading = no Speeds | 1x1695.468, 1x1698.632 Models | 2xIntel(R) Pentium(R) CPU G2030 @ 3.00GHz Caches | 2x3072 KB# Memory ##################################################### Total | 3.8G Free | 1.5G Used | physical = 1.7G, swap allocated = 3.9G, swap used = 2.2G, virtual = 3.9G Buffers | 563.1M Caches | 1.7G Dirty | 76 kB UsedRSS | 2.5G Swappiness | 60 DirtyPolicy | 20, 10 DirtyStatus | 0, 0

pt-summary

Page 15: Percona Toolkit for Effective MySQL Administration

Locator Size Speed Form Factor Type Type Detail ========= ======== ================= ============= ============= =========== ChannelA-DIMM0 4096 MB 1333 MHz DIMM DDR3 Synchronous ChannelB-DIMM0 {EMPTY} Unknown DIMM Unknown None # Top Processes ############################################## PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2588 dhanase+ 20 0 1943588 111592 22592 S 20.0 2.8 8:06.26 gnome-shell 2925 dhanase+ 20 0 595064 20384 2332 S 6.7 0.5 10:11.08 skype 1 root 20 0 189044 4152 2148 S 0.0 0.1 0:02.40 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.20 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+ 7 root 20 0 0 0 0 R 0.0 0.0 0:07.49 rcu_sched 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 0:06.04 rcuos/0# Notable Processes ########################################## PID OOM COMMAND 1443 -17 sshd# The End ####################################################

pt-summary

Page 16: Percona Toolkit for Effective MySQL Administration

pt-mysql-summary

Page 17: Percona Toolkit for Effective MySQL Administration

pt-mysql-summary

● It summarizes the status and configuration of a MySQL database server.

$ pt-mysql-summary # Percona Toolkit MySQL Summary Report ####################### System time | 2016-01-08 23:22:20 UTC (local TZ: IST +0530)# Instances ################################################## Port Data Directory Nice OOM Socket ===== ========================== ==== === ====== 3306 /var/lib/data 0 0 /tmp/mysql.sock# MySQL Executable ########################################### Path to executable | /usr/local/mysql/bin/mysqld Has symbols | Yes# Report On Port 3306 ######################################## User | root@[email protected] Time | 2016-01-09 04:52:19 (IST) Hostname | [email protected] Version | 5.6.26 MySQL Community Server (GPL) Built On | linux-glibc2.5 x86_64 Started | 2016-01-08 21:42 (up 0+07:09:24) Databases | 4 Datadir | /var/lib/data/ Processes | 4 connected, 1 running Replication | Is not a slave, has 0 slaves connected Pidfile | /var/lib/data/localhost.pid (exists)

Page 18: Percona Toolkit for Effective MySQL Administration

# Processlist ################################################

Command COUNT(*) Working SUM(Time) MAX(Time) ------------------------------ -------- ------- --------- --------- Query 1 1 0 0 Sleep 3 0 60000 25000

User COUNT(*) Working SUM(Time) MAX(Time) ------------------------------ -------- ------- --------- --------- asdfasfd 1 0 0 0 root 2 1 0 0 test 1 0 0 0

Host COUNT(*) Working SUM(Time) MAX(Time) ------------------------------ -------- ------- --------- --------- localhost 4 1 0 0

db COUNT(*) Working SUM(Time) MAX(Time) ------------------------------ -------- ------- --------- --------- NULL 4 1 0 0

State COUNT(*) Working SUM(Time) MAX(Time) ------------------------------ -------- ------- --------- --------- 3 0 0 0 init 1 1 0 0

pt-mysql-summary

Page 19: Percona Toolkit for Effective MySQL Administration

# Configuration File ######################################### Config File | /etc/my.cnf

[mysqld]basedir = /usr/local/mysqldatadir = /var/lib/dataport = 3306server_id = 4socket = /tmp/mysql.sockinnodb_flush_method = O_DIRECTdefault-storage-engine = innodbsql_mode = NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES # The End ####################################################

pt-mysql-summary

Page 20: Percona Toolkit for Effective MySQL Administration

pt-query-digest

Page 21: Percona Toolkit for Effective MySQL Administration
Page 22: Percona Toolkit for Effective MySQL Administration

pt-query-digest

● It analyzes MySQL queries from slow, general, and binary log files.

● It can also analyze queries from SHOW PROCESSLIST.

● By default, queries are grouped by fingerprint and reported in descending order of query time (i.e. the slowest queries first).

● If no FILES are given, the tool reads STDIN.

● The optional DSN is used for certain options like --since and --until.

Page 23: Percona Toolkit for Effective MySQL Administration

$ pt-query-digest mysql-slow.log-20160108

# 490ms user time, 60ms system time, 26.68M rss, 215.63M vsz# Current date: Sat Jan 2 10:27:51 2016# Hostname: Cent_Node2# Files: /data/log/mysql-slow.log-20160102# Overall: 468 total, 15 unique, 0.01 QPS, 0.16x concurrency _____________# Time range: 2016-01-01 10:00:07 to 22:55:06# Attribute total min max avg 95% stddev median# ============ ======= ======= ======= ======= ======= ======= =======# Exec time 7460s 2s 369s 16s 144s 47s 3s# Lock time 107ms 0 2ms 227us 568us 237us 185us# Rows sent 4.14M 0 35.27k 9.07k 22.45k 11.74k 0.99# Rows examine 1.11G 0 274.27M 2.43M 1.20M 17.07M 1.14M# Rows affecte 7 0 1 0.01 0 0.12 0# Bytes sent 751.05M 11 19.56M 1.60M 4.26M 2.29M 3.88k# Query size 466.51k 6 9.31k 1020.74 1.78k 921.93 1.33k

pt-query-digest

Page 24: Percona Toolkit for Effective MySQL Administration

# Profile# Rank Query ID Response time Calls R/Call V/M Item# ==== ================== =============== ===== ======== ===== ===========# 1 0x813031B8BBC3B329 5189.9245 69.6% 154 33.7008 13... COMMIT# 2 0x8618BCB1F7B137BF 709.1529 9.5% 3 236.3843 36.47 SELECT master.organization master.organization_devices transaction_log.tx_master transaction_log.tx_result master.geo_heirarchy_type master.geo_heirarchy# 3 0xA0CA1BBEDCD1B80D 442.7764 5.9% 156 2.8383 0.16 SELECT master.organization_attribute master.user_attribute master.organization_devices master.user_attribute master.organization master.user master.mapping_organization_address master.address master.user master.address master.mapping_organization_address# MISC 0xMISC 298.8991 4.0% 16 18.6812 0.0 <9 ITEMS>

Explanations:

Column Meaning============ ==========================================================Rank The query's rank within the entire set of queries analyzedQuery ID The query's fingerprintResponse time The total response time, and percentage of overall totalCalls The number of times this query was executedR/Call The mean response time per executionV/M The Variance-to-mean ratio of response timeItem The distilled query

pt-query-digest

Page 25: Percona Toolkit for Effective MySQL Administration

# Query 1: 0.00 QPS, 0.14x concurrency, ID 0x813031B8BBC3B329 at byte 722464# Scores: V/M = 133.51# Time range: 2016-07-01 11:14:29 to 21:33:15# Attribute pct total min max avg 95% stddev median# ============ === ======= ======= ======= ======= ======= ======= =======# Count 32 154# Exec time 69 5190s 2s 232s 34s 213s 67s 6s# Lock time 0 0 0 0 0 0 0 0# Rows sent 0 0 0 0 0 0 0 0# Rows examine 0 0 0 0 0 0 0 0# Rows affecte 0 0 0 0 0 0 0 0# Bytes sent 0 1.65k 11 11 11 11 0 11# Query size 0 924 6 6 6 6 0 6# String:# Databases transactio... (105/68%), collection... (10/6%)... 8 more# Hosts 10.240.175.35 (137/88%), 10.240.213.146 (11/7%)... 1 more# Last errno 0# Users appuser (137/88%), np_user (11/7%)... 1 more# Query_time distribution# 1us# 10us# 1ms# 10ms# 1s ################################################################# 10s+ #######################################commit\G

pt-query-digest

Page 26: Percona Toolkit for Effective MySQL Administration

pt-query-digest

● One best tool a DBA should have.

● The queries can be visualized using Box anemometer.

● It over tcpdump too.

Page 27: Percona Toolkit for Effective MySQL Administration

pt-duplicate-key-checker

Page 28: Percona Toolkit for Effective MySQL Administration

pt-duplicate-key-checker

● It examines MySQL tables for duplicate or redundant indexes.

● Connection options are read from MySQL option files.

● It also gives the suggestions to drop those duplicate indexes.

● Dropping index can help you to save disk space.

● Optimizer can have better statistics to choose the right index if duplicates were removed.

Page 29: Percona Toolkit for Effective MySQL Administration

$ pt-duplicate-key-checker --database test# ######################################################################### test.t1 # ########################################################################

# name_id is a duplicate of name_idx# Key definitions:# KEY `name_id` (`Name`)# KEY `name_idx` (`Name`),# Column types:# `name` varchar(20) default null# To remove this duplicate index, execute:

ALTER TABLE `test`.`t1` DROP INDEX `name_id`;

# ######################################################################### Summary of indexes # ########################################################################

# Size Duplicate Indexes 69# Total Duplicate Indexes 1# Total Indexes 2

pt-duplicate-key-checker

Page 30: Percona Toolkit for Effective MySQL Administration

pt-table-checksum

Page 31: Percona Toolkit for Effective MySQL Administration

pt-table-checksum

● It performs an online replication consistency check by executing checksum queries on the master and slave.

● The optional DSN specifies the master host.

Steps :

● First it checks for replication lag.● Select datas in small chunks. ( mostly based on PK )● Make checksum on chunk with replace into select.● Displays the output.

Page 32: Percona Toolkit for Effective MySQL Administration

$ pt-table-checksum u=pt_check,p=pt@check,P=3306 --set-vars innodb_lock_wait_timeout=200 --no-check-binlog-format –databases=test --tables=t1

TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE01-07T05:37:58 0 3 7365324 98 0 55.747 test.t1

pt-table-checksum

Page 33: Percona Toolkit for Effective MySQL Administration

pt-table-sync

Page 34: Percona Toolkit for Effective MySQL Administration

pt-table-sync

● Synchronizes the data between the master and slave servers.

● It is a very effective tool and it can be used along with pt-table-checksum to make the data sync.

Steps :

● First it checks for replication lag and delay. ● Make the datas into small chunks. ● Then synchronizes the data.

Page 35: Percona Toolkit for Effective MySQL Administration

$ pt-table-sync --sync-to-master u=pt_check,p=pt@check,P=3306,h=192.168.1.2 --databases=test --tables=t1 --print > t1.sql

● The t1.sql contains the missing transactions from master.

● Import the sql file to slave. And then re-run the pt-table-checksum.

$ pt-table-checksum u=pt_check,p=pt@check,P=3306 --set-vars innodb_lock_wait_timeout=200 --no-check-binlog-format –databases=test --tables=t1

TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE01-07T05:49:24 0 0 7365324 98 0 46.374 test.t1

● The data gets consistent.

pt-table-sync

Page 36: Percona Toolkit for Effective MySQL Administration

pt-kill

Page 37: Percona Toolkit for Effective MySQL Administration

pt-kill

● It kill MySQL queries that match certain criteria.

● You can specify by busy-time, match-command,interval etc.,

● It can be deamonized too.

For busy queries (Print):$ pt-kill --busy-time 60 --print

For busy queries (Kill):$ pt-kill --busy-time 60 --kill

For sleep process :$ pt-kill --match-command Sleep --print

Page 38: Percona Toolkit for Effective MySQL Administration

pt-online-schema-change

Page 39: Percona Toolkit for Effective MySQL Administration

pt-online-schema-change

● It helps in modifying the table structure without much locking.

● Modifying,altering,adding a column and altering the table engine

can be easily done by this pt-osc.

● Support the table with foreign keys

● Support Galera Cluster too.

● 4X slower than normal alter.

● Data is copied to new table in small chunks.

Note : With latest MySQL 5.6 / 5.7 most of the DDL operation has become

online.

Page 40: Percona Toolkit for Effective MySQL Administration

Process Flow in Online Schema Change

Page 41: Percona Toolkit for Effective MySQL Administration

Altering Engine :

$ pt-online-schema-change --alter "ENGINE=INNODB" D=world,t=test --execute

Creating triggers...Created triggers OK.Copying approximately 3081121 rows...Copying `world`.`test`: 0% 01:15:42 remainCopying `world`.`test`: 1% 01:04:17 remainCopying `world`.`test`: 2% 01:02:28 remainCopying `world`.`test`: 3% 01:00:34 remain.......Copying `world`.`test`: 97% 03:05 remainCopying `world`.`test`: 97% 02:35 remainCopying `world`.`test`: 99% 00:54 remainCopying `world`.`test`: 99% 00:23 remain2016-01-04T14:46:14 Copied rows OK.2016-01-04T14:46:14 Analyzing new table...2016-01-04T14:46:27 Swapping tables...2016-01-04T14:46:27 Swapped original and new tables OK.2016-01-04T14:46:27 Dropping old table...2016-01-04T14:46:35 Dropped old table `world`.`_test_old` OK.2016-01-04T14:46:36 Dropping triggers...2016-01-04T14:46:36 Dropped triggers OK.Successfully altered `world`.`test`.

pt-online-schema-change

Page 42: Percona Toolkit for Effective MySQL Administration

How it works ?

● Create a new table with modified structure.

● Creates triggers for data changes. ( Insert/Update/delete )

● Copies data in small chunks.

● Swaps the table.

● Drop the old table and triggers.

pt-online-schema-change

Page 43: Percona Toolkit for Effective MySQL Administration

More Tools

pt-archiver pt-kill pt-slave-findpt-collect pt-log-player pt-slave-restartpt-mext pt-stalkpt-deadlock-logger pt-diskstats pt-slave-delay pt-visual-explainpt-index-usage

Note: Please read the manual and test it before using it on production.

pt-online-schema-change pt-pmp pt-fifo-split pt-query-advisor pt-tcp-modelpt-find pt-trendpt-fk-error-logger pt-show-grants pt-upgradept-heartbeat pt-sift pt-variable-advisor

Page 44: Percona Toolkit for Effective MySQL Administration

Thank You