A TokuDB Migration Case Study - Percona...The Pythian Group Inc., 2017 A TokuDB Migration Case Study...

The Pythian Group Inc., 2017

A TokuDB Migration Case StudyRonald BradfordSenior MySQL Database Consultant

Percona Live Dublin - Sep 2017

1


Overview

• Background (TokuDB, requirements, environment)

• Evaluation (Other options, Backups, Release cadence)

• Production Upgrade Considerations

• TokuDB Outcomes and Future Benefits

• Post Migration Situations

• Wrapup

2


Background

3


What is TokuDB?

• MySQL transactional storage engine• Developed at MIT, owned now by Percona

• Available in Percona & MariaDB

• Based on a Fractal Tree Index

• Most recognized for high compression capabilities

https://en.wikipedia.org/wiki/TokuDBhttps://www.percona.com/software/mysql-database/percona-tokudb

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_intro.html

4

https://en.wikipedia.org/wiki/TokuDB

https://www.percona.com/software/mysql-database/percona-tokudb

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_intro.html


Why consider TokuDB?

• Evaluation of a high compression solution• 1+TB InnoDB footprint very quickly

▪ Primary table 350M rows▪ Largest child table 5B rows

• Limited VM environment management

• Long data retention requirement

5


Other Options Evaluated

• NoSQL Solutions

• TokuDB Compression (multiple levels)

• InnoDB Compression

• RocksDB Compression

• Stay with no compression

• Subsequently• Architectural improvements

6


Environment in place

• Already using Percona 5.6.26

• Running on RedHat 6.6

• Percona Server installed via rpm

7

Very easy to get started


Requirements to consider TokuDB

• Upgrade to 5.6.35 first

• Install the TokuDB package (rpm)

• Implement Dependencies • jemalloc

• selinux

• Foreign Keys are not supported

8


Change of backup process

• Xtrabackup does not work with TokuDB

• TokuDB offers a hot backup• Required wrapper work

• Longer term - Reverted to using LVM snapshots• mylvmbackup

9


Evaluation

10


Upgrade to current version

• Percona 5.6.35• Multiple bug fixes

• Interesting Hot Backup issue fixed in 5.6.36

11

https://www.percona.com/doc/percona-server/5.6/release-notes/release-notes_index.html

https://www.percona.com/doc/percona-server/5.6/release-notes/release-notes_index.html


Installation

• TokuDB Engine

• TokuDB Backup Plugin

• rpm install

• OS requirements (SELinux)

• OS dependencies (jemalloc - MySQL restart)

• Enable engine (ps_tokudb_admin)

• Enable backup plugin (second MySQL restart)

https://www.percona.com/doc/percona-server/5.6/tokudb/toku_backup.html

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_installation.html

12

https://www.percona.com/doc/percona-server/5.6/tokudb/toku_backup.html



Installation How To

• Try out a TokuDB environment

vagrant init centos/6; vagrant up --provider virtualboxvagrant sshcurl -sL http://bit.ly/EM_tokudb | sh -

# curl -sL https://gist.githubusercontent.com/bradfordpythian/60036b603cf1973c29a0516eead55bc4/raw/099bc193b9b5d1381c258d4459ca8e75d9bb5edd/install_tokudb.sh | sh -

https://gist.github.com/bradfordpythian/60036b603cf1973c29a0516eead55bc4

13

https://gist.github.com/bradfordpythian/60036b603cf1973c29a0516eead55bc4


14

# This snippet performs the following for RedHat/CentOS/OL 6.x system## - Clean Installation of Percona Server 5.6.26# - Clean Installation of TokuDB for said version (e.g. if already exists on server)# - Configuration and verification of TokuDB#…

# Download and Install Percona Servercd /tmpwget https://www.percona.com/downloads/Percona-Server-5.6/Percona-Server-5.6.26-74.0/binary/redhat/6/x86_64/Percona-Server-5.6.26-74.0-r32f8dfd-el6-x86_64-bundle.tartar xvf Percona-Server-5.6.26-74.0-r32f8dfd-el6-x86_64-bundle.tarsudo rpm -ivh Percona-Server-client-56-5.6.26-rel74.0.el6.x86_64.rpm Percona-Server-server-56-5.6.26-rel74.0.el6.x86_64.rpm Percona-Server-shared-56-5.6.26-rel74.0.el6.x86_64.rpmrpm -qa | grep -i perconamysql --version

# Start and Verify Percona Serversudo service mysql startsudo chkconfig mysql onmysql -uroot -e "SELECT VERSION(),@@hostname;"mysql -uroot -e "SHOW ENGINES" | grep -i tokudb

cd /tmp# See https://centos.pkgs.org/6/epel-x86_64/jemalloc-3.6.0-1.el6.x86_64.rpm.htmlwget http://dl.fedoraproject.org/pub/epel/6/x86_64//jemalloc-3.6.0-1.el6.x86_64.rpmsudo rpm -ivh jemalloc-3.6.0-1.el6.x86_64.rpm

# jemalloc >= 3.3.0 dependencysudo rpm -ivh Percona-Server-tokudb-56-5.6.26-rel74.0.el6.x86_64.rpm

if [ `sestatus | grep "^Current.*enforcing" | wc -l` -eq 1 ]then sudo setenforce permissive sudo sed -i "s/^SELINUX=enforcing/SELINUX=permissive/" /etc/selinux/configfi

sudo service mysql restartsudo ps_tokudb_admin --enable -urootmysql -uroot -e "SELECT VERSION(), @@tokudb_version, @@hostname;"mysql -uroot -e "SHOW ENGINES" | grep -i tokudb

mysql -uroot -e "CREATE SCHEMA IF NOT EXISTS test;USE test; DROP TABLE IF EXISTS t1; CREATE TABLE t1(i1 INT) ENGINE=TokuDB;SHOW CREATE TABLE t1;"


TokuDB rpm installation

15

$ sudo rpm -ivh Percona-Server-tokudb-56-5.6.26-rel74.0.el6.x86_64.rpmwarning: Percona-Server-tokudb-56-5.6.26-rel74.0.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID cd2efd2a: NOKEYPreparing... ########################################### [100%] 1:Percona-Server-tokudb-5########################################### [100%]

* This release of Percona Server is distributed with TokuDB storage engine. * Run the following script to enable the TokuDB storage engine in Percona Server:

ps_tokudb_admin --enable -u <mysql_admin_user> -p[mysql_admin_pass] [-S <socket>] [-h <host> -P <port>]

* See http://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_installation.html for more installation details

* See http://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_intro.html for an introduction to TokuDB




ps_tokudb_admin Error Checking

16

$ sudo ps_tokudb_admin --enable -urootChecking SELinux status...ERROR: SELinux is in enforcing mode and needs to be disabled (or put into permissive mode) for TokuDB to work correctly.

...

$ sudo ps_tokudb_admin --enable -urootChecking SELinux status...INFO: SELinux is in permissive mode.

Checking if Percona Server is running with jemalloc enabled...ERROR: Percona Server is not running with jemalloc, please restart mysql service to enable it and then run this script...


ps_tokudb_admin Execution

17

$ sudo ps_tokudb_admin --enable -urootChecking SELinux status...INFO: SELinux is in permissive mode.

Checking if Percona Server is running with jemalloc enabled...INFO: Percona Server is running with jemalloc enabled.

Checking transparent huge pages status on the system...INFO: Transparent huge pages are enabled (should be disabled).

Checking if thp-setting=never option is already set in config file...INFO: Option thp-setting=never is not set in the config file. (needed only if THP is not disabled permanently on the system)

Checking TokuDB engine plugin status...INFO: TokuDB engine plugin is not installed.

Disabling transparent huge pages for the current session...INFO: Successfully disabled transparent huge pages for this session.

Adding thp-setting=never option into /etc/my.cnfINFO: Successfully added thp-setting=never option into /etc/my.cnf

Installing TokuDB engine...INFO: Successfully installed TokuDB engine plugin.


Compression types

• DEFAULT (zlib)

• FAST (quicklz)

• SMALL (lzma)

• Snappy

• Uncompressed

18

CREATE SCHEMA IF NOT EXISTS test;USE test;DROP TABLE IF EXISTS t;CREATE TABLE t (i1 SERIAL) ENGINE=InnoDB;SHOW CREATE TABLE t\GALTER TABLE t ENGINE=TokuDB;SHOW CREATE TABLE t\GALTER TABLE t ENGINE=TokuDB, ROW_FORMAT=TOKUDB_LZMA;SHOW CREATE TABLE t\G

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html#compression-details

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html#compression-details


Compression Analysis

• Disk footprint savings• Low 85%

• Medium 88%

• High 91%

• Simulation Load• 4-11% variance

• Compression Time (migration)• Larger execution variance

19


TokuDB Assessment

• For different compression levels• Time to migrate production sized data (see later issues)

• Disk footprint

• Benchmark primary INSERT path (master/child)

• Benchmark point SELECT path

• Some benchmarks execution times better, some not

• SELECT performance was lower

• INSERT/UPDATE performance was higher

20


Downgrading an Altered table

21

# worksALTER TABLE demo ENGINE=TokuDB;ALTER TABLE demo ENGINE=InnoDB;

# FailsALTER TABLE demo ENGINE=TokuDB ROW_FORMAT=tokudb_lzma;ALTER TABLE demo ENGINE=InnoDB;


TokuDB Filesystem

• Messy under the covers

22

-rw-rw---- 1 mysql mysql 32768 Mar 9 19:32 tokudb.directory-rw-rw---- 1 mysql mysql 16384 Mar 9 19:25 tokudb.environment-rw------- 1 mysql mysql 0 Mar 9 19:25 __tokudb_lock_dont_delete_me_data-rw------- 1 mysql mysql 0 Mar 9 19:25 __tokudb_lock_dont_delete_me_environment-rw------- 1 mysql mysql 0 Mar 9 19:25 __tokudb_lock_dont_delete_me_logs-rw------- 1 mysql mysql 0 Mar 9 19:25 __tokudb_lock_dont_delete_me_recovery-rw------- 1 mysql mysql 0 Mar 9 19:25 __tokudb_lock_dont_delete_me_temp-rw-rw---- 1 mysql mysql 32768 Mar 9 19:32 tokudb.rollback

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_files_and_file_types.html

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_files_and_file_types.html


Backup/Restore Testing (TokuDB Hot Backup)

• Backup

• Restore

• Start Instance

• Run testing (no issues)

• Restart instance

• Crash

• RTFM

Always part of a good migration plan

23


Backup/Restore Testing (TokuDB Hot Backup)

• All tables (in two schema’s migrated to TokuDB)

You must disable InnoDB asynchronous IO if backing up InnoDB tables with TokuBackup. Otherwise you will have inconsistent, unrecoverable backups. The appropriate setting is innodb_use_native_aio=0.

• mysql schema has InnoDB tables

• sys schema has InnoDB table

24

https://www.percona.com/doc/percona-server/5.6/tokudb/toku_backup.html#limitations-and-known-issues

https://www.percona.com/doc/percona-server/5.6/tokudb/toku_backup.html#limitations-and-known-issues


MySQL 5.6 metadata InnoDB tables

25

mysql > select table_schema,table_name from information_schema.tables where engine='InnoDB';+--------------+-----------------------------------+| table_schema | table_name |+--------------+-----------------------------------+| mysql | innodb_index_stats || mysql | innodb_table_stats || mysql | slave_master_info || mysql | slave_relay_log_info || mysql | slave_worker_info || sys | sys_config |+--------------+-----------------------------------+6 rows in set (0.18 sec)

Significantly more in Percona 5.7


TokuDB Hot Backups still crashing

• New intermediate tables are still InnoDB• default_storage_engine

26

mysql > select table_schema,table_name from information_schema.tables where engine='InnoDB';+--------------+-----------------------------------+| table_schema | table_name |+--------------+-----------------------------------+| SCHEMA | _tmp_xxx_xxx_xxx || mysql | innodb_index_stats || mysql | innodb_table_stats || mysql | slave_master_info || mysql | slave_relay_log_info || mysql | slave_worker_info || sys | sys_config |+--------------+-----------------------------------+7 rows in set (0.18 sec)


Production UpgradeConsiderations

27


TokuDB/InnoDB Memory Management

• Allocating Memory• innodb_buffer_pool_size

• tokudb_cache_size

• Finding a balance, before, during and after• Resizing is a reboot, i.e. an outage

• Pre migration, and post migration

28

https://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_quickstart.html#considerations-to-run-tokudb-in-production

5.7 (one of two buffers) https://www.pythian.com/blog/experiments-mysql-5-7s-online-buffer-pool-resize/



https://www.pythian.com/blog/experiments-mysql-5-7s-online-buffer-pool-resize/


Upgrade/Downgrade path with production

• Production is a 3 server system (internal DC using VMWare)• Master

• Slave

• Backup Slave

29



• Minimal downtime migration (3 server solution)

• InnoDB -> InnoDB, InnoDB

• InnoDB -> TokuDB, InnoDB (reduced HA)

• InnoDB -> TokuDB -> InnoDB

• TokuDB -> InnoDB, InnoDB (outage)

• TokuDB -> TokuDB, InnoDB (reduced HA)

• TokuDB -> TokuDB, TokuDB

30

The Pythian Group Inc., 2017 31



• Migration (6 server solution)

• Added three additional servers• Master

• Slave

• Backup Slave

• TokuDB Future Master▪ TokuDB Future Slave▪ TokuDB Future Backup Slave

32

The Pythian Group Inc., 2017 33



• Migration become an application connection switch• Stop Application

• Set Master to read only

• Monitor

• Remaster InnoDB servers as slaves to new TokuDB Master

• Remove TokuDB Master read only

• Point Application to TokuDB Master

34


TokuDB Instrumentation (5.6)

36

$ mysql -e "SHOW ENGINE TOKUDB STATUS"#322 lines..locktree: memory size 0locktree: memory size limit 4294967296locktree: number of times lock escalation ran 18locktree: time spent running escalation (seconds) 221.892852locktree: latest post-escalation memory size 34747727locktree: number of locktrees open now 122locktree: number of pending lock requests 0locktree: number of locktrees eligible for the STO 0locktree: number of times a locktree ended the STO early 1349352locktree: time spent ending the STO early (seconds) 68.863206locktree: number of wait locks 417104locktree: time waiting for locks 1316392841821locktree: number of long wait locks 328846locktree: long time waiting for locks 1306669270761locktree: number of lock timeouts 324318locktree: number of waits on lock escalation 36locktree: time waiting on lock escalation 443789501locktree: number of long waits on lock escalation 36locktree: long time waiting on lock escalation 443789501...


TokuDB Outcomes and Possible FutureFeatures

38


Total Space Savings

39


TokuDB Process Information

mysql> SHOW PROCESSLIST;...| 336 | Fetched about 136788000 rows, loading data still remains | 670 | Loading of data about 6.6% done | 848 | Loading of data about 37.9% done

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html#progress-tracking

40



TokuDB Process Information (2)

mysql> SELECT * FROM information_schema.processlist where command != 'Sleep';

STATE: Queried about 500000 rows STATE: Creating sort index STATE: Queried about 2447542 rows, Updated about 27000 rows

mysql> SHOW SLAVE STATUS\G

Slave_SQL_Running_State: processing commit of transaction, 49152 out of 50000

41




TokuDB Online Hot Alter

• For the right type of ALTER there is no blocking• On Master

• In master session

• On Slave

• For adding indexes• No DML blocking on Master

• Session only blocking

• No DML blocking on slave

42

https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#hot-column-add-delete-expand-and-rename-hcader https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#hot-index-creation https://github.com/percona/tokudb-engine/wiki/Hot-schema-change-with-TokuDB

https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#hot-column-add-delete-expand-and-rename-hcader

https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#hot-column-add-delete-expand-and-rename-hcader

https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#hot-index-creation

https://github.com/percona/tokudb-engine/wiki/Hot-schema-change-with-TokuDB


Other Types of Alters

• Change of nullability• Evaluation of gh-ost

• Issues to due tablesize (5B rows)

43


Read Free Replication Capability (not used)

Master

• binlog_format=ROW

Slave

• read_only=1

• tokudb_rpl_unique_checks=0

• tokudb_rpl_lookup_rows=0

44

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html#read-free-replication https://www.percona.com/blog/2014/09/25/tokudb-read-free-replication-details-and-use-cases/

https://github.com/percona/tokudb-engine/wiki/Replication-with-TokuDBhttps://github.com/percona/tokudb-engine/wiki/Testing-Read-Free-Replication

https://www.percona.com/doc/percona-server/LATEST/tokudb/using_tokudb.html#read-free-replication

https://www.percona.com/blog/2014/09/25/tokudb-read-free-replication-details-and-use-cases/

https://github.com/percona/tokudb-engine/wiki/Replication-with-TokuDB

https://github.com/percona/tokudb-engine/wiki/Testing-Read-Free-Replication


Clustered Indexes (not used)

• Fast Insertions and Richer Indexes

• Clustering Secondary Indexes• Extended parser syntax

▪ CLUSTERING KEY▪ UNIQUE CLUSTERING KEY¶

45

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html#clustering-secondary-indexes

https://www.percona.com/doc/percona-server/5.6/tokudb/using_tokudb.html


Post Migration Observations

46


Using JSON in Percona 5.7

• Evaluation of using JSON• JSON datatype. Fine!

• Virtual column. NOT FINE

47

CREATE TABLE test.t1(id BIGINT NOT NULL, meta JSON NOT NULL) ENGINE=tokudb;ALTER TABLE test.t1 ADD name VARCHAR(40) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(meta,'$.Name')));ERROR 3106 (HY000): 'Specified storage engine' is not supported for generated columns.


Long running transactions

• Future code deployments unveiled issues

• Experimented with per session tokudb_lock_timeout

48

DELETE mFROM m, dWHERE d.id = m.id

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction


Failing Replication

49

2017-06-01 12:59:03 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ...2017-06-01 12:59:12 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ...2017-06-01 12:59:21 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ...2017-06-01 12:59:30 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ... 2017-06-01 12:59:39 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ...2017-06-01 12:59:48 29813 [Warning] Slave SQL: Error 'Lock wait timeout exceeded; try restarting transaction' on query ...2017-06-01 12:59:48 29813 [ERROR] Slave SQL: Slave SQL thread retried transaction 10 time(s) in vain, giving up. Consider raising the value of the slave_transaction_retries variable. Error_code: 12052017-06-01 12:59:48 29813 [Warning] Slave: Lock wait timeout exceeded; try restarting transaction Error_code: 1205

mysql> SHOW SLAVE STATUS\G

Last_Error: Slave SQL thread retried transaction 10 time(s) in vain, giving up. Consider raising the value of the slave_transaction_retries variable.

Failing slave query was a single row

unique key update.


Optimizer Issue (5.6)

• Batch process (runs in chunks)• At 80M of 350M rows

• Switched from PK scan to table scan

• Solution was to hint an UPDATE• USING INDEX (PRIMARY)

50

Is this TokuDB specific or MySQL?


Wrap-up

51


In Review

• Met the need at the time• Diskspace• Workload

• Was easy to consider• No application changes• No major software changes (already Percona 5.6)

• Only leveraging just one feature of TokuDB

• Lots to learn• Instrumentation• Tuning

52


THANK YOUPlease rate this presentation if you have the Percona App installed

[email protected]

© 2016 Pythian. Confidential

53

A TokuDB Migration Case Study - Percona...The Pythian Group Inc., 2017 A TokuDB Migration Case Study...

Documents

Transcript of A TokuDB Migration Case Study - Percona...The Pythian Group Inc., 2017 A TokuDB Migration Case Study...