Webinar slides: Managing MySQL Replication for High Availability

Copyright 2015 Severalnines AB

Managing MySQL Replication for High Availability

February 2, 2016

Krzysztof Książek

Severalnines

[email protected]

1

mailto:[email protected]

2

Your host & some logistics

I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!

Feel free to ask any questions in the Questions section of this application or via the Chat box.

You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.


! MySQL replication and ClusterControl (Demo)

! Deployment

! Topology changes

! Important metrics

! How to deal with

! Schema changes

! Topology changes

! Potential issues in MySQL replication

3

Agenda


! Available since 3.23

! Proven, robust, based on binary logs

! GTID introduced in MySQL 5.6

! Multi-threaded replication introduced in 5.6, vastly improved in 5.7

! Asynchronous or semisynchronous

! Backbone of majority of large MySQL deployments

4

MySQL Replication


MySQL replication and ClusterControl

5


! Deployment process

! Add slave

! Promote slave

! Recover slave

! Monitoring of the replication environment

6




7

DEMO


Deployment scenarios for replication

8


! Single DC

! Master - slave

! Intermediate master

! Master - master

! Multi-DC

! Master - slave, over WAN

! Active - passive

! WAN replication with Galera or MySQL Cluster

9




10

Master - slaveIntermediate master



11

Master - master, active - standbyTwo synchronous clustersconnected over WAN


How to deal with schema changes?

12


! Schema changes in MySQL

! Online DDL’s

! not-so-online DDL’s

! Rolling schema upgrades

! external tools like pt-online-schema-change

13



! With MySQL replication lag is a serious limiting factor in DDL’s

! Online DDL will be online only on the master

! Replication will be locked for the duration of the DDL

! Even when you use Multi-Threaded Slave

! Direct alters are suitable only for small tables

! unless you can accept replication lag, of course

14



! Rolling Schema upgrades - run DDL from the bottom to the top of replication chain

! SET SESSION sql_log_bin=0 to avoid errant transactions

! Once you reach the master, failover and then alter the old master

! Schema changes have to be compatible, you’ll be running DML’s on a mix of altered and not altered tables

! Schema changes have to be compatible, you’ll be reading from a mix of altered and not altered tables

15



! pt-online-schema-change - part of the Percona Toolkit

! Create a new table with desired schema

! Setup triggers to copy data from old to the new table

! Copy the data from old to new table in batches

! Rename table once the data has been copied

16




17


! Point-in-time change, you can use for any DDL

! Replication-aware, it can monitor lag and throttle itself

! Introduces additional load, slow down your system

! It has issues with foreign keys

! Can’t work with tables which have triggers

! Won’t work with tables without PK and unique key

18



! Otherwise pt-online-schema-change is a very reliable tool

! Schema change may take a long time but it’s throttling itself and can be paused at any time

! One of the most important tools in the DBA arsenal

! Sometimes it’s the only way to run a schema change without downtime

19



How to deal with topology changes?

20


! If you use GTID - it’s easy

! GTID allows for a flexible topology change

! CHANGE MASTER TO … MASTER_AUTO_POSITION=1;

! You can slave off every node in the replication chain

! Be aware of errant transactions, though

! http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions

21


http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions


! If you do not use GITD, things are more complex

! Slave knows only a binlog position of itself and it’s master

! You can use binary and relay logs to identify position of a given transaction in the replication chain but it’s a complex and error prone process

! To avoid it, you have to stop writes on a involved subset of hosts

22




23


! First step requires to slave DB3 and DB4 off DB2

! You need to enable log-slave-updates on DB2

! Stop DB2, DB3 and DB4 at the same position using START SLAVE UNTIL …

! Start it until a beginning of binary log two files later

24



! Locate your first position in binlog using mysqlbinlog tool

! START SLAVE UNTIL master_log_file='mysql-bin.000121', master_log_pos=4;

! Run FLUSH LOGS on the master two times to rotate logs - slaves should end up stopped at the same position

! Check the position on DB2 (SHOW MASTER STATUS)

! Use it to slave DB3 and DB4 off DB2

25



! We should finally reach the desired topology

! Next step - setup master - master replication between DB1 and DB2

! Confirm that DB2 is not taking writes other than the replication

! Use server_id in binary logs for that - you should see only DB1

! If it does writes, you’re in trouble - investigate and eliminate the culprit

26



! Once confirmed writes are not hitting DB2, execute CHANGE MASTER TO … on DB1 pointing it to the DB2 and using any recent coordinates

! Monitor Exec_Master_Log_Pos in the SHOW SLAVE STATUS on DB1, it should be stable. If it does increase, something is still writing to DB2.

! Once all is set up, you are ready for a failover

27



Potential issues in MySQL replication

28


! Lag is inevitable part of MySQL replication, async or semisync, it doesn’t matter

! Lag can be caused by large number of writes and DDL’s

! Multi-threaded slave + GTID is a good choice to minimize lag caused by writes (for DDL’s - use pt-online-schema-change)

! Hardware has to follow too, especially I/O subsystem

29



! Data drift may happen even if you take all necessary precautions

! Use ROW binary log format for the best data consistency

! Use read_only and super_read_only on slaves to minimize possibilities of errant writes

! Use pt-table-checksum and pt-table-sync to keep an eye on the data consistency

! When in doubt, rebuild a slave from the source of truth (master)

30



! Errant transactions can ruin your day

! Those are transactions executed on a slave but not on a master

! read_only and super_read_only can help to avoid them

! GTID_SUBSET(), GTID_SUBTRACT() will help you to detect them

! We’ve covered the topic here:http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions

31


http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions


! More blogs featuring MySQL replication:

! http://severalnines.com/blog/become-mysql-dba-blog-series-common-operations-schema-changes

! http://severalnines.com/blog/become-mysql-dba-blog-series-common-operations-replication-topology-changes

! http://severalnines.com/blog/become-mysql-dba-blog-series-live-migration-using-mysql-replication

! http://severalnines.com/tutorials/mysql-replication-high-availability-tutorial

! Contact: [email protected]

32

Thank You!

http://severalnines.com/blog/become-mysql-dba-blog-series-common-operations-schema-changes

http://severalnines.com/blog/become-mysql-dba-blog-series-common-operations-replication-topology-changes

http://severalnines.com/blog/become-mysql-dba-blog-series-live-migration-using-mysql-replication

http://severalnines.com/tutorials/mysql-replication-high-availability-tutorial

mailto:[email protected]

Webinar slides: Managing MySQL Replication for High Availability

Internet

Transcript of Webinar slides: Managing MySQL Replication for High Availability