Webinar slides: Replication Topology Changes for MySQL and MariaDB

1

Your host & some logistics

I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!

Feel free to ask any questions in the Questions section of this application or via the Chat box.

You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.

Copyright 2015 Severalnines AB

Replication Topology Changes for MySQL and MariaDB

September 29, 2015

Krzysztof Książek

Severalnines

[email protected]

2

mailto:[email protected]


! We want to help all non-DBA people who look after MySQL infrastructures

! Share tips and good practices

! Watch the replays of the previous webinars on our Slideshare page

! http://www.slideshare.net/Severalnines/videos

3

“Become a MySQL DBA” series

http://www.slideshare.net/Severalnines/videos


! MySQL Replication topology changes

! using GTID

! using regular replication

! Failover process

! Using MaxScale for automatic re-routing of queries

! Other external tools useful when dealing with failover

4

Agenda


MySQL Replication topology changes

5


! Maintenances

! Adding an index on a large table

! Delete data and then rebuild a large table to reclaim disk space

! Reslaving nodes for a planned failover

! Reslaving nodes after master crash

6

Why do we make changes in a replication chain?


! GTID allows for a flexible topology changes

! CHANGE MASTER TO … MASTER_AUTO_POSITION=1;

! You can slave off every node in the replication chain

! Be aware of errant transactions, though

! http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions

7

Replication topology changes - GTID

http://www.severalnines.com/blog/mysql-replication-and-gtid-based-failover-deep-dive-errant-transactions


! Far less flexible than GTID

! Slave knows only a binlog position of itself and it’s master

! You can use binary and relay logs to identify position of a given transaction in the replication chain but it’s a complex and error prone process

! To avoid it, you have to stop writes on a involved subset of hosts

8

Replication topology changes - no GTID



9


! First step requires to slave DB3 and DB4 off DB2

! You need to enable log-slave-updates on DB2

! Stop DB2, DB3 and DB4 at the same position using START SLAVE UNTIL …

! Start it until a beginning of binary log two files later

10



! Locate your first position in binlog using mysqlbinlog tool

! START SLAVE UNTIL master_log_file='mysql-bin.000121', master_log_pos=4;

! Run FLUSH LOGS on the master two times to rotate logs - slaves should end up stopped at the same position

! Check the position on DB2 (SHOW MASTER STATUS)

! Use it to slave DB3 and DB4 off DB2

11



! We should finally reach the desired topology

! Next step - setup master - master replication between DB1 and DB2

! Confirm that DB2 is not taking writes other than the replication

! Use server_id in binary logs for that - you should see only DB1

! If it does writes, you’re in troubles - investigate and eliminate the culprit

12



! Once confirmed writes are not hitting DB2, execute CHANGE MASTER TO … on DB1 pointing it to the DB2 and using any recent coordinates

! Monitor Exec_Master_Log_Pos in the SHOW SLAVE STATUS on DB1, it should be stable. If it does increase, something is still writing to DB2.

! Once all is set up, you are ready for a failover

13



Failover process

14


! Exact failover process may depend strongly on the infrastructure that you use

! In short - it’s a process of moving writes from one database node to another

! It may or may not be graceful, depending on your application and tools that you use

15

Failover Process


! Requires downtime period for a master switch

! Not ideal solution but easiest to implement and it will work with every application that can be stopped

! We are assuming the topology we reached in our previous slides, with failover about to happen from DB1 to DB2

16

Offline failover


! As a first step, you need to stop all activity in your application - you don’t want to have any writes hitting db

! You can monitor it using SHOW MASTER STATUS on the master

! You can monitor binary logs to locate and identify writes that hit the database

! Com_* counters can also be monitored

17

Offline failover


! Next step - you need to repoint your application to the new master, DB2 in our example

! Depending on your setup you can:

! Change DNS entries

! Set read_only values accordingly on MySQL hosts

! Use your proxy’s settings to redirect connections from one host to the other

18

Offline failover


! Once you pointed writes to a new master, you may want to perform some tests before going back live

! All the time, ensure that any writes are happening only on the correct host - new master, DB2 in our case

! Do not use any ‘hardcoded’ IP’s in your scripts - this may become a serious problem when you will be performing a failover

19

Offline failover


! Moving writes without downtime period - planned failover

! Stop the writes on the master, failover, move the writes to the new master - sounds familiar?

! Main problem is - how to do it gracefuly?

! FTWRL + read_only = 1? May take too long

! read_only = 1? Will kill current connections

! Drop the user + wait few seconds + read_only = 1? Still not ideal but may be acceptable

20

Online failover


! Proxies may help significantly in this regard - single point of config change, immediate result

! Caching reads and writes can also be useful

! Application will have to be able to handle errors and rollbacks as transition may but also may not be graceful

! Some of the tools can help you to make it graceful, without application seeing any error at all

21

Online failover


Using MaxScale to reroute queries

22


! MaxScale can implement read-write split

! MaxScale can be used with MySQL replication backend

! It monitors health of the backend nodes

! It detects replication topology changes

! Makes easier to handle any changes like failover

23



! For reads, most of the failures are being handled gracefully

! For writes, not yet - errors will be thrown and transactions rolled back, application has to handle them

! But it should be handling them anyway, right?

! Experimental support for handling failovers (along with reslaving) in the latest MaxScale

! No easy way of performing a planned failover, though

24


Confidential


External tools which help handling failover process


! Handles the failover process for you

! Both automatic failover and manual one

! Can be used as a standalone solution or a part of a set of scripts or tools

! Integrates nicely with other tools in a manual failover process

! Can work with GTID and regular replication

26

MHA


! Part of MySQL Utilities from Oracle

! Allows user to perform a manual failover

! Works only with GTID replication

! Checks for errant transactions

! Can be easily combined with other scripts through --exec-before and --exec-after

27

mysqlrpladmin


! New addition to the ‘database-aware’ proxy family

! Works as a gateway for all connections to databases

! Performs read/write split based on user-defined set of rules

! Gives you ability to perform a sub second graceful failover in case of a master-master replication.

! Works great with external scripts for handling topology changes

28

ProxySQL


! Idea is simple - all queries go through proxy to backend nodes

! You can use failover script to switch a backend node into a ‘failover’ mode

! Proxy will not forward more connections to it, it will “buffer” new connections

! Your failover script can wait till the current transactions finish or timeout will be reached and transactions can be killed

! In a next step you can reconfigure the replication topology as needed

29

ProxySQL


! Once there are no transactions running and new topology has been set up, proxy can be reconfigured to move writes to another backend

! Depends on how long are your transactions and how long timeout is set, failover can take even a fraction of a second

! Application will see temporary slowdown but should not see any errors

! Additional scripting is required - ProxySQL doesn’t monitor replication topology on it’s own

30

ProxySQL


! More blogs in “Become a MySQL DBA” series:

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

! Contact: [email protected]

31

Thank You!

http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

mailto:[email protected]

Webinar slides: Replication Topology Changes for MySQL and MariaDB

Technology

Transcript of Webinar slides: Replication Topology Changes for MySQL and MariaDB