GTID Replication - WordPress.com of Contents 5 3 GTID Replication Implementation 4 GTID Fail-over,...
Transcript of GTID Replication - WordPress.com of Contents 5 3 GTID Replication Implementation 4 GTID Fail-over,...
GTID ReplicationImplementation and Troubleshooting
Abdel-MawlaGharieb2015/04/15
Who am I ?
• Abdel-Mawla Gharieb (Moll)
• From Alexandria, Egypt.
• Graduated in 2008.
• I like sports.
• Playing piano.
• 7 years of experience.
• Sr. DB Engineer at
2
Rakuten, Inc.
• Founded in Feb 1997.
• Largest e-commerce company in Japan.
• 4th Largest e-commerce company in the world.
• More than 11 thousands employees worldwide.
• In 2008, started expanding outside Japan.
• No.1 Internet company in the world is our target.
3
Rakuten Group4
Table of Contents5
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
1 What is GTID?
2 GTID Benefits
Table of Contents6
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
What is GTID?
• GTID: Global Transaction IDentifier.
• Unique among all servers inside the replication
• Consists of two parts {source_id:transaction_id}:
– Source_id: Server’s UUID
– Transaction_id: Sequence number.
7
b1ee1e40-de96-11e4-b3a7-f4c709d514e2:10
Table of Contents 8
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
GTID Benefits
• Setting up MySQL Replication is so simple now!
• Consistency is guaranteed between master and slaves.
• Simple to determine inconsistency.
• Simple to do Point in Time Recovery (PiTR).
• Useful to get consistent backup in MTS.
• Fail-over process become much easier.
• Automatic fail-over scripts are easy to implement.
9
GTID Benefits10
Master A Master B
Slave
mastera-binlog.73847547645
masterb-binlog.69893457845
slave-binlog.482847645
CHANGE MASTER TOMASTER_HOST=‘Master_B’,MASTER_LOG_FILE= ??,MASTER_LOG_POS= ??;
Classic Replication
GTID Benefits11
Master A Master B
Slave
uuid_A:1973 uuid_A:1973
uuid_A:1973CHANGE MASTER TOMASTER_HOST=‘Master_B’,MASTER_AUTO_POSITION = 1;
GTID Replication
Table of Contents12
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
GTID Replication Implementation
GTID important variables• gtid-mode: ON|OFF• enforce-gtid-consistency: Prevent executing the non-transactionally safe
statements…– CREATE TABLE … SELECT statements.– CREATE TEMPORARY TABLE statements inside transactions.– Transactions or statements which update both transactional and non-
transactional tables.
• gtid-purged: Transactions have been purged from the binary logs.• gtid-executed: Transactions already executed on the server.• gtid-next: GTID value for the next transaction.
13
GTID Replication Implementation
• Fresh installation
• Adding slave to an existing GTID replication
14
GTID Replication Implementation
Fresh installation
• Master’s side configuration.
• Slave’s side configuration.
15
Fresh installation
Master’s side configuration• Add the following variables to my.cnf:
– server-id = 1
– log-bin = mysql-bin
– gtid-mode = ON
– enforce-gtid-consistency
– log-slave-updates
16
Fresh installation
Master’s side configuration• Restart MySQL to apply the configuration changes:
• Create a MySQL user to be used by the slave:
17
SQL> GRANT REPLICATION SLAVE ON *.* TO ‘repl’@’slave_host’ IDENTIFIED BY ‘p@ssw0rd’;
shell> service mysql restart
Fresh installation
Slave’s side configuration• Add the following variables to my.cnf:
– server-id = 2
– log-bin = mysql-bin
– gtid-mode = ON
– enforce-gtid-consistency
– log-slave-updates
18
Fresh installation
Slave’s side configuration• Restart MySQL to apply the configuration changes:
• Execute the CHANGE MASTER TO command:
19
shell> service mysql restart
SQL> CHANGE MASTER TOMASTER_HOST=‘master_ip’,MASTER_PORT=3306,MASTER_USER=‘repl’,MASTER_PASSWORD=‘p@ssw0rd’,MASTER_AUTO_POSITION=1;
Fresh installation
Slave’s side configuration• Start the replication:
20
SQL> START SLAVE;
Fresh installation
Slave’s side configuration• Checking the replication status:
– SQL> SHOW SLAVE STATUS|GSlave_IO_State: Waiting for master to send eventMaster_Host: 127.0.0.1Master_User: replMaster_Port: 3306. . .Slave_IO_Running: YesSlave_SQL_Running: Yes. . .Retrieved_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Executed_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Auto_Position: 1
21
GTID Replication Implementation
Adding slave to an existing
GTID replication
22
Adding slave to an existing GTID replication
• Backup the master server:
– shell> mysqldump -u root -p --all-databases --flush-privileges --single-transaction --flush-logs --triggers --routines --events --hex-blob >/path/to/backupdir/full_backup-$TIMESTAMP.sql
• Start MySQL service on the slave with GTID variables.
• Restore the master backup.
• Execute change master to with MASTER_AUTO_POSITION=1
• Start the slave “START SLAVE;”
23
Adding slave to an existing GTID replication
• How does the slave get the correct GTID value to start replication from?!!
mysqldump knows about GTID!!
24
---- GTID state at the beginning of the backup--SET @@GLOBAL.GTID_PURGED='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-7';
Table of Contents25
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
Table of Contents26
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
Migration to GTID Replication (5.6)
• Make the master server read only: SET GLOBAL read_only=ON;
• Allow all slaves to catch up with the master.
• Shutdown MySQL service on ALL servers.
• ADD the GTID variables.
• Add read_only in the master’s and skip_slave_start in the slaves’.
• Start MySQL service on ALL servers.
• Issue CHANGE MASTER TO MASTER_AUTO_POSITION=1; on all slaves and start them.
• Make the master writable: SET GLOBAL read_only=OFF;
27
Migration to GTID Replication (5.6)
Shutting down all servers at the same time means the GTID migration requires downtime. Why shutdown then?
– gtid_mode is a read only variable.
– Different gtid_mode values among replication servers is not acceptable.
Any workarounds ?!• Facebook and Booking.com patches.
• Facebook patch is ported in Percona Server starting from 5.6.22-72.0 (http://www.percona.com/blog/2015/02/10/online-gtid-rollout-now-available-percona-server-5-6/)
28
Migration to GTID Replication (5.7)
New changes in GTID variables
• enforce_gtid_consistency: Dynamic & ENUM (OFF|ON|WARN).
• gtid_mode: Dynamic & values (OFF|OFF_PERMISSIVE|ON_PERMISSIVE|ON)
29
Migration to GTID Replication (5.7)
enforce_gtid_consistency• OFF: non-transactionally safe statements are OK.
• WARN: non-transactionally safe statements will raise a warning but also OK.
• ON: non-transactionally safe statements will raise errors and not be executed.
30
Migration to GTID Replication (5.7)
GTID_MODE• OFF: Master will not generate GTIDs, slave will not replicate GTIDs.
• OFF_PERMISSIVE: same like OFF but the slave can replicate both GTID and non GTID transactions.
• ON: Master generates GTIDs only, slave replicates GTIDs only.
• ON_PERMISSIVE: same like ON but the slave can replicate both GTID and non GTID transactions.
31
Migration to GTID Replication (5.7)
Online migration
• SET GLOBAL ENFORCE_GTID_CONSISTENCY = WARN; on all servers• Check after a while if some warnings were printed or not.• SET GLOBAL ENFORCE_GTID_CONSISTENCY = ON; on all servers• SET GLOBAL GTID_MODE = OFF_PERMISSIVE; on all servers• SET GLOBAL GTID_MODE = ON_PERMISSIVE; on all servers• Wait for the following actions:
32
Migration to GTID Replication (5.7)
Online migration– Ongoing transactions require gtid-free were finished.
– All gtid-free transactions were replicated on all slaves.
– Binary log files that contain gtid-free events were expired.
• SET GLOBAL GTID_MODE = ON; on all servers
• STOP SLAVE;CHANGE MASTER TO MASTER_AUTO_POSITION=1;START SLAVE; on all slaves
33
Table of Contents34
1 What is GTID?
2 GTID Benefits
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
GTID Replication Troubleshooting
• Skip a transaction.
• Re-build a broken slave.
35
GTID Replication Troubleshooting
Skip a transaction
36
Skip a transaction
• Sample errors:
37
Last_SQL_Error: Could not execute Write_rows event on table test.t1; Duplicate entry '4'forkey'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000304, end_log_pos285
Skip a transaction
• Sample errors:
38
Last_SQL_Error: Could not execute Delete_rows event on table test.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000304, end_log_pos 688
Last_SQL_Error: Could not execute Update_rows event on table test.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000304, end_log_pos 492
Skip a transaction
Problem solution• Check which transaction is causing the problem:
39
SQL> SHOW SLAVE STATUSG ..Retrieved_Gtid_Set: b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-5Executed_Gtid_Set : b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-4Auto_Position: 1
Skip a transaction
Problem solution• Inject an empty transaction:
40
SQL> SET GTID_NEXT='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:5';SQL> BEGIN;COMMIT;SQL> SET GTID_NEXT='AUTOMATIC';SQL> START SLAVE;
GTID Replication Troubleshooting
Re-build a broken slave.
41
Re-build a broken slave
• Sample error:
42
Last_IO_Errno: 1236Last_IO_Error : Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.'
Re-build a broken slave
Problem solution• Backup the master’s database:
43
shell> mysqldump -u root -p --all-databases --flush-privileges –single-transaction --master-data=2 --flush-logs --triggers --routines --events –hex-blob >/path/to/backupdir/full_backup-$TIMESTAMP.sql
shell> head -n 50 /path/to/backupdir/full_backup-$TIMESTAMP.sql|grep PURGED SET @@GLOBAL.GTID_PURGED='b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-8';
Re-build a broken slave
Problem solution• Restore the backup on the broken slave:
44
shell> mysql -u root -p < /path/to/backupdir/full_backup-$TIMESTAMP.sqlERROR 1840 (HY000): @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty
Re-build a broken slave
Problem solution• Empty the GTID_EXECUTED on the slave:
• Restore the backup again
45
SQL> RESET MASTER;
shell> mysql -u root -p < /path/to/backupdir/full_backup-$TIMESTAMP.sql
Re-build a broken slave
Problem solution• Make sure that GTID_EXEUCTED is correct:
46
SQL> SHOW GLOBAL VARIABLES LIKE 'gtid_executed';+---------------------+---------------------------------------------------------------+| Variable_name | Value |+---------------------+---------------------------------------------------------------+| gtid_executed | b1ee1e40-de96-11e4-b3a7-f4c709d514e2:1-8 |+---------------------+---------------------------------------------------------------+1 row in set (0.00 sec)
Re-build a broken slave
Problem solution• Start the slave:
47
SQL> START SLAVE;
Before Using GTID …
• Migration to GTID replication requires downtime (5.6).
• Non-transactionally safe statements will not be executed (5.6).
• MySQL performance in GTID replication is a little bit slower.
• MariaDB GTID is not compatible with MySQL server or Percona server.
• Errant transactions might cause troubles.
• New protocol, new problems and new solutions.
48
Conclusion …49
3 GTID Replication Implementation
4 GTID Fail-over, Practical Demo
5 Migration to GTID Replication
6 GTID Replication Troubleshooting
1 What is GTID?
2 GTID Benefits