Easy MySQL Replication Setup and Troubleshooting

Easy MySQL ReplicationSetup and Troubleshooting

Bob BurgessSalesforce ExactTarget Marketing Cloud (Radian6)

• Introduction• Description• Configuration Items• Commands• Setup• Beyond the Basics• Troubleshooting Faults• Gotchas• Closing

Slides

• I will post these slides on the conference site and slideshare

• No need to copy them down

Who Am I?

• MySQL user since 2007• Radian6: Social Media Monitoring• Now part of

Salesforce ExactTarget Marketing Cloud• Large Data Set• Dozens of Databases • Replication is CRUCIAL for us

Why Am I Doing This?

• Replication isn’t hard• Show simplest replication case• Share how we deal with common problems

Our First Replication

• Set up a separate database for running reports, to separate back-end from user-facing activity

• Alexander Rubin of MySQL Support in 2008

What is “Replication”?

Why Replication?

• Database Failure• Share Read Traffic (Scale-Out)• Reporting/Analytics Databases• Network Load• Backups

Replication Compared

• DRBD– Block-level– Replica is unusable until switched to

• Galera / Percona Cluster– More complicated– Not WAN-Friendly for High Transaction Rate

• Tungsten Replicator• Others..?

Replication Described

Master Replica

Master

table1table2

Update table1 set...Insert into table2...

Binlogs:•Events in commit order•Contains timestamps and auto-increment IDs

client

Replica

table1table2

IO Thread SQL Thread

log_bin=/data/log/mysql-bin

Config items

Master

table1table2

client

Replica

table1table2

server_id=1 server_id=2

Config items

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commands

• change master to• start slave• stop slave• reset slave• show slave status\G• show master status• mysqlbinlog• grant replication slave

Replication Commandschange master to... Which server? Port? Credentials? Log file & position?

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commands• change master to...

Which server? Port? Credentials? Log file & position?•change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;

Replication Commands

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commandschange master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;

(1) Master

table1table2

client

Replica (2)

table1table2

IO Thread

Replication Commandsstart slave;

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commandsstop slave;

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commandsreset slave;

(1) Master

table1table2

client

Replica (2)

table1table2

Replication Commands• show slave status\G• Manual troubleshooting• Use in scripts for alarming

LAG=$(mysql -s -e’show slave status\G’|grep ‘Seconds_Behind_Master’|cut -f2 -d:| tr -d ‘ ‘)

IO_UP=$(mysql -s -e’show slave status\G’|grep ‘Slave_IO_Running’|cut -f2 -d:| tr -d ‘ ‘)

Slave_IO_State: Waiting for master to send event

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error:

Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Slave_IO_State: Waiting for

master to send event

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

What it’s doing

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0

Master_Server_Id: 1

Where it’s connected

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3

Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Where the IO Thread has read up to

Master

table1table2

client

Replica

table1table2

Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843

Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

Where the SQL Thread has executed up to (in relaylog)

Master

table1table2

client

Replica

table1table2

Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548

Relay_Master_Log_File: mysql-bin.000001

Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

Last_Error: Skip_Counter: 0

Exec_Master_Log_Pos: 23456234

Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Where the SQL Thread has executed up to (corresponding position in Master’s binlog)

Master

table1table2

client

Replica

table1table2

Relay_Master_Log_File: mysql-bin.000001 Exec_Master_Log_Pos: 23456234

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001

Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

If the IO and SQL Threads are running

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes

Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Filters in effect

Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key:

Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Lag (how long ago was timestamp of latest-processed query?)

Can be “NULL”

Last_Errno: 0

Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error:

Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Error (SQL Thread)

Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Error (IO Thread)

Basic Setup Walk-Through

• Unique server_id values

my.cnf on master[mysqld]server_id=1

my.cnf on replica[mysqld]server_id=2

• Replication UserOn Master:

grant replication slave on *.* to ‘repl’ identified by ‘pass’;

• Point to the correct binlog position• On master:show master status;

...once both databases are equal• On Replica:change master to ...

• Start it up• On replica:start slave;

• Check it• On replica:show slave status\G

And that’s it.

Limiting Tables or Schemas

• If you don’t need all the tables• In my.cnf:replicate-do-table=schema.tableNamereplicate-skip-table=schema.otherTable

• If you don’t need all the schemas• In my.cnfreplication-do-db=schemaOnereplication_skip_db=schemaTwo

Limiting Tablesreplicate-do-table=schema.tableName

(1) Master

table1table2

client

Replica (2)

table1table2

IO Thread SQL Threadwith filter

Limiting Columns or Rows

• If you don’t need all columns• Subset of table for reporting or analytics

• Original tablecreate table x (a int primary key, b int, c int) engine=InnoDB;

• Blackhole table on replica like master tablecreate table x (a int, b int, c int) engine=blackhole;

• Target table on replicacreate table y (a int primary key, b int) engine=InnoDB;

• Before-insert trigger on replica to put data into second “real” table

• Only columns a and b, and only for b=1create trigger x_bi before insert on x for each row insert into y (a, b) select new.a, new.b from dual where new.b=1;

(1) Master

table1table2

client

Replica (2)

Multi-Level

• Local and Remote DR Site• Spreading load over many servers

M R M R

Multi-Level

• Configlog_slave_updates=1

Multi-LevelMaster Replica

table1table2

mysql IO Thread SQL Thread

Replica

table1table2

Types of Binlogs

• Statement-Based– Contains the actual statements run– Data drift is more likely (code modified by triggers,

etc)• Row-based– Replicates changes for each row– Mostly unreadable by humans– Unaffected by how the change is done (data drift less

likely)– mysqlbinlog -v

Fault: Lag

• Long-running transactions• High concurrency on master becomes single

session on replica• show slave status\G• Easy to alarm on

Fault: Stopping from Error

• Any error stops SQL Thread!• Duplicate Key• Unknown Function• slave_exec_mode=idempotent

...suppresses duplicate-key and no-key-found errors

• show slave status\G

Faults: Can’t Connect to Master

• Fault shows in slave status as soon as you start slave.

• Did you create replication slave user?• Correct user/password?• Correct hostname/port?• Master is running and reachable?

Fault: Can’t Find Binlog

• Clearly stated in the slave status• Can happen if replica has been stopped (or is

lagged) and the binlogs are removed from the master

• Execute from archiveOR• Rebuild replica and restart from a known good

binlog position

Faults: Configuration

• replication-do-table• replication-do-schema• log_slave_updates

Faults: server_id

• Must be unique!• If equal: “Seconds behind” jumps everywhere.

Gotcha: functions

• Functions will be run on the replica even if only appearing in a select on the master

• Make sure all functions exist on replica DB, even if only a dummy version

create function myFtn (a int, b int) returns int deterministic return 0;

Gotcha: Reconnection

• When a replica is stopped and reset, until the database is restarted, replication may reconnect to a master after “reset slave”

change master to master_port=1234;

Gotcha: Master Crash

• Replication becomes unstable if master crashes• Error: unknown instruction at position xxx• Often need to use mysqlbinlog to find a

suitable position in the binlog to restart replication from

stop slave; reset slave;change master to ...;start slave;

Gotcha: MySQL Versions

• Replication not completely compatible between versions (especially later=>earlier)

• 5.5 master to 5.1 replica: character set error “sometimes”

• Stops replication

Impossible

• One replica can have only one master(until 5.7)

• Can’t change table or schema filters dynamically

• Can’t put binlogs back on master server, for recovery purposes

The Book

Alternatives

• Tungsten Replicator from Continuent• Others?

Questions?

Thank you!

Bob Burgessrburgess@salesforce.com

Don’t forget your session evaluation!

Easy MySQL Replication Setup and Troubleshooting

Data & Analytics

Transcript of Easy MySQL Replication Setup and Troubleshooting

MySQL Replication: Latest Developments · MySQL 5.1.30 row-based replication MySQL 4.0.12 MySQL 4.1.7 MySQL 5.5.8 semi-sync replication MySQL 5.6.10 crash-safe replication metadata

MySQL Replication Tutorial

MYSQL TROUBLESHOOTING

MySQL Replication Troubleshooting · MySQL Replication Troubleshooting ... Error during thread ... #160822 14:15:11 server id 1 end_log_pos 1124 CRC32 0x82cb7a14 Query thread_id=9

MySQL Replication Advanced Features - FOSDEM · MySQL Replication MySQL 3.23 •Initial Statement Replication Implemented MySQL 4.0 •Split IO Thread and SQL Thread MySQL 5.1 •Row

MySQL Replication Update MySQL 5.5 (GA) & MySQL 5.6.2 (Dev ...akipp.se/larsthalmann.com/110412.thalmann.MySQL_UC_2011.pdf · MySQL Replication Update MySQL 5.5 (GA) & MySQL 5.6.2

MySQL Enterprise Monitor€¦ · mysql-monitor.log MySQL log file. Replication 1.dot The calculated MySQL server replication structure preferences.properties MySQL Enterprise Monitor

Troubleshooting mysql-tutorial

MySQL Replication: What's New In MySQL 5.7 and MySQL 8 › ... › MySQL8_Replication.pdf · MySQL Database Replication: Some Notes •Logical replication log recording master changes

Managing MySQL Replication - Jeremy D. Zawodnyjeremy.zawodny.com/mysql/mysqlcon-2003/MySQLReplication.pdf · Managing MySQL Replication MySQL User Conference Jeremy Zawodny Yahoo!

MySQL Group Replication & MySQL InnoDB Cluster€¦ · MySQL Group Replication Developed by Oracle Generally Available in MySQL 5.7.17 on December 2016 MySQL InnoDB Cluster as Solution

Cisco Prime Network Registrar IPAM 8.1.3 MySQL Database ... · Troubleshooting MySQL 21 1 MySQL Database Replication Overview The MySQL Database Replication Guide is designed to help

High Availability Using MySQL Group Replication · 2020-06-19 · MySQL Group Replication •What is MySQL Group Replication? “Update everywhere replication plugin for MySQL with

Troubleshooting MySQL Performance

Diagnosing MySQL Replication Failures

MySQL Replication Tutorial Presentation (pdf) - O'Reilly …assets.en.oreilly.com/1/event/2/MySQL Replication Tutorial... · MySQL Replication Tutorial Lars Thalmann Technical lead

MySQL replication & cluster

MySQL 5.6 Replication Features

MySQL 5.7 Replication News

Reliable MySQL Using Replication