Galera Cluster Best Practicesfor DBAs and DevOps

Philip StoevCodership Oy

Agenda• A very quick overview of Galera Cluster• Ongoing monitoring of the cluster and detection of

bottlenecks• Backup strategies• Selecting the optimal State Snapshot Transfer (SST)

method

Galera Cluster OverviewSynchronous

– each transaction is immediately replicated on all nodes at commit– no stale slaves

Multi-Master– read from and write to any node– automatic transaction conflict detection

Replication– a copy of the entire dataset is available on all nodes– new nodes can join automatically

For MySQL– based on a modified version of MySQL (5.5, 5.6 with 5.7 coming up)– InnoDB storage engine

And more …• Recovers from node failures within seconds• Data consistency protections

– avoids reading stale data– prevents unsafe data modifications

• Cloud and WAN support

Monitoring Galera Cluster

General Principles of Monitoring• Galera Cluster is first and foremost MySQL + InnoDB:

– all traditional advice still applies (e.g. slow query log)– InnoDB still does most of the heavy lifting (e.g. I/O)

• Galera reports metrics via SHOW GLOBAL STATUS– FLUSH STATUS resets counters

• Galera reports events via the MySQL server error log– good idea to check you logs or log rotation scripts

• Monitor each node separately for maximum visibility

Monitoring the NetworkEspecially in WAN or complex network environments:• monitor the health of the network using a third-party tool:

– round-trip (ping) times– bandwidth utilization

• monitor each network link between any two segments separately– Galera assumes all links perform equally well

Detecting Issues With Cluster Health• SHOW GLOBAL STATUS variables:

MySQL [test]> show status like '%wsrep%';+------------------------------+-------------------------------------------+| Variable_name | Value |+------------------------------+-------------------------------------------+| wsrep_ready | ON || wsrep_cluster_status | Primary (or non-Primary) || wsrep_local_state_comment | Synced (or Donor/Desynced, Joiner) |

| wsrep_cluster_size | 3 |+------------------------------+-------------------------------------------+

Detecting Galera-Specific BottlenecksMySQL [test]> show global status like '%wsrep%';+------------------------------+--------------------------------------------+| Variable_name | Value |+------------------------------+--------------------------------------------+| wsrep_flow_control_paused | 0.100000 || wsrep_flow_control_sent | 123 || wsrep_local_bf_aborts | 77 || wsrep_local_cert_failures | 24 |+------------------------------+--------------------------------------------+

In case of BF aborts or cert failures:• try increasing wsrep_retry_autocommit• move conflicting workload to just one node

Detecting Galera-Specific Bottlenecks #2• SHOW PROCESSLIST:

MySQL [test]> show processlist;+----+------+-----------+------+---------+------+----------+---------------------------+| Id | User | Host | db | Command | Time | State | Info |+----+------+-----------+------+---------+------+----------+---------------------------+...| 5 | root | localhost | test | Query | 10 | query end| INSERT INTO t1 VALUES (1) |...3 rows in set (0.05 sec)

Backups

Backups• Multiple nodes do not remove the need for backups

– backups also help in case of human error• All nodes in Galera have the same information, so

backing up one node is sufficient• Galera does not use the binary log files, so generate

and back those up only if you have a specific need• Practicing the restore operation is highly recommended

Performance Considerations• Backups slow down the node being backed up

– which in turn can cause the entire cluster to slow down• or reduce the number of available nodes in the cluster

– can affect quorum calculations• you can have an async slave and take backups from it

– Just do not let it fall far behind the master cluster

XtraBackupNon-blocking backup for InnoDB• --galera-info option records the precise position

when the backup was taken in a file named xtrabackup_galera_info

• a short lock is still taken

Backup Procedure with a Dedicated Node• Make sure backup node will get no write traffic

– make sure all existing sessions are disconnected• Temporarily disconnect a node from the cluster

– SET GLOBAL wsrep_provider = ‘none’• Perform backup• Restore the value of wsrep_provider; check for wsrep_ready=ON

Restoring Single Node from BackupOption #1:

– wipe the data directory clean and restart– let it rejoin the cluster via SST– the operation will involve a donating node as well

Option #2:– use a backup that was recently taken– wipe the data directory clean and restore data– put the information from xtrabackup_galera_info into grastate.dat

# GALERA saved stateversion: 2.1uuid: 021a77ed-80cf-11e6-9e8e-6249ad0d3a57seqno: 1234cert_index:

– restart node and it will catch up via IST

Restoring Entire Cluster #1Note: a new logical cluster is being createdEasiest approach:

1. Restore data on one node and restart first node with –wsrep-new-cluster

2. Start one more node. Node rejoins via SST, with first node as donor

3. Restart two more nodes, first two nodes serve as donors, and so forth.

Restoring Entire Cluster #2Optimized approach:1. Restore backup on first node, bootstrap 1-node

cluster. Prevent new transactions from being issued.2. Restore backup on other nodes and create

grastate.dat on them:# GALERA saved stateversion: 2.1uuid: <new-cluster-uuid>seqno: 0cert_index:

SST

General Principles• It is always best to avoid SST altogether by sizing

gcache.size appropriately• SST choice is determined by:

– availability of a dedicated donating node– encryption requirements (SST encryption is configured

separately from other cluster operations)– bandwidth utilization

SST Methods• Rsync (the default)

– causes the donor to block new transactions • XtraBackup

– Donor remains operational• Mysqldump

– logical, rather than physical database copy

Configuration without a Dedicated Donor• Use xtrabackup-v2 SST method

– requires the installation of XtraBackup– there is still performance impact on the donor– encryption, compression are available

Configuration with Dedicated Donor• rsync may be the best method to use:

– available on any machine– compression and partial file transfers are possible

• use wsrep_sst_donor to specify donor node to use:– takes a list of node names as configured with

wsrep_node_name– if the list terminates with a comma, other nodes can also be

used for donors if none of the nodes from the list are available• if using an arbitrator, consider switching it to a full

Galera node that can then act as a dedicated donor

Questions

• Please use the Question/Chat box in the GoToWebinar panel

• Ideas welcome for future webinars

Thank You

http://www.galeracluster.com

Discussion group:

codership-team@googlegroups.com