MySQL backup and restore best practices

Roman VynarSeptember 21, 2015

Introduction

This is a hands-on tutorial on setting up the different types of MySQL backups and recovering from various disasters.

Prerequisites:

• good knowledge of Linux, Shell and MySQL

• be familiar with mysqldump and MySQL binlogs

• a laptop with Virtualbox installed

2

Topics to be covered

• Planning, strategies and tools for MySQL backups

• Ensuring the replication consistency

• Role of the delayed slaves

• Binary backups with Percona Xtrabackup and mylvmbackup

• Logical backups with mysqldump and mydumper

• Binlog backups with mysqlbinlog

• Encryption of backups

• Off-site backups

• Monitoring of backups and its importance

• Saving disk space

• Backing up Galera node

• Various recovery scenarios including disaster recovery

3

Virtualbox preparation

There are two pre-installed virtual machines:

• master.vm - MySQL master

• slave.vm - MySQL slave

Copy Virtualbox Appliance from USB stick provided to your laptop

Double-click on Appliance.ova to import it into Virtualbox. It includes 2 VMs.

4

Virtualbox network

Each instance has 2 network adapters:

• Host-only adapter

• NAT

Configure host-only network from the main menu: Virtualbox > Preferences > Network > Host-only Networks > “vboxnet0” or “Virtualbox Host-Only Ethernet Adapter” > edit and set: 192.168.56.1 / 255.255.255.0Windows users only: open Setting > Network and click OK to re-save host-only network adapter.

5

Starting instances

Internal IP addresses statically assigned:• master.vm - 192.168.56.201• slave.vm - 192.168.56.202

• Both instances are running CentOS 6.7 and have the necessary packages pre-installed.

• Unix and MySQL root password: abc123

Launch both instancesVerify network connectivity

6

Pre-installed packages

• EPEL repo:rpm -Uvh http://mirror.omnilance.com/epel/6/i386/epel-release-6-8.noarch.rpm

• Percona repo:rpm -Uvh http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm

• Packages:yum install Percona-Server-server-56 Percona-Server-client-56 Percona-Server-shared-56 percona-xtrabackup percona-toolkit mylvmbackup nsca-client qpressrpm -ivh files/mydumper-0.9.0-179.el6.x86_64.rpm

7

Pre-configuration

We have already installed Percona Server 5.6, setup the replication master > slave and placed the basic /etc/my.cnf and credentials file /root/.my.cnf.

Also we have loaded a sample database “employees” from https://launchpad.net/test-db

Under /root/files/ there are a few scripts to simplify some of our tutorial tasks.

Verify replication status of slave.vm.

8

Factors to consider for backups

• Application bugs

• Software misconfigurations

• Hardware failures

• People mistakes

• Access breach

• Disasters

9

Some examples of problems

• A DBA runs DELETE query on the wrong data set

• Your application has a bug that overwrites data

• A table or entire schema was dropped by accident

• InnoDB corruption bug that propagates via replication

• Your database or application was hacked

• A disk failed and RAID array does not recover

• A hurricane puts a few feet of water in your datacenter

10

Planning and strategies

• Perform both logical and binary backups

• Store backups on more than one server

• Store copies of your backups off-site

• Estimate the backup retention

• Do capacity planning

• Monitor backups

• Verify replication consistency periodically

• Test your backups

11

Tools for MySQL backups

Base tools:

• mysqldump

• mysqlbinlog

• mydumper / myloader

• Percona Xtrabackup

• mylvmbackup

Supporting tools:

• s3cmd

• qpress

• gpg

12

Replica is not a backup

• A replication slave is not a backup!

• A slave is the right place to take backups from.

• A delayed slave can be used for a really fast partial data recovery.

13

Ensuring replication consistency

pt-table-checksumhttp://www.percona.com/doc/percona-toolkit/2.1/pt-table-checksum.html

Checksum master from any node:pt-table-checksum h=192.168.56.201

Verify no table diffs:pt-table-checksum h=192.168.56.201 --replicate-check-only

14

Let’s create some discrepancy in replication.

Run UPDATE query on the slave:mysql> UPDATE employees.dept_manager SET from_date=CURDATE() WHERE emp_no=110022;

Checksum master again, just this one table:pt-table-checksum h=192.168.56.201 --tables employees.dept_manager

Fix it by running pt-table-sync on the master:pt-table-sync --execute --replicate percona.checksums --sync-to-master 192.168.56.202

Re-checksum the table in question:pt-table-checksum h=192.168.56.201 --tables employees.dept_manager

15

Create and fix a replication diff

Role of delayed slaves

pt-slave-delayhttp://www.percona.com/doc/percona-toolkit/2.1/pt-slave-delay.html

Let’s delay the slave:pt-slave-delay --delay 1h 192.168.56.202

Check out SHOW SLAVE STATUS

16

Percona Xtrabackup is an open source, free MySQL hot backup software that performs non-blocking backups for InnoDB.

• Backups that complete quickly and reliably

• Uninterrupted transaction processing during backups

• Savings on disk space and network bandwidth

• Automatic backup verification

• Higher uptime due to faster restore time

17

Binary backups: Percona Xtrabackup

• Setting up new slaves

• When we lose an entire server due to hardware failure, corruption, etc.

• When the majority of data on the server was lost

• Basically when restoring may take less time that trying to load a logical backup or perform InnoDB recovery

18

When do we restore from Xtrabackup?

From now on, all the tasks we will be performing on the slave until further notice.

mkdir -p /backups/xtrabackup

On demand, increase open_file_limit:ulimit -n 65536

Run Xtrabackup:innobackupex --compress --rsync --slave-info /backups/xtrabackup

ls -la /backups/xtrabackup/

19

Creating a backup with Xtrabackup

Percona Xtrabackup supports incremental backups, which means that it can copy only the data that has changed since the last full backup.

Do some change on Master for testing purpose:mysql> UPDATE employees.dept_manager SET from_date=CURDATE();mkdir /backups/xtrabackup_incinnobackupex --slave-info --incremental --incremental-basedir=<path to full> /backups/xtrabackup_incCompare size of full vs incremental backup:du -sh /backups/xtrabackup*/*

20

Incremental backup with Xtrabackup

mylvmbackup is a tool for quickly creating full physical backups of a MySQL server's data files

http://www.lenzg.net/mylvmbackup/

It performs the actual backup by accessing the file system directly. It is also a requirement that the MySQL server's data directory resides on an LVM volume and the same volume group has as much free space for a snapshot.

21

Binary backups: mylvmbackup

When do we restore from mylvmbackup:

• When the MySQL datadir is huge

• Need to restore all at once

Let’s create a backup:

mkdir /backups/mylvmbackup

mylvmbackup --vgname=Mysql_VG --lvname=data --lvsize=1G --backupdir=/backups/mylvmbackup

It will create a compressed backup while your MySQL server is running.

22

Using mylvmbackup

• Simplest way of doing logical backups.

• Dumps tables in a single thread

• Available with a standard MySQL distribution

• Can perform online backup for InnoDB tables

mkdir /backups/mysqldump

mysqldump --single-transaction --all-databases | gzip > /backups/mysqldump/today.sql.gz

23

Logical backups: mysqldump

mydumper is a tool used for backing up MySQL database servers much faster than the mysqldump.

• Parallelism (hence, speed) and performance (avoids expensive character set conversion routines, efficient code overall)

• Easier to manage output (separate files for tables, dump metadata, etc, easy to view/parse data)

• Consistency - maintains snapshot across all threads, provides accurate master and slave log positions, etc

• Manageability - supports PCRE for specifying database and tables inclusions and exclusions

24

Logical backups: mydumper/myloader

• Restoring a single table

• Restoring a single schema or rolling forward a single schema to a point in time (involves using binlogs)

• Restoring data in multiple threads

• Creating a live copy of database

25

When do we restore from mydumper?

Create backup:

mkdir -p /backups/mydumper/today

mydumper --outputdir=/backups/mydumper/today --host=localhost --compress --kill-long-queries --verbose=3 --build-empty-files --triggers --events —routines

Note, mydumper supports full schema dump including SP, views, events and triggers starting the version 0.9.0.

26

Create mydumper data backup

mydumper backup structure

• metadata

• db1-schema-create.sql(.gz)

• db1.table1-schema.sql(.gz)

• db1.table1-schema-triggers.sql(.gz)

• db1.table1-schema-view.sql(.gz)

• db1-schema-post.sql(.gz) (functions, SP, events)

• db1.table1.sql(.gz)

• …• similar for all other tables and databases

27

• /backups/mydumper/today/mysql-schema-create.sql.gz

• /backups/mydumper/today/mysql.db-schema.sql.gz

• /backups/mydumper/today/mysql.db.sql.gz

• /backups/mydumper/today/mysql.user-schema.sql.gz

• /backups/mydumper/today/mysql.user.sql.gz

• …

• /backups/mydumper/today/metadata

28

mydumper backup structure

• mysqlbinlog --read-from-remote-server --raw --stop-never

• Mirror the binlogs on the master to a second server

• Works with older 5.1/5.5 servers also

• Roll forward backups even after losing the master

• Useful for point-in-time recovery or disaster recovery

• Easy to backup off-site

• Takes very little resources to run. Can run about anywhere with disk space and writes out binlog files sequentially.

29

Binlog backups: mysqlbinlog 5.6

• Ensure it stays running, restart it if it appears to be hanging• Optionally, verify the file size is the same on master and

slave• Re-transfer files that are partially transferred• Compress the files after successful transfer

mkdir -p /backups/binlogscd /backups/binlogsmysqlbinlog --raw --read-from-remote-server --stop-never --verify-binlog-checksum --host=192.168.56.201 --stop-never-slave-server-id=999 mysql-bin.000001Compare what was downloaded and master binlogs

30

Pulling binlogs with mysqlbinlog

• To prevent information disclosure, NEVER upload your backups off-site unencrypted

• GPG tools can help you to achieve a strong level of encryption:

• Encrypt file: gpg --batch --yes --encrypt --always-trust --quiet --recipient=my@backups.local --output=file.sql.gpg file.sql

• Decrypt file:gpg --batch --decrypt --passphrase-fd=0 --quiet --output=file.sql file.sql.gpg

31

Encryption of backups

Generate GPG key backup@slave.vm for our backups:gpg --gen-keyTo gain enough entropy you can run Xtrabackup a few times:innobackupex --compress --rsync --slave-info /backups/xtrabackup

Encrypt mydumper backup folder:/root/files/encrypt.sh

Decrypt folder:/root/files/decrypt.sh

Backup GPG keys once:gpg --export --armor backup@slave.vm > public_key.ascgpg --export-secret-keys --armor backup@slave.vm > private_key.asc

Keep passphrase and keys backup in the save place!

32

Encrypting backup

Amazon S3 is a reasonably priced data storage service. Ideal for off-site file backups, file archiving, web hosting and other data storage needs.

With S3 you can set bucket lifecycle properties so data over X days is archived to Glacier and data over Y days is removed.

33

Off-site backups to Amazon S3

s3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers.

It is ideal for batch scripts and automated backup upload to S3, triggered from cron, etc.http://s3tools.org/s3cmd

34

s3cmd

• s3cmd --configure

• enable https

• Check bucket permissions:s3cmd info s3://bucket/Beware, check there is no ACL for http://acs.amazonaws.com/groups/global/AuthenticatedUsers, it makes your data public!

• Set lifecycle rule for the bucket as appropriate

• Upload backup:s3cmd --server-side-encryption sync backupdir/ s3://bucket/path/

35

Using s3cmd

• Always monitor backup status

• Employ passive checks (sending reports)

• Consider checking freshness of backups

• Some kind of heartbeat for mysqlbinlog needed

• Most of issues with backups are due to problems with FLUSH TABLE WITH READ LOCK or long running queries. So MySQL monitoring itself is recommended.

36

Monitoring of backups and its importance

With Nagios, you can employ NSCA alerts:sh -x /root/files/monitor.sh

Nagios option “freshness_threshold” could be set for a service check so if your NSCA report has not checked in within a certain period it will alert you.

define service{

use passive-service

host_name slave.vm

service_description Backup xtrabackup

freshness_threshold 129600

check_command check_dummy!36 hours

}

37

Monitoring backups with Nagios

• Enable compression

• Perform incremental backups with Xtrabackup

• Hardlink files:

If md5sum(file1) == md5sum(file2):

cp -al file1 file2

Create one more mydumper backup to have two at least

Check out and run the script:/root/files/hardlink.sh

du -sh /backups/mydumper/*

38

Saving disk space

Backing up Galera node

It is recommended to turn on MySQL variable wsrep_desync prior taking a backup from Galera/Percona Xtradb Cluster node to disable sending out flow control messages:mysql> SET GLOBAL wsrep_desync = 1;

And turn off back after the backup is complete and wsrep_local_recv_queue is low:

mysql> SET GLOBAL wsrep_desync = 0;

This ensures the whole cluster won’t get frozen for a long time during and after the backup process.

39

Partial data recovery:

• Recovering a single table or dataset

• Usually involves using a logical backup

• Sometimes applying selective events further from the binlogs too

Full data recovery:

• Can be done from any type of full backup, just the speed matters

Point-in-time recovery (disaster scenario):

• Involves full data recovery and replaying the binlogs to the desired point in time

40

Recovery scenarios

Update table employees.dept_emp table using some garbage values or imagine a statement was accidentally executed without a where clause. On the master, run:mysql> UPDATE employees.dept_emp SET to_date=CURDATE();

Restore the whole table from the slave:mysql -h 192.168.56.201 employeesmysql> TRUNCATE TABLE employees.dept_emp;zcat /backups/mydumper/today/employees.dept_emp.sql.gz | mysql -h 192.168.56.201 employees

Verify data.

41

Partial data recovery: a single table reload

Let’s now create a copy of one database on the slave using myloader.

myloader --host=localhost --source-db=employees --threads=8 --queries-per-transaction=10 --verbose=3 --database=employees_copy --directory /backups/mydumper/today

If you want myloader to replicate data, add --enable-binlog option.

42

Partial data recovery from mydumper

How myloader restores from backup

1. create DB

2. create tables and dummy views as empty MyISAM tables

3. restore data

4. restore SP, functions, events

5. drop dummy views and reload them normally

6. restore triggers

43

Full data recovery from Xtrabackup

service mysql stop

rm -rf /data/mysql

cp -r /backups/xtrabackup/<path> /data/mysql

innobackupex --decompress /data/mysql/

innobackupex --apply-log --use-memory=256M /data/mysql/

chown -R mysql:mysql /data/mysql

service mysql start

cat /data/mysql/xtrabackup_slave_info

mysql> CHANGE MASTER TO MASTER_LOG_FILE='<BINLOG>', MASTER_LOG_POS=<POS>, MASTER_HOST='192.168.56.201', MASTER_USER='repl', MASTER_PASSWORD='abc123';

mysql> START SLAVE;

44

Reloading from incremental Xtrabackup

service mysql stoprm -rf /data/mysqlcp -r /backups/xtrabackup/<path to full> /data/mysqlinnobackupex --decompress /data/mysql/innobackupex --apply-log --redo-only --use-memory=256M /data/mysql/innobackupex --apply-log --redo-only --use-memory=256M /data/mysql/ --incremental-dir=/backups/xtrabackup_inc/<path to incremental>

If you want to apply more incremental backups, repeat the last step for them. It is important that you do this in the chronological order in which the backups were done. You can consult to xtrabackup_checkpoints file in the directory of each one for order.

45

Reloading from incremental Xtrabackup

innobackupex --apply-log --use-memory=256M /data/mysql/chown -R mysql:mysql /data/mysqlservice mysql startcat /data/mysql/xtrabackup_slave_infomysql> CHANGE MASTER TO MASTER_LOG_FILE='<BINLOG>', MASTER_LOG_POS=<POS>, MASTER_HOST='192.168.56.201', MASTER_USER='repl', MASTER_PASSWORD='abc123';mysql> START SLAVE;mysql> SHOW SLAVE STATUS\G

46

Full data recovery from mylvmbackup

service mysql stop

rm -rf /data/mysql

cd /backups/mylvmbackup

tar zxf <backup file>

mv backup/mysql /data/mysql

service mysql start

mysql> SHOW SLAVE STATUS\G

47

Imagine we have only one server master.vm and all the next actions we will be doing on it.

Create Xtrabackup backup:innobackupex --compress --rsync /root

Do some notable changes in MySQL and flush logs for testing, e.g.:mysql> DROP TABLE employees.salaries;mysql> FLUSH LOGS;

Pull binlogs locally using mysqlbinlog simulating remote downloading:mkdir /root/binlogscd /root/binlogsmysqlbinlog --raw --read-from-remote-server --stop-never --verify-binlog-checksum --host=localhost --stop-never-slave-server-id=999 mysql-bin.000001

Check and compare if all the binlogs were downloaded

48

Point-in-time recovery

Stop mysqlbinlog Destroy MySQL:service mysql stop rm -rf /var/lib/mysqlRestore from Xtrabackup:mv /root/<path> /var/lib/mysql innobackupex --decompress /var/lib/mysql innobackupex --apply-log --use-memory=256M /var/lib/mysql chown -R mysql:mysql /var/lib/mysqlservice mysql startVerify that MySQL data is not the recent, i.e. at the point of taking Xtrabackup.

49

Point-in-time recovery

Roll forward the binlogs from backup:cat /var/lib/mysql/xtrabackup_binlog_info mysqlbinlog --start-position=<POS> /root/binlogs/<starting binlog> | mysql

Load all the binlogs after this one till the last we have.

Check out that all the recent changes were applied after time Xtrabackup was done. We have completed the disaster recovery 😇

The same way we can restore to any point in time, this is limited to how old full backup we have, binary or logical, and presence of all the related binlog files.

50

Point-in-time recovery

Percona Backup Service

Let the Percona Managed Services team worry about your MySQL backups and data recovery! https://www.percona.com/services/managed-services/remote-dba

• Reliable MySQL Backups

• Efficient MySQL Data Recovery

• Cost Effective and Highly Flexible

51

Questions?

Thank you for attending!

52