Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup...

25
ACCELERATING MYSQL WITH FLASHBLADE A FRAMEWORK FOR FASTER BACKUP & RECOVERY USING FLASHBLADE TECHNICAL WHITE PAPER

Transcript of Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup...

Page 1: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

ACCELERATING MYSQL WITH FLASHBLADE A FRAMEWORK FOR FASTER BACKUP & RECOVERY USING FLASHBLADE

TECHNICAL WHITE PAPER

Page 2: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

2

TABLE OF CONTENTS

INTRODUCTION ........................................................................................................................................ 3

THE PROBLEM .......................................................................................................................................... 3

THE SOLUTION ......................................................................................................................................... 3

OVERVIEW ................................................................................................................................................. 4

AUDIENCE ................................................................................................................................................. 4

MYSQL OVERVIEW ................................................................................................................................... 4

SOLUTION DESIGN .................................................................................................................................. 4

Solution Overview .............................................................................................................................. 4

Environment Overview ..................................................................................................................... 5

Server Configuration ......................................................................................................................... 5

Virtual Machine Configuration ....................................................................................................... 6

MySQL Configuration ......................................................................................................................... 7

FlashBlade Configuration ................................................................................................................ 8

BACKUP AND RESTORE TESTS AND RESULTS ................................................................................ 10

Logical Backup Test .......................................................................................................................... 10

Restore Logical Backups Test ....................................................................................................... 13

Physical Backup Test ........................................................................................................................ 15

Restore Physical Backups Test ..................................................................................................... 18

BEST PRACTICES FOR MYSQL ON FLASHBLADE ............................................................................ 21

CONCLUSION .......................................................................................................................................... 23

REFERENCES .......................................................................................................................................... 24

ABOUT THE AUTHOR ............................................................................................................................ 25

Page 3: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

3

THE PROBLEM

Enterprise usage of MySQL may range anywhere from a handful to hundreds of databases. With the advent of

virtualization, usage has been steadily increasing. In addition to the growing number of MySQL databases, the

growth of data that feeds these databases has been staggering.

This poses a key challenge to database administrators, not only in managing databases but in backing up databases.

DBAs have to perform backups periodically and restores as needed within agreed service levels, and those SLAs

change frequently to meet business requirements.

THE SOLUTION

FlashBlade™ is a ground-breaking scale-out flash storage system from Pure Storage®. Engineered with a massively

parallel architecture from software to hardware, it has emerged as the industry-leading solution for use cases like

artificial intelligence and modern analytics that require the highest performance. Many customers have tapped this

performance to modernize their backup and recovery infrastructure.

The current FlashBlade system can support a read rate of 16 GBps and a write rate of 4.5 GBps, in a single 4U chassis.

The sustained write rate of 4.5 GBps is equivalent to a backup rate of 15 TB/hour, which is well above the normal

required backup rate for many customers. The restore rates from FlashBlade can be higher than 45 TB/hour with

concurrent restore operations.

MySQL customers can now direct their MySQL backups to FlashBlade to accelerate backup and restore significantly.

Customers can realize these backup/restore rates without requiring any exotic and costly software on their hosts for

block differencing purposes.

While the performance of FlashBlade is very impressive, even more attractive is its non-disruptive scalability.

The capacity and performance of FlashBlade can be scaled in a non-disruptive manner, simplifying the operational

procedures of a customer’s backup/recovery process. The system can be scaled in seconds, without running special

commands or extra cables. Additionally, FlashBlade's in-line compression feature reduces capacity requirements by

almost 66% for most typical databases – driving costs down significantly while not sacrificing performance. Allowing

the storage system to handle compression also saves dramatically on host CPU cycles.

INTRODUCTION

MySQL is the preferred database platform for the majority of websites worldwide,

as well as thousands of corporate web-based applications. This open-source

database is ubiquitous: most SaaS providers and top social media companies

have deployed MySQL as the database of choice for their key services. As MySQL

deployments and data volumes grow, the importance of efficient backup and

recovery infrastructure becomes critical.

Page 4: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

OVERVIEW

The purpose of this technical white paper is to provide techniques and guidelines to rapidly backup and restore

MySQL databases on FlashBlade. The goal of this document is to help MySQL administrators to meet their backup/

recovery service level objectives.

AUDIENCE

The target audience for this document includes but is not limited to MySQL database administrators, storage

administrators, IT managers, system architects, sales engineers, field consultants, professional services, and partners

who are looking to design and deploy MySQL backups on FlashBlade. A working knowledge of MySQL, Linux®, server,

storage, and networks is assumed but is not a prerequisite to read this document.

MYSQL OVERVIEW

MySQL is very simple to use and at the same time provides the flexibility and scalability to support data growth.

One of the key reasons enterprises choose MySQL is for its total cost of ownership (TCO). Even with the commercial

Enterprise version, there are no up-front licensing costs, which means lower capital spending.

MySQL databases can be backed up in two ways:

• Physical backups of directories and files that store database contents. These backups are faster, as they

involve file copying without conversion. They are portable only to other servers that have similar OS and

MySQL database characteristics. Physical backup tools include mysqlbackup of MySQL Enterprise Backup

from Oracle, Percona XtraBackup (a free, open source, online backup solution for MySQL), MariaDB, and

Percona Server for MySQL.

• Logical backups of database structure and content. These are slower than physical backups, as the server

must access database information and convert it to a logical format. The advantage of logical backups is

that they are machine-independent and highly portable. mysqldump is the preferred and most prevalent tool

used by DBAs to perform logical backup and restore of MySQL databases.

SOLUTION DESIGN

Solution Overview

The solution to accelerate MySQL data protection, be it physical or logical, is to enable parallelism at the application

level and to use the Pure Storage FlashBlade storage system as the target and source, respectively, for backup and

restore activity.

FlashBlade is a scale-out NAS system that is engineered with massively parallel architecture. On top of the

performance gained from flash technology, enabling parallel connections to the FlashBlade system from the

host across different IP addresses and ports can better utilize the compute and network resources of the

blades in the FlashBlade system.

4

Page 5: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

5

In this paper, we tested both the physical and logical backup and restore of a MySQL database on FlashBlade.

For physical backup and restore, we chose Percona Xtrabackup, while for logical backup and restore we chose

mysqldump. For the sake of clarity, we performed only full backups and restores in our tests.

Environment Overview

As part of this solution, we wanted to simulate a real-world scenario in which numerous MySQL databases are

distributed across multiple storage systems and backed up to the same target at the same time.

The setup includes 20 virtual machines running MySQL on Red Hat Linux 7.4 across four ESX servers. VMFS

datastores were setup across three Pure Storage FlashArrays (//M20, //M50, and //X70) that were connected to these

ESX servers. The 20 virtual machines were distributed across these three storage systems (4 on //M20, 8 on //M50,

and 8 on //X70).

Each virtual machine ran MySQL with a source database size of 70 GB. As the backups were to be performed on

to FlashBlade, volumes/filesystems from FlashBlade were mounted over NFS protocol on these virtual machines.

As the backups were to be targeted to a single destination, the same NFS filesystems were mounted across all

virtual machines.

FIGURE 1. MySQL databases distributed across multiple storage systems and backed up to the same target

Server Configuration

Four Intel® CPU based Cisco UCS B-series B200 M4 blade servers were deployed to host the ESX server. Each server

had a Cisco UCS VIC 1340 card, and they were connected by four ports from each Cisco Fabric extender of the Cisco

UCS chassis to the Cisco Fabric Interconnect, which was in turn connected to the Cisco MDS 9148S for upstream

connectivity to the Pure Storage FlashArrays. The server configuration is described in Table 1.

Page 6: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

COMPONENT DESCRIPTION

PROCESSOR 4 X INTEL XEON E5-2609 V4 1.7 GHZ (2 CPUS WITH 8 CORES EACH)

MEMORY 64 GB @ 2.4GHZ (2 X 32GB)

HBA4 X 10G PORTS ON CISCO UCS VIC 1340 (UCSB-MLOM-40G-03) 40GBPS

NIC6 INTERFACES ON CISCO UCS VIC 1340

5 CONFIGURED - 1 FOR PUBLIC, 4 FOR BACKUP

UCS FIRMWARE (ACROSS ALL COMPONENTS) 3.1 (2B)

TABLE 1. UCS Blade configuration

Four data VIPs (virtual IPs) were created on FlashBlade, each one of them configured to the VLAN (201 to 204).

The ESX servers were configured with five virtual switches, one for public traffic and the remaining four for FlashBlade

traffic. The four virtual switches for backup traffic in ESX were tagged with corresponding VLANs (201 to 204).

Virtual Machine Configuration

Each VM was configured with 4 vCPUs and 4 GB of memory.

FIGURE 2. Typical VM configuration

The VM was configured with two disks. Red Hat O.S was installed on the first disk and the MySQL database data

on the second disk. The second disk was carved out from one of the three Pure FlashArrays (//M20, //M50, //X70).

The VM was configured with five network adapters; one was used for the public network and another four for the

6

Page 7: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

backup network. The NFS filesystem from FlashBlade, where the MySQL backup was to be stored, was mounted

across these four interfaces. Four private IP addresses were configured for the four interfaces.

MySQL Configuration

The open source MariaDB variant of MySQL was installed on the Red Hat Linux 7.4 virtual machines.

+----------------------------+----------------------+

| Variable_name | Value |

+----------------------------+----------------------+

| innodb_version | 5.5.52-MariaDB-38.3 |

| protocol_version | 10 |

| slave_type_conversions | |

| version | 5.5.56-MariaDB |

| version_comment | MariaDB Server |

| version_compile_machine | x86_64 |

| version_compile_os | Linux |

+----------------------------+----------------------+

For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we

used the utility innobackupex, which is a perl wrapper for the xtrabackup tool, as innobackupex allows additional

restore features like copy-back or move-back that would copy or move the backup files to the datadir location.

As of Percona Xtrabackup 2.3+, the features of innobackupex are incorporated into xtrabackup and

innobackupex may be deprecated in the next major release.

The following are some of the key parameters that were configured in the my.cnf file.

datadir = /m01/data

innodb_file_per_table=1

The datadir parameter was configured to point to the Pure FlashArray volume where the MySQL database was

to be stored.

7

Page 8: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

The innodb_file_per_table was enabled to store the tables and indexes in a separate .ibd datafile, as opposed

to a single, big, monolithic datafile. There are various advantages to file-per-table tablespaces – like better file

management, reduced fragmentation, faster truncate, etc. – but the key one in this context is acceleration of the

backup process, and specifically with physical backups through XtraBackup. XtraBackup allows multiple files

to be backed up concurrently using the parallel option.

FlashBlade Configuration

COMPONENT DESCRIPTION

FLASHBLADE 14 X 17TB BLADES

CAPACITY224 TB RAW

151.63 TB USABLE (WITH NO DATA REDUCTION)

CONNECTIVITY 4 X 40 GB/S ETHERNET (DATA)

1 GB/S ETHERNET (MANAGEMENT PORT)

PHYSICAL 4U

SOFTWARE PURITY//FB 2.1.1

TABLE 2. FlashBlade configuration

The FlashBlade network settings were configured with four subnets across four VLANs.

FIGURE 3. FlashBlade network settings

8

Page 9: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

For the MySQL backup target, we created two NFS filesystems, each with a size of 10TB. Logical backups were sent

to the NFS filesystem named mysql-backup-01, and physical backups were sent to the NFS filesystem named mysql-

backup-02. We created two NFS filesystems to identify the data reduction rate (if any) on FlashBlade specific to the

type of backups.

FIGURE 4. Creating NFS filesystems

Even though the target for the logical backup is a single filesystem, mounting it on a single mount point across all

VMs on the same interface would make that underlying adapter on the ESX a bottleneck and limit bandwidth to

1 GB/s (10 Gbps). Thus, to maximize bandwidth, we mounted the same filesystem across four interfaces on all 20

virtual machines across the four subnets.

Similarly, the target filesystem for physical backups was mounted on four interfaces on all 20 VMs.

Note: Be sure to size the NFS filesystem(s) to hold the MySQL backup. Ideally, the filesystems should be

sized based on the retention and frequency of backups. Of course, you will benefit from the data reduction

offered by FlashBlade; the data reduction rate will vary based on the type of data. As the space is always thin-

provisioned, there is no penalty for creating a larger-sized filesystem.

9

Page 10: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

BACKUP AND RESTORE TESTS AND RESULTS

Our backup and restore tests included the following four tests across logical and physical backup mechanisms.

They were tested singly and on 20 virtual machines to showcase the scalability aspect of the solution.

1. Logical backup using mysqldump

2. Restore logical backup

3. Physical backup using Percona Xtrabackup

4. Restore physical backup using Percona Xtrabackup

Logical Backup Test

Each VM was configured with backup scripts like the one below, directing the mysqldump output on to FlashBlade

across different mount points. The mysqldump commands were run in parallel to overcome the single threaded

limitation of mysqldump, especially if run at database level. Since the virtual machine was configured with four vCPUs,

we limited the parallel threads to four.

As the test environment had only one database (dss), we had to run the logical backups at table level. If you have

multiple databases, you can run mysqldump in parallel at the database level.

The backups are organized under the same NFS filesystem by the servername, which is passed as a parameter

whether it was invoked from a single virtual machine or across all 20 of them.

mysqldump -u root -q dss table1 > /b01/backup/$server/table1.sql &

mysqldump -u root -q dss table2 > /b02/backup/$server/table2.sql &

mysqldump -u root -q dss table3 > /b03/backup/$server/table3.sql &

mysqldump -u root -q dss table4 > /b04/backup/$server/table4.sql &

wait

mysqldump -u root -q dss table5 > /b01/backup/$server/table5.sql &

mysqldump -u root -q dss table6 > /b02/backup/$server/table6.sql &

mysqldump -u root -q dss table7 > /b03/backup/$server/table7.sql &

mysqldump -u root -q dss table8 > /b04/backup/$server/table8.sql &

Even though the same filesystem (mysql-backup-01) was mounted across four mount points, the four mount points

were configured across different subnets, enabling network aggregations that can yield higher bandwidth.

LOGICAL BACKUP ON A SINGLE VIRTUAL MACHINE

The backup script was run on a single virtual machine to get a baseline. It took 9 minutes 11 seconds to backup 56.4 GB

(actual data, not including white spaces) of the MySQL database, yielding a throughput of 104 MB/s. One option to

increase throughput is to increase the number of parallel mysqldump sessions, provided you have available compute

resources (as in our case). We were limited by the 4 vCPUs at the virtual machine level.

10

Page 11: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

The FlashBlade GUI shows bandwidth usage of around 110 MB/s for most of the backup activity.

FIGURE 5. Bandwidth usage on FlashBlade

LOGICAL BACKUP ON 20 VIRTUAL MACHINES

The backup script listed in the earlier section was run across all 20 virtual machines. The 20 virtual machines

were driving a total of 1.1 TB of backup data into FlashBlade across the four interfaces.

mysqldump -u root -q dss table1 > /b01/backup/$server/table1.sql &

mysqldump -u root -q dss table2 > /b02/backup/$server/table2.sql &

mysqldump -u root -q dss table3 > /b03/backup/$server/table3.sql &

mysqldump -u root -q dss table4 > /b04/backup/$server/table4.sql &

wait

mysqldump -u root -q dss table5 > /b01/backup/$server/table5.sql &

mysqldump -u root -q dss table6 > /b02/backup/$server/table6.sql &

mysqldump -u root -q dss table7 > /b03/backup/$server/table7.sql &

mysqldump -u root -q dss table8 > /b04/backup/$server/table8.sql &

The 1.1 TB of backup across 20 virtual machines took 14 minutes to complete, yielding a throughput of 1.34 GB/s.

The following screenshot shows bandwidth metrics on FlashBlade. The 20 virtual machines were generating

around 1.51 GB/s of write bandwidth for most of the activity.

FIGURE 6. Bandwidth usage on FlashBlade

11

Page 12: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

The following screenshots show read bandwidth metrics during the same time across the three FlashArrays on

which MySQL databases were hosted.

//M20

//M50

//X70

FIGURE 7. Bandwidth usage on three FlashArrays hosting MySQL databases

Meanwhile, FlashBlade has reduced the backup data by 2.1:1. The actual physical storage space consumed is

544.05 Gb for the 1.1 TB of the full backup.

FIGURE 8. Physical storage space consumed on FlashBlade

12

Page 13: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

Restore Logical Backups Test

Similar to the backup, the restore was performed by running mysql commands in parallel. As the logical backups were

performed at the table level, the restore was included in the database (dss) where the tables are restored.

RESTORE OF A LOGICAL BACKUP ON A SINGLE VIRTUAL MACHINE

The following restore script was run on the single virtual machine to restore the dss database at the table level.

As part of the restore, MySQL creates a table and inserts the records into the table.

Note: If you are restoring at the database level on a remote server using sql script, be sure to create the

database before the restore, unless the sql dump was created with the --database option.

mysql -u root -q dss < /b01/backup/$server/table1.sql &

mysql -u root -q dss < /b02/backup/$server/table2.sql &

mysql -u root -q dss < /b03/backup/$server/table3.sql &

mysql -u root -q dss < /b04/backup/$server/table4.sql &

wait

mysql -u root -q dss < /b01/backup/$server/table5.sql &

mysql -u root -q dss < /b02/backup/$server/table6.sql &

mysql -u root -q dss < /b03/backup/$server/table7.sql &

mysql -u root -q dss < /b04/backup/$server/table8.sql &

To restore 70 GB worth of database on a single virtual machine, it took 37 minutes 21 seconds at a read throughput

of 32 MB/s on FlashBlade, while averaging around 100 MB/s on the FlashArray the database is restored to. This

sub-optimal throughput issue was not due to the infrastructure, but caused by MySQL. The restore process performs

inserts of records one at a time from the sql file and, as such, is not benefitting from any bulk operations.

FIGURE 9. Bandwidth usage on FlashBlade

FIGURE 10. Bandwidth usage on FlashArray

13

Page 14: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

RESTORE OF A LOGICAL BACKUP ON 20 VIRTUAL MACHINES

As with the single virtual machine, the following restore script was run across 20 virtual machines, and the

server name was passed as the parameter to restore the database from the right set of backups.

mysql -u root -q dss < /b01/backup/$server/table1.sql &

mysql -u root -q dss < /b02/backup/$server/table2.sql &

mysql -u root -q dss < /b03/backup/$server/table3.sql &

mysql -u root -q dss < /b04/backup/$server/table4.sql &

wait

mysql -u root -q dss < /b01/backup/$server/table5.sql &

mysql -u root -q dss < /b02/backup/$server/table6.sql &

mysql -u root -q dss < /b03/backup/$server/table7.sql &

mysql -u root -q dss < /b04/backup/$server/table8.sql &

The restore of MySQL databases across 20 virtual machines took 56 minutes 19 seconds, which was more than

four times the backup time. Overall 1.1 TB was read from FlashBlade at a throughput of 341 MB/s and restored on to

three FlashArrays.

The following screenshot shows read bandwidth from FlashBlade when the restore was performed.

FIGURE 11. Bandwidth usage on FlashBlade

The following screenshots show write bandwidth metrics across the three FlashArrays to which the MySQL

database was restored.

//M20

14

Page 15: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

//M50

//X70

FIGURE 12. Bandwidth usage on three FlashArrays to which the MySQL database is restored

Overall, FlashBlade is significantly underutilized, even while a logical backup and restore of 20 virtual machines was

performed, leaving room for more consolidation of high bandwidth workloads.

Physical Backup Test

For the physical backup test, Percona Xtrabackup was configured on the virtual machine. Xtrabackup provides an

option to run the backups in parallel, but it requires the innodb_file_per_table to be enabled, which stores the table in

a separate .ibd file.

The following is the command that was run to perform a full backup of the database. To differentiate the backups

and ascertain if there are any differences in data reduction rates, the backups were sent to a different NFS filesystem

(mysql-backup-02). If you do not pass the --no-timestamp option, the backups will be created in a timestamped

directory (YYYY-MM-DD_hh-mm-ss).

innobackupex --defaults-file=/etc/my.cnf --user=root --parallel=4 /x01/xtrabackup/$server

Unlike the logical backup, where we were able to run parallel commands and direct the output to four different mount

points to take advantage of the separate interfaces, Xtrabackup doesn’t allow multiple destinations. Thus, we rotated

the destination for every virtual machine between /x01, /x02, /x03, and /x04. As they are mounted on separate

subnets, we were able to use available network interfaces to spread out backup traffic.

15

Page 16: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

Alternatively, we could have mounted the same filesystem on the same mount point /x01, but mounted it across

different subnets in every virtual machine to accomplish the same behavior.

By using all four interfaces across the 20 virtual machines, we were able to use the available bandwidth and not

overwhelm a single interface, which would have limited our bandwidth to 10 Gbps (1 GB). Meanwhile, physical backup

on a single virtual machine would be limited to 10 Gbps or 1 GB. In our setup, the virtual machine’s compute would

become the bottleneck before the network bandwidth is saturated at 10 Gbps.

PHYSICAL BACKUP ON A SINGLE VIRTUAL MACHINE

The full backup of a MySQL database with a size of 70 GB took 152 seconds at a throughput of 471 MB/s, which is over

4 times the performance of the logical backup.

FIGURE 13. Bandwidth usage on FlashBlade

The improvement in performance over logical backup is due to the parallel option and to the implementation of

Xtrabackup’s backup feature.

PHYSICAL BACKUP ON TWENTY VIRTUAL MACHINES

The backup command was run across all 20 virtual machines at the same time. The only difference in the backup

across all the virtual machines is the destination directory. We rotated the destinations from /x01 to /x04 across the 20

virtual machines.

innobackupex --defaults-file=/etc/my.cnf --user=root --parallel=4 /x01/xtrabackup/$server

The backup of MySQL databases across 20 virtual machines containing 1.35 TB took 359 seconds at a throughput

of 3.85 GB/s.

16

Page 17: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

FIGURE 14. Bandwidth usage on FlashBlade

As seen from the above FlashBlade bandwidth metrics, concurrent backups from 20 virtual machines pushed

the write bandwidth to 4 GB/s for most of the activity.

The following screenshots show the bandwidth metrics across the three FlashArrays (//X70, //M50, and //M20).

As //X70 and //M50 were hosting 8 virtual machines each, the read bandwidth was hovering over 1.6 GB/s for most

of the activity while //M20 was hovering around 750MB/s.

//M70

//M50

//X20

FIGURE 15. Bandwidth usage on three FlashArrays to which the MySQL database is restored

17

Page 18: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

Meanwhile, the data reduction on FlashBlade for the physical backups was also 2.1:1 – like the logical backups.

FIGURE 16. Bandwidth usage on FlashBlade

Restore Physical Backups Test

Percona Xtrabackup has an extra step prior to restoring a backup: it prepares the backup for restore. The prepare step

makes the files consistent at a single instant in time by undoing all uncommitted transactions and playing back the

committed transactions from the transaction logs. The prepare operation can be performed on any machine, and not

necessarily on the originating server. It can be done right after the backup or just before the restore.

To prepare the backup with innobackupex, the --apply-log option should be used along with the backup directory.

If the backup was taken without the --no-timestamp option, be sure to provide the timestamped directory name.

innobackupex --apply-log /x01/xtrabackup/$server/$1

Once the backup is prepared, it can be restored. The backup can be restored back to the same datadir location or it

can be copied to another location.

innobackupex --defaults-file=/etc/my.cnf --copy-back /x01/xtrabackup/$server/$1

The --copy-back option copies the contents from the backup directory to the datadir directory. The datadir must

be empty for the --copy-back option, as it will not over write the existing directory unless the ---force-non-empty-

directories option is specified.

Once the restore is complete, the files ownership should be updated to mysql before starting the MySQL instance.

# chown –R mysql:mysql /m01/data

Restore timing and performance is generally limited by the target FlashArray as backup data is read out of FlashBlade

and written into the FlashArray. As we have multiple FlashArrays, FlashBlade can deliver higher performance, since a

15-blade system can deliver read bandwidth in the range of 16GB/s.

RESTORE OF A PHYSICAL BACKUP ON A SINGLE VIRTUAL MACHINE

As part of the Xtrabackup restore, the files are copied back to the datadir location. There are not many options to fine-

tune restore performance at the Xtrabackup level.

18

Page 19: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

19

The prepare step of the restore took 2 seconds, and the actual restore of 70 GB of backup data from a single virtual

machine took 219 seconds at a throughput of 327 MB/s.

In the screenshots below, the FlashBlade GUI shows average read bandwidth while the FlashArray GUI displays write

bandwidth on the target FlashArray to which the data is written. The read performance from FlashBlade is directly

dependent on the write performance on the target FlashArray.

FIGURE 17. Bandwidth usage on FlashBlade

FIGURE 18. Bandwidth usage on FlashArray

RESTORE OF A PHYSICAL BACKUP ON TWENTY VIRTUAL MACHINES

The prepare and restore processes were performed at the same time on 20 virtual machines. Similar to the backup

process, the restores were performed from the same NFS filesystem, but from different network interfaces to spread

the bandwidth across the four interfaces.

To restore 1.35 TB of data across 20 virtual machines, it took 8 minutes 27 seconds, yielding a throughput of 2.73 GB/s.

As seen from the FlashBlade bandwidth metrics, the read bandwidth hovered close to 3 GB/s for most of the restore

and started coming down based on activity at the individual host level.

FIGURE 19. Bandwidth usage on FlashBlade

Page 20: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

20

The following screenshots show write bandwidth on the three target FlashArrays.

//M20

//M50

//X70

FIGURE 20. Bandwidth usage on three FlashArrays to which the MySQL database is restored

The target FlashArrays are not anywhere close to their maximum potential write performance: their write performance

for this test is purely dependent on the performance of XtraBackup and its implementation. However, XtraBackup

restore performance is over six times that of the logical restores through the sql dump file. Meanwhile, FlashBlade has

a lot more room left to serve other demanding workloads that can stress both reads and writes.

Page 21: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

21

BEST PRACTICES FOR MYSQL ON FLASHBLADE

ENABLE PARALLELISM

To increase read and write bandwidth on FlashBlade, use client level parallelization techniques as applicable.

This increases the number of connections to FlashBlade.

USE MULTIPLE NETWORK INTERFACES

To enhance network bandwidth, be sure to have multiple network interfaces on the client. These multiple interfaces

can be configured on a single subnet or on multiple subnets.

SINGLE SUBNET

IO performance in a MySQL environment with multiple interfaces and a single subnet is limited to the speed of the first

interface that is picked by the OS. This is because the OS returns the first path when multiple paths are available on

the subnet, and thus traffic is always routed through the first path.

For example, in the setup below, FlashBlade has a single IP address on which the NFS filesystem will be mounted from

the client, and the client has two interfaces.

Client interface NFS server

ens7f0 10.21.108.193 10.21.108.10

ens7f1 10.21.108.194 10.21.108.10

[root@mysql-m50-01 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 10.21.108.1 0.0.0.0 UG 0 0 0 ens7f0

10.21.108.0 0.0.0.0 255.255.255.0 U 0 0 0 ens7f0 <-- first route

10.21.108.0 0.0.0.0 255.255.255.0 U 0 0 0 ens7f1 <-- route ignored

As per the routing table, traffic can go through the first interface (ens7f0) as the destination, and the mask fits for both

routes. The OS will invariably choose the first route.

In this case, to enhance bandwidth, it is recommended to use NIC bonding at the client level.

MULTIPLE SUBNETS

Use a separate subnet for each interface. With multiple subnets, there is no need to bond the network interfaces to

aggregate bandwidth across the available interfaces. The routing will be automatic in the case of multiple subnets.

Client interface NFS server

ens7f0 10.21.108.193 10.21.108.10

ens7f1 10.21.107.194 10.21.107.10

Page 22: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

22

[root@mysql-m50-01 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 10.21.108.1 0.0.0.0 UG 0 0 0 ens7f0

10.21.108.0 * 255.255.255.0 U 0 0 0 ens7f0

10.21.107.0 * 255.255.255.0 U 0 0 0 ens7f1

In this case, these are two dynamic routes and, based on the traffic, the route is selected automatically.

As such, if you decide to use multiple subnets, they should be configured on both the client and the FlashBlade side.

Multiple subnets can be configured in the FlashBlade GUI under Network Settings.

LINUX MOUNT OPTIONS

To mount the NFS filesystem on Linux, use the following mount options. Do not specify the rsize or wsize options,

as the system can get the default offered by FlashBlade, which is 524288.

rw, bg, nointr, hard, tcp, vers=3, nolock, noac, actimeo=0

MYSQL CONFIGURATIONS

For InnoDB, enable the innodb_file_per_table that stores each table in its own .ibd file, which helps in space

management, speeds up truncate table operations, reduces fragmentation, and improves backup and restore time

when used with MySQL Enterprise Backup or Percona XtraBackup.

Page 23: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

23

CONCLUSION

Businesses need a scalable and performant storage system to meet their recovery time objectives in the context

of a shrinking backup time window. Pure Storage FlashBlade provides a scalable, highly available, data reducible,

and performant solution to meet enterprise MySQL backup and restore requirements. The logical backup tests of

MySQL across 20 virtual machines illustrated FlashBlade write rates of over 1.5GB/s (limited only by MySQL’s backup

tool implementation), while the physical backup tests across 20 virtual machines yielded a rate of 3.85 GB/s by

taking advantage of the available bandwidth in FlashBlade. Similarly, the restore rates of 20 virtual machines using

physical backups yielded 2.73GB/s of read bandwidth from FlashBlade – which had a lot more room left to support

additional workloads.

Having FlashBlade as the backup target alone won’t provide stellar results unless the source system is capable of

reading and writing data out and in at the required rates. Without adequate speed, the source system becomes the

bottleneck. However, complementing FlashBlade, Pure’s FlashArray system provides the necessary performance,

with scalable read bandwidth for backup activity and comparable write bandwidth for restore activity.

Page 24: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

24

REFERENCES

The following documents and links were referred to in preparing this document.

1. Pure Storage Community pages

https://support.purestorage.com

2. MySQL documentation

https://dev.mysql.com/doc/

3. Percona Xtrabackup

https://www.percona.com/software/mysql-database/percona-xtrabackup

Page 25: Accelerating MySQL with FlashBlade - Pure Storage · For the physical backups, Percona Xtrabackup 2.2.13 was installed on top of the MySQL software. In this paper, we used the utility

25

ABOUT THE AUTHOR

Somu Rajarathinam is the Pure Storage Oracle Solutions Architect

responsible for defining database solutions based on the company’s

products, performing benchmarks, and developing reference

architectures for Oracle databases on Pure.

Somu has over 20 years of Oracle database experience, including

as a member of Oracle Corporation’s Systems Performance and

Oracle Applications Performance Groups. His career has also

included assignments with Logitech, Inspirage, and Autodesk, ranging

from providing database and performance solutions to managing

infrastructure, to delivering database and application support, both

in-house and in the cloud.

Web: www.somu.us

Twitter: @purelydb

[email protected] | 800-379-PURE | @PURESTORAGE

© 2017 Pure Storage, Inc. All rights reserved.

Pure Storage, the “P” Logo, FlashBlade, and Pure1 are trademarks or registered trademarks of Pure Storage, Inc. in the U.S. and other countries. Cisco UCS, and Cisco Nexus are registered trademarks of Cisco Systems, Inc. in the U.S. and other countries. Red Hat Enterprise Linux is the registered trademark of Red Hat, Inc. in the U.S. and other countries. MySQL is the trademark of Oracle, MariaDB Corporation AB and others.

The Pure Storage product described in this documentation is distributed under a license agreement and may be used only in accordance with the terms of the agreement. The license agreement restricts its use, copying, distribution, decompilation, and reverse engineering. No part of this documentation may be reproduced in any form by any means without prior written authorization from Pure Storage, Inc. and its licensors, if any.

THE DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. PURE STORAGE SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE.

ps_wp25p_mysql-fast-backup-and-restore-with-flashblade_ltr_01