AWS + MySQL

Ryan HuddlestonDirector of Remote DBA

Percona Live New York 2012

www.percona.com

Agenda● Options for Running MySQL in AWS

● RDS or EC2● IO Performance● Cloud Watch and Percona Monitoring

Plugins● HA Options

www.percona.com

Why AWS● Starting up new nodes is very fast● Only use what is needed and can scale

quickly to meet demand● No upfront capital investment● Can concentrate on writing applications not

on running datacenters, ordering and racking equipment, cable management, redundant network provider contracts

www.percona.com

Amazon RDS or EC2● RDS - Relational Database Service

● Easy to setup without MySQL expertise● Automated Backups● Automated failover during hardware failure● Create read replicas with a few clicks● Ready to connect to mysql over port 3306

● EC2 - Elastic Compute Cloud● More flexible● Can be more cost effective

www.percona.com

Amazon RDS● Cost is 30% higher than EC2 instances● Meant to be a Black Box● High availability using equivalent to DRDB

○ Probably is DRDB underneath○ Recovery time dependent on Innodb Crash

recovery time○ You have to pay 2x the instance cost for HA

and cannot use the other node for queries ○ Backups can use the secondary instance but

beware, it can slow down the master.

www.percona.com

RDS Operation Issues● Migrating out of RDS involves mysqldump and

reload. If your data get's too large the move will be very painful

● No access to binary logs therefore no ability to replicate out of RDS. Also no ability to troubleshoot issues by looking through binary logs (replication logs)

● Binary backups not possible on read replica and snapshots can impact performance. As of 9-26 may be possible to mitigate via provisioned-IOPS option

www.percona.com

RDS Operation Issues - 2● No super_priv, so can't kill queries by other users● Can be difficult for DBA's to troubleshoot● Major upgrades involve dumping and reloading

entire database (e.g. 5.1 to 5.5)● If replication on a read replica breaks, you must

destroy/recreate that instance

www.percona.com

Amazon RDS - cont● You still need DBA's, it doesn't tune your SQL● You may get by with no DBA's if databases is

low to medium usage● If you're having IO or CPU issues, you must rely

on cloud watch metrics to tell you this● As of September 26th you now have provision-

IOPS volumes in RDS to mitigate IO issues○ With RDS you can only buy provisioned IOPS,

1000 per 100 GB which may be too much or little depending on workload

www.percona.com

New node with RDS

www.percona.com

RDS maintenance windows

www.percona.com

Amazon EC2● Very flexible for DBA's that know how to put the

pieces together● Can use Xtrabackup and send to S3 for Backups

○ Options to make it very low impacting (--rsync)○ You must monitor that backups are working

● Upgrading storage or Instance types takes manual work to move data around

● You are in control of IOPS to each volume you can have a 100GB storage with 10,000 IOPS (10x 10GB disk via software RAID 0)

www.percona.com

Amazon EBS

If you RAID-0 more that one EBS volume above you should get EBS optimized instances

www.percona.com

Amazon EC2● You must upgrade MySQL manually, though also

allows for major upgrades with minimal downtime● You must install an HA solution such as Percona

XtraDB Cluster, MMM, MHA, DRDB etc○ You must make sure the secondary master(s)

are up and being monitored properly and is located in a secondary AZ

www.percona.com

Percona Server ● With EC2 You can use alternative forks of

MySQL such as Percona Server that solves real operational issues

○ Drop table locks table cache mutex (innodb_lazy_drop_table)

○ Buffer pool restore at startup (innodb_buffer_pool_restore_at_startup)

○ Speed up ALTER and mysqldump loads (expand_fast_index_creation)

○ cache statistics between restarts (innodb_use_sys_stats_table)

○ innodb_adaptive_flushing_method

www.percona.com

Amazon IO● Amazon historically has had a lot of problems

with database workloads● Databases need consistent IO to work well● Normal EBS (Elastic Block Store) volumes can

do ~100 IOPS○ IO capability fluctuates based on other users

in EBS○ Workarounds included stripping many EBS

volumes together via RAID 0 or even RAID 10

www.percona.com

Amazon IO - cont● In August 2012 AWS announced Provisioned

IOPS EBS volumes● $.10 per month per provision IOPS● $100 per month for 1000 IOPS● Reliable and can allows you to serve a

significant amount of traffic ● Can stripe together volumes via RAID-0 to

go above 1000 IOPS. ○ The same thing RDS is doing under the hood

www.percona.com

Amazon IO Best Practices● Purchase Provisioned-IOPS volumes for MySQL

instances● Start with 1000 IOPS and go from there● Can greatly improve reliability and predictability

of the Database in AWS● In RDS things can get expensive. For example if

you are currently using typically ~200 IOPS but have 800GB of data. Provisioned-IOPS forced you to purchase 8000 IOPS which is $800 per month (twice that for multi-AZ deployments)

www.percona.com

Amazon Best Practices● Use the Ephemeral drive (but not for data)

○ mysql tempdir○ mysql-slow logs especially when

long_query_time=0○ backups, store some local copies, using --

compress on xtrabackup. May be faster than pulling from S3 in a pinch

● Use Xtrabackup for binary backups and send to S3. Also look at Glacier if you need to retain backups for longer periods

www.percona.com

S3 Backups example/usr/bin/innobackupex --compress --rsync /ephemeral_drive/xtrabackup/

NOW=$(date +%y%m%d)

cd /ephemeral_drive/xtrabackup/

LATEST=`ls -1t | head -1`

tar -cf - $LATEST | split -b 500MiB - "/ephemeral_drive/xtrabackup/S3/xtradb_split-$NOW.gz_"

s3cmd --delete-removed --no-check-md5 sync /ephemeral_drive/xtrabackup/S3/ s3://mybucket/mysqldump/

● s3cmd from http://s3tools.org

www.percona.com

Monitoring in AWS

● CloudWatch● Nagios + Percona Monitoring Plugins

Simplicity vs Flexibility

www.percona.com

AWS CloudWatch - RDS● Binlog disk usage● CPU utilization - key metric● DB connections● Free memory● Free disk space● Read and Write IOPS - key metric● Read and Write Throughput● Read and Write Latency● Swap Usage● Replication Lag - only on read replica

www.percona.com

AWS CloudWatch

● Create Alarms● Limited time ranges on Graphs

www.percona.com

Percona Monitoring Plugins● Innodb Deadlocks pt-deadlock-logger + pmp-

check-mysql-deadlocks● Threads Running:

○ pmp-check-mysql-status -x Threads_running -w 40 -c 80

● Processes in bad states:○ pmp-check-mysql-processlist

● Long Running Transactions that haven't committed, pmp-check-mysql-innodb

● Status of Percona XtraDB Cluster:○ pmp-check-mysql-status_pxc

www.percona.com

Percona Monitoring Plugins + Nagios

● Replication Running● Replication Lag:

○ pmp-check-mysql-replication-delay plus pt-heartbeat

● Free Memory● Disk space● Anything you can think of

● Investigating integrating CloudWatch into Nagios via AWS API's

www.percona.com

Percona Monitoring Plugins + Nagios

www.percona.com

Other Monitoring Tools● pt-stalk

○ RainGauge ○ https://github.com/box/RainGauge

● Box/Anemometer○ https://github.com/box/Anemometer

www.percona.com

High availability● PXC - Percona XtraDB Cluster

○ Plans to test configurations having nodes split between different AZ's

● MHA/MMM○ one monitoring node

● Corosync/Pacemaker○ Investigating better ways to integrate with tools like

MHA○ must use unicast in AWS○ Can't do vip failover○ Investigating using similar methods to PXC + HaProxy

www.percona.com

Conclusions● AWS is better able to handle database workloads now

with provisioned IOPS● EC2 is more cost effective and flexible if you know what

you're doing● RDS is great for low-medium sized databases where

there are no dedicated DBAs● Combination and CloudWatch and Nagios/Percona

Monitoring Plugins may work well in Amazon● Amazon's multiple availability zones is a nice feature.

Run your HA across without the traditional problems with remote data centers for DR○ Percona XtraDB Cluster may be a good option here

www.percona.com

Questions?

www.percona.com

Percona LiveLondon

Visit:http://www.percona.com/live/london-2012/

London UK,Dec 4-5

Registration is Open!

www.percona.com

Annual Percona LiveMySQL Conference and Expo

The Hyatt Regency Hotel, Santa Clara, CA

April 22nd-25th, 2013 Registration

and Call for Papers

are Open! Visit:

http://www.percona.com/live/mysql-conference-2013/