CUSTOM NON-RDS MULTI-AZ MYSQL REPLICATION

Cloudpack Night #6

5/17/2013

ABOUT ME (@IJIN)

• Michael H. Oshita

• Japanese American

• In Japan since 2002

• Software/Infra/Cloud Engineer

• http://ijin.github.io

CERTIFIED TODAY!

COMPONENTS

• AWS VPC (Routing-Based HA Pattern)

• MySQL Master-Slave Replication

• MHA (Master HA Manager for MySQL)

• no RDS!!

MHA

Master

Slave

Slave

Manager

Manager detects failure on master

Master

Master

Slave

Manager

MHASlave is promoted to master after binlogs are applied

binlogbinlog

binlog

"Starting master failover.""* Phase 1: Configuration Check Phase..\n""* Phase 2: Dead Master Shutdown Phase..\n" ==> Delete routing table (master_ip_failover)"* Phase 3: Master Recovery Phase..\n" "* Phase 3.1: Getting Latest Slaves Phase..\n" "* Phase 3.2: Saving Dead Master's Binlog Phase..\n" "* Phase 3.3: Determining New Master Phase..\n" ==> Change routing table (master_ip_failover)"* Phase 4: Slaves Recovery Phase..\n"

MHAFAILOVER SEQUENCE

“SHARE A PRIVATE IP ACROSS AVAILABILITY ZONES”

http://d.hatena.ne.jp/c9katayama/20111225/1324837509

#ヤマン ++

• Storage engines & distributions

• Instance types

• Fast

• Efficient

WHY NOT RDS?→ FLEXIBILITY!!

• Percona Server (XtraDB)

• MariaDB

• TokuDB

• Mroonga

STORAGE ENGINES & DISTRIBUTIONS

• hi1.xlarge

• m3.xlarge

• c1.xlarge

• etc.

INSTANCE TYPES

• Routing Table changes pretty quickly

• Total failover in roughly <20s (RDS takes 3-6m)

• Non-DNS based

• Apps can be configured with a single IP

• no need to worry about internal DNS problems

FAST FAILOVER

• Minimum of 2 (web+db) servers

• Warm up (replication booster)

• Fine tuning

• Server Logs

EFFECIENT

DEMO

CONSIDERATIONS

• Backups → Xtrabackup, copy binlogs to s3

• Point in Time Recovery → Automate with Chef

• Provisioning Replicas → Automate with Chef

• Some learning curve

• API backplane a SPoF

THANK YOU!