Experiences with Percona Cluster

20
Experiences with Percona Cluster PRESENTED BY Yashada Jadhav and Trey Raymond April 14, 2015

Transcript of Experiences with Percona Cluster

Page 1: Experiences with Percona Cluster

E x p e r i e n c e s w i t h P e r c o n a C l u s t e r

▪  PRESENTED BY Yashada Jadhav and Trey Raymond | April 14, 2015

Page 2: Experiences with Percona Cluster

About Trey

2 Yahoo Confidential & Proprietary

▪  Sr. MySQL Engineer with a background in shouting about keys per page

▪  10 years experience with MySQL in a variety of environments

▪  Responsible for company schema design standards and SQL best practices

▪  Active helping folks and bantering in the freenode #mysql community

▪  In his free time, enjoys playing with his dog and receiving single malt scotch from conference attendees

Page 3: Experiences with Percona Cluster

About Yashada

3 Yahoo Confidential & Proprietary

▪  MySQL DevOps Engineer with a background in database design and performance tuning.

▪  4+ years of experience on various flavors of relational databases.

▪  In her spare time, she enjoys listening to music and going to concerts. She appreciates sarcasm a lot too!

Page 4: Experiences with Percona Cluster

About Yahoo

4 Yahoo Confidential & Proprietary

▪  Yahoo is focused on making the world’s daily habits inspiring and entertaining. ▪  By creating highly personalized experiences for our users, we keep people

connected to what matters most to them, across devices and around the world. ▪  In turn, we create value for advertisers by connecting them with the audiences

that build their businesses ▪  More than 1B monthly active users across Yahoo and Tumblr ▪  More than 575M mobile monthly active users across Yahoo and Tumblr

Page 5: Experiences with Percona Cluster

Ad Products Team

5 Yahoo Confidential & Proprietary

▪  Thousands of Production Servers ▪  OLTP systems & Data marts ▪  Database Design and Architecture ▪  Capacity Planning and Performance Reviews ▪  24x7 Monitoring and Operational Support

Mission Statement: Delivering scalable and cost efficient data services through innovation and automation powering Yahoo Products

Page 6: Experiences with Percona Cluster

MySQL at Yahoo

6 Yahoo Confidential & Proprietary

▪  MySQL powers many mission-critical products within Advertising and User space across Desktop and Mobile

▪  Multiple production configurations based on product requirement ▪  DBaaS setup for multiple products ▪  Yahoo Sports: Mobile friendly ▪  Flickr: Sharded across thousands of servers ▪  Hot:Hot, Hot:Warm Configurations ▪  Versions range from Percona Server 5.1 to 5.6 including Percona XtraDB

Cluster ▪  Operating systems running customized RHEL 5.6 to 6.5

Page 7: Experiences with Percona Cluster

PXC Implementations at Yahoo

7 Yahoo Confidential & Proprietary

▪  Yahoo Commerce Central - An open cloud platform that lets developers reach merchants on several E-Commerce platforms

Page 8: Experiences with Percona Cluster

• Hot:Hot Configuration • Scale up to future growth • Reduced cost of service

• Easy integration with Yahoo

products

8 Yahoo Confidential & Proprietary

PXC at Yahoo

Page 9: Experiences with Percona Cluster

›  API: Visitor statistics of merchant sites ›  UI: Inventory of all products

•  70% OLAP, 20% OLTP, 10% reporting •  1000 – 7000 QPS

›  Metadata service: Billing and Subscription Management •  90% OLTP load, 50% reads 50% writes •  150-500 QPS

9 Yahoo Confidential & Proprietary

YCC – Product Architecture

Page 10: Experiences with Percona Cluster

▪  3 Clusters – API, Proxy and MDB spread across 3 DCs.

▪  DNS Based Load Balancer based on proximity and round robin

10 Yahoo Confidential & Proprietary

YCC – Database Architecture

Page 11: Experiences with Percona Cluster

Configuration Specifics

11 Yahoo Confidential & Proprietary

▪  PXC version: 5.5.33.2 ▪  wsrep_provider_options: gmcast.listen_addr=tcp://

0.0.0.0:13306;evs.keepalive_period=PT3S;evs.send_window=1024;evs.user_send_window=512;evs.inactive_check_period=PT10S;evs.suspect_timeout=PT30S;evs.inactive_timeout=PT1M;evs.install_timeout=PT1M;gcache.size=8G;ist.recv_addr=tcp://_IPADDR_:2222

▪  transaction_isolation: REPEATABLE-READ ▪  wsrep_retry_autocommit: 3 ▪  wsrep_slave_threads: 8 ▪  wsrep_sst_receive_address: _HOSTNAME_:9999

Page 12: Experiences with Percona Cluster

What do we monitor?

12 Yahoo Confidential & Proprietary

▪  wsrep_local_state

▪  wsrep_cluster_size

▪  wsrep_ready

▪  wsrep_cluster_status

▪  Long running queries

▪  Queries > 300 seconds for “wsrep in pre-commit stage”

▪  pt-query-digest

Page 13: Experiences with Percona Cluster

State Transfers

13 Yahoo Confidential & Proprietary

▪  OLAP cluster cache sizes were too less which triggered SST

▪  Database size > 1T

▪  SST is a pain!

▪  FTWRL at the end of SST causes front end downtimes

▪  We increased the cache sizes to avoid this issue.

Page 14: Experiences with Percona Cluster

OLTP vs OLAP Uptime

14 Yahoo Confidential & Proprietary

OLAP Uptime

OLTP Uptime

Page 15: Experiences with Percona Cluster

OLTP vs OLAP QPS

15 Yahoo Confidential & Proprietary

OLAP QPS

OLTP QPS

Page 16: Experiences with Percona Cluster

OLTP vs OLAP Deep Dive

16 Yahoo Confidential & Proprietary

▪  Database size ▪  Transactions per second

Page 17: Experiences with Percona Cluster

PXC Operations

17 Yahoo Confidential & Proprietary

▪  Sensitive to Network Failures ▪  Identical Hardware ▪  XtraBackup from a live node cause downtime ▪  Deadlocks and Cluster Stalls with Multi Node Writes ▪  Cluster is sensitive to all nodes ▪  Transaction sizes matter ▪  Race conditions

Page 18: Experiences with Percona Cluster

Dual Master Hot Warm Set up

18 Yahoo Confidential & Proprietary

▪  Read Write Traffic Splitting ▪  Better use of capacity across master and slaves ▪  Subject to replication lags and delays ▪  More suited for OLTP applications

Page 19: Experiences with Percona Cluster

PXC – Bugs we hit

19 Yahoo Confidential & Proprietary

▪  Node consistency issues with foreign key grammar https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1260713

▪  Non Deterministic Execution of Transactions from Same Threads https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1217653

Page 20: Experiences with Percona Cluster

Contact us

20 Yahoo Confidential & Proprietary

▪ Trey Raymond [email protected]

▪ Yashada Jadhav [email protected]