Twitter, Inc.

MySQL @Twitter: No More Forkin’ -Migrating to MySQL Community Version

Twitter, Inc.

MySQL @Twitter • Background• Migration Process• Migration Challenges• Post-Deployment Challenges• Future Plans• Q&A

http://despicableme2mkt103.blogspot.com/2013/06/welcome-minions.html

Twitter, Inc.

Celebrity @katyperry (109m) US President @BarackObama (102m) Comedian @TheEllenShow (77.7m) Website @YouTube (71.9m) Sports Figure @Cristiano (72m) Tech Figure @BillGates (45.7m)

https://www.pinterest.co.uk/pin/615796949022229370

Most Twitter Followers

•No - These clusters do not contain tweets •No - I don’t have access to tweets and user profiles •Yes - These clusters are used by internal applications and some external-facing applications / services.

Twitter, Inc.

Answers to you questions

Twitter, Inc.

Background • History of MySQL @Twitter • Rationale for Twitter Fork • List of Custom Changes • Reasons for Abandoning Fork

History of MySQL @TwitterTwitter, Inc.

October 2005 - MySQL 5.0 Community Version Released March 2006 - First Tweet from Jack Dorsey

November 2008 - MySQL 5.1 Community Version Released

December 2010 - MySQL 5.5 ReleasedSeptember 2011 - Start of Twitter Fork - MySQL 5.5.16 Twitter VersionMarch 2012 - MySQL 5.5.21 Twitter Version ReleasedFebruary 2013 - MySQL 5.6.10

Community Version Released June 2013 - Start of MySQL 5.6 Twitter Version BuildMarch 2014 - MySQL 5.6.16 Twitter Version Released

October 2015 - MySQL 5.7 Community Version Released

January 2017 - Last Release of MySQL Twitter Version (5.6.35)

Q4 2016 - Twitter Evaluation of MySQL 5.7 Community Version

Rationale for Twitter Fork • General Operability by the DBA Team • Agility to Change • Observability • Memory Management • Predictability • Performance • Auditability

Twitter, Inc.

Source: https://confluence.twitter.biz/display/MYSQL/Home

List of Custom Changes • User name length increased from 16 to 32 (Twitter MySQL

5.5.39) • innodb_deadlock_check (Twitter MySQL 5.5.35) • Improve memory allocation, especially under NUMA (Twitter

MySQL 5.5.19) • Add partitioning functions for YMD and YMDH (Twitter MySQL

5.5.23)

Twitter, Inc.

Reasons for Abandoning Our Branch • Cost • Leverage Community Support and Agility • MySQL 5.7 New Features

• Automatic Password Expiration • JSON Data Type Support • Multi-Source Replication • And others

Twitter, Inc.

https://tenor.com/view/running-workout-minions-cardio-gif-4349585

Twitter, Inc.

#MigrationProcess • MySQL Clusters @Twitter • Migration Methodology (High-Level) • Migration Methodology (Server Upgrade) • Deployment Schedule and Timeline

MySQL Clusters @Twitter • Sandboxes / Staging Clusters - 240 • Multi-Tenant Clusters - 100 • Single-Tenant Clusters - 410 • Revenue Clusters - 220 • Gizzard Clusters - 3400

Twitter, Inc.

https://www.pinterest.com/pin/494481234062937503/

Migration Methodology In-place Upgrade

vs. Logical Upgrade

(Uninstall / Reinstall / Restore)

Twitter, Inc.

Migration Methodology (High-Level) • Create MySQL 5.7 bridge from a standby slave using

mysql_twtr_to_community - dump source (standby), scan dump for incompatibilities, load to destination (bridge)

• Perform checksum between the bridge and standby (pt-table-checksum)

• Upgrade 1 slave within the cluster and restore from the MySQL 5.7 bridge

• After having 1 slave in MySQL 5.7 for a while and no issues encountered, upgrade remaining slaves

• Upgrade standby • Promote standby to master (point of no return) • Upgrade old master to MySQL 5.7

Twitter, Inc.

Migration Methodology (Server Upgrade) • Block host from receiving traffic and unmonitor • Monitor performance of remaining slaves, check for increase in

total queries and slow queries • Uninstall old version of MySQL • Install MySQL 5.7 • Restore databases from another MySQL 5.7 slave

Twitter, Inc.

Deployment Timeline/Schedule •2016 Q4 - MySQL 5.7 Migration Gap Analysis / Strategy Development •May 2017 - Developed Migration Scripts •June 2017 - Build Staging/Sandbox Environments (24) •October 2017 - Upgrade Multi-Tenant Clusters (10) •November/December 2017 - Upgrade Single-Tenant Clusters (41) •February 2018 to March 2018 - Upgrade Revenue Clusters (After SB)

Twitter, Inc.

Twitter, Inc.

#MigrationChallenges

• Logistic Challenges• Technical Challenges

https://giphy.com/gifs/minions-lightbulb-dJz8PK0eMNXdS

Migration Logistic Challenges • Multi-tenant cluster coordination • Unable to find cluster owners • Holidays / Moratoriums / Year-End Vacations (Black Friday,

Christmas, New Year, Year-End)

Twitter, Inc.

http://minionnation.tumblr.com/post/107236814057/happy-new-year-from-the-minions

https://tenor.com/view/shopping-minions-black-friday-sho-gif-4698509

https://giphy.com/gifs/christmas-despicable-me-minions-bCEkUOFPlCSha

Migration Technical Challenges • pt-show-grants - password not included anymore in IDENTIFIED

BY • pt-table-checksum results issues • Missing user grants • Twitter-specific system variables • Slave capacity • Cluster too active (bridge unable to catch-up) • Partition by timestamp • GTID issues

Twitter, Inc.

Migration Technical Challenges pt-show-grants

•MySQL 5.6 •GRANT USAGE *.* TO <User> IDENTIFIED BY PASSWORD <Password>

•MySQL 5.7 •GRANT USAGE *.* TO <User>

Twitter, Inc.

Solution: Created our own script to extract user grants that returns the same format both for MySQL 5.6 and MySQL 5.7

Migration Technical Challenges pt-table-checksum results issues Issue: Error checksumming table <Table Name>: Possible infinite loop detected!

Twitter, Inc.

Solution: Compared row count between standby and bridge for all tables in all databases.

Migration Technical Challenges Missing User Grants

Twitter, Inc.

Solution: Extract user grants from master and apply to upgraded slaves.

Migration Technical Challenges Twitter-Specific System Variables Issue: Unable to install MySQL 5.7 due to unknown configuration setting.

Twitter, Inc.

Solution: Remove offending system variable from the puppet attribute of the host (example: innodb_deadlock_check)

Migration Technical Challenges Slave Capacity

Twitter, Inc.

Bridge MySQL 5.7

Solution: Add more slaves (temporarily / permanently) to replace the slave to be upgraded.

Standby MySQL 5.6

Master MySQL 5.6

Slave MySQL 5.6

Slave MySQL 5.6

Slave MySQL 5.6

Migration Technical Challenges Cluster too active Issue: Bridge unable to catch-up after restoring from slave due to the master receiving too much activity.

Twitter, Inc.

Solution: Turn durability off

Migration Technical Challenges Partition by timestamp Issue: Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed.

Twitter, Inc.

Solution: Change data type of partition column to datetime

Cause: In MySQL 5.7, only unix_timestamp function is allowed to partition a table by RANGE based on the value of a timestamp column.

Migration Technical Challenges GTID Issues

Error 'When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation.

Twitter, Inc.

GTID Issue

Twitter, Inc.

MasterStandby

Slave Slave Slave

West Coast Data Center

BridgeRead-Only = 0Read-Only = 1Read-Only = 1

pt-heartbeat (super user)

GTID Issue •Connect bridge to master?

Twitter, Inc.

MasterStandby

Slave Slave Slave

West Coast Data CenterBridge

MySQL 5.7

Read-Only = 0Read-Only = 1 Read-Only = 1

pt-heartbeat (super user)

GTID Issue •Make the standby read-write?

Twitter, Inc.

MasterStandby

Slave Slave Slave

West Coast Data Center

BridgeRead-Only = 0Read-Only = 0Read-Only = 1

pt-heartbeat (super user)

GTID Issue - Solution

Twitter, Inc.

MasterStandby

Slave Slave Slave

West Coast Data CenterBridge

MySQL 5.6

Read-Only = 0Read-Only = 1 Read-Only = 0

pt-heartbeat (super user) pt-heartbeat

(super user)

BridgeMySQL 5.7Read-Only = 1

Twitter, Inc.

Post-Deployment Challenges • Compatibility Issues• Application Performance• Post-Migration Realizations• Recommendations

Compatibility Issues

•New Key Words and Reserved Words •LOAD DATA LOCAL INFILE •Incorrect usage of UNION and LIMIT

Twitter, Inc.

Compatibility Issues New Key Words and Reserved Words

• Account, Channel, Encryption, Filter, Instance • SELECT * FROM virtual

Twitter, Inc.

Solution: • Put table names within quotes: SELECT * FROM `virtual` • Rename table to avoid the use of a reserved word

Compatibility Issues LOAD DATA LOCAL INFILE •LOAD DATA LOCAL INFILE <File Name> INTO TABLE <Table Name> FIELDS TERMINATED BY ’,’ ENCLOSED BY ‘“‘ (<Fields List>);

•Error: Invalid utf8 character string

Cause: Change of default value of innodb_strict_mode changed from OFF to ON.

Twitter, Inc.

Solution #2: Convert input file to utf8 using iconv$ iconv —from-code=ISO-8859-1 --to-code=UTF-8 [Input File] > [Output File]

Solution #1: Add “CHARACTER SET latin1” LOAD DATA LOCAL INFILE <File Name> INTO TABLE <Table Name> CHARACTER SET latin1 FIELDS TERMINATED BY ’,’ ENCLOSED BY ‘“‘ (<Fields List>);

Compatibility Issues Incorrect usage of UNION and LIMIT

Twitter, Inc.

(SELECT SQL_CALC_FOUND_ROWS * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause>) AS sorted LIMIT 1 OFFSET 700) UNION ALL (SELECT FOUND_ROWS(), 'Total Count', <Empty Columns>)

Output: ID Type Date Status 84720 My Tweets 2018-03-05 Unknown 6 Total Rows

SELECT SQL_CALC_FOUND_ROWS * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause>) AS sorted LIMIT 1 OFFSET 700 UNION ALL SELECT FOUND_ROWS(), 'Total Count', <Empty Columns>

Output: ID Type Date Status 84720 My Tweets 2018-03-05 Unknown 1233 Total Rows

Compatibility Issues Incorrect usage of UNION and LIMIT

Solution: Query re-write

SELECT * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause> LIMIT 1 OFFSET 700) AS sorted UNION ALL SELECT COUNT(*), <Empty Columns> FROM <table> WHERE <WHERE Clause>

Twitter, Inc.

Application Performance Slow Analytics Queries with Derived Tables SELECT <Columns>, SUM(<Column>) FROM (SELECT <Columns> FROM <Table1> LEFT JOIN <Table2> ON <Condition1> LEFT JOIN <Table3> ON <Condition2> LEFT JOIN (SELECT <Columns> FROM <Table4>) ON <Condition3>) WHERE <WHERE Clause> GROUP BY <GROUP BY Clause>

•MySQL 5.6 : 3-4 Minutes •MySQL 5.7 : 27-29 Minutes

Twitter, Inc.

Solution: set session optimizer_switch=‘derived_merge=off’; Execution Time: 50 seconds

Twitter, Inc.

Post-Migration Realizations

• Unused Clusters • Grouped Clusters • Use of LOAD DATA LOCAL INFILE • Load Balancer Parameter Set-up • Slave Load Threshold

Twitter, Inc.

Future Plans (Next 12 Months)

•MySQL HA •MySQL 8.0 •Vitess

http://codinghelptech.com/Database/MySQL-8---Roles

•Test, test, test •Work with customers early in the process •Inform customers of the progress/schedule •Timing is everything (from scheduling to standby promotion to moratorium)

•Transparency is key •Explain Plan is your friend •Request for help as necessary

Twitter, Inc.

#Recommendations

#Recommendations •Test, test, test •Work with customers early in the process •Inform customers of the progress/schedule •Timing is everything (from scheduling to standby promotion to moratorium)

•Transparency is key •Explain Plan is your friend •Request for help as necessary

Twitter, Inc.

MySQL @Twitter - Q&A

Twitter, Inc.

https://giphy.com/gifs/happy-excited-applause-BQAk13taTaKYw

What month and year was Jack Dorsey’s first tweet?

What was Jack Dorsey’s first tweet?

What was MySQL Twitter version’s last release version?

Who has the most number of Twitter followers?

What 3 Percona toolkit tools / utilities were mentioned during the presentation?

Twitter, Inc.

Time for Twitter Swags

MySQL @Twitter

We’re Hiring #jointheflock

Twitter, Inc.