Greetz Case Study: Migrating from Oracle RAC to MariaDB Galera Cluster
Transcript of Greetz Case Study: Migrating from Oracle RAC to MariaDB Galera Cluster
IT STRATEGY
Who’s that guy?
• Jan van der Veen (31) • Techie with Multimedia design
background • Working for Greetz since 2006 • Started as freelance Flash
developer • Joined as lead front-end
developer (1st employee with fixed contract)
• CTO since October 2014
Migrating from Oracle RAC to MariaDB Galera Cluster
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Request for proposal - Objectives
• Improve System Availability, Scalability & Performace • Reduce Outage Events through a highly replicated infra
(>2 nodes), to be able to scale during peak moments and an highly performant database environment.
• Move towards an nearly real time improved procedure for Disaster Recovery environment.
• Provide Realtime Data available to the Datawarehouse (Business Intelligence reporting & analysis) environment.
• Reduce Overall TCO of software licencing and support • Reduce licencing cost while scaling and pressure on forced
upgrades. • Employ lower-end hardware (i.e. Remove dependency on SAN
storage)
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Request for proposal – Partner selection
• Suggest on the open source solution selection (MySQL vs. MariaDB).
• Design a sound future looking high-available storage and database architecture (High availability, scalability, data replication, Disaster Recovery, Data warehouse & BI requirements).
• Define the approach, support the project implementation and production rollout.
• Once the implementation is completed, provide ongoing Day-to-day Operational Support of the live production environment.
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Request for proposal – Partner selection
• RFP sent to SkySQL, Percona and Novaccent; • SkySQL selected as an implementation partner;
• Draft proposal fit our needs perfectly; • Services organization to support after migration;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Existing environment
• Infrastructure landscape
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Existing environment
• Application landscape • Classic n-tiered web application; • Java-based; • Wicket web application framework; • Jersey REST API’s; • Spring 2.5.x; • Hibernate ORM;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Existing environment
• Business intelligence
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Proposed environment • Maria DB 5.5; • Galera Cluster;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Hardware & OS
• High-powered commodity server hardware; • Low redundancy within server; • 3-node setup for redundancy;
• Dell r420 PowerEdge rack server; • 64 GB memory; • 4x Intel Xeon E5-2430 @ 2.20GHz (6 cores); • 2x Samsung 840 PRO SSD 512 GB (RAID 0 – striping); • CentOS 6.0 Linux;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Implementation
• Localize all PL/SQL queries in the application (limited) and replace with generic HQL queries;
• Data migration project to extract data from Oracle to CSV and import into MariaDB;
• Setup test infrastructure; • MariaDB cluster setup next to Oracle environment and
switch on-the-fly; • Rehearse & performance test;
• Benchmark & compare;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Launch
• Start at midnight 14-9 • Planned go-live 12:00
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Launch
• Synchronization between cluster nodes 2 & 3 failed • Node 2 went down, node 3 was incomplete • Only 1 node up and running • Synchronization to node 2 restarted • Rollback or wait another hour?
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Launch • Synchronization to node 2 successful; • Decided to go live with only 2 nodes; • Website back online @ 14:46;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
After launch
• Synchronization issue turned out to be a bug; • Solved by SkySQL/Galera within 2 weeks; • System stable ever since;
• Well… almost…
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
After launch
• Stable, as long as you don’t do this on your database… ALTER TABLE customerimage ADD COLUMN createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL), ADD COLUMN forInternalPurpose CHAR(1) DEFAULT 'N' NOT NULL; UPDATE customerimage SET createdAt = NOW();
• First release on MariaDB @ 9-10-2013; • Updated several million records in one transaction; • Replication locked up the cluster in commit phase; • Database unresponsive, thread couldn’t be killed; • Restarted process, cluster dead; • Call up the remote DBA and initiate the roll-back procedure; • Site was offline until late afternoon;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Recommendations
• ORM’s, yeah! • Application monitoring is a must-have;
• AppDynamics or New Relic; • Expect the unexpected; • Have the entire cluster running in a test environment before moving
it to production; • Data migration validation;
• Build in checkpoints; • Verify data through random samples;
• Perform large DDL’s on clusters seperately; • Rollback procedure;
• Not only theory, execute! • Backup stored on regular hard disk;
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster
Where to go from here?
• Micro-service architecture; • Split up business logic layer into separate web-service
components; • Deploy services separately, each with their own DB-
schema; • Continuous delivery;
• Feature-based deployment strategy; • Push-button deploys with automated roll-back support;