Greetz Case Study: Migrating from Oracle RAC to MariaDB Galera Cluster

28
IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Transcript of Greetz Case Study: Migrating from Oracle RAC to MariaDB Galera Cluster

IT STRATEGY

Migrating from Oracle RAC to MariaDB Galera Cluster

IT STRATEGY

Who’s that guy?

• Jan van der Veen (31) • Techie with Multimedia design

background • Working for Greetz since 2006 • Started as freelance Flash

developer • Joined as lead front-end

developer (1st employee with fixed contract)

• CTO since October 2014

Migrating from Oracle RAC to MariaDB Galera Cluster

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Request for proposal - Objectives

• Improve System Availability, Scalability & Performace • Reduce Outage Events through a highly replicated infra

(>2 nodes), to be able to scale during peak moments and an highly performant database environment.

• Move towards an nearly real time improved procedure for Disaster Recovery environment.

• Provide Realtime Data available to the Datawarehouse (Business Intelligence reporting & analysis) environment.

• Reduce Overall TCO of software licencing and support • Reduce licencing cost while scaling and pressure on forced

upgrades. • Employ lower-end hardware (i.e. Remove dependency on SAN

storage)

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Request for proposal – Partner selection

• Suggest on the open source solution selection (MySQL vs. MariaDB).

• Design a sound future looking high-available storage and database architecture (High availability, scalability, data replication, Disaster Recovery, Data warehouse & BI requirements).

• Define the approach, support the project implementation and production rollout.

• Once the implementation is completed, provide ongoing Day-to-day Operational Support of the live production environment.

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Request for proposal – Partner selection

• RFP sent to SkySQL, Percona and Novaccent; • SkySQL selected as an implementation partner;

• Draft proposal fit our needs perfectly; • Services organization to support after migration;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Existing environment

• Infrastructure landscape

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Existing environment

• Application landscape • Classic n-tiered web application; • Java-based; • Wicket web application framework; • Jersey REST API’s; • Spring 2.5.x; • Hibernate ORM;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Existing environment

• Business intelligence

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Proposed environment • Maria DB 5.5; • Galera Cluster;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Hardware & OS

• High-powered commodity server hardware; • Low redundancy within server; • 3-node setup for redundancy;

• Dell r420 PowerEdge rack server; • 64 GB memory; • 4x Intel Xeon E5-2430 @ 2.20GHz (6 cores); • 2x Samsung 840 PRO SSD 512 GB (RAID 0 – striping); • CentOS 6.0 Linux;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Implementation

• Localize all PL/SQL queries in the application (limited) and replace with generic HQL queries;

• Data migration project to extract data from Oracle to CSV and import into MariaDB;

• Setup test infrastructure; • MariaDB cluster setup next to Oracle environment and

switch on-the-fly; • Rehearse & performance test;

• Benchmark & compare;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Implementation

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Implementation

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Launch

• Start at midnight 14-9 • Planned go-live 12:00

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Launch

• Synchronization between cluster nodes 2 & 3 failed • Node 2 went down, node 3 was incomplete • Only 1 node up and running • Synchronization to node 2 restarted • Rollback or wait another hour?

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Launch • Synchronization to node 2 successful; • Decided to go live with only 2 nodes; • Website back online @ 14:46;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

After launch

• Synchronization issue turned out to be a bug; • Solved by SkySQL/Galera within 2 weeks; • System stable ever since;

• Well… almost…

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

After launch

• Stable, as long as you don’t do this on your database… ALTER TABLE customerimage ADD COLUMN createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL), ADD COLUMN forInternalPurpose CHAR(1) DEFAULT 'N' NOT NULL; UPDATE customerimage SET createdAt = NOW();

• First release on MariaDB @ 9-10-2013; • Updated several million records in one transaction; • Replication locked up the cluster in commit phase; • Database unresponsive, thread couldn’t be killed; • Restarted process, cluster dead; • Call up the remote DBA and initiate the roll-back procedure; • Site was offline until late afternoon;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Recommendations

• ORM’s, yeah! • Application monitoring is a must-have;

• AppDynamics or New Relic; • Expect the unexpected; • Have the entire cluster running in a test environment before moving

it to production; • Data migration validation;

• Build in checkpoints; • Verify data through random samples;

• Perform large DDL’s on clusters seperately; • Rollback procedure;

• Not only theory, execute! • Backup stored on regular hard disk;

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Nowadays

• New Relic

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Nowadays

• Observium

IT STRATEGY Migrating from Oracle RAC to MariaDB Galera Cluster

Where to go from here?

• Micro-service architecture; • Split up business logic layer into separate web-service

components; • Deploy services separately, each with their own DB-

schema; • Continuous delivery;

• Feature-based deployment strategy; • Push-button deploys with automated roll-back support;