MySQL Cluster - doag.org & NoSQL Combined ... How Does MySQL Cluster Manager Help? Example:...

MySQL Cluster Wann brauche ich das?

Mario Beck

Principal Sales Consultant

[email protected]

The presentation is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into any

contract. It is not a commitment to deliver any

material, code, or functionality, and should not be

relied upon in making purchasing decisions.

The development, release, and timing of any

features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

40% DATA

GROWTH PER YEAR

5.3BN MOBILE SUBS IN

2010 (78% PENETRATION)

13K iPHONE APPS DOWNLOADED PER MINUTE

370K CALL MINUTES EVERY 60 SECONDS

$1TR BY 2014

$700BN IN 2011

2.1BN USERS

8X DATA GROWTH IN 5 YRS

70+ NEW DOMAINS EVERY 60 SECONDS

100K TWEETS PER MINUTE

750M USERS

20M APPS PER DAY

600 NEW VIDEOS

EVERY 60 SECONDS

EXTREME WRITE SCALABILITY REAL TIME USER EXPERIENCE

ROCK SOLID RELIABILITY ELIMNATE BARRIERS TO ENTRY

Driving new Database Requirements

EXTREME WRITE SCALABILITY REAL TIME USER EXPERIENCE

ROCK SOLID RELIABILITY ELIMNATE BARRIERS TO ENTRY

No Trade-Offs

Transactional Integrity

Complex Queries

Standards & Skillsets

No Trade-Offs: Cellular Network

HLR / HSS

Billing, AuC, VLR

AuC, Call

Routing, Billing

Location

Updates

Pre & Post Paid

• Massive volumes of write traffic

• <3ms database response

• Downtime & lost transactions = lost $

MySQL Cluster in Action: http://bit.ly/oRI5tF

No Trade-Offs: eCommerce

• Integrated Service Provider

platform

• eCommerce

• Payment processing

• Fulfillment

• Supports 1k+ manufacturers &

18k retail partners

• Requirements

• Scaling, On-Demand

• HA: failures & on-line upgrades

• High batch & real time loads

• Low TCO: capex and opex

http://mysql.com/customers/view/?id=1080

No Trade-Offs: Flight Control

• US Navy aircraft carriers

• Consolidated flight operations

management system

• Maintenance records

• Fuel loads

• Weather conditions

• Flight deck plans

• Requirements

• No Single Points of Failure

• Complete redundancy

• Small footprint, harsh environment

• 4 x MySQL Cluster nodes,

Linux and Windows

MySQL User Conference Session: http://bit.ly/ogeid3

ACID Compliant Relational Database • SQL & NoSQL interfaces

Write-Scalable & Real-Time • Distributed, multi-master, auto-sharding, optimized in-memory structures & indices

99.999% Availability • Shared-nothing, integrated clustering & sub-second recovery, local & geographic

replication, on-line operations

Low Barriers to Entry • Open-source, elastic, multiple APIs, management tools, commodity hardware

MySQL Cluster Architecture

Data Nodes

Node Group 1

F1

F3

F3

F1

No

de

1

No

de

2

Node Group 2

F2

F4

F4

F2

No

de

3

No

de

4

Application Nodes

Cluster

Mgmt

Cluster

Mgmt

REST

JPA

MySQL Cluster - Extreme Resilience

Data Nodes

Node Group 1

F1

F3

F3

F1

No

de

1

No

de

2

Node Group 2

F2

F4

F4

F2

No

de

3

No

de

4

Application Nodes

Cluster

Mgmt

Cluster

Mgmt

REST JPA

Handling Scheduled Maintenance On-Line Operations

• Scale the cluster (add nodes on-line)

• Repartition tables

• Upgrade / patch servers & OS

• Upgrade / patch MySQL Cluster

• Back-Up

• Evolve the schema on-line, in real-time

Auto-Sharding with High Availability

• Transparent to the application

• No need for application-layer sharding logic

• Partitioning based on hashing all or part of the primary key

• Each node stores primary fragment for 1 partition and back-up fragment for another

• Transparency maintained during failover, upgrades and scale-out

• No need to limit application to single-shard transactions

Best Practice : Primary Keys

• ALWAYS DEFINE A PRIMARY KEY ON THE TABLE!

• A hidden PRIMARY KEY is added if no PK is specified. BUT..

• .. NOT recommended

• The hidden primary key is for example not replicated (between Clusters)!!

• There are problems in this area, so avoid the problems!

• So always, at least have id BIGINT AUTO_INCREMENT PRIMARY KEY

• Even if you don't “need” it for you applications

Scale-Out on Commodity Hardware

• 8 Intel servers • Dual-6-core CPUs @2.93 GHz

• 24GB RAM

• 2 Data Nodes per server

• flexAsync benchmark • 16 parallel threads, each issuing 256

simultaneous transactions

• Read / Write 100KB attribute

• Access via NDB API

4.3M

QPS

Copyright 2011 Oracle Corporation 18

MySQL Cluster 7.2.1 DMR

• 70x Higher Complex Query Performance: Adaptive Query Localization

• Native memcached API

• MySQL 5.5 Server Integration

Enabling Next Generation Web Services

• Multi-Site Clustering

• Simplified Active / Active Replication

Enhancing Cross Data Center Scalability

• Consolidated Privileges

Simplifying Provisioning & Administration

Adaptive Query Localization Scaling Distributed Joins

• Perform Complex Queries

across Shards • JOINs pushed down to data nodes

• Executed in parallel

• Returns single result set to MySQL

• Opens Up New Use-Cases

• Real-time analytics

• Recommendations engines

• Analyze click-streams

mysqld

Data Nodes

mysqld

A

Q

L

The existence, content and timing of future releases described here is included for information only and may be changed at Oracles discretion.

October 3rd, 2011

Data Nodes

70x More

Performance

DON’T COMPROMISE

FUNCTIONALITY TO SCALE-OUT !!

MySQL Cluster 7.2 AQL Test Query Web-Based Content Management System


Data

Node1

Data

Node2

MySQL

Server

MySQL Cluster 7.2 70x Speedup! (DM)

• 70x Speedup for real

customer JOIN across

11 tables

• Must Analyze tables for

best results

mysql> ANALYZE

TABLE <tab-name>;

7.2 = 1.26s

7.1 = 87.23s

7.2

analyzed

= 65.3s


October 3rd, 2011

Memcached Key-Value API

• Persistent, Scalable, HA

Back-End to memcached • No application changes: re-

uses standard memcached

clients & libraries

• Consolidate Caching &

Database Tiers • Eliminate cache invalidation

• Simpler re-use of data across

services

• Improved service levels

• Flexible Deployment • Schema or Schema-less

storage


October 3rd, 2011

New

NoSQL

Access

API Flexibility: SQL & NoSQL Combined

• SQL: Complex, relational queries

• HTTP / memcached: Key-Value web services

• Java: Enterprise Apps

• NDB API: Real-time services

Mix

&

Match

Data Nodes

NDB API

Clients

Native memcached HTTP/REST

JDBC / ODBC

PHP / PERL

Python / Ruby


October 3rd, 2011

MySQL 5.5 Server Integration

• Exploit Latest Server / Storage

Engine Developments within a

Single Application • InnoDB: Foreign Keys, XA Transactions,

Large Rows

• MySQL Cluster: HA, High Write Rates, Real-

Time

• Reduces Complexity, Simplifies

DevOps

• MySQL 5.5 • 3x higher performance

• Improved partitioning, diagnostics, availability,

etc.


October 3rd, 2011

Enhanced Cross Data Center Scalability Geographic Replication

• Replicating complete

clusters across data

centers

• DR & data locality

• No passive resources

• Simplified Active /

Active Replication

• Eliminates requirement

for application & schema

changes

• Transaction-level

rollback

Geographic

Replication


October 3rd, 2011

Enhanced Cross Data Center Scalability Multi-Site Clustering

• Split data nodes & node

groups across data

centers

• Synchronous replication

and auto-failover between

sites

• Improved heartbeating to

handle network partitions

• Extends Disaster

Recovery Options

• Active/Active with no

need for conflict

handling

Data Node 1

Synchronous

Replication

Data Node 3

Data Node 2

Data Node 4


October 3rd, 2011


October 3rd, 2011

Simplified Provisioning & Maintenance User Privilege Consolidation

Monitoring &

Recovery

High

Availability

Operation

Automated

Management

Reducing TCO and creating a more agile, highly

available database environment

MySQL Cluster Manager


How Does MySQL Cluster Manager Help?

Example: Initiating upgrade from MySQL Cluster 6.3 to 7.1

• 1 x preliminary check of cluster state

• 8 x ssh commands per server

• 8 x per-process stop commands

• 4 x scp of configuration files (2 x mgmd & 2 x

mysqld)

• 8 x per-process start commands

• 8 x checks for started and re-joined processes

• 8 x process completion verifications

• 1 x verify completion of the whole cluster.

• Excludes manual editing of each configuration

file.

Total: 46 commands -

2.5 hours of attended operation

Before MySQL Cluster Manager With MySQL Cluster Manager

upgrade cluster --package=7.1 mycluster;

Total: 1 Command -

Unattended Operation

• Results • Reduces the overhead and complexity

of managing database clusters

• Reduces the risk of downtime resulting from administrator error

• Automates best practices in database cluster management


Creating & running your first Cluster

- the “manual” way (without MCM)

• Up & running in 10-15 minutes using Quick Start guides from

http://dev.mysql.com/downloads/cluster/ • Versions for Linux, Windows & Solaris


Download & Extract

• edelivery.oracle.com

• www.mysql.com

• dev.mysql.com

Configure

• Cluster-wide “config.ini”

• Per-mysqld “my.cnf”

Start processes

• Management Nodes

• Data Nodes

• MySQL Servers

http://dev.mysql.com/downloads/cluster/

New in MySQL Cluster Manager 1.1.2 Bootstrap single host Cluster

1. Download MCM from edelivery.oracle.com:

• Package including Cluster

2. Unzip

3. Run agent, define, create & start Cluster!

$> bin\mcmd –bootstrap MySQL Cluster Manager 1.1.2 started

Connect to MySQL Cluster Manager by running "D:\Andrew\Documents\MySQL\mcm\bin\mcm" -a NOVA:1862

Configuring default cluster 'mycluster'...

Starting default cluster 'mycluster'...

Cluster 'mycluster' started successfully

ndb_mgmd NOVA:1186

ndbd NOVA

ndbd NOVA

mysqld NOVA:3306

mysqld NOVA:3307

ndbapi *

Connect to the database by running "D:\Andrew\Documents\MySQL\mcm\cluster\bin\mysql" -h NOVA -P 3306

-u root

4. Connect to Cluster & start using database

To bootstrap with Cluster 7.2 replace contents of mcm/cluster directory

http://www.clusterdb.com/mysql-cluster/mysql-cluster-manager-1-1-2-creating-a-cluster-is-now-trivial





























MySQL Enterprise Monitor 2.3


MySQL Cluster in Action Web Reference Architectures

Session Management eCommerce

MySQL Master

Content Management

Slave 1

MySQL Master

Analytics Slave 2 Slave 3 Slave 4 Slave 5

Slave 6 Slave 7 Slave 8 Slave 9 Slave 10

Slave N

Slave 2

Node Group 2

F2

F4

Nod

e 3

N

od

e 4

F2

F4

Node Group 1

F1

F3

Nod

e 3

N

od

e 4

F1

F3

MySQL Cluster Data Nodes

MySQL Servers

Node Group 2

F2

F4

Nod

e 3

N

od

e 4

F2

F4

Node Group 1

F1

F3

Nod

e 3

N

od

e 4

F1

F3

MySQL Cluster Data Nodes

MySQL Servers

Slave 1 Slave 3

Data

Refinery Memcache / Application Servers

Distributed

Storage

XOR • 4 x Data Nodes: 6k

page hits per second

• Each page hit

generating 8 – 12

database operations

Whitepaper: http://www.mysql.com/why-mysql/white-papers/mysql_wp_high-availability_webrefarchs.php

Getting Started

Learn More

Evaluate MySQL Cluster 7.2 Bootstrap a Cluster!

Scaling Web Databases Guide www.mysql.com/cluster/

Download Today http://dev.mysql.com/downloads/cluster/

Download, No Obligation https://edelivery.oracle.com/




Summary

Scale Web Services with

Carrier-Grade Availability

Don’t Trade Functionality for Scale

Evaluate the Development Milestone Today


Data Node 1

F1

F2

Data Node2

F2

F1

Application Nodes

Data Nodes

Cluster

Mgmt

Scale the Cluster Capacity & Performance

Data Nodes

Node Group 1

F1

F3

F3

F1

No

de

1

No

de

2

Node Group 2

F2

F4

F4

F2

No

de

3

No

de

4

Application Nodes

Cluster

Mgmt

Cluster

Mgmt

Scale the Cluster Capacity & Performance

Node Failure Detection & Self-Healing Recovery

On-Line Scaling & Maintenance

• Can also update schema on-line

• Upgrade hardware & software with no downtime

• Perform back-ups on-line

1. New node group added

2. Data is re-partitioned

3. Redundant data is deleted

4. Distribution is switched to share load with new node group

Only MySQL Can…..

blend the agility & innovation of the web….

….with the trust & capability of the network.

MySQL Cluster - doag.org & NoSQL Combined ... How Does MySQL Cluster Manager Help? Example:...

Documents

Transcript of MySQL Cluster - doag.org & NoSQL Combined ... How Does MySQL Cluster Manager Help? Example:...