MariaDB Enterprise Cluster Enterprise Cluster Introduction Architecture Installation State Transfer...

MariaDB Enterprise Cluster

MariaDB Training


Introduction

ArchitectureInstallation

State Transfer

CaveatsMulti-Master Conflicts

Schema Changes

Back-UpsRecovery

Load Balancing

Advanced TopicsConclusion

Introduction

[email protected]

Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Enterprise Cluster Administration

Introducing MariaDB AbFounders from MySQL the

Company and Community Funded by Founders, Employees,

and Venture Capital About 100 Employees Several former MySQL Employees

and Community Members in over 14 Countries

3

[email protected]


Personal IntroductionsInstructor

Name and Background

Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics

4

[email protected]


Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site

Location of Rest Rooms Smoking Areas Snacks and Drinks

LVC Classes Chat with Everyone

5

[email protected]


Course OutlineOverview Architecture Installation State Transfer Caveats Multi-Master Conflicts Schema Changes Back-Ups Recovery

Load Balancing Advanced Topics

6


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Architecture

[email protected]


Advantages of GaleraParallel Slave Applying Practically No Slave Lag Instant Trivial Failover Automatic Node Provisioning Works Well in WAN

8

[email protected]


MariaDB Replication — Server Centric Style

Asynchronous Replication No Conflict Detection Multi-Master Replication

If Node C Crashes, Does Cluster Survive?

If Node B Crashes and Clients Switch to C, How does Node B Rejoin?

Which Node has Data X? How do you Back-Up Cluster?

9

MasterServer

SlaveServer

Node A

Node C

Node B

[email protected]


Data Doesn’t Belong to a Node Nodes Belong to Data Data is Synchronized among

Two or More Servers

Galera Approach — Data Centric Style

10

Server 1 Server 2 Server 3 Server N

DataSet

[email protected]


Galera ApproachGalera Nodes are Anonymous — All are Equal Galera Cluster is One Large Distributed Master A DataSet Needs an Identifier DataSet Identier is a Cluster Identifier

00295a79-9c48-11e2-bdf0-9a916cbb9294

11

DataSet Cluster

[email protected]


Global Transaction Identifier (GTID)DataSet plus Sequence of Atomic

Changes equals GTID

12

DataSet00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

64201

[email protected]


Global Transaction Identifier (GTID)

13

00295a79-9c48-11e2-bdf0-9a916cbb9294:0

00295a79-9c48-11e2-bdf0-9a916cbb9294:1

00000000-0000-0000-0000-000000000000:-1

Initial DataSet

First Change and Transaction

Undefined GTID

[email protected]



14

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201

8182213e-7c1e-11e2-a6e2-080027635ef5:12345

Galera GTID

MySQL 5.6 GTID

0-10-12345

MariaDB 10 GTID(server identifier : transaction processed by server)

(data & cluster Identifier : data change in cluster)

(domain - server identifier - data change in asynchronous cluster)

[email protected]



15

8182213e-7c1e-11e2-a6e2-080027635ef5:12345 8182213e-7c1e-11e2-a6e2-080027635ef5:12346 8182213e-7c1e-11e2-a6e2-080027635ef5:12347

Visible in MySQL 5.6

New Master Promoted

f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:1 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:2 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:3

[email protected]



16

0-10-12345 0-10-12346 0-10-12347

Visible in MariaDB 10

New Master Promoted

0-20-12348 0-20-12349 0-20-12350

[email protected]



17

00295a79-9c48-11e2-bdf0-9a916cbb9294:64201 00295a79-9c48-11e2-bdf0-9a916cbb9294:64202 00295a79-9c48-11e2-bdf0-9a916cbb9294:64203

Visible in Galera

New Master Promoted

00295a79-9c48-11e2-bdf0-9a916cbb9294:64204 00295a79-9c48-11e2-bdf0-9a916cbb9294:64205 00295a79-9c48-11e2-bdf0-9a916cbb9294:64206

[email protected]


Master or SlaveNot a Node Role or Function A Relation Between a Node and a Client

18

master1slave2

slave1slave2

slave1master2

slave1slave2

slave1slave2

Cluster

master for client1 master for client1

client2client1

[email protected]


Cluster Addresswsrep_cluster_address

19

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

10.0.0.5

Documentation on Galera Cluster Addresses: https://mariadb.com/kb/en/mariadb/galera-cluster-address/

https://mariadb.com/kb/en/mariadb/galera-cluster-address/

[email protected]


Cluster Address

20

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

gcomm://10.0.0.6

10.0.0.5handshake

[email protected]


Cluster Address

21

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3 10.0.0.4

10.0.0.5

[email protected]


Cluster Address

22

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3

10.0.0.4

10.0.0.5

[email protected]


Cluster Addresswsrep_cluster_address = gcomm://node1,node2

Try to Connect to Members (node1, node2)

Can only Join a Running Cluster

23

10.0.0.5

[email protected]


Node Synchronization (State Transfer)

24

10.0.0.1

10.0.0.610.0.0.2

10.0.0.3

10.0.0.4

10.0.0.5

[email protected]



25

joined

syncedsynced

desync

synced

undefined

[email protected]



26

catch-up

UNDEFINED

JOINER

JOINED

SYNCHED

catch-up

SYNCHED

DONOR

JOINED

SYNCHED

New Node Cluster Old Node

Verify Node Synched

Donor Found — Wait Be a Donor

Transfer Missing Data (Private Channel)

[email protected]


Avoiding Split BrainDistinguishing Server Crash from Network Failure in

Shared Nothing Architecture Decision Algorithm Used to Avoid Spilt Brain Absolute Majority Needed in Galera Uneven Number of Nodes Safer

27

[email protected]


Primary Component

28

Primary

[email protected]


Primary Component

29

Primary

Non-Primary

Continues Working

[email protected]


Primary Component

30

Primary

Non-Primary

Tries to ReconnectContinues Working

[email protected]


Primary Component

31

Primary

[email protected]


Split Brain

32

Non-Primary Non-Primary

Split Brain Possible with Even Number of Nodes

[email protected]


Synchronous PenaltiesGalera Copies Data Buffer to All Cluster Members on COMMIT from Client (~1 RTT added latency)

Connection throughput equals 1/RTT trx/sec Total throughput equals 1/RTT trx/sec ✕ #connections A Given Row Can't be Modified More Than 1/RTT

times a second

33

Round Trip Time (RTT) is Length of Time for a Signal to be sent and Receipt of Acknowledgement

[email protected]


Slave Lag in Galera

34

Client Master Node

START TRANSACTION

COMMIT Replicate Write Set

OK

Slave Node

COMMIT

PROCESS

APPLY

COMMIT

Acknowledge

SELECT (stale data)

Certify

[email protected]


NuancesUses Only InnoDB Primary Keys are Necessary Commit Latency Transaction Size Limited to 2GB — Unlimited in Future Versions DEADLOCK on COMMIT AUTO_INCREMENT Handled Differently Cluster Performance Limited to Slowest Node No Cluster-Wide Read Locks No XA Transactions

35


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Installation

[email protected]


Installing from a RepositoryMariaDB Repository Configuration Tool (https://downloads.mariadb.org/mariadb/repositories)

Available for all Major Distributions

37

Downloading MariaDB & Galera: https://mariadb.com/kb/en/mariadb/getting-started-with-mariadb-galera-cluster/

https://downloads.mariadb.org/mariadb/repositories

https://mariadb.com/kb/en/mariadb/getting-started-with-mariadb-galera-cluster/

[email protected]


Installing MariaDB Galera Cluster

38

$ yum install epel-release

Install Extra Packages for Enterprise Linux

$ yum install MariaDB-server

Install Galera Enabled MariaDB Server

$ yum install socat

Install Network Copying Tool

[email protected]


Additional Utilities

39

XtraBackup Useful for SST — Not Supported in MariaDB 10.2

$ yum install xtrabackup

XtraBackup Repo: http://www.percona.com/doc/percona-xtrabackup/installation/yum_repo.html

Percona Toolkit

$ yum install percona-toolkit

http://www.percona.com/doc/percona-xtrabackup/installation/yum_repo.html

[email protected]


Linux Configuration

40

SELINUX=permissive

[Service] TimeoutStartSec="30 min"

cat /selinux/enforce echo 0 > /selinux/enforce

Start Timeout Settings: https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStartSec= SELinux Configuration: http://galeracluster.com/documentation-webpages/selinux.html

SELinux — Disable Enforce Set Server to SELinux Mode Set Start Timeout for systemd

(CentOS 7+)Excerpt from /etc/selinux/config

systemctl daemon-reload

Executed from Command-Line

https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStartSec=

http://galeracluster.com/documentation-webpages/selinux.html

[email protected]


Linux FirewallPorts Used

Port 3306 — Client Connections to Nodes Port 4567 — Replication Protocol Port 4568 — Incremental State Transfers (IST) Port 4444 — Snapshot State Transfer (SST; socat)

IST & SST Methods may have Additional Connectivity Requirements

mysqldump requires mysql client connections (port 3306) between Nodes

Rsync and XtraBackup use Netcat (nc) between Nodes

41

iptables -L /etc/init.d/iptables stop chkconfig iptables off

Executed from Command-Line

[email protected]


MariaDB Configuration

42

[mysqld] max_connections=1024 binlog_format=ROW innodb_buffer_pool_size=200M

innodb_log_file_size=100M innodb_flush_log_at_trx_commit=2 default-storage-engine=innodb innodb_autoinc_lock_mode=2

Only InnoDB Tables will be Replicated --enforce-storage-engine=innodb

Documentation on Enforcing Storage Engine Setting: https://mariadb.com/kb/en/mariadb/server-system-variables/#enforce_storage_engine Documentation on InnoDB Flush Log at Transaction Commit Setting: https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-system-variables/#innodb_flush_log_at_trx_commit

https://mariadb.com/kb/en/mariadb/server-system-variables/#enforce_storage_engine

https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-system-variables/#innodb_flush_log_at_trx_commit

[email protected]


MariaDB Configuration — Galera Settings

43

wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="our_cluster" wsrep_node_name=node1 wsrep_cluster_address="gcomm://192.168.56.91,192.168.56.90,192.168.56.89"

wsrep_node_address=192.168.56.91 wsrep_slave_threads=1 wsrep_sst_method=rsync wsrep_sst_auth=galera:galera

wsrep_on=ON

[email protected]


Start First Node

44

systemctl start mariadb galera_new_cluster

mysql -u root -p -e /

"SHOW VARIABLES LIKE 'wsrep_provider_options"

Start galera1 as First Node Use SHOW STATUS and SHOW VARIABLES to List Galera Options

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; SHOW GLOBAL VARIABLES LIKE 'wsrep%';

Executed from Command-Line on First Node

Executed from mysql Client on First Node


[email protected]


Configure Galerawsrep_* Variables Configure MariaDB

Replication Behavior wsrep_provider_options is a Collection

of Variables that Control Galera Replication Plugin Behavior

45

Galera System Variables: https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/ Galera Status Variables: https://mariadb.com/kb/en/mariadb/galera-cluster-status-variables/

https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/

https://mariadb.com/kb/en/mariadb/galera-cluster-status-variables/

[email protected]


Default Enterprise Cluster Related Values

46

base_host = 192.168.56.92

base_port = 4567

cert.log_conflicts = no

evs.causal_keepalive_period = PT1S

evs.debug_log_mask = 0x1

evs.inactive_check_period = PT0.5S

evs.inactive_timeout = PT15S

evs.info_log_mask = 0

evs.install_timeout = PT15S

evs.join_retrans_period = PT1S

evs.keepalive_period = PT1S

evs.max_install_timeouts = 1

evs.send_window = 4

evs.stats_report_period = PT1M

evs.suspect_timeout = PT5S

evs.use_aggregate = true

evs.user_send_window = 2

evs.version = 0

evs.view_forget_timeout = PT5M

gcache.dir = /var/lib/mysql/

gcache.keep_pages_size = 0

gcache.mem_size = 0

gcache.name = /var/lib/mysql//galera.cache

gcache.page_size = 128M

gcache.size = 128M

gcs.fc_debug = 0

gcs.fc_factor = 1

gcs.fc_limit = 16

gcs.fc_master_slave = NO

gcs.max_packet_size = 64500

gcs.max_throttle = 0.25

gcs.recv_q_hard_limit = 9223372036854775807

gcs.recv_q_soft_limit = 0.25

gcs.sync_donor = NO

gmcast.listen_addr = tcp://0.0.0.0:4567

gmcast.mcast_addr =

gmcast.mcast_ttl = 1

gmcast.peer_timeout = PT3S

gmcast.time_wait = PT5S

gmcast.version = 0

gmcast.segment = 0

ist.recv_addr = 192.168.56.92

pc.checksum = true

pc.ignore_quorum = false

pc.ignore_sb = false

pc.linger = PT20S

pc.npvo = false

pc.version = 0

pc.weight = 1

protonet.backend = asio

protonet.version = 0

replicator.causal_read_timeout = PT30S

replicator.commit_order = 3

Galera System Variables: https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/ Galera Status Variables: https://mariadb.com/kb/en/mariadb/galera-cluster-status-variables/

https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/

https://mariadb.com/kb/en/mariadb/galera-cluster-status-variables/

[email protected]


Start Other NodesSecond and Third Nodes must be Provided an Address

for Connecting to the Cluster Use wsrep_cluster_address with IP Address of Node

and at Least One Other Node

47

wsrep_cluster_address = gcomm://192.168.56.101,192.168.56.102,192.168.56.103

Excerpt from Configuration File of Each Node

systemctl start mariadb

Executed from Command-Line on Second and Third Node

[email protected]


Cluster Address — wsrep_cluster_address

Write All Nodes Planned for Cluster in Address String

Don't Leave wsrep_cluster_address=gcomm:// in my.cnf

The wsrep_urls Parameter for mysqld_safe is Deprecated

48

wsrep_cluster_address=gcomm://10.0.0.1,10.0.0.2, ...

Galea Cluster Address: https://mariadb.com/kb/en/mariadb/galera-cluster-address/

https://mariadb.com/kb/en/mariadb/galera-cluster-address/

[email protected]


MariaDB Configuration

49

[mysqld] logbin binlog_format=ROW

wsrep_on=ON wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="our_cluster" wsrep_node_name=node2 wsrep_cluster_address="gcomm://192.168.56.91,192.168.56.92,192.168.56.93" wsrep_node_address=192.168.56.92

wsrep_slave_threads=1 wsrep_sst_method=rsync wsrep_sst_receive_address=192.168.56.92:3306 wsrep_sst_auth=galera:rover123

[email protected]


Start Other NodesJoin the Second and Third

Nodes to Cluster

50

SHOW GLOBAL STATUS LIKE 'wsrep%';



Executed from Command-Line on Second Node

Executed from mysql Client on Second Node

SHOW GLOBAL STATUS LIKE 'wsrep%';

Executed from mysql Client on Third Node

Executed from Command-Line on Third NodeNode 3

Node 2

[email protected]


Initial Cluster StartupAll Nodes Should Be Running Now and

Consistent Test Replication

51

SELECT * FROM table1;

CREATE TABLE test.table1 (col1 INT UNSIGNED KEY);

INSERT INTO test.table1 VALUES (1),(2),(3);

SELECT * FROM test.table1; INSERT INTO table1

VALUES (4);

Executed from mysql Client on Node 1



1

2

3

[email protected]


Sysbench Benchmarking ToolInstall RPM Packages

To Build a Database for Testing: sysbench [options] prepare

To Run a Test on Database: sysbench [options] run

Create a Database

52

Set Up SysBench: https://mariadb.com/kb/en/mariadb/sysbench-benchmark-setup/ Project Page: https://launchpad.net/sysbench

$ sysbench --test=sysbench_tests/db/common.lua --mysql-host=node1 \

--mysql-user=test --mysql-db=sbtest --oltp-table-size=250000 prepare

$ sysbench --test=sysbench_tests/db/oltp.lua --mysql-host=node1 \

--mysql- user=test --mysql-db=sbtest --oltp-table-size=250000 \

--report-interval=1 --max-requests=0 --tx-rate=10 run | grep tps

$ wget www.lefred.be/files/sysbench-0.5-3.el6_.x86_64.rpm

$ rpm -ivh www.lefred.be/files/sysbench-0.5-3.el6_.x86_64.rpm

https://mariadb.com/kb/en/mariadb/sysbench-benchmark-setup/

https://launchpad.net/sysbench


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


State Transfers

[email protected]


Node Recovery ProcedureA Node Determines it's not in Synch Communicates with Other Nodes to

Recover Determines its Current Cluster State to

Determine Transfer Method Required

54

Documentation on Node Recovery: http://galeracluster.com/documentation-webpages/recovery.html

http://galeracluster.com/documentation-webpages/recovery.html

[email protected]


Recovery MethodsIncremental State Transfer (IST)

Partial Recovery (Delta's) of Write Sets

Snapshot State Transfer (SST) Full Recovery of Data

Error Log shows Method Used and Outcome Check Error Logs on Joiner and Donor Nodes

55

[email protected]


Incremental State TransferIncremental State Transfer is very Effective gcache.size Parameter Defines Size of Write Sets kept for

IST Gcache is mmap File

Available Disk Space is Upper Limit for Size Allocation

Used if Joining Node had a Previous State Later than Start of Cluster

Only Possible after Graceful Shutdown Changes are Tracked by gcache — galera.cache on File System and in

Memory

56

Documentation on IST: http://galeracluster.com/documentation-webpages/statetransfer.html#incremental-state-transfer-ist

http://galeracluster.com/documentation-webpages/statetransfer.html#incremental-state-transfer-ist

[email protected]


Galera Saved Stategrastate.dat is Replication State File

Residing in Database System Data Directory — Included in Snapshot

uuid is Universal Unique Identifier seqno is Sequence Number of Last

Transaction Value Greater than 0 Indicates Proper

Shutdown Value -1 means No Known Sequence

Position (Server Crashed or Running)

57

# GALERA saved state version: 2.1 uuid: 82e240bb-6709-11e2-0800-619463ebd7b6

seqno: -1

[email protected]


Snapshot State TransferUsed to Send Full Database State Set wsrep_sst_method to Dictate Method to Use

mysqldump rsync (default) xtrabackup xtrabackup-v2 — Recommended

New Joining Nodes use SST

58

Documentation on SST: http://galeracluster.com/documentation-webpages/statetransfer.html#state-snapshot-transfer-sst

http://galeracluster.com/documentation-webpages/statetransfer.html#state-snapshot-transfer-sst

[email protected]


Forced SST — Expediting Full Recovery

Force SST by Deleting all Files in datadir

Stop All Nodes and Synchronize from galera1

59

systemctl stop mariadb

rm -rf /var/lib/mysql/*


cd /var/lib/mysql rm -rf * systemctl start mariadb

systemctl start mariadb tail -f mysqld.err

Executed from Command-Line on Node 2



1

2

3

[email protected]


XtraBackup SST — Alternate Full Recovery

Full Recovery without Downtime on Donor

Requires Installing and Appropriate Credentials

Add datadir to Configuration File

60

yum install xtrabackup

[server]

datadir = /var/lib/mysql

XtraBackup SST: https://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/xtrabackup_sst.html

GRANT PROCESS, RELOAD,

LOCK TABLES, REPLICATION CLIENT ON *.* TO 'xtrabackup'@localhost IDENTIFIED BY 'galera123';



Excerpt from my.cnf on Node 1

1

2

3

https://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/xtrabackup_sst.html

[email protected]


Configuring XtraBackup SSTStop Second Node and Edit its

MariaDB Configuration File Configure XtraBackup SST and

Rejoin Node to Cluster

61


[server] wsrep_sst_method = xtrabackup-v2 wsrep_sst_auth = galera:galera

rm -rf /var/lib/mysql/* systemctl start mariadb


Excerpt from my.cnf on Node 2


1

2

3

[email protected]


Verify IST WorksStop Second Node and Rejoin it to

Cluster Check Error Log for IST Messages Regularly Verify SST Works

62


journalctl -xe


CREATE TABLE test.testeroo

(col1 INT);





1

2

3

4


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Caveats

[email protected]


CaveatsConsistent Reads Primary Key is Required Dealing with AUTO_INCREMENT

Detecting Slow Nodes Parallel Replication MyISAM Replication XA Transactions

64

[email protected]


Slave Lag in Galera

65

Client Master Node

START TRANSACTION

COMMIT Replicate changeset

OK

Slave Node

COMMIT

PROCESS

APPLY

COMMIT

Acknowledgement

SELECT (waiting due to wsrep_sync_wait)

Certification

[email protected]


Consistent Reads SettingsSet wsrep_sync_wait to 1

Every Read (e.g., SELECT) Waits for Slave Queue to be Applied Fully

Set Timeout for Maximum Causal Read Wait replicator.causal_read_keepalive

Read Locks only on Local Node

66

[email protected]


Spotting Causality FailureTry Work Load to Spot Causality Failure Configure Read Causality and Test Again

67

for i in {1..1000}; do echo -n -e "\nRound $i "; mysql -ugalera -pgalera -h0 -P11001 test -e"INSERT INTO t VALUES($i)"; ret=$(mysql --batch -ugalera -pgalera -h0 -P11002 test -e"SELECT * FROM t WHERE i=$i");

if [ "$ret" = "" ]; then echo -n "FAILED" fi done

galera2-mariadb> SET SESSION wsrep_sync_wait = 1;

[email protected]


Dealing with AUTO_INCREMENTGalera Provides Automatic Control of AUTO_INCREMENT

Galera will adjust auto_increment_increment and auto_increment_offset variables so Inserts in Separate Nodes will Interleave

Increments AUTO_INCREMENT by Number of Nodes

68

wsrep_auto_increment_control = ON

Galera & AUTO_INCREMENT: https://mariadb.org/auto-increments-in-galera/

Excerpt from my.cnf File

https://mariadb.org/auto-increments-in-galera/

[email protected]


Controlling AUTO_INCREMENTStart First and Second Node Create AUTO_INCREMENT Table and Insert Rows through All Nodes Start Third Node and Insert More Rows

69

SHOW VARIABLES LIKE 'auto_increment%';

INSERT INTO a (n) VALUES ('n2'),('n2'),('n2');


SELECT * FROM a;

CREATE TABLE a

(i INT PRIMARY KEY AUTO_INCREMENT,

n VARCHAR(10));

INSERT INTO a (n)

VALUES ('n1'),('n1'),('n1');



INSERT INTO a (n) VALUES ('n3'),('n3'),('n3');

SELECT * FROM a;Executed from mysql Client on Node 1




1

2

3

4

[email protected]


Controlling AUTO_INCREMENTDisable AUTO_INCREMENT Control in

galera2 Stop and Start galera3

70

SET GLOBAL wsrep_auto_increment_control=0;






Executed from Command-Line on Node 3Executed from Command-Line on Node 3

1

2

3

4

5

Execute on Both Node 1 and 2

[email protected]


Detecting a Slow NodeNodes should have Equal Capacity — Use Equivalent Hardware

Galera Implements Flow Control to Protect Cluster from Slave Queue Growth

71

wsrep_flow_control_sent Number of Times Node Asked for Flow Control

wsrep_flow_control_recvd Number of Times Node Received Flow Control Stop Signal

wsrep_flow_control_paused Fraction of Time Node had to Pause for Flow Control

wsrep_local_recv_queue Length of Slave Queue

Slow Nodes in Galera: http://galeracluster.com/documentation-webpages/detectingaslownode.html

http://galeracluster.com/documentation-webpages/detectingaslownode.html

[email protected]


Slave Queue TuningNodes should have Equal Capacity — Use Equivalent Hardware Galera Implements Flow Control to Protect Cluster

from Slave Queue Growth A Value Greater than 0.02 for wsrep_flow_control_paused indicates a Performance Problem in Cluster

72

Managing Galera Flow: http://galeracluster.com/documentation-webpages/managingfc.html

http://galeracluster.com/documentation-webpages/managingfc.html

[email protected]


Slave Queue Tuningwsrep_slave_threads

Number of Parallel Threads Applying to Slave Queue Slave Thread Count = 1 to 3 x Logical CPU Cores

gcs.fc_limit High Limit for Flow Control (default 16) Flow Control Stops when Reached 5 x wsrep_slave_threads

gcs.fc_factor Factor times Limit is Lower Limit (default 1; recommended 0.8) Flow Control Resumes when Slave Queue Reduced to this

Level

73

[email protected]


Flow ControlConfigure Flow Control Limit Low

on Second Node Push Heavy Load against the First

Node and Check Status on Second

74

sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \ --mysql-user=root --mysql-password=fido123 \ --mysql-db=sbtest --oltp-table-size=25000 \ --report-interval=5 --max-requests=0 --num-threads=10 run

SET GLOBAL wsrep_provider_options = 'gcs.fc_limit=2';



[email protected]


Parallel ReplicationAny Number of Slave Applier

Threads can be Started Slave Control Assigns Write

Sets for Slave Appliers Slave Control is on ROW

Level Applying Only is Parallel Commit Order is Dictated

75

MariaDB

Slave Thread Slave Thread

Slave Control

Slave Queue

[email protected]


True Parallel ApplyingEvery Application Benefits Works on Row Level — Not on

Database or Table

76

SET GLOBAL wsrep_slave_threads = n;

Set to Number of Slaves that makes Sense

Monitor wsrep_cert_deps_distance Maximum about 4 times Number of

CPU Cores

[email protected]


True Parallel ApplyingStart Heavy Load on First

Node Monitor CPU on Second

Node with top Configure More Slave

Threads

77

sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \

--mysql- user=root --mysql-password=fido123 \

--mysql-db=sbtest --oltp-table-size=25000 \

--report-interval=5 --max-requests=0 --num-threads=10 run

SET GLOBAL wsrep_slave_threads = 2; SHOW PROCESSLIST;

SET GLOBAL wsrep_slave_threads = 4; SHOW PROCESSLIST;




Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Multi-Master Conflicts

[email protected]


Multi-Master ConflictsGalera Uses Optimistic Concurrency Control When Two Transactions Modify same Row on

Different Nodes at Same Time, One Transaction must Abort

Victim Transaction will get Deadlock Error (1213) upon COMMIT

Application should Retry Deadlocked Transactions Not All Applications have Retrying Logic Built-In

79

[email protected]


Retry TransactionsGalera can Retry autocommit Transaction for Client

Application within MariaDB Server MariaB will Not Return Deadlock Error — will Retry

Silently Transaction Set wsrep_retry_autocommit to Number of Retries

before Returning Deadlock Error Retrying Applies Only to autocommit Transactions

since Not Safe for Multi-Statement Transactions

80

[email protected]


DiagnosticsSet wsrep_log_conflicts to send Information on each Cluster

Conflict to MariaDB Error Log Use Cert.log_conflicts to Log Conflicting Transaction

Database Hot-Spots Rows to which Many Transactions want to Write Simultaneously Patterns like Queue or ID Allocation can be Hot-Spots

Monitor Status Variables wsrep_local_bf_aborts wsrep_local_cert_failures

Set wsrep_debug so All Conflicts will be Logged

81

[email protected]


Diagnose ConflictsCheck if Application Logic can be

Changed to get Deadlock Exception and Apply Retrying Logic in Application

Try Setting wsrep_retry_autocommit Limit Number of Master Nodes or

Change to Master-Slave Model Treat Writes Only to Hot-Spot Table as

Master-Slave, if can Limit Access to Table

82

tail -f /var/lib/mysql/node1.err

SET GLOBAL wsrep_log_conflicts=1; SET GLOBAL wsrep_provider_options =

'cert.log_conflicts=1';

SHOW LOCAL STATUS LIKE 'wsrep_local%';




1

2

3

[email protected]


Create Ad-Hoc ConflictAny Command after Conflict will

Result in Deadlock Error

83

CREATE TABLE c (i INT PRIMARY KEY, j INT); INSERT INTO c VALUES (1,0),(2,0); BEGIN; UPDATE c SET j=1 WHERE i=1;

UPDATE c SET j=2 WHERE i=1;

COMMIT; BEGIN; UPDATE c SET j=1 WHERE i=1;

UPDATE c SET j=2 WHERE i=1;

SELECT * FROM c;






1

2

3

4

5


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Schema Changes

[email protected]


Schema UpgradesDDL is Non-Transactional

Bad for Replication

Galera has Two Methods for DDL Total Order Isolation (TOI) Rolling Schema Upgrade (RSU)

Use wsrep_osu_method to choose Option pt-online-schema-change provides Lockless Schema

Upgrade Use TOI Mode Careful of Foreign Keys

85

[email protected]


Total Order IsolationDDL is Replicated Up-Front Each Node gets the DDL Statement and

Must Process DDL at Same Slot in Transaction Stream

Galera Isolates and Locks Entire Cluster for Duration of DDL Processing

86

Documentation on Isolation Levels: http://galeracluster.com/documentation-webpages/isolationlevels.html

http://galeracluster.com/documentation-webpages/isolationlevels.html

[email protected]


Rolling Schema UpgradeDDL is Not Replicated Galera will Remove Node from Replication for

Duration of DDL Processing When done with DDL, Node will get Missed

Transactions (e.g., IST) DBA should Roll RSU Operation over All Nodes Requires Backward Compatible Schema Changes Use Only under Certain Conditions

Planned SQL is Not Conflicting SQL will Not Generate Inconsistency

87

[email protected]


Schema Upgrade StrategyBest Practices

Plan Upgrades Try to make Backwards Compatible Rehearse Upgrades Determine DDL Execution Time Use RSU if Possible

ALTER TABLE to Create New AUTO_INCREMENT

Column will Cause Problems Every Node has Different AUTO_INCREMENT and Offset

Settings

88

[email protected]


Load Testing a Schema UpgradeStart Moderate sysbench Load

89

sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \

--mysql-user=root --mysql-password=fido1123 \ --mysql-db=sbtest --oltp-table-size=25000 \ --report-interval=5 --max-requests=0 --tx-rate=10 run

Issue Some DDL under TOI

ALTER TABLE sbtest

ADD COLUMN (m int UNSIGNED KEY);


ALTER TABLE sbtest ADD COLUMN (n int UNSIGNED);

CREATE TABLE m (i int);


CREATE TABLE n (i int);



1

2

3

4

[email protected]


Test Schema UpgradeIssue some DDL under RSU

90

SET GLOBAL wsrep_OSU_method=RSU; ALTER TABLE sbtest DROP COLUMN m;

SHOW CREATE TABLE sbtest; SET GLOBAL wsrep_OSU_method=RSU; ALTER TABLE sbtest DROP COLUMN n;

DROP TABLE m;

DROP TABLE n;

1

2

3

4






Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Back-Ups with Galera

[email protected]


Galera for Back-UpsAll Galera Nodes are Continuously Up-to-Date Best Practices

Dedicate a Reference Node for Back-ups Assign Global Transaction Identifier with Back-up

Possible Methods Desychronize a Node for Back-up XtraBackup

92

[email protected]


Back-Ups with Global Transaction IdentifierGlobal Transaction Identifier (GTID) marks Position

in Cluster Transaction Stream Backup with Known GTID makes Utilizing IST

Possible when Joining New Nodes Recovering a Node Provisioning New Nodes

Use XtraBackup's wrapper innobackupex with --galera-info

93

[email protected]


Back-Up by Desynchronizing a Node

Galera Replication

94

Server 1 Server 2 Server 3

Load Balancing Isolate Back-Up Node

[email protected]


Back-Up by Desynchronizing a NodeDesynchronize a Node

from Group (Enable wsrep_desync)

95

Desynchronize

Galera Replication


Load Balancing

[email protected]



96

Galera Replication


Load Balancing

FLUSH TABLES WITH READ LOCK;

Block Changes

[email protected]


Back-Up by Desynchronizing a NodeRead GTID from Status

and Assign to Back-Up (wsrep_cluster_uuid,

wsrep_last_committed)

97

Back-Up

Galera Replication


Load Balancing

[email protected]


Performing Back-Up — After Desynchronizing Node

Make Logical Backup mysqldump mydumper

Physical Back-Up Copy with cp LVM Snapshot

98

[email protected]



99

Galera Replication


Load Balancing

UNLOCK TABLES; Replicate Changes

[email protected]


Back-Up by Desynchronizing a NodeDisable wsrep_desync

100

Galera Replication


Load Balancing

[email protected]


XtraBackupHot Back-Up Method — Can be Used Any Time Simple and Efficient Use --galera-info Option to get Global Transaction

Identifier Logged into Separate Galera Information File

101


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Recovery

[email protected]


Recovering from DowntimeScheduled Maintenance — Clean Shut-Downs

Unexpected Cluster Failures Power Outage Network Outage Out of Synch Nodes

103

[email protected]


Recovering from Clean ShutdownCluster Start Order is Important — Always in

Reverse Order Start First the Node with Latest Writes or

Changes If Start Instead another Bootstrap Node, Force

SST for Subsequent Nodes

SST Initiated Unless All Nodes are in Same State (GTID)

Shutdown Cleanly All Nodes Cluster can Shrink to One Node if Node

Shutdowns are Graceful

104

symtemctl stop mariadb

Execute on All Three Nodes

[email protected]


RecoveryGalera GTID (seqno) is Written in Persistent InnoDB

Storage after each Commit Galera will use InnoDB GTID if seqno in grastate is

Undefined (-1)

seqno is Undefined in Case of Crash

Persistent GTID can be Queried with galera_recovery_information

Old Non systemd Installations Use --wsrep_recover Use InnoDB GTID if Unsure which Node was Last Updated

105

[email protected]


Split BrainShutdown Third Node and Kill Second Node Make One Node the Primary Component (PC)

Never have More than One Primary Component

106

systemdtl stop mariadb

Node 1 changes to Non-Primary State to Avoid Split Brain

SET GLOBAL wsrep_provider_options = 'pc.bootstrap=1';

systemdtl start mariadb


killall -9 mysqld




systemdtl start mariadb


1

2

3

4

5

[email protected]


Split Brain ScenariosEven Number of Nodes

Two Data Center Configurations Two-Node Configuration

Weighted Nodes with One-Node Majority (3 + 1 + 1)

107

Cautionpc.ignore_sb is Dangerous

SET GLOBAL wsrep_provider_options =

'pc.ignore_sb=1';

SET GLOBAL wsrep_provider_options = 'pc.weight=3';


[email protected]


Two-Node Cluster and Split Brainwsrep_provider_options="pc.ignore_sb"

108

VIP

Clients


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Load Balancing

[email protected]


Load Balancing ToolsUse glbd Daemon for Load Balancing through a

Daemon process Use libglb to Load Balance in Application Process

Need to Set Two Environment Variables to use libglb LD_PRELOAD=<path-to-libglb> GLB_OPTIONS='options string'

MaxScale is an Intelligent Proxy that

110

Documentation on Galera Load Balancer: https://mariadb.com/kb/en/mariadb/galera-load-balancer/ Project Page: http://www.codership.com/downloads/glb/

$ LD_PRELOAD=src/.libs/libglb.so \ GLB_OPTIONS="--random 3306 192.168.0.1 192.168.0.2 192.168.0.3" \

mysql -u root —p fido123 -h 127.0.0.1 -P 3306

https://mariadb.com/kb/en/mariadb/galera-load-balancer/


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Advanced Galera Topics

[email protected]


Weighted QuorumBetter Control on Primary & Non-

Primary State in Case of Failure For Link Failure, MAIN remains

Primary Manual Intervention to make DR

Primary

112

Documentation on Weighted Quorum: http://galeracluster.com/documentation-webpages/weightedquorum.html

server MAIN1: pc.weight=1 server MAIN2: pc.weight=1

server DR1: pc.weight=0 server DR2: pc.weight=0

http://galeracluster.com/documentation-webpages/weightedquorum.html

[email protected]


MulticastingValue of Multi-Casting — WAN vs. LAN

Recommended to set Multicast before Bootstrapping Cluster

Problems may occur if Migrating from Unicast to Multicast

113

$> ip route add 224.0.0.0/4 dev eth0

[server] wsrep_provider_options="gmcast.mcast_addr=239.192.0.11" wsrep_cluster_address="gcomm://239.192.0.11"

[email protected]


Multiple Galera Cluster (Asynchronous)Multi-Site Replication is Possible

with Galera Additional Caution Required

All Tables must have Primary Keys Manual Management of

AUTO_INCREMENT

Use Same server_id for All

114

[server]

wsrep_auto_increment_control=0 auto_increment_increment = 16 auto_increment_offset = 1,2,3,4,5,6,7,8,9(…) server_id = 1234

[email protected]


WAN ReplicationLink Failures happens More Frequently No Need to Arbitrate Always — Increase the

Timeouts

Writes Stall for Longer Time while Link is Down — Monitor the Network

115

wsrep_provider_options =

"evs.keepalive_period = PT3S; evs.suspect_timeout = PT30S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M"

Defaults: evs.keepalive_period = PT1S evs.suspect_timeout = PT5S evs.inactive_timeout = PT15S evs.install_timeout = PT15S

[email protected]


Cluster NotificationsCluster can Trigger Notifications Notification can be Sent to MaxScale Script API for Notification Syntax wsrep_notify_cmd Defines the Script to Handle

Notifications Load Balancer Configuration Monitoring

116

[email protected]


Cluster NotificationsNotification Script (/usr/bin/

wsrep_notify.sh)

117

# Command to Call when Node Status or Cluster Membership Changes.

# Will be Passed All or Some of Following Options:

# --status - New Status of Node

# --uuid - UUID of Cluster

# --primary - Whether Component is Primary ("yes" or "no")

# --members - Comma-Separated List of Members

# --index - Index of Node in List

echo "$@" >> /tmp/wsrep_notifications.log

[email protected]


Cluster NotificationsConfigure the Second Node to Call

the Notification Handler Cause Cluster State Changes

118

[server] wsrep_notify_cmd = /usr/bin/wsrep_notify.sh

tail -f /tmp/wsrep_notifications.log




Excerpt from Configuration File on Node 2

1

2

3

[email protected]


Master & Slave SettingsUse if Writing only to One

Master Set wsrep_sync_wait to 1

Can Release Flow Control before Pausing Replication

119

SET GLOBAL VARIABLE

wsrep_provider_options = "gcs.fc_limit = 256; gcs.fc_factor = 0.99; gcs.fc_master_slave = yes";

Defaultsgcs.fc_limit = 16gcs.fc_factor = 0.5fc_master_slave = NO


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Conclusions

[email protected]


On-Line ResourcesDocumentation Knowledge Base Open-Source Community

Post Questions on Forums Ask Questions on IRC — FreeNode

121

MariaDB Documentation and Knowledge Base: http://mariadb.com/kb

http://mariadb.com/kb

[email protected]


Other Training CoursesPublic In-Person Courses Private In-Person Courses Live Virtual Training Courses Self-Paced On-Line Courses

122

Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions

https://mariadb.learnupon.com/store

https://mariadb.learnupon.com/store/sessions


Introduction


State Transfer


Schema Changes

Back-UpsRecovery

Load Balancing


Thanks for Participating

MariaDB Enterprise Cluster Enterprise Cluster Introduction Architecture Installation State Transfer...

Documents

Transcript of MariaDB Enterprise Cluster Enterprise Cluster Introduction Architecture Installation State Transfer...