3 / 147 · MySQL 8.0 InnoDB Cluster is a very easy High Availability solution for MySQL Frédéric...

3 / 147

MySQL HA so easy ? That's insane !

MySQL 8.0 InnoDB Cluster is a very easy High Availability solution forMySQL

Frédéric Descamps - MySQL Community Manager - Oracle

4 / 147

Save the date !

5 / 147

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended forinformation purpose only, and may not be incorporated into any contract. It is not acommitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features orfunctionality described for Oracle´s product remains at the sole discretion of Oracle.

6 / 147

about me - http://about.me/lefred

Who am I ?

7 / 147

Frédéric Descamps@lefredMySQL EvangelistHacking MySQL since 3.23devops believerliving in Belgium 🇧🇪http://lefred.be

8 / 147

MySQL InnoDB Cluster: vision "A single product - MySQL - with high availability and scaling features baked in;providing an integrated end-to-end solution that is easy to use."

9 / 147

MySQL InnoDB Cluster: vision "A single product - MySQL - with high availability and scaling features baked in;providing an integrated end-to-end solution that is easy to use."

InnoDB

cluster

Ease-of-Use

Extreme Scale-Out

Out-of-Box Solution

Built-in HA

High Performance

Everything Integrated

10 / 147

One Product: MySQL

All components created togetherTested togetherPackaged together

Easy to Use

One client: MySQL ShellEasy packagingIntegrated orchestrationHomogenous servers

Flexible and Modern

SQL and NoSQL togetherProtocol BuffersAsynchronous APIDeveloper friendly

Support Read/Write Scale Out

Shared clustersFederated system of N replica sets

supporting cross shardoperations

Each replica set manages a shard

MySQL InnoDB Cluster: goals

11 / 147

MySQL InnoDB Cluster Architecture

12 / 147

Let's start with the demo, shall we ?

13 / 147

14 / 147

15 / 147

16 / 147

Group Replication: heart of MySQL InnoDBCluster

17 / 147

Group Replication: heart of MySQL InnoDBCluster

18 / 147

MySQL Group Replication

but what is it ?!?

19 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQL

20 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theory

21 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theoryGR allows to write on all Group Members (cluster nodes) simultaneously whileretaining consistency

22 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theoryGR allows to write on all Group Members (cluster nodes) simultaneously whileretaining consistencyGR implements conflict detection and resolution

23 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theoryGR allows to write on all Group Members (cluster nodes) simultaneously whileretaining consistencyGR implements conflict detection and resolutionGR allows automatic distributed recovery

24 / 147

but what is it ?!?

GR is a plugin for MySQL, made by MySQL and packaged with MySQLGR is an implementation of Replicated Database State Machine theoryGR allows to write on all Group Members (cluster nodes) simultaneously whileretaining consistencyGR implements conflict detection and resolutionGR allows automatic distributed recoverySupported on all MySQL platforms !!

Linux, Windows, Solaris, OSX, FreeBSD

25 / 147

And what does it means for users ?

26 / 147

And what does it means for users ?no longer necessary to handle server fail-over manually or with a complicated script

27 / 147

And what does it means for users ?no longer necessary to handle server fail-over manually or with a complicated scriptGR provides fault tolerance

28 / 147

And what does it means for users ?no longer necessary to handle server fail-over manually or with a complicated scriptGR provides fault toleranceGR enables update-everywhere setups

29 / 147

And what does it means for users ?no longer necessary to handle server fail-over manually or with a complicated scriptGR provides fault toleranceGR enables update-everywhere setupsGR handles crashes, failures, re-connetcs automatically

30 / 147

And what does it means for users ?no longer necessary to handle server fail-over manually or with a complicated scriptGR provides fault toleranceGR enables update-everywhere setupsGR handles crashes, failures, re-connetcs automaticallyAllows an easy setup of a MySQL service high available !

31 / 147

Default = Single Primary ModeBy default, MySQL InnoDB Cluster runs in Single Primary Mode.

32 / 147

mysql> show global variables like 'group_replication_single_primary_mode';+---------------------------------------+-------+| Variable_name | Value |+---------------------------------------+-------+| group_replication_single_primary_mode | ON |+---------------------------------------+-------+

33 / 147

mysql> show global variables like 'group_replication_single_primary_mode';+---------------------------------------+-------+| Variable_name | Value |+---------------------------------------+-------+| group_replication_single_primary_mode | ON |+---------------------------------------+-------+

In Single Primary Mode, a single member acts as the writable master (PRIMARY) andthe rest of the members act as hot-standbys (SECONDARY).

The group itself coordinates and configures itself automatically to determine whichmember will act as the PRIMARY, through a leader election mechanism.

34 / 147

35 / 147

MySQL RouterMySQL Router is lightweight middleware that provides transparent routing between yourapplication and backend MySQL Servers. It can be used for a wide variety of use cases,such as providing high availability and scalability by effectively routing database trafficto appropriate backend MySQL Servers.

36 / 147

MySQL RouterMySQL Router is lightweight middleware that provides transparent routing between yourapplication and backend MySQL Servers. It can be used for a wide variety of use cases,such as providing high availability and scalability by effectively routing database trafficto appropriate backend MySQL Servers.

MySQL Router doesn´t require any specific configuration. It configures itselfautomatically (bootstrap) using MySQL InnoDB Cluster´s metadata.

37 / 147

ProxySQL has native support for GroupReplication which makes it a good choicefor advanced users.

ProxySQLIf you need some specific features that are not yet available in MySQL Router, liketransparent R/W splitting, then you can use your software of choice.

We are also collaborating with ProxySQL.

38 / 147

MySQL ShellThe MySQL Shell is an interactive Javascript, Python, or SQL interface supportingdevelopment and administration for the MySQL Server and is a component of the MySQLServer. You can use the MySQL Shell to perform data queries and updates as well asvarious administration operations.

39 / 147

MySQL Shell (2)The MySQL Shell provides:

40 / 147

Both Interactive and Batch operations

41 / 147

Both Interactive and Batch operationsDocument and Relational Models

42 / 147

Both Interactive and Batch operationsDocument and Relational ModelsCRUD Document and Relational APIs via scripting

43 / 147

Both Interactive and Batch operationsDocument and Relational ModelsCRUD Document and Relational APIs via scriptingTraditional Table, JSON, Tab Separated output results formats

44 / 147

Both Interactive and Batch operationsDocument and Relational ModelsCRUD Document and Relational APIs via scriptingTraditional Table, JSON, Tab Separated output results formatsMySQL Standard and X Protocols

45 / 147

Both Interactive and Batch operationsDocument and Relational ModelsCRUD Document and Relational APIs via scriptingTraditional Table, JSON, Tab Separated output results formatsMySQL Standard and X Protocolsand more...

46 / 147

New since MySQL 8.0.11The Shell improved a lot and uses all the enhancements part of MySQL 8:

47 / 147

set persists

48 / 147

set persists

restart

This makes the remote administration very easy and cloud friendly.

49 / 147

don´t forget, this is a HA solution !

50 / 147

Thank you !

Any Questions ?

share your 💕 for MySQL on social media using @MySQL #DataOpsBarcelona

more at http://lefred.be/content/category/mysql/group-replication/http://mysqlhighavailability.com/

51 / 147

want to know more?

52 / 147

GR is nice, but how does it work ?

53 / 147

it´s just ...

54 / 147

it´s just ...

55 / 147

it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply ofthe changes are local to each nodes and asynchronous.

56 / 147

it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply ofthe changes are local to each nodes and asynchronous.

not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustratethis...

57 / 147

58 / 147

59 / 147

60 / 147

61 / 147

62 / 147

63 / 147

64 / 147

65 / 147

66 / 147

67 / 147

68 / 147

69 / 147

MySQL Group Replication (full transaction)

70 / 147

71 / 147

72 / 147

73 / 147

74 / 147

75 / 147

76 / 147

77 / 147

78 / 147

MySQL Group Communication System (GCS)MySQL Xcom protocol

79 / 147

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State Machine

80 / 147

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol

81 / 147

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocolits task: deliver messages across the distributed system:

82 / 147

atomically

83 / 147

atomicallyin Total Order

84 / 147

atomicallyin Total Order

MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.

85 / 147

Group Replication : Total Order Delivery - GTID

86 / 147

87 / 147

88 / 147

89 / 147

90 / 147

91 / 147

92 / 147

93 / 147

94 / 147

95 / 147

96 / 147

97 / 147

98 / 147

99 / 147

100 / 147

101 / 147

102 / 147

103 / 147

104 / 147

Group Replication: return commitAsynchronous Replication:

105 / 147

Group Replication: return from commit (2)Semi-Sync Replication:

106 / 147

Group Replication: return from commit (3)Group Replication:

107 / 147

Does this mean we can have a distant node andalways let it ack later ?

108 / 147

109 / 147

There will be always a penalty to pay. For example if you have 3 members, every 3rd writethe system will be affected by the lartency... why ?

110 / 147

There will be always a penalty to pay. For example if you have 3 members, every 3rd writethe system will be affected by the lartency... why ?

That´s because the system has to wait for the noop (single skip message) from the“distant” node.

More info: http://lefred.be/content/mysql-group-replication-about-ack-from-majority/

111 / 147

Group Replication : Optimistic LockingGroup Replication uses optimistic locking

112 / 147

during a transaction, local (InnoDB) locking happens

113 / 147

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)

114 / 147

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification

115 / 147

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification

Let´s first have a look at the traditional locking to compare.

116 / 147

Traditional locking

117 / 147

Traditional locking

118 / 147

Traditional locking

119 / 147

Traditional locking

120 / 147

Traditional locking

121 / 147

Traditional locking

122 / 147

Optimistic Locking

123 / 147

Optimistic Locking

124 / 147

Optimistic Locking

125 / 147

Optimistic Locking

126 / 147

Optimistic Locking

127 / 147

Optimistic Locking

128 / 147

Optimistic Locking

The system returns error 149 as certification failed:

ERROR 1180 (HY000): Got error 149 during COMMIT

129 / 147

Such conflicts happen only when using multi-primary group !

130 / 147

CertificationCertification is the process that only needs to answer the following unique question:

131 / 147

can the write (transaction) be applied ?

132 / 147

can the write (transaction) be applied ?based on unapplied earlier transactions

133 / 147

can the write (transaction) be applied ?based on unapplied earlier transactionssuch conflicts must come for other members/nodes

134 / 147

happens on every member/node

135 / 147

happens on every member/nodeshould be deterministic on every node

136 / 147

happens on every member/nodeshould be deterministic on every noderesults are not reported to the group

137 / 147

pass: enter in the apply queue

138 / 147

pass: enter in the apply queuefail: drop the transaction

139 / 147

serialized by the total order in GCS/XCOM + GTID

140 / 147

serialized by the total order in GCS/XCOM + GTIDcost is based on trx size (# rows & # keys)

141 / 147

serialized by the total order in GCS/XCOM + GTIDcost is based on trx size (# rows & # keys)first committer (and ack from majority) wins rule

142 / 147

Drawbacks of optimistic lockinghaving a first-committer-wins system means conflicts will more likely happen whenwriting on multiple members with:

143 / 147

large transactions

144 / 147

large transactions

long running transactions

145 / 147

large transactions

long running transactions

hotspot records

146 / 147

Thank you !

Any Questions ?

share your 💕 for MySQL on social media using @MySQL #DataOpsBarcelona

more at http://lefred.be/content/category/mysql/group-replication/http://mysqlhighavailability.com/

147 / 147

3 / 147 · MySQL 8.0 InnoDB Cluster is a very easy High Availability solution for MySQL Frédéric...

Documents

Transcript of 3 / 147 · MySQL 8.0 InnoDB Cluster is a very easy High Availability solution for MySQL Frédéric...

Descamps v. United States, 133 S. Ct. 2276 (2013)

MANAGED MySQL SERVER - CenturyLink...MySQL SERVER • Procure and manage MySQL Enterprise Edition Software Licenses • Support MySQL Versions 5.0, 5.1 and 5.6 • Install MySQL software

MySQL X Protocol - Percona€¦ · What is the MySQL X Protocol? In April MySQL 5.7.12 introduces MySQL DocumentStore noSQL API to access JSON data in MySQL MySQL x plugin in the

Recidivist Enhancements after Descamps June 2014

Descamps v. United States (2013)

Securing your MySQL/MariaDB Data - MySQL Expert - AWS ...ronaldbradford.com/presentations/Securing-your-MySQL-MariaDB-data... · Securing your MySQL/MariaDB Data Ronald ... MySQL

NoSQL + SQL = MySQL - doag.org · MySQL is the most popular database . MySQL Evolution •MySQL 5.1 •MySQL 5.5 ... Intro: One Giant Leap for SQL: MySQL 8.0 Released “This is a

MySQL Enterprise Monitor€¦ · mysql-monitor.log MySQL log file. Replication 1.dot The calculated MySQL server replication structure preferences.properties MySQL Enterprise Monitor

PHP and MySQL php-mysql PHP and MySQL

MySQL Whitepaper MySQL HA Solutions

Why Use MySQL · 2.MySQL Cluster Community Edition 3.MariaDB/Percona Server • Commercial 1.MySQL Classic Edition 2.MySQL Standard Edition 3.MySQL Enterprise Edition 4.MySQL Cluster

1 / 284 - · PDF fileMySQL InnoDB Cluster & Group Replication in a Nutshell: Hands-On Tutorial Percona Live 2017 - Santa Clara Frédéric Descamps - MySQL Community Manager - Oracle

Why MySQL Enterprise Edition? MySQL Enterprise Edition ...€¦ · MySQL Enterprise Audit Adds regulatory compliance to MySQL applications (HIPAA, Sarbanes-Oxley, PCI, etc.) MySQL

Manage MySQL like a devops sysadmin - percona.com€¦ · Manage MySQL like a devops sysadmin Webinar – Oct 2012 Frédéric Descamps. Who am I ... (starting by aligning devs and

MySQL InnoDB Cluster · Linux High Availability Cluster with DRBD and MySQL. MySQL NDB Cluster. Oracle MySQL Cloud Service. MySQL with Solaris Cluster. Galera Cluster for MySQL. Amazon

11-9540 Descamps v United States - Amazon Web Servicessblog.s3.amazonaws.com/wp-content/uploads/2012/12/Descamps-Brief-for... · no. 11-9540 in the supreme court of the united states

1 / 152 - Percona · MySQL InnoDB Cluster & Group Replication in a Nutshell: Hands-On Tutorial Percona Live Europe 2017 - Dublin Frédéric Descamps - MySQL Community Manager - Oracle

MySQL Replication Update MySQL 5.5 (GA) & MySQL …assets.en.oreilly.com/1/event/56/MySQL Replication Update... · MySQL Replication Update MySQL 5.5 ... Be sure that slave has received

SNO+ solar neutrino program - underground.physics.berkeley.edu · JinPing - SNO+ Solar Freija Descamps, 09/06/2014 SNO+ solar neutrino program Freija Descamps for the SNO+ collaboration

Post-Descamps World Paresh Patel, Federal Public Defender, D.Md.