1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...

1 / 191

2 / 1912 / 191

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purpose only, and may not beincorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features or functionality described for Oracle´s productremains at the sole discretion of Oracle.

Copyright @ 2019 Oracle and/or its affiliates. All rights reserved.

3 / 191

about.me/lefred

Who am I ?


4 / 191

Frédéric Descamps@lefredMySQL EvangelistManaging MySQL since 3.23devops believerliving in Belgium 🇧🇪https://lefred.be


5 / 191

https://lefred.be/

Group Replication: heart of MySQL InnoDB Cluster


6 / 191

Group Replication: heart of MySQL InnoDB Cluster


7 / 191

MySQL Group Replication

but what is it ?!?


8 / 191

GR is a plugin for MySQL, made by MySQL and packaged withMySQLGR is an implementation of Replicated Database StateMachine theoryPaxos based protocolGR allows to write on all Group Members (cluster nodes)simultaneously while retaining consistencyGR implements conflict detection and resolutionGR allows automatic distributed recovery

Automatic failoverAutomatic membership configuration

Adding/removing membersNetwork partitions, failures

Prevents data lossSupported on all MySQL platforms !!

Linux, Windows, Solaris, OSX, FreeBSDCompletely Open Source - GPL ! No license required to haveHigh Availability


but what is it ?!?


9 / 191

Group Replication is nice, but how does it work ?


10 / 191


it´s just ...


11 / 191


it´s just ...


12 / 191


it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes andasynchronous.


13 / 191


it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes andasynchronous.

not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustrate this...


14 / 191


15 / 191



16 / 191



17 / 191



18 / 191



19 / 191



20 / 191



21 / 191



22 / 191



23 / 191



24 / 191



25 / 191



26 / 191



27 / 191

MySQL Group Replication (full transaction)


28 / 191



29 / 191



30 / 191



31 / 191



32 / 191



33 / 191



34 / 191



35 / 191



36 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocol


37 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State Machine


38 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol (a variant of Mencius)


39 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol (a variant of Mencius)its task: deliver messages across the distributed system:


40 / 191


atomically


41 / 191


atomicallyin Total Order (Writes)


42 / 191


atomicallyin Total Order (Writes)

MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.


43 / 191

Architecture, Stack, Core, ...



44 / 191

MySQL Group Replication - Architecture

Node Types:

R: Traffic routers/proxies: MySQL Router, HAProxy, ProxySQL...M: mysqld nodes participating in MySQL Group Replication


45 / 191

MySQL Group Replication - Stack


46 / 191

Server calls into the plugin through a generic interface

Most server internals are hidden from the plugin

Plugin interacts with the server through a generic interface

Replication plugin determines the fate of the commitoperation through a well defined server interfaceThe plugin makes use of the relay log infrastructure toinject changes in the receiving server

MySQL Group Replication - Core


47 / 191

Maintaining distributed execution contextDetecting and Resolving conflictsHandling distributed recovery

Detect membership changesDonate state if neededCollect state if needed

Proposing transactions to other membersReceiving and handling transactions from other membersDeciding the ultimate fate of transactions

commit or rollback

MySQL Group Replication - GR Plugin


48 / 191

The communication API (and binding) is responsible for:

Abstracting the underlying group communication system implementation fromthe plugin itselfMapping the interface to a specific group communication system implementation

The Group Communication System engine:

Paxos implementation (Similar to Paxos Mencius)Building block to provide distributed agreement between servers

MySQL Group Replication - GCS


49 / 191

Total Order

GTID generation


50 / 191

MySQL Group Replication uses MySQL replication framework bydesign:

binary logs


51 / 191


binary logs

relay logs


52 / 191


binary logs

relay logs

GTIDs: Global Transaction IDs


53 / 191

How does Group Replication handle GTIDs ?There are two ways of generating GTIDs:


54 / 191


AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses thesource server UUID, on Group Replication, the group name is used.


55 / 191


AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses thesource server UUID, on Group Replication, the group name is used.

ASSIGNED: the user assigns manually a GTID through SET GTID_NEXT to the transaction. This is common to any replicationformat and the id is assigned before the transaction starts.


56 / 191


57 / 191

Group Replication : Total Order Delivery - GTID


58 / 191



59 / 191



60 / 191



61 / 191



62 / 191



63 / 191



64 / 191



65 / 191



66 / 191



67 / 191



68 / 191



69 / 191



70 / 191



71 / 191



72 / 191



73 / 191



74 / 191



75 / 191



76 / 191

Group Replication : GTIDThe previous example was not totally in sync with reality. In fact, a writer allocates a block of GTID and when we have multiplewrites (multi-primary mode) all writers will use GTID sequence numbers in their allocated block.

The size of the block is defined by group_replication_gtid_assignment_block_size (default to 1M)


77 / 191

Group Replication : GTIDExample:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355


78 / 191



New write on an other node:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355,1000354


79 / 191





Let's write on the third node:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355:1000354:2000354


80 / 191





Let's write on the third node:


And writing back on the first one:



81 / 191


82 / 191

done !

Return from Commit


83 / 191

Group Replication: return from commitAsynchronous Replication:


84 / 191

Group Replication: return from commit (2)Semi-Sync Replication:


85 / 191

(*): eventual

Group Replication: return from commit (3)Group Replication (*):


86 / 191

(*): before



87 / 191

(*): after



88 / 191

Does this mean we can have a distant node and always let it ack later?


89 / 191


NO!


90 / 191


NO!

Because the system has to wait for the noop (single skip message) from the “distant” node where latency is higher

The size of the GCS consensus messages window can be get and set from UDF functions:group_replication_get_write_concurrency(), group_replication_set_write_concurrency()

mysql> select group_replication_get_write_concurrency();+-------------------------------------------+| group_replication_get_write_concurrency() |+-------------------------------------------+| 10 |+-------------------------------------------+


91 / 191

Event HorizonGCS Write Consensus Concurrency


92 / 191


group replication write concurrency


93 / 191




94 / 191




95 / 191



96 / 191



97 / 191



98 / 191



99 / 191


100 / 191

conflict

Optimistic Locking


101 / 191

Group Replication : Optimistic LockingGroup Replication uses optimistic locking


102 / 191


during a transaction, local (InnoDB) locking happens


103 / 191


during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)


104 / 191


during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification


105 / 191


during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification

Let´s first have a look at the traditional locking to compare.


106 / 191

Traditional locking


107 / 191

Traditional locking


108 / 191

Traditional locking


109 / 191

Traditional locking


110 / 191

Traditional locking


111 / 191

Traditional locking


112 / 191

Optimistic Locking


113 / 191

Optimistic Locking


114 / 191

Optimistic Locking


115 / 191

Optimistic Locking


116 / 191

Optimistic Locking


117 / 191

Optimistic Locking


118 / 191

Optimistic Locking

The system returns error 149 as certification failed:

ERROR 1180 (HY000): Got error 149 during COMMIT


119 / 191

Optimistic Locking (2)The conflict detection can happen at two different stage:

Only one of the two transactions was sent to the Group and the other one is still running (local).

If both transaction were sent to the Group at almost the same time and both reached GCS/XCOM.


120 / 191

Optimistic Locking (3)

Only one transaction was sent to GCS and the other conflicting one is still local:

In this case, it's the high priority transaction mechanism of InnoDB that kills the local one:

If you try any statement in the transaction's session you will see:

ERROR: 1213: Deadlock found when trying to get lock; try restarting transaction

If you try to commit the transaction you will see:

ERROR: 1180: Got error 149 - 'Lock deadlock; Retry transaction' during COMMIT


121 / 191

Optimistic Locking (4)

Both transactions were sent to GCS:

The second transaction (the conflicting one) will return:

ERROR 3101 (40000): Plugin instructed the server to rollback the current transaction.


122 / 191

Such conflicts happen only when using multi-primary group !

not totally true in MySQL < 8.0.13 when failover happens


123 / 191

Drawbacks of optimistic lockinghaving a first-committer-wins system means conflicts will more likely happen when writing on multiple members with:


124 / 191


large transactions


125 / 191


large transactions

long running transactions


126 / 191


large transactions

long running transactions

hotspot records


127 / 191


128 / 191

Configurable Consistency Guarantees

Consistency Levels


129 / 191

mysql> show variables like 'group_replication_consistency';+-------------------------------+----------+| Variable_name | Value |+-------------------------------+----------+| group_replication_consistency | EVENTUAL |+-------------------------------+----------+

Consistency: EVENTUAL (default)By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read thesame data on another node, it is eventually there.


130 / 191

mysql> show variables like 'group_replication_consistency';+-------------------------------+----------+| Variable_name | Value |+-------------------------------+----------+| group_replication_consistency | EVENTUAL |+-------------------------------+----------+

Consistency: EVENTUAL (default)By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read thesame data on another node, it is eventually there.

Since MySQL 8.0.16, we have to possibility to set thesynchronization point at read or at write or both (globally or for asession).


131 / 191

Consistency: BEFORE (READ)


132 / 191

Consistency: AFTER (WRITE)


133 / 191

Consistency: BEFORE_AND_AFTER


134 / 191


135 / 191

can the transaction be committed ?

Certification


136 / 191

CertificationCertification is the process that only needs to answer the following unique question:


137 / 191


can the write (transaction) be committed ?


138 / 191


can the write (transaction) be committed ?based on yet to be applied transactions


139 / 191


can the write (transaction) be committed ?based on yet to be applied transactionssuch conflicts must come for other members/nodes


140 / 191



happens on every member/node and is deterministic


141 / 191



happens on every member/node and is deterministicresults are not reported to the group (does not require a new communication step)


142 / 191




pass: commit/queue to appy


143 / 191




pass: commit/queue to appyfail: rollback/drop the transaction


144 / 191





serialized by the total order in GCS/XCOM + GTID


145 / 191





serialized by the total order in GCS/XCOM + GTIDcost is based on trx size (# rows & # keys)


146 / 191

terminology

Write vs Writeset


147 / 191

Let's illustrate a table:


148 / 191

Now let's make a change


149 / 191

and at commit time:


150 / 191

WritesetsContain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies thatneed to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.


151 / 191

WritesetsContain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies thatneed to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.

WritesCalled also write values, refers to the actual changes. Write values are also gathered during transaction execution.


152 / 191

Writeset - examples+-------+-----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+-----------+------+-----+---------+-------+| id | binary(1) | NO | PRI | NULL | || name | binary(2) | YES | | NULL | |+-------+-----------+------+-----+---------+-------+


153 / 191


mysql> insert into t2 values (1,2);


154 / 191



pke: PRIMARY | test | t2 | 1 | 1 hash: 11853456929268668462


155 / 191




mysql> update t2 set name=3 where id=1;


156 / 191





pke: PRIMARY | test | t2 | 1 | 1 hash: 10002085147685770725pke: PRIMARY | test | t2 | 1 | 1 hash: 10002085147685770725


157 / 191

Writeset - examples (2)+-------+-----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+-----------+------+-----+---------+-------+| id | binary(1) | NO | PRI | NULL | || name | binary(2) | YES | UNI | NULL | || name2 | binary(1) | YES | | NULL | |+-------+-----------+------+-----+---------+-------+


158 / 191


mysql> insert into t3 values (1,2,3);


159 / 191






161 / 191

Certification


164 / 191


165 / 191

Enhanced support for large transactions

Message Fragmentation


166 / 191

Fragmenting an outgoing message

(1). If the message size exceeds the maximum size that the user allows(group_replication_communication_max_message_size), the member fragments the message into chunksthat do not exceed the maximum size.

(2). The member broadcasts each chunk to the group, i.e. forwards each chunk individually to XCom.


167 / 191

Reassembling an incoming message - first fragment

(2). The members conclude that the incoming message is actually a fragment of a bigger message.

(3). The members buffer the incoming fragment because they conclude the fragment is a chunk of a still-incomplete message.(Fragments contain the necessary metadata to reach this conclusion.)


168 / 191

Reassembling an incoming message - second fragment

(5). Same as step 3.

(6). Same as step 4.


169 / 191

Reassembling an incoming message - last fragment

1. The members conclude that the incoming message is actually a fragment of a bigger message.

2. The members conclude that the incoming fragment is the last chunk missing, reassemble the original, whole message, andprocess it.


170 / 191

Houston we have a problem !

Flow Control


171 / 191

Flow ControlIn Group Replication, every member send statistics about its queues (applier queue and certification queue) to the other members.Then every node decide to slow down or not if they realize that one node reached the threshold for one of the queue.


172 / 191


So when group_replication_�ow_control_mode is set to QUOTA on the node seeing that one of the othermembers of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculatedbased on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over thequota” messages from the last period.


173 / 191


So when group_replication_�ow_control_mode is set to QUOTA on the node seeing that one of the othermembers of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculatedbased on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over thequota” messages from the last period.

This mean that the threshold is NOT decided on the node being slow, but the node writing a transaction checks its threshold flowcontrol values and compare them to the statistics from the other nodes to decide to throttle or not.


174 / 191

Flow Control - on writer

>quota


175 / 191

Flow Control - on all members


176 / 191

Flow Control - configuration variablesAs in MySQL 8.0.16:

+-----------------------------------------------------+-------+| Variable_name | Value |+-----------------------------------------------------+-------+| group_replication_�ow_control_applier_threshold | 25000 || group_replication_�ow_control_certi�er_threshold | 25000 || group_replication_�ow_control_hold_percent | 10 || group_replication_�ow_control_max_quota | 0 || group_replication_�ow_control_member_quota_percent | 0 || group_replication_�ow_control_min_quota | 0 || group_replication_�ow_control_min_recovery_quota | 0 || group_replication_�ow_control_mode | QUOTA || group_replication_�ow_control_period | 1 || group_replication_�ow_control_release_percent | 50 |+-----------------------------------------------------+-------+


177 / 191

transaction's lifecycle in Group Replication

Summary


178 / 191

begin;


179 / 191

begin;update table1 set c = 999 where id =2;


180 / 191

begin;update table1 set c = 999 where id =2;update table1set b = "eee"where id = 3;


181 / 191


commit;

clie

nt b

lock

s on

com

mit.

..


182 / 191


commit;

clie

nt b

lock

s on

com

mit.

..

writesets+ gtid_event+ write values


183 / 191


commit;

clie

nt b

lock

s on

com

mit.

..


certify


184 / 191


commit;

clie

nt b

lock

s on

com

mit.

..


certifycertify


185 / 191


commit;

clie

nt b

lock

s on

com

mit.

..


certifycertify

certify


186 / 191


commit;

commit finalized


certifycertify

certify

+ GTID

binlog


187 / 191


commit;

commit finalized


certifycertify

certify

+ GTID

binlog

+ GTID


188 / 191


commit;

commit finalized


certifycertify

certify

+ GTID

binlog

+ GTID

+ GTIDrelaylog

relaylog


189 / 191


commit;

commit finalized


certifycertify

certify

+ GTID

binlog

+ GTID

+ GTIDrelaylog

relaylog

binlog

binlog


190 / 191

Thank you !

Any Questions ?


191 / 191

1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...

Documents

Transcript of 1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...