Proxysql ha plam_2016_2_keynote

Percona Live Europe 2016Use ProxySQL to Improve Your

MySQL High Availability SolutionMarco Tusa Manager Consulting

Amsterdam, Netherlands | October 3 – 5, 2016

2

● Marco “The Grinch” • Open source enthusiast

About me

3

What is HA

3

What is HA9 0. 0 0 0 % (36 days) MySQL Replication 9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD 9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive) 9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes 9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication 9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication 9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum 9 9. 9 9 9 % (5 minutes) MySQL Cluster

4

What is HA

5

What is HA

6

Objectively What we have is:

6

Objectively What we have is:

MySQL Group Replication

7

The telephone game

8

All talking

9

Black box:

10

Use the Right Tool for the Job…

11

What we need

12

ProxySQL

13

ProxySQL

14

Understand the basics

14


Write / Read (Master/Slave)

14



Multiple Writers

14



Multiple WritersSharding &

Geographic Distribution

14





Mirroring

14





Mirroring

Query re-write

15

Understand the basics – how this is done?

15

Servers group by function


15

Servers group by functionHost Groups


15


Rules define what, where and how


15


Rules define what, where and howQuery Rules


15



Special Grouping for Replication


15



Special Grouping for ReplicationReplication Host Group


16

HostGroup (HG) & Replication HG


17

Query rules


18

Scheduler


19

Basic scenario

19

Basic scenario

HG: 10U: app_RW

HG: 11U: app_R

HG: 12U: ETL_R

20

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('etl_R','test',1,12,’etl_schema'); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;

Basic scenario - Users

21

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.5',10,3306,1000); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.6',11,3306,1000000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.7',11,3306,1000000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.5',11,3306,1,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,10); INSERT INTO mysql_replication_hostgroups VALUES (10,11); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Basic scenario – Servers / HG

22

insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1); insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1); insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1); insert into mysql_query_rules (username,destination_hostgroup,active) values('etl_R',12,1);

LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

Basic scenario – Query rules

23

Basic scenario – with ProxySQL

23


Writes

23


Writes

Reads

23


Writes

ReadsReads

23


ReadsReads

24

READ_ONLY = 0 WRITE HG READ_ONLY = 1 READ HG

ProxySQL – Replication HG

READ_ONLY=0 READ_ONLY=1 READ_ONLY=1

HG 11 HG 11HG 10 & HG 11

24

READ_ONLY = 0 WRITE HG READ_ONLY = 1 READ HG

ProxySQL – Replication HG

READ_ONLY=0 READ_ONLY=1

HG 10 & HG:11 HG 11

READ_ONLY=0

HG 10 & HG 11

25

ProxySQL + MHA (or any controller)

Writes

ReadsReads

26

No movable part like:


26

No movable part like:VIP


26


DNS (Route 53 or similar)


26


DNS (Route 53 or similar)No Crazy scripts to manage other proxy (ie HAProxy)


26


DNS (Route 53 or similar)No Crazy scripts to manage other proxy (ie HAProxy)

Better distributed HA


27

What timing with MHA?

27

MHA discovery time (ping x 3 times) -> ~9 sec


27

MHA discovery time (ping x 3 times) -> ~9 secMHA recovery (binlog) variable time


27


MHA Slave shift + Master READ-ONLY < 2 sec


27


MHA Slave shift + Master READ-ONLY < 2 sec

ProxySQL will react as soon as READ_ONLY=0


28

Nothing all defaults

master_ip_failover_script=master_ip_failover master_ip_online_change_script=master_ip_online_change

Test it:masterha_master_switch --master_state=alive --conf=/etc/mha.cnf --orig_master_is_new_slave --interactive=0 --running_updates_limit=0

What is needed with MHA?

29

Given no VIP or centralize element.

Why better distributed HA?

30

Integration with PXC\Galera

31


32


32

Integration with PXC\Galera read_only wsrep_status wsrep_rejectqueries wsrep_donorrejectqueries wsrep_connected wsrep_desinccount wsrep_ready wsrep_provider wsrep_segment Number of nodes in by segment Retry loop

32


Online Offline_soft Offline_hard Shunned

32


Online Offline_soft Offline_hard Shunned

But we also have Hostgroups Replication HG

33


34


HG: 10

HG: 10

HG: 12HG: 11

HG: 11HG: 12

35

Multiple writer vs Single writer


35

Multiple writer vs Single writerMulti writer based on weight and Hostgroups


35


Need Replication HG and READ_ONLY


35


Need Replication HG and READ_ONLYFailover is done without the need of performing actions on the nodes


35



Failover is done modifying the READ_ONLY value


35



Failover is done modifying the READ_ONLY valueSingle WRITER node isolation not guarantee


35



Failover is done modifying the READ_ONLY valueSingle WRITER node isolation not guarantee

Single WRITER node isolation is guarantee


36

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');

insert into mysql_users (username,password,active,default_hostgroup,default_schema) values (‘appDR_R','test',1,12,’etl_schema'); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;

Integration with PXC\Galera - Users

37

insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1);

insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1);

insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1);

insert into mysql_query_rules (username,destination_hostgroup,active) values(‘appDR_R',12,1); LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

Integration with PXC\Galera – QR

38

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)

VALUES ('192.168.1.5',10,3306,1000000000); 192.168.1.6 … 10000; 192.168.1.7 … 10

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.5',11,3306,1); 192.168.1.6 … 100000; 192.168.1.7 … 100000

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100); 192.168.1.9 … 100; LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Integration with PXC\Galera – HG MW

39

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.5',10,3306,100);

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.5',11,3306,1); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.6',11,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.7',11,3306,100);

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100); 192.168.1.9 … 100; INSERT INTO mysql_replication_hostgroups VALUES (10,11); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Integration with PXC\Galera HG SW

40

ProxySQL is configured Now what?

Integration with PXC

40

ProxySQL is configured Now what?

Example scripthttps://github.com/Tusamarco/proxy_sql_toolsFull guide: https://goo.gl/vcDFZx


41

Set the script: INSERT INTO scheduler (id,active,interval_ms,filename,arg1) values (10,0,2000,"/var/lib/proxysql/galera_check.pl","-u=admin -p=admin -h=192.168.1.50 -H=500:W,501:R -P=3310 --execution_time=1 --retry_down=2 --retry_up=1 --main_segment=1 --debug=0 --log=/var/lib/proxysql/galeraLog"); LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK; update scheduler set active=1 where id=10; LOAD SCHEDULER TO RUNTIME;


42

+--------------+-------------+--------+------------+---------+ | hostgroup_id | hostname | status | weight | comment | +--------------+-------------+--------+------------+---------+ | 10 | 192.168.1.5 | ONLINE | 1000000000 | | | 10 | 192.168.1.6 | ONLINE | 10000 | | | 10 | 192.168.1.7 | ONLINE | 10 | | | 11 | 192.168.1.5 | ONLINE | 1 | | | 11 | 192.168.1.6 | ONLINE | 100000 | | | 11 | 192.168.1.7 | ONLINE | 100000 | | | 12 | 192.168.1.8 | ONLINE | 100 | | | 12 | 192.168.1.9 | ONLINE | 100 | | +--------------+-------------+--------+------------+---------+

Integration with PXC MW

43

+--------------+-------------+--------+------------+---------+ | hostgroup_id | hostname | status | weight | comment | +--------------+-------------+--------+------------+---------+ | 10 | 192.168.1.5 |OFFLINE_SOFT | 1000000000 | 10_W_11_R_retry_up=0;10_W_11_R_retry_down=0; | 10 | 192.168.1.6 | ONLINE | 10000 | | | 10 | 192.168.1.7 | ONLINE | 10 | | | 11 | 192.168.1.5 | OFFLINE_SOFT | 1 | ... | | 11 | 192.168.1.6 | ONLINE | 100000 | | | 11 | 192.168.1.7 | ONLINE | 100000 | | | 12 | 192.168.1.8 | ONLINE | 100 | | | 12 | 192.168.1.9 | ONLINE | 100 | | +--------------+-------------+--------+------------+---------+


44

+---------------------------------+-------+| Variable_name | Value |+---------------------------------+-------+| wsrep_sst_donor_rejects_queries | ON |+---------------------------------+-------+


44


ProxySQL Think all is ok


44


ProxySQL Think all is ok

Application writes will fail


45

+--------------+-------------+--------+------------+---------+ | hostgroup_id | hostname | status | weight | comment | +--------------+-------------+--------+------------+---------+ | 10 | 192.168.1.6 | ONLINE | 10000 | | | 10 | 192.168.1.7 | ONLINE | 10 | | | 11 | 192.168.1.6 | ONLINE | 100000 | | | 11 | 192.168.1.7 | ONLINE | 100000 | | | 12 | 192.168.1.8 | ONLINE | 100 | | | 12 | 192.168.1.9 | ONLINE | 100 | | | 9010 | 192.168.1.5 | ONLINE | 1000000000 | | | 9011 | 192.168.1.5 | ONLINE | 1 | | +--------------+-------------+--------+------------+---------+


46

WRITE Fail-over is automatically perform by ProxySQL

following the weight Until an active node exists the HG


47

WRITE Fail-over will not happen

Until READ_ONLY variable is modified on a reader HG node.

WRITER if DONOR will never be OFFLINE_SOFT

Integration with PXC SW

48

Integration with PXC\Galera SW manual

49

Integration with PXC\Galera SW Auto

50

Change state to: OFFLINE_SOFT = ~2 sec

HostGroup 9000 = ~2 sec

ProxySQL Shunned = ~1.2 sec (5 fails in 1sec)

Failover (auto) = ~3 sec

Integration with PXC - Timing

51

Avoid SPOF

52

Cost in time

53

Cost in time

54

So what.. 9 0. 0 0 0 % (36 days) MySQL Replication 9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD 9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive) 9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes 9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication 9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication 9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum 9 9. 9 9 9 % (5 minutes) MySQL Cluster

55

Brief summary

55

Improve service flexibility with r/w split;

Brief summary

55

Improve service flexibility with r/w split;Allow simple way to manage access to specific servers by

usage (like DW-ETL);

Brief summary

55


usage (like DW-ETL);Implement sharding;

Brief summary

55



Allow Query revrite/query fix/blocking without need to modify code;

Brief summary

55




Improve failover time;

Brief summary

55




Improve failover time;Reduce master downtime impact on reads;

Brief summary

55




Improve failover time;Reduce master downtime impact on reads;

Not SPOF (as vip)

Brief summary

58

Contacts

To contact Me

[email protected]

[email protected]

To follow me

http://www.tusacentral.net/

http://www.percona.com/blog/

https://www.facebook.com/marco.tusa.94

@marcotusa

http://it.linkedin.com/in/marcotusa/

“Consulting = No mission refused!”

http://www.tusacentral.net/

https://www.facebook.com/marco.tusa.94

Proxysql ha plam_2016_2_keynote

Presentations & Public Speaking

Transcript of Proxysql ha plam_2016_2_keynote