ProxySQL hand-on - Percona...databases-000webhost-lt 3 7 Hostgroups Manager Management of servers...

ProxySQL hand-on René Cannaò

ProxySQL

Frankfurt, 5th Nov 2018

René Cannaò

ProxySQL

Frankfurt, 5th Nov 2018

2

A bit about ProxySQL LLCWe provide services to help build, support

as well as improve the performance &

reliability of your Cloud-Based or On-

Premise MySQL infrastructure:

• ProxySQL Development

• Remote Consulting

• ProxySQL Support Services

• ProxySQL Training

We provide services to help build, support

as well as improve the performance &

reliability of your Cloud-Based or On-

Premise MySQL infrastructure:

• ProxySQL Development

• Remote Consulting

• ProxySQL Support Services

• ProxySQL Training

3

A bit about me…

Rene Cannao

• Founder of ProxySQL

• MySQL DBA

Rene Cannao

• Founder of ProxySQL

• MySQL DBA

4

Agenda

● What is ProxySQL?● Features● Fundamentals● What is new?● Internals● Admin● Clustering● Rules● Mirroring● Failover● New protocols● GTID casual reads

● What is ProxySQL?● Features● Fundamentals● What is new?● Internals● Admin● Clustering● Rules● Mirroring● Failover● New protocols● GTID casual reads

5

Other sessions

HA and clustering solution: ProxySQL as an intelligent router for Galera and Group Replication

Wednesday 7th November , 2018 @11:20AM

ProxySQL Adaptive query routing based on GTID tracking

Wednesday 7th November, 2018 @4:30PM

HA and clustering solution: ProxySQL as an intelligent router for Galera and Group Replication

Wednesday 7th November , 2018 @11:20AM

ProxySQL Adaptive query routing based on GTID tracking

Wednesday 7th November, 2018 @4:30PM

Overview of ProxySQL

7

What is ProxySQL

The MySQL data stargateThe MySQL data stargate

8

ProxySQL Architecture Overview

Data gatewayClients connect to ProxySQLRequests are evaluatedActions are performed

9

Main motivations

empower the DBAsimprove operationunderstand and improve performancecreate a proxy layer to shield the databaseHigh performance and High Availability

10

Some of the most interesting features:● on-the-fly rewrite of queries

● caching reads outside the database server

● connection pooling and multiplexing

● complex query routing and read/write split

● load balancing

● real time statistics

● monitoring

● data masking

● multiple instances on same ports

11

Some of the most interesting features:● high availability and scalability

● seamless failover

● firewall

● query throttling

● query timeout

● query mirroring

● runtime reconfiguration

● scheduler

● support for Galera/PXC and Group Replication

12

Some of the most interesting features:● support for millions of users

● support for tens of thousands of database servers

● native ProxySQL Clustering solution

● support for ClickHouse as a backend

● support for Aurora

● SSL support for frontend

● SSLv1.2

● native Support for Galera

● causal reads using GTID

14

Hostgroups and Query Routing

All backends are grouped into hostgroups

Hostgroups have logical functionalities

All backends are grouped into hostgroups

Hostgroups have logical functionalities

15

Hostgroups example #1

HostGroup0 (HG0): Write masters

HostGroup1( HG1): Read slaves

Read/Write split

16

Hostgroups example #2

HG0: main write mastersHG1: main read slavesHG2: reporting slavesHG3: ad-hoc queries slavesHG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site serversHG7: test servers

HG8 : mirror for traffic on HG0HG9 : mirror for traffic on HG1

HG0: main write mastersHG1: main read slavesHG2: reporting slavesHG3: ad-hoc queries slavesHG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site serversHG7: test servers

HG8 : mirror for traffic on HG0HG9 : mirror for traffic on HG1

17

Basic designHG0

HG2

HG1

ProxySQL

APP1

APP2

APP3

18

Basic designHG0

HG2

HG1

APPAPP

ProxySQL

Very low latency usingUnix Domain Socket (?)

19

One ProxySQL per host

20

One ProxySQL per host

The most common setupIt scales to hundreds or thousands of nodeTry this solution before anything else!

21

Silos approach

22

Clustered ProxySQL Architecture

DB DBDB

APP

DB

PROXYSQL

DB DB

APP

PROXYSQL

APP

PROXYSQL

APP

PROXYSQL

PROXYSQL PROXYSQL

DBMANAGER

PROXYSQL

Clustered Proxy Layer

23

Clustered ProxySQL at scaleTested with:

● 8 app servers with 3k clients’ connections each (24k total)● 4 middle layer proxysqls processing 4k connections each from local

proxysqls (16k total)● 256 backends/shard (meaning 256 routing rules) processing 600

connections each (150k total)

Single ProxySQL was tested with up to 950k connections

ProxySQL 1.4 is able to process up to 750k QPS

Tested with:

● 8 app servers with 3k clients’ connections each (24k total)● 4 middle layer proxysqls processing 4k connections each from local

proxysqls (16k total)● 256 backends/shard (meaning 256 routing rules) processing 600

connections each (150k total)

Single ProxySQL was tested with up to 950k connections

ProxySQL 1.4 is able to process up to 750k QPS

24

ProxySQL at scaleReal setups:

Over 1000 proxysql instances in a single network

Over 100 mysql servers configured on a single proxysql instances

Over 2 millions unique users

Routing with over 100 thousands shards

Real setups:

Over 1000 proxysql instances in a single network

Over 100 mysql servers configured on a single proxysql instances

Over 2 millions unique users

Routing with over 100 thousands shards

ProxySQL Internals

26

ProxySQL Modules in 1.3.x

HG0

HG1

APP1ProxySQL

Queries cache

Query Processor

APP2

APP3

Thr

ead

#1

Thr

ead

#2

Thr

ead

#3

Users Auth

ConnectionPool

HostgroupManager

Admin

Monitoring

27

ProxySQL Modules in 1.4.x

HG0

HG1

APP1ProxySQL

Queries cache

Query Processor

APP2

APP3

Thr

ead

#1

Thr

ead

#2

Thr

ead

#3

Users AuthConnection

Pool

HostgroupManager

Admin

MonitoringClickHouse Server

ClickHouse Auth

Cluster

SQLite3 Server

Th

read

#3

idle

s

Thr

ead

#2 id

les

Thr

ead

#1

idle

s Purge Thread

28

Thread pool in MySQL

Threads in ProxySQL are known as "MySQL Threads"Fixed number of worker threads (configurable)All threads listen on the same port(s)Client connections are not shared between threadsAll threads perform their own network I/OUses "poll()"... (does it scale?)

29

Threads never share client connections

Pros:Thread contention is reducedNo need for synchronizationEach thread calls "poll()"

Cons:Possibly imbalanced load

Thread 1

Thread 2

Thread 3

30

poll() vs. epoll()

"poll()" is O(N)"epoll()" is O(1)"epoll()" scales better than "poll()"

Why does ProxySQL use "poll()"?It is faster than "epoll()" for fewer connections (~1000)Performance degrades when there are a lot of connections

31

ProxySQL Auxiliary Threads

Each worker thread has an auxiliary threadWorker thread uses "poll()"Auxiliary thread uses "epoll()"Worker thread passes idle connections to auxiliary threadWhen a connections becomes active auxiliary thread passes connection to the worker thread

Solution scales to almost 1 million connections

Thread 1

Aux 1

Thread 2

Aux 2

Thread 3

Aux 3

32

MySQL Thread Overview

MySQL Session

MySQL Session

MySQL Session

MySQL Session

MySQL Thread

MySQL Session

MySQL Session

All other Modules:• Query Processor• Query Cache• Hostgroups Manager• Authentication• Others

For low contention, threads independently:Track internal metricsStore values for mysql-XXX variablesStore a copy of the defined query rules

33

Queries Processor

Based on Queries Rules

Defines what to cache

Defines the hostgroup target

Timeout/delay

Firewall

Mirroring

Rewrite queries

Based on Queries Rules

Defines what to cache

Defines the hostgroup target

Timeout/delay

Firewall

Mirroring

Rewrite queries

34

Queries rules

Complex rules to match incoming traffic:● regex on query● regex on digest text● username● schemaname● Source IP address● Bind IP address/port● digest

Rules can be chained

Complex rules to match incoming traffic:● regex on query● regex on digest text● username● schemaname● Source IP address● Bind IP address/port● digest

Rules can be chained

35

Queries Cache and Rewrite

Caching on the wireInternal key/value storageIn memory onlyPattern basedExpired by timeout

Rewrite on the wireRegex match/replace on query on digest textOptionally cached or mirrored

Caching on the wireInternal key/value storageIn memory onlyPattern basedExpired by timeout

Rewrite on the wireRegex match/replace on query on digest textOptionally cached or mirrored

36

Users Authentication

Credentials stored in the proxy

User login always possible (even without backends)

Max connections

Login credentials are encrypted

Extremely scalable, up to millions of users:

https://www.percona.com/live/e17/sessions/scaling-million-databases-000webhost-lt

Credentials stored in the proxy

User login always possible (even without backends)

Max connections

Login credentials are encrypted

Extremely scalable, up to millions of users:

https://www.percona.com/live/e17/sessions/scaling-million-databases-000webhost-lt

37

Hostgroups Manager

Management of servers

Track servers status

Manages the connections pool

Management of servers

Track servers status

Manages the connections pool

38

Connections Pool

Reduced the overhead of creating new connections, and are recycled when not in use

One to many connections

Multiplexing & maximum connections

Auto-reconnect and automatic re-execution of queries

Failover management

Reduced the overhead of creating new connections, and are recycled when not in use

One to many connections

Multiplexing & maximum connections

Auto-reconnect and automatic re-execution of queries

Failover management

39

Connections Pool

40

Connections Pool

41

Connection Pool

Replies to ping.From hundreds of millions of ping per day, to 0:

https://www.percona.com/blog/2017/09/01/life360-used-proxysql-lower-database-load/

Replies to ping.From hundreds of millions of ping per day, to 0:

https://www.percona.com/blog/2017/09/01/life360-used-proxysql-lower-database-load/

42

Purge Thread

Unsafe connections are reset instead of being dropped

Drastically reduce the need to establish new connections

In ProxySQL 2.0 : each worker thread performs most of the reset

Unsafe connections are reset instead of being dropped

Drastically reduce the need to establish new connections

In ProxySQL 2.0 : each worker thread performs most of the reset

43

Auto-reconnect and re-execution

Automatic detection of failures

Graceful handling

Auto-reconnect when possible

Pause until a backend becomes available

Re-execution of queries

Automatic detection of failures

Graceful handling

Auto-reconnect when possible

Pause until a backend becomes available

Re-execution of queries

44

Multiplexing

Reduce the number of connections against mysqld (configurable)

Many clients connections (tens of thousands) can use few backend connections (few hundreds)

Tracks connection status (transactions, user variables, temporary tables, etc)

Order by waiting time

Reduce the number of connections against mysqld (configurable)

Many clients connections (tens of thousands) can use few backend connections (few hundreds)

Tracks connection status (transactions, user variables, temporary tables, etc)

Order by waiting time

45

No ProxySQL:

ProxySQL:

46

No ProxySQL: ProxySQL:

47

ProxySQL:No ProxySQL:

49

Multiplexing

https://www.percona.com/blog/2017/09/01/life360-used-proxysql-lower-database-load/https://www.percona.com/blog/2017/09/01/life360-used-proxysql-lower-database-load/

50

MySQL Threads and Connection Pool

MyHGM is a shared resource so it can cause contention when accessed by MySQL Threads

MySQL_ThreadMySQL_Thread

MyHGM

51

Thread Connection Cache

MySQL_ThreadMySQL_Thread

MyHGMConnectionsCache

ConnectionsCache

ProxySQL Web UI

53

Multiplexing (see it in web UI)

54

Multiplexing (see it in web)

55

More metrics on web UI

56


57


58

Monitoring Module

It monitors backends and collects metrics

Monitors replication lag and shun hosts

Monitors read_only (and super_read_only and innodb_read_only in 2.0) variables (replication hostgroups)

Ping and terminates unresponsive nodes

It monitors backends and collects metrics

Monitors replication lag and shun hosts

Monitors read_only (and super_read_only and innodb_read_only in 2.0) variables (replication hostgroups)

Ping and terminates unresponsive nodes

59

Monitoring Module

Built in support for asynchronous replication: second behind master and pt-heartbeat

Support for any type of clustering solution via external scripts(Galera/PXC , and any heterogeneous replication setup)

In ProxySQL 1.4, support for Group Replication

In ProxySQL 2.0, support for Galera

Built in support for asynchronous replication: second behind master and pt-heartbeat

Support for any type of clustering solution via external scripts(Galera/PXC , and any heterogeneous replication setup)

In ProxySQL 1.4, support for Group Replication

In ProxySQL 2.0, support for Galera

Admin Module

61

Admin Interface

Allows runtime configuration

Exports internal statuses

It uses MySQL protocol

Configuration possible from any client/tool using MySQL API

Covered during the tutorial

Allows runtime configuration

Exports internal statuses

It uses MySQL protocol

Configuration possible from any client/tool using MySQL API

Covered during the tutorial

62

Admin Interface

DiskMemoryRuntime

DiskMemoryRuntime

Runtime

Memory

DiskConfig

63

Try it!

Source code on GitHub:https://github.com/sysown/proxysql/

Forum:https://groups.google.com/forum/#!forum/proxysql

Tutorials on:http://www.proxysql.com

Source code on GitHub:https://github.com/sysown/proxysql/

Forum:https://groups.google.com/forum/#!forum/proxysql

Tutorials on:http://www.proxysql.com

https://github.com/sysown/proxysql/

https://groups.google.com/forum/#!forum/proxysql

http://www.proxysql.com/



http://www.proxysql.com/

Demo Environment

65

Demo

Code on GitHub:https://github.com/renecannao/ProxySQL-Tutorial-PLSC2018

Some hosts provided: Username: plscPasswords: proxysql

Code on GitHub:https://github.com/renecannao/ProxySQL-Tutorial-PLSC2018

Some hosts provided: Username: plscPasswords: proxysql

ProxySQL Cluster

67

ProxySQL Cluster

Introduced in 1.4.2

Support for MySQL Query Rules, MySQL Users, MySQL Servers and ProxySQL Servers (no Global Variables, no Scheduler)

A group of instances constantly check each other for changes in configuration

A configuration change applied in a proxy is pulled by the other proxies

Allow to keep multiple ProxySQL instances with the same configuration(s)

Introduced in 1.4.2

Support for MySQL Query Rules, MySQL Users, MySQL Servers and ProxySQL Servers (no Global Variables, no Scheduler)

A group of instances constantly check each other for changes in configuration

A configuration change applied in a proxy is pulled by the other proxies

Allow to keep multiple ProxySQL instances with the same configuration(s)

68

ProxySQL Cluster

● Core Nodes● Satellite Nodes

Largest setup we know:

1) 3 core nodes

2) 900 satellite nodes

69

ProxySQL Cluster

Conflict resolution is timestamp based• No order• Last win

No quorum (in the roadmap)

Highly scalable: Core nodes vs satellite nodes

Conflict resolution is timestamp based• No order• Last win

No quorum (in the roadmap)

Highly scalable: Core nodes vs satellite nodes

70

ProxySQL Cluster - Configmysql> SET admin-cluster_username='radmin';mysql> SET admin-cluster_password='radmin';mysql> LOAD ADMIN VARIABLES TO RUNTIME;mysql> SAVE ADMIN VARIABLES TO DISK;

mysql> INSERT INTO proxysql_servers (hostname) VALUES ('proxysql1'),('proxysql2'),('proxysql3');mysql> LOAD PROXYSQL SERVERS TO RUNTIME;mysql> SAVE PROXYSQL SERVERS TO DISK;

mysql> SET admin-cluster_username='radmin';mysql> SET admin-cluster_password='radmin';mysql> LOAD ADMIN VARIABLES TO RUNTIME;mysql> SAVE ADMIN VARIABLES TO DISK;

mysql> INSERT INTO proxysql_servers (hostname) VALUES ('proxysql1'),('proxysql2'),('proxysql3');mysql> LOAD PROXYSQL SERVERS TO RUNTIME;mysql> SAVE PROXYSQL SERVERS TO DISK;

mysql> SHOW VARIABLES LIKE 'admin-cluster%';+---------------------------------------------------+--------+| Variable_name | Value |+---------------------------------------------------+--------+| admin-cluster_username | radmin || admin-cluster_password | radmin || admin-cluster_check_interval_ms | 1000 || admin-cluster_check_status_frequency | 10 || admin-cluster_mysql_query_rules_diffs_before_sync | 3 || admin-cluster_mysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_users_diffs_before_sync | 3 || admin-cluster_proxysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_query_rules_save_to_disk | true || admin-cluster_mysql_servers_save_to_disk | true || admin-cluster_mysql_users_save_to_disk | true || admin-cluster_proxysql_servers_save_to_disk | true |+---------------------------------------------------+--------+12 rows in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'admin-cluster%';+---------------------------------------------------+--------+| Variable_name | Value |+---------------------------------------------------+--------+| admin-cluster_username | radmin || admin-cluster_password | radmin || admin-cluster_check_interval_ms | 1000 || admin-cluster_check_status_frequency | 10 || admin-cluster_mysql_query_rules_diffs_before_sync | 3 || admin-cluster_mysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_users_diffs_before_sync | 3 || admin-cluster_proxysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_query_rules_save_to_disk | true || admin-cluster_mysql_servers_save_to_disk | true || admin-cluster_mysql_users_save_to_disk | true || admin-cluster_proxysql_servers_save_to_disk | true |+---------------------------------------------------+--------+12 rows in set (0.00 sec)

71

ProxySQL Cluster - Monitoring

stats_proxysql_servers_checksums

stats_proxysql_servers_metrics

stats_proxysql_servers_checksums

stats_proxysql_servers_metrics

ProxySQL Cluster DEMO

MySQL Query rules

74

Query rewriting overview

SQL-aware

Regex-based

flexible

SQL-aware

Regex-based

flexible

75

Query rewriting examples

read/write splitting

Data masking

Sharding

read/write splitting

Data masking

Sharding

76

Query rewriting flow

77

Match Related Tables

mysql_users• active• fast_forward(bypass)

mysql_users• active• fast_forward(bypass)

mysql_query_rules• username• schemaname• flagIN/OUT• client_addr• proxy_addr• proxy_port• digest• match_digest• match_pattern• negate_match_pattern

mysql_query_rules• username• schemaname• flagIN/OUT• client_addr• proxy_addr• proxy_port• digest• match_digest• match_pattern• negate_match_pattern

78

Action related fields

flagOUTreplace_patterndestination_hostgroupcache_ttltimeoutretriesdelaymirror_flagOUTmirror_hostgroupOK_msgerror_msglogmultiplexapply

flagOUTreplace_patterndestination_hostgroupcache_ttltimeoutretriesdelaymirror_flagOUTmirror_hostgroupOK_msgerror_msglogmultiplexapply

active

New in 2.0 :gtid_from_hostgroupcache_empty_resultcache_timeout

active

New in 2.0 :gtid_from_hostgroupcache_empty_resultcache_timeout

79

3 layers for ProxySQL tables

DiskMemoryRuntime

DiskMemoryRuntime

Runtime

Memory

DiskConfig

80

Precautions

Confirm the query hits with stats_mysql_query_rulesdouble check rules and rewrite resultshold off on writing rules to disk until you're sure they are working as expectedhave a rollback plan

• load from disk, • then load from memory to runtime

test select query rules and rewrites on a slaves firsttest mutable rules on a mirror or other throw away db.

Confirm the query hits with stats_mysql_query_rulesdouble check rules and rewrite resultshold off on writing rules to disk until you're sure they are working as expectedhave a rollback plan

• load from disk, • then load from memory to runtime

test select query rules and rewrites on a slaves firsttest mutable rules on a mirror or other throw away db.

81

mysql> SELECT username, default_hostgroup FROM mysql_users\G*************************** 1. row *************************** username: rootdefault_hostgroup: 01 row in set (0.00 sec)

mysql> SELECT match_digest, destination_hostgroup FROM mysql_query_rules\G*************************** 1. row *************************** match_digest: ^SELECT.*FOR UPDATEdestination_hostgroup: 0*************************** 2. row *************************** match_digest: ^SELECTdestination_hostgroup: 12 rows in set (0.00 sec)

mysql> SELECT username, default_hostgroup FROM mysql_users\G*************************** 1. row *************************** username: rootdefault_hostgroup: 01 row in set (0.00 sec)

mysql> SELECT match_digest, destination_hostgroup FROM mysql_query_rules\G*************************** 1. row *************************** match_digest: ^SELECT.*FOR UPDATEdestination_hostgroup: 0*************************** 2. row *************************** match_digest: ^SELECTdestination_hostgroup: 12 rows in set (0.00 sec)

Example - Simple R/W

This example is a simple one, NOT for production use

For production configuration guide, see:https://proxysql.com/blog/configure-read-write-split

Query Rewriting with ProxySQLDEMO

Click to add textClick to add text

ProxySQL Upgradewith zero downtime

84

Multiple instances running on the same port

Requires Linux Kernel 3.9

Use different datadir (optionally different config file)

Close the listener for a process

Drain connections from one node

Wait all the connections to pass over

Requires Linux Kernel 3.9

Use different datadir (optionally different config file)

Close the listener for a process

Drain connections from one node

Wait all the connections to pass over

Mirroring with ProxySQL


86

What is mirroring?

87

What is mirroring?

88

Why mirror queries?

Validate performance on a different server using different hostgroups.

Validate performance of query rewrite or schema change

Pre-fetch slave replication (Replication Booster)

Validate performance on a different server using different hostgroups.

Validate performance of query rewrite or schema change

Pre-fetch slave replication (Replication Booster)

89

How to mirror

mysql>SHOW CREATE TABLE mysql_query_rules\G*********************** 1. row ************************ table: mysql_query_rulesCreate Table: CREATE TABLE mysql_query_rules (

rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0,username VARCHAR,schemaname VARCHAR,flagIN INT NOT NULL DEFAULT 0,

*snip*mirror_flagOUT INT UNSIGNED,mirror_hostgroup INT UNSIGNED,error_msg VARCHAR,

*snip*)

mysql>SHOW CREATE TABLE mysql_query_rules\G*********************** 1. row ************************ table: mysql_query_rulesCreate Table: CREATE TABLE mysql_query_rules (

rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0,username VARCHAR,schemaname VARCHAR,flagIN INT NOT NULL DEFAULT 0,

*snip*mirror_flagOUT INT UNSIGNED,mirror_hostgroup INT UNSIGNED,error_msg VARCHAR,

*snip*)

90

Mirroring flow

91

Mirroring - example #1

mysql> SELECT username, destination_hostgroup, mirror_hostgroup, mirror_flagOUT FROM mysql_query_rules WHERE username='plam_mirror'\G****************** 1. row ********************** username: pl_mirrordestination_hostgroup: 3 mirror_hostgroup: 4 mirror_flagOUT: NULL

mysql> SELECT username, destination_hostgroup, mirror_hostgroup, mirror_flagOUT FROM mysql_query_rules WHERE username='plam_mirror'\G****************** 1. row ********************** username: pl_mirrordestination_hostgroup: 3 mirror_hostgroup: 4 mirror_flagOUT: NULL

92

Mirroring - example #2

Mirroring Demo


94

Mirroring caveats

Can be dropped: mysql-mirror_max_concurrency,

mysql-mirror_max_queue_length

Can be executed out of order

No error handling of failed queries

Can be dropped: mysql-mirror_max_concurrency,

mysql-mirror_max_queue_length

Can be executed out of order

No error handling of failed queries

Failover with ProxySQL

96

Failover highlights

improve failover time as perceived by the application

prevent errors sent to the application

perform transparent database failovers: gracefully redirecting traffic without the

application knowing

existing applications do not have to be rewritten to autoreconnect since

connections are not lost from failovers

improve failover time as perceived by the application

prevent errors sent to the application

perform transparent database failovers: gracefully redirecting traffic without the

application knowing

existing applications do not have to be rewritten to autoreconnect since

connections are not lost from failovers

97

Failover scenario - without ProxySQL

98

Failover scenario - without ProxySQL

99

Failover scenario - with ProxySQL

100


101


102


103

Failover without ProxySQL (10 minute outage)

https://www.nylas.com/blog/growing-up-with-mysql/

104

Failover with ProxySQL (10 second outage)

https://www.nylas.com/blog/growing-up-with-mysql/

105

Failover - MySQL Servers

mysql> SELECT hostgroup_id, hostname, status FROM mysql_servers WHERE hostname IN ('mysql1, 'mysql2’)\G*************************** 1. row *************************** hostgroup_id: 0

hostname: mysql1 status: ONLINE*************************** 2. row *************************** hostgroup_id: 1

hostname: mysql2 status: ONLINE2 rows in set (0.00 sec)

mysql> SELECT hostgroup_id, hostname, status FROM mysql_servers WHERE hostname IN ('mysql1, 'mysql2’)\G*************************** 1. row *************************** hostgroup_id: 0

hostname: mysql1 status: ONLINE*************************** 2. row *************************** hostgroup_id: 1

hostname: mysql2 status: ONLINE2 rows in set (0.00 sec)

106

Failover - Replication Hostgroups

mysql> SELECT * FROM mysql_replication_hostgroups\G*************************** 1. row ***************************writer_hostgroup: 0reader_hostgroup: 1 comment:1 row in set (0.00 sec)

mysql> SELECT * FROM mysql_replication_hostgroups\G*************************** 1. row ***************************writer_hostgroup: 0reader_hostgroup: 1 comment:1 row in set (0.00 sec)

107

ProxySQL Failover ProTIP

ProxySQL does not handle promotion or re-slaving

External process needed, such as MHA or Orchestrator

ProxySQL does not handle promotion or re-slaving

External process needed, such as MHA or Orchestrator

Failover Demo


Some New Features


110

ClickHouse Server

New in 1.4.3 – experimental (not fully completed, subject to change)

Support for ClickHouse: https://clickhouse.yandex/

Fast, really fast data analytics using MySQL client

Users management

Supports only one backendSupports only DDL and SELECT (no INSERT)


Support for ClickHouse: https://clickhouse.yandex/

Fast, really fast data analytics using MySQL client

Users management

Supports only one backendSupports only DDL and SELECT (no INSERT)

https://clickhouse.yandex/

https://clickhouse.yandex/

111

ClickHouse Server

112

ClickHouse Server

113

SQLite3 Server


Support for SQLite3 : https://www.sqlite.org/

Fully transactional, in-memory

Same users as MySQL backends

A lot of features in the roadmap


Support for SQLite3 : https://www.sqlite.org/

Fully transactional, in-memory

Same users as MySQL backends

A lot of features in the roadmap

https://www.sqlite.org/

https://www.sqlite.org/

114

You-Name-It Server

What other service should become part of ProxySQL ?What other service should become part of ProxySQL ?

115

Causal read with GTID

Feature introduced in 2.0

Ensures that reads sent to slave returns what the client has written on master

Requires GTID and MySQL 5.7+

Feature introduced in 2.0

Ensures that reads sent to slave returns what the client has written on master

Requires GTID and MySQL 5.7+

116

116

ProxSQL Read / Write Split

117

117

ProxySQL Binlog Reader

118

118

ProxySQL Binlog Reader

119

Hundreds of thousands of shardsFeature introduced in 1.4.7Simpler version of mysql_query_rulesUsed as a hash table

mysql> SELECT username,schemaname,flagIN, destination_hostgroup dest FROM mysql_query_rules_fast_routing;

+----------+------------+--------+------+

| username | schemaname | flagIN | dest |

+----------+------------+--------+------+

| userA | shard1 | 0 | 1 |

| userA | shard2 | 0 | 1 |

| userA | shard3 | 0 | 1 |

| userA | shard4 | 0 | 1 |

| userA | shard5 | 0 | 2 |

| userA | shard6 | 0 | 2 |

| userA | shard7 | 0 | 2 |

+----------+------------+--------+------+

Feature introduced in 1.4.7Simpler version of mysql_query_rulesUsed as a hash table

mysql> SELECT username,schemaname,flagIN, destination_hostgroup dest FROM mysql_query_rules_fast_routing;

+----------+------------+--------+------+

| username | schemaname | flagIN | dest |

+----------+------------+--------+------+

| userA | shard1 | 0 | 1 |

| userA | shard2 | 0 | 1 |

| userA | shard3 | 0 | 1 |

| userA | shard4 | 0 | 1 |

| userA | shard5 | 0 | 2 |

| userA | shard6 | 0 | 2 |

| userA | shard7 | 0 | 2 |

+----------+------------+--------+------+

120

Rate My Session

121

Thank you!

• Please remember to report feature requests and bug reports:


• Community support can be found on our forum:


• Useful blog articles are available at our site: http://proxysql.com/blog

• Visit us at http://proxysql.com/support for subscription and support options

• Please remember to report feature requests and bug reports:


• Community support can be found on our forum:


• Useful blog articles are available at our site: http://proxysql.com/blog

• Visit us at http://proxysql.com/support for subscription and support options

ProxySQL hand-on - Percona...databases-000webhost-lt 3 7 Hostgroups Manager Management of servers...

Documents

Transcript of ProxySQL hand-on - Percona...databases-000webhost-lt 3 7 Hostgroups Manager Management of servers...