How to Manage Scale-Out Environments with MariaDB MaxScale

43
MariaDB MaxScale Dipti Joshi Director Product Management [email protected]

Transcript of How to Manage Scale-Out Environments with MariaDB MaxScale

Page 1: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title Slide (Dark)

MariaDB MaxScale

Dipti Joshi Director Product [email protected]

Page 2: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

2

MariaDB MaxScale

Latest GA Version 2.1Current Beta Version 2.2

Part of MariaDB TX and AX

Offering

Next Generation Database Proxy for

SCALABILITY

SECURITY

HIGH AVAILABILITY

DATA STREAMING

Page 3: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

MariaDB MaxScale Concept

DATABASE SERVERS

MASTER

SLAVES

Binlog Cache

Insulates client applications from the complexities

of backend database cluster

Simplify replication from database

to other databases

CLIENT

PROTOCOL SUPPORT

AUTHENTICATION

DATABASE MONITORING

LOAD BALANCING & ROUTING

QUERY TRANSFORMATION & LOGGING

Flexible, easy to write plug-ins for

Generic Core

MULTI-THREADED

E-POLL BASED

STATELESS

SHARES THE THREAD POOL

Page 4: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Core and Plugins

Filter

Client

Protocol

Protocol

Filter Filter Router

Server State

Monitor

Parser

Core

updates

monitors

uses

Backend

Page 5: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Scalability

Transaction Scaling to support user growth

and simplify applications

Connection Rate Limitation

Query Cache

LOAD BALANCING TO

MARIADB MASTER/SLAVE

MARIADB GALERA CLUSTER

AURORA CLUSTER

Replication Scaling to support web-scale applications’ user base

BINLOG SERVER FOR HORIZONTAL SCALING OF SLAVES IN MASTER/SLAVE ARCHITECTURE

Multi-tenant database scaling to transparently grow tenants and data volume

SCHEMA SHARDING

HANDLE LARGE AMOUNT OF READ OPERATIONS WITH HIGH PERFORMANCE

Page 6: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Two ContentPowerPoint Default

Routing based on

QUERY TYPES

QUERY PATTERNS

DATABASE SERVER STATE

REPLICATION LAGS

Load balancing across database clusters

MARIADB MASTER/SLAVE

MARIADB GALERA

MARIADB MULTI-MASTER

AWS AURORA

Scaling

Scale database environment without application impact

Minimize maintenance downtime

Connection Based Routing

readwrite

MASTER SLAVES

read write

Statement Based Routing (Read-Write Split)

readwrite

MASTER SLAVES

read & write

Page 7: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Scaling: Tunability

Route all reads to master subsequent to a write for a configurable amount of time or number of operations

Route queries to only those slave that are less than configurable replication lag behind master

Route all queries that match a pattern(regex) to specific servers

Weighted routing

Page 8: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Two ContentPowerPoint Default

Scaling: Multi-tenant Database

Each tenant with its own schema

Multi-tenant database hosting

Without impacting existing user base

Scale the database with user base and data volume growth

Multi-tenant Database Routing(Schema Sharding)

shard n...shard 2shard 1

Page 9: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Scaling: Binlog Replication

Large scale Master/Slave Replication Clusters

Transparent MariaDB binlog replication relay

Horizontally Scale Slaves without Master Overload

SLAVES

MASTER

SLAVES

Binlog CacheBinlog Cache

Page 10: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Query Cache

in memory LRU cache.

Clients

Database Servers

Query Cache Filter

CACHE QUERY RESULTS IN MAXSCALE FOR CONFIGURABLE TIMEOUT

FOR CACHED QUERIES RETURN RESULTS FROM CACHE

Handle large amount of read operations with high performance

Page 11: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale 2.1 Performance Gain

NUMBER OF CONNECTIONS

up to

2.8 x Performance

GainQ

PSQ

PS

read write

read only

Page 12: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Security in MaxScale

Black & White List

Connection Rate Limitation

End to End SSL

Database Firewall Filter for SQL Injection protection

Selective Data Masking HIPPA/PCI Compliance

Maximum Rows Returned Limit DDoS Protection

Transport layer security between applications, proxy & databases

DDoS Protection

LDAP/GSSAPI Authentication Support

Encrypted Binlog Server Files

SSL between binlog server and Master/Slave

Page 13: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

QUERY FAILED: 1141ERROR: Required WHERE/HAVING clause is missing

rule safe_select deny no_where_clause on_queries select

rule safe_cust_select deny regex '.*from.*customers.*'

user %app-user@% match all rules safe_cust_select safe_select

Security

DATABASE FIREWALL FILTER

SELECT * FROM CUSTOMERS;

MaxScale

Database Servers

1

2

3Database Firewall Filter

Allow/Block queries thatMATCH A SET OF RULESMATCH RULES FOR SPECIFIED USERSMATCH ON• date/time• a WHERE clause• query type• column match• a wildcard or regular expression or function name

Protect against SQL injection

Prevent unauthorized data access

Prevent data damage

Page 14: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Security

SELECT Name, creditcardNum, balance FROM customerTbl

WHERE id=1001

Name creditcardNum balance--------------------------------------- John Smith xxxxxxxxxx 1201.07

Database Servers

Client

HIPPA/PCI Compliance: Selective Data Masking based on column name

DATABASE NAME, TABLE NAME CLASSIFIER MAY BE PROVIDED• commerceDb.customerTbl.creditcardNum• customerTbl.creditcardNum• credicardNum

Page 15: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Two ContentPowerPoint Default

Security

DDoS ProtectionMAXIMUM ROWS FILTER

• Return zero rows to client if number of rows in result set exceeds configured max limit

• Return zero rows to client if the size of result set exceeds configured max size in KB

Max Rows Limit = 500 NumRows Returned > MaxRows Limit

QUERY FAILED: 1141ERROR: No rows returned

51

QUERY

4 MaxRowsLimit FILTER

Clients

NumRows returned = 100032

Database Servers

QUERY

Page 16: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Two ContentPowerPoint Default

Security

Slaves

Master

Slaves

SSL

SSL

SSL

SSL

Secured Binlog ServerENCRYPT BINLOG SERVER FILES on MaxScale

SSL between binlog server and Master/Slave

Secured user access LDAP/GSSAPI for secured single sign-on across OS platforms(windows, linux), applications and databases

Binlog Cache Binlog Cache

Page 17: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Operational Use Cases

Dynamically configure server, listener, monitor

Dynamically configure database firewall rules

Query Duplication

Launchable scripts

DUPLICATE QUERY BETWEEN PRODUCTION AND STAGING FOR UPGRADE VERIFICATION

DUPLICATE QUERY BETWEEN DATABASE AND EXTERNAL APPLICATIONS FOR QUERY ANALYTICS

Query logging CLUSTER WIDER QUERY TUNING

Query translation TRANSLATE QUERIES SYNTAX FROM ONE PROTOCOL TO ANOTHER

ASYNCHRONOUSLY UPGRADE APPLICATIONS AND DATABASE

INVOKE EXTERNAL OPERATIONAL TOOLS FOR SERVER DOWN/UP EVENT MANAGEMENT

MANAGE FAILOVER - PROMOTE SLAVE TO MASTER UPON FAILURE

Page 18: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

Binlog conversion to Avro

Streaming of Change Data Capture events (binlog) to big data environments

Data Streaming

Change Data Capture Clients

EMR

MariaDB ColumnStore

Big Data Platforms

AVRO or JSON events

Streaming

MariaDBMaster

Binlog to AVRO conversion

Page 19: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title Only (Blue)

MariaDB MaxScale 2.2

MariaDB 10.2 Support

REST based management

PAMD Authentication

More performance enhancements

MaxScale CDC to MariaDB ColumnStore Data Adapter

Page 20: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and Text(2 Column)

Demo

● Read Write Split

● Max Connections Limit - Connection Rate Limitation

● Data Masking

Page 21: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title Only (Dark)

Learn more

Download https://mariadb.com/downloads/mariadb-tx/maxscale

Documentationhttps://mariadb.com/kb/en/mariadb-enterprise/maxscale/

Blogshttps://mariadb.com/blog-tags/maxscale

Product Pagehttps://mariadb.com/products/technology/maxscale

Webinars https://mariadb.com/resources/webinars?combine=MaxScale

Reach [email protected]

Page 22: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Thank You (Dark)

Thank you

Page 23: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Section Header (Blue)

Example

Read Write Splitting

Page 24: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Read/Write SplittingMariaDB Replication + R/W Split Routing

Each application server uses only 1 connection

MaxScale monitors the state of each node and only applies operations on available

slaves

MaxScale creates 2 connections, one for R/W on the master node and one for R/O load balanced on the slave nodes

MaxScale

R/WSplitting

Page 25: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title Only (Dark)

Install MariaDB MaxScale

How to set up MariaDB MaxScaleDirections here for reference:https://mariadb.com/kb/en/mariadb-enterprise/setting-up-mariadb-maxscale/

Install the package relevant to your

distribution

Step 1

Create the required users in your MariaDB or

MySQL Replication cluster

Step 2

Create a MariaDB MaxScale

configuration file

Step 3

Page 26: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Threads

• MariaDB MaxScale configuration is held in an ini file, /etc/maxscale.cnf.

• This must be manually created, a template file does exist within the /usr/share/maxscale directory.

• A global, maxscale, section is included within every MariaDB MaxScale configuration file.

• Various MariaDB MaxScale wide parameters are set in the file.

• The most important of these is the number of threads that MariaDB MaxScale will use to execute the code that forwards requests and handles responses for clients.

[maxscale]

threads=4

Page 27: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Service

• The first step is to create a service for our Read/Write Splitter.

• Create a section in your MariaDB MaxScale configuration file and set the type to service.

[Splitter Service]

type=service

Page 28: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration readwritesplit Module

• The router we need to use for this configurationis the readwritesplit module.

• The server names given here are actually the names of server sections in the configuration file and not the physical hostnames or addresses of the servers.

[Splitter Service]

type=service

router=readwritesplit

servers=dbserv1,

dbserv2, dbserv3

Page 29: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Passwords

• The final step in the service sections is to add the username and password that will be used to populate the user data from the database cluster.

• There are two options for representing the password, either plain text or encrypted passwords.

• In order to use encrypted passwords a set of keys must be generated that will be used by the encryption and decryption process. To generate the keys, use the maxkeys command and pass the name of the secrets file in which the keys are stored.

maxkeys

/var/lib/maxscale/

.secrets

Page 30: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – More Passwords

• Once the keys have been created the maxpasswd command can be used to generate the encrypted password.

maxpasswd

plainpassword

96F99AA1315BDC3604B0

06F427DD9484

Page 31: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Splitter Service

• The username and password, either encrypted or plain text, are stored in the service section using the user and passwd parameters.

[Splitter Service]

type=service

router=readwritesplit

servers=dbserv1, dbserv2, dbserv3

user=maxscale

passwd=96F99AA1315BDC3604B006F427

DD9484

Page 32: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Listener Service

• Listening ports must be associated with the service in order to allow network connections.

• This is done by creating a series of listener sections.

• A service may have multiple listeners. [Splitter Listener]

type=listener

service=Splitter Service

Page 33: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – More Listener Service

• A listener must also define the protocol module it will use for the incoming network protocol, currently this should be the MySQLClient protocol for all database listeners.

• The listener may then supply a network port to listen on and/or a socket within the file system.

[Splitter Listener]

type=listener

service=Splitter Service

protocol=MySQLClient

port=3306

socket=/tmp/ClusterMaster

Page 34: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Servers

• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.

• The next stage is the configuration is to define the server information.

[dbserv1]

type=server

address=192.168.2.1

port=3306

protocol=MySQLBackend

Page 35: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Servers

• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.

• The next stage is the configuration is to define the server information.

[dbserv2]

type=server

address=192.168.2.2

port=3306

protocol=MySQLBackend

Page 36: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Servers

• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.

• The next stage is the configuration is to define the server information.

[dbserv3]

type=server

address=192.168.2.3

port=3306

protocol=MySQLBackend

Page 37: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Replication Monitor

• In order for MariaDB MaxScale to monitor the servers using the correct monitoring mechanisms a section should be provided that defines the monitor to use and the servers to monitor.

• Once again a section is created with a symbolic name for the monitor, with the type set to monitor.

• Parameters are added for the module to use, the list of servers to monitor and the username and password to use when connecting to the servers with the monitor.

[Replication Monitor]

type=monitor

module=mysqlmon

servers=dbserv1, dbserv2, dbserv3

user=maxscale

passwd=96F99AA1315BDC3604B006F427

DD9484

Page 38: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

MaxScale Configuration – Services and Listener Section

• As with the password definition in the server either plain text or encrypted passwords may be used.

• The final stage in the configuration is to add the option service which is used by the maxadmin command to connect to MariaDB MaxScale for monitoring and administration purposes. This creates a service section and a listener section.

[CLI]

type=service

router=cli

[CLI Listener]

type=listener

service=CLI

protocol=maxscaled

socket=default

Page 39: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title and ContentPowerPoint Default

Starting MariaDB MaxScale

• Upon completion of the configuration process MariaDB MaxScale is ready to be started for the first time. This may either be done manually by running the maxscale command or via the service interface.

% maxscale

or

% service maxscale start

Page 40: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

Services------------------+----------------+--------+---------------Service Name | Router Module | #Users | Total Sessions------------------+----------------+--------+---------------Splitter Service | readwritesplit | 1 | 1CLI | cli | 2 | 2------------------+----------------+--------+---------------

List Services

Check the error log in /var/log/maxscale to see if any errors are detected in the configuration file and to confirm MariaDB MaxScale has been started. Also the maxadmin command may be used to confirm that MariaDB MaxScale is running and the services, listeners etc., have been correctly configured.

% maxadmin list services

Page 41: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

Servers

List Servers

% maxadmin list servers

----------+--------------+-------+-------------+---------------Server | Address | Port | Connections | Status ----------+--------------+-------+-------------+---------------dbserv1 | 192.168.2.1 | 3306 | 0 | Running, Slavedbserv2 | 192.168.2.2 | 3306 | 0 | Running, Masterdbserv3 | 192.168.2.3 | 3306 | 0 | Running, Slave----------+--------------+-------+-------------+---------------

Page 42: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

Listeners

List Listeners

MariaDB MaxScale is now ready to start accepting client connections and routing them to the master or slaves within your cluster. • Other configuration options are available that

can alter the criteria used for routing, these include monitoring the replication lag within the cluster and routing only to slaves that are within a predetermined delay from the current master or using weights to obtain unequal balancing operations.

• These options may be found in the MariaDB MaxScale Configuration Guide. More detail on the use of maxadmin can be found in the document.

% maxadmin list listeners

-----------------+-----------------+-----------+-------+--------Service Name | Protocol Module | Address | Port | State-----------------+-----------------+-----------+-------+--------Splitter Service | MySQLClient | * | 3306 | RunningCLI | maxscaled | localhost | 6603 | Running-----------------+-----------------+-----------+-------+--------

Page 43: How to Manage Scale-Out Environments with MariaDB MaxScale

LAYOUT

Title OnlyPowerPoint Default

Testing the Read - Write Splitting

A simple select statement routes to one of the slaves

A grouped transaction could contain writes, so is considered as one. It should route to the master

MariaDB> select @@hostname;+------------+| @@hostname |+------------+| dbserve1 |+------------+MariaDB> start transaction;MariaDB> select @@hostname;+------------+| @@hostname |+------------+| dbserve2 |+------------+