How to Manage Scale-Out Environments with MariaDB MaxScale
-
Upload
mariadb-corporation -
Category
Software
-
view
101 -
download
5
Transcript of How to Manage Scale-Out Environments with MariaDB MaxScale
LAYOUT
Title OnlyPowerPoint Default
2
MariaDB MaxScale
Latest GA Version 2.1Current Beta Version 2.2
Part of MariaDB TX and AX
Offering
Next Generation Database Proxy for
SCALABILITY
SECURITY
HIGH AVAILABILITY
DATA STREAMING
LAYOUT
Title OnlyPowerPoint Default
MariaDB MaxScale Concept
DATABASE SERVERS
MASTER
SLAVES
Binlog Cache
Insulates client applications from the complexities
of backend database cluster
Simplify replication from database
to other databases
CLIENT
PROTOCOL SUPPORT
AUTHENTICATION
DATABASE MONITORING
LOAD BALANCING & ROUTING
QUERY TRANSFORMATION & LOGGING
Flexible, easy to write plug-ins for
Generic Core
MULTI-THREADED
E-POLL BASED
STATELESS
SHARES THE THREAD POOL
LAYOUT
Title and ContentPowerPoint Default
MaxScale Core and Plugins
Filter
Client
Protocol
Protocol
Filter Filter Router
Server State
Monitor
Parser
Core
updates
monitors
uses
Backend
LAYOUT
Title and ContentPowerPoint Default
Scalability
Transaction Scaling to support user growth
and simplify applications
Connection Rate Limitation
Query Cache
LOAD BALANCING TO
MARIADB MASTER/SLAVE
MARIADB GALERA CLUSTER
AURORA CLUSTER
Replication Scaling to support web-scale applications’ user base
BINLOG SERVER FOR HORIZONTAL SCALING OF SLAVES IN MASTER/SLAVE ARCHITECTURE
Multi-tenant database scaling to transparently grow tenants and data volume
SCHEMA SHARDING
HANDLE LARGE AMOUNT OF READ OPERATIONS WITH HIGH PERFORMANCE
LAYOUT
Two ContentPowerPoint Default
Routing based on
QUERY TYPES
QUERY PATTERNS
DATABASE SERVER STATE
REPLICATION LAGS
Load balancing across database clusters
MARIADB MASTER/SLAVE
MARIADB GALERA
MARIADB MULTI-MASTER
AWS AURORA
Scaling
Scale database environment without application impact
Minimize maintenance downtime
Connection Based Routing
readwrite
MASTER SLAVES
read write
Statement Based Routing (Read-Write Split)
readwrite
MASTER SLAVES
read & write
LAYOUT
Title and ContentPowerPoint Default
Scaling: Tunability
Route all reads to master subsequent to a write for a configurable amount of time or number of operations
Route queries to only those slave that are less than configurable replication lag behind master
Route all queries that match a pattern(regex) to specific servers
Weighted routing
LAYOUT
Two ContentPowerPoint Default
Scaling: Multi-tenant Database
Each tenant with its own schema
Multi-tenant database hosting
Without impacting existing user base
Scale the database with user base and data volume growth
Multi-tenant Database Routing(Schema Sharding)
shard n...shard 2shard 1
LAYOUT
Title and ContentPowerPoint Default
Scaling: Binlog Replication
Large scale Master/Slave Replication Clusters
Transparent MariaDB binlog replication relay
Horizontally Scale Slaves without Master Overload
SLAVES
MASTER
SLAVES
Binlog CacheBinlog Cache
LAYOUT
Title and ContentPowerPoint Default
Query Cache
in memory LRU cache.
Clients
Database Servers
Query Cache Filter
CACHE QUERY RESULTS IN MAXSCALE FOR CONFIGURABLE TIMEOUT
FOR CACHED QUERIES RETURN RESULTS FROM CACHE
Handle large amount of read operations with high performance
LAYOUT
Title and ContentPowerPoint Default
MaxScale 2.1 Performance Gain
NUMBER OF CONNECTIONS
up to
2.8 x Performance
GainQ
PSQ
PS
read write
read only
LAYOUT
Title and ContentPowerPoint Default
Security in MaxScale
Black & White List
Connection Rate Limitation
End to End SSL
Database Firewall Filter for SQL Injection protection
Selective Data Masking HIPPA/PCI Compliance
Maximum Rows Returned Limit DDoS Protection
Transport layer security between applications, proxy & databases
DDoS Protection
LDAP/GSSAPI Authentication Support
Encrypted Binlog Server Files
SSL between binlog server and Master/Slave
LAYOUT
Title and ContentPowerPoint Default
QUERY FAILED: 1141ERROR: Required WHERE/HAVING clause is missing
rule safe_select deny no_where_clause on_queries select
rule safe_cust_select deny regex '.*from.*customers.*'
user %app-user@% match all rules safe_cust_select safe_select
Security
DATABASE FIREWALL FILTER
SELECT * FROM CUSTOMERS;
MaxScale
Database Servers
1
2
3Database Firewall Filter
Allow/Block queries thatMATCH A SET OF RULESMATCH RULES FOR SPECIFIED USERSMATCH ON• date/time• a WHERE clause• query type• column match• a wildcard or regular expression or function name
Protect against SQL injection
Prevent unauthorized data access
Prevent data damage
LAYOUT
Title and ContentPowerPoint Default
Security
SELECT Name, creditcardNum, balance FROM customerTbl
WHERE id=1001
Name creditcardNum balance--------------------------------------- John Smith xxxxxxxxxx 1201.07
Database Servers
Client
HIPPA/PCI Compliance: Selective Data Masking based on column name
DATABASE NAME, TABLE NAME CLASSIFIER MAY BE PROVIDED• commerceDb.customerTbl.creditcardNum• customerTbl.creditcardNum• credicardNum
LAYOUT
Two ContentPowerPoint Default
Security
DDoS ProtectionMAXIMUM ROWS FILTER
• Return zero rows to client if number of rows in result set exceeds configured max limit
• Return zero rows to client if the size of result set exceeds configured max size in KB
Max Rows Limit = 500 NumRows Returned > MaxRows Limit
QUERY FAILED: 1141ERROR: No rows returned
51
QUERY
4 MaxRowsLimit FILTER
Clients
NumRows returned = 100032
Database Servers
QUERY
LAYOUT
Two ContentPowerPoint Default
Security
Slaves
Master
Slaves
SSL
SSL
SSL
SSL
Secured Binlog ServerENCRYPT BINLOG SERVER FILES on MaxScale
SSL between binlog server and Master/Slave
Secured user access LDAP/GSSAPI for secured single sign-on across OS platforms(windows, linux), applications and databases
Binlog Cache Binlog Cache
LAYOUT
Title and ContentPowerPoint Default
Operational Use Cases
Dynamically configure server, listener, monitor
Dynamically configure database firewall rules
Query Duplication
Launchable scripts
DUPLICATE QUERY BETWEEN PRODUCTION AND STAGING FOR UPGRADE VERIFICATION
DUPLICATE QUERY BETWEEN DATABASE AND EXTERNAL APPLICATIONS FOR QUERY ANALYTICS
Query logging CLUSTER WIDER QUERY TUNING
Query translation TRANSLATE QUERIES SYNTAX FROM ONE PROTOCOL TO ANOTHER
ASYNCHRONOUSLY UPGRADE APPLICATIONS AND DATABASE
INVOKE EXTERNAL OPERATIONAL TOOLS FOR SERVER DOWN/UP EVENT MANAGEMENT
MANAGE FAILOVER - PROMOTE SLAVE TO MASTER UPON FAILURE
LAYOUT
Title OnlyPowerPoint Default
Binlog conversion to Avro
Streaming of Change Data Capture events (binlog) to big data environments
Data Streaming
Change Data Capture Clients
EMR
MariaDB ColumnStore
Big Data Platforms
AVRO or JSON events
Streaming
MariaDBMaster
Binlog to AVRO conversion
LAYOUT
Title Only (Blue)
MariaDB MaxScale 2.2
MariaDB 10.2 Support
REST based management
PAMD Authentication
More performance enhancements
MaxScale CDC to MariaDB ColumnStore Data Adapter
LAYOUT
Title and Text(2 Column)
Demo
● Read Write Split
● Max Connections Limit - Connection Rate Limitation
● Data Masking
LAYOUT
Title Only (Dark)
Learn more
Download https://mariadb.com/downloads/mariadb-tx/maxscale
Documentationhttps://mariadb.com/kb/en/mariadb-enterprise/maxscale/
Blogshttps://mariadb.com/blog-tags/maxscale
Product Pagehttps://mariadb.com/products/technology/maxscale
Webinars https://mariadb.com/resources/webinars?combine=MaxScale
Reach [email protected]
LAYOUT
Thank You (Dark)
Thank you
LAYOUT
Section Header (Blue)
Example
Read Write Splitting
LAYOUT
Title and ContentPowerPoint Default
Read/Write SplittingMariaDB Replication + R/W Split Routing
Each application server uses only 1 connection
MaxScale monitors the state of each node and only applies operations on available
slaves
MaxScale creates 2 connections, one for R/W on the master node and one for R/O load balanced on the slave nodes
MaxScale
R/WSplitting
LAYOUT
Title Only (Dark)
Install MariaDB MaxScale
How to set up MariaDB MaxScaleDirections here for reference:https://mariadb.com/kb/en/mariadb-enterprise/setting-up-mariadb-maxscale/
Install the package relevant to your
distribution
Step 1
Create the required users in your MariaDB or
MySQL Replication cluster
Step 2
Create a MariaDB MaxScale
configuration file
Step 3
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Threads
• MariaDB MaxScale configuration is held in an ini file, /etc/maxscale.cnf.
• This must be manually created, a template file does exist within the /usr/share/maxscale directory.
• A global, maxscale, section is included within every MariaDB MaxScale configuration file.
• Various MariaDB MaxScale wide parameters are set in the file.
• The most important of these is the number of threads that MariaDB MaxScale will use to execute the code that forwards requests and handles responses for clients.
[maxscale]
threads=4
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Service
• The first step is to create a service for our Read/Write Splitter.
• Create a section in your MariaDB MaxScale configuration file and set the type to service.
[Splitter Service]
type=service
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration readwritesplit Module
• The router we need to use for this configurationis the readwritesplit module.
• The server names given here are actually the names of server sections in the configuration file and not the physical hostnames or addresses of the servers.
[Splitter Service]
type=service
router=readwritesplit
servers=dbserv1,
dbserv2, dbserv3
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Passwords
• The final step in the service sections is to add the username and password that will be used to populate the user data from the database cluster.
• There are two options for representing the password, either plain text or encrypted passwords.
• In order to use encrypted passwords a set of keys must be generated that will be used by the encryption and decryption process. To generate the keys, use the maxkeys command and pass the name of the secrets file in which the keys are stored.
maxkeys
/var/lib/maxscale/
.secrets
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – More Passwords
• Once the keys have been created the maxpasswd command can be used to generate the encrypted password.
maxpasswd
plainpassword
96F99AA1315BDC3604B0
06F427DD9484
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Splitter Service
• The username and password, either encrypted or plain text, are stored in the service section using the user and passwd parameters.
[Splitter Service]
type=service
router=readwritesplit
servers=dbserv1, dbserv2, dbserv3
user=maxscale
passwd=96F99AA1315BDC3604B006F427
DD9484
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Listener Service
• Listening ports must be associated with the service in order to allow network connections.
• This is done by creating a series of listener sections.
• A service may have multiple listeners. [Splitter Listener]
type=listener
service=Splitter Service
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – More Listener Service
• A listener must also define the protocol module it will use for the incoming network protocol, currently this should be the MySQLClient protocol for all database listeners.
• The listener may then supply a network port to listen on and/or a socket within the file system.
[Splitter Listener]
type=listener
service=Splitter Service
protocol=MySQLClient
port=3306
socket=/tmp/ClusterMaster
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Servers
• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.
• The next stage is the configuration is to define the server information.
[dbserv1]
type=server
address=192.168.2.1
port=3306
protocol=MySQLBackend
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Servers
• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.
• The next stage is the configuration is to define the server information.
[dbserv2]
type=server
address=192.168.2.2
port=3306
protocol=MySQLBackend
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Servers
• An address parameter may be given if the listener is required to bind to a particular network address when using hosts with multiple network addresses.
• The next stage is the configuration is to define the server information.
[dbserv3]
type=server
address=192.168.2.3
port=3306
protocol=MySQLBackend
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Replication Monitor
• In order for MariaDB MaxScale to monitor the servers using the correct monitoring mechanisms a section should be provided that defines the monitor to use and the servers to monitor.
• Once again a section is created with a symbolic name for the monitor, with the type set to monitor.
• Parameters are added for the module to use, the list of servers to monitor and the username and password to use when connecting to the servers with the monitor.
[Replication Monitor]
type=monitor
module=mysqlmon
servers=dbserv1, dbserv2, dbserv3
user=maxscale
passwd=96F99AA1315BDC3604B006F427
DD9484
LAYOUT
Title and ContentPowerPoint Default
MaxScale Configuration – Services and Listener Section
• As with the password definition in the server either plain text or encrypted passwords may be used.
• The final stage in the configuration is to add the option service which is used by the maxadmin command to connect to MariaDB MaxScale for monitoring and administration purposes. This creates a service section and a listener section.
[CLI]
type=service
router=cli
[CLI Listener]
type=listener
service=CLI
protocol=maxscaled
socket=default
LAYOUT
Title and ContentPowerPoint Default
Starting MariaDB MaxScale
• Upon completion of the configuration process MariaDB MaxScale is ready to be started for the first time. This may either be done manually by running the maxscale command or via the service interface.
% maxscale
or
% service maxscale start
LAYOUT
Title OnlyPowerPoint Default
Services------------------+----------------+--------+---------------Service Name | Router Module | #Users | Total Sessions------------------+----------------+--------+---------------Splitter Service | readwritesplit | 1 | 1CLI | cli | 2 | 2------------------+----------------+--------+---------------
List Services
Check the error log in /var/log/maxscale to see if any errors are detected in the configuration file and to confirm MariaDB MaxScale has been started. Also the maxadmin command may be used to confirm that MariaDB MaxScale is running and the services, listeners etc., have been correctly configured.
% maxadmin list services
LAYOUT
Title OnlyPowerPoint Default
Servers
List Servers
% maxadmin list servers
----------+--------------+-------+-------------+---------------Server | Address | Port | Connections | Status ----------+--------------+-------+-------------+---------------dbserv1 | 192.168.2.1 | 3306 | 0 | Running, Slavedbserv2 | 192.168.2.2 | 3306 | 0 | Running, Masterdbserv3 | 192.168.2.3 | 3306 | 0 | Running, Slave----------+--------------+-------+-------------+---------------
LAYOUT
Title OnlyPowerPoint Default
Listeners
List Listeners
MariaDB MaxScale is now ready to start accepting client connections and routing them to the master or slaves within your cluster. • Other configuration options are available that
can alter the criteria used for routing, these include monitoring the replication lag within the cluster and routing only to slaves that are within a predetermined delay from the current master or using weights to obtain unequal balancing operations.
• These options may be found in the MariaDB MaxScale Configuration Guide. More detail on the use of maxadmin can be found in the document.
% maxadmin list listeners
-----------------+-----------------+-----------+-------+--------Service Name | Protocol Module | Address | Port | State-----------------+-----------------+-----------+-------+--------Splitter Service | MySQLClient | * | 3306 | RunningCLI | maxscaled | localhost | 6603 | Running-----------------+-----------------+-----------+-------+--------
LAYOUT
Title OnlyPowerPoint Default
Testing the Read - Write Splitting
A simple select statement routes to one of the slaves
A grouped transaction could contain writes, so is considered as one. It should route to the master
MariaDB> select @@hostname;+------------+| @@hostname |+------------+| dbserve1 |+------------+MariaDB> start transaction;MariaDB> select @@hostname;+------------+| @@hostname |+------------+| dbserve2 |+------------+