Les fonctionnalites mariadb
Transcript of Les fonctionnalites mariadb
© MariaDB. Company Confidential.
© MariaDB. Company Confidential.
MariaDB
© MariaDB. Company Confidential.
MariaDB Versions
© MariaDB. Company Confidential.
MariaDB 10.0
Scalability● Advanced parallel replication● Sharding● MaxScale proxy (future)
Performance● Enhanced optimization● Improved and special purpose storage engines● Carefully tuned and enhanced server internals● Advanced performance monitoring
Availability ● HA clustering - integrating Galera cluster● More online operations, less planned downtime
NoSQL● Interoperable storage engines such as Cassandra and Connect● Dynamic columns and JSON processing● HandlerSocket API
Operations ● Comprehensive diagnostics built-in to the DB● APIs and open architecture for easier integration
Security● Role-based access control● Authentication plugins● Sophisticated auditing capabilities
© 2014, MariaDB Corp.
Global Transaction ID (GTID)
● New MariaDB exclusive global event ID unique across multiple independent replication streams.
○ DomainID added to SeqNum-ServerID to uniquely label replication events.
○ Slaves save their replication status in a crash-safe table, transactionally synced to the slave’s binlog.
○ Replication streams always strictly ordered, but independent streams may be interleaved on the slave.
● Much simpler failover to new master with complex topologies.
● Supports multi-source and parallel replication.
A1
A2
A3
A4
A5
B1
B2
B3
B4
B5
A1
A2
B1
A3
B2
B3
A4
A 4
B 3
Crash-safeReplication State
XXDomainID
32-bit
YYYYSeqNum
64-bit
ZZServerID
32-bit
MariaDB GTID
© 2014, MariaDB Corp.
● Sponsored by Google.● Allows slaves to
process update eventsin parallel.○ In same consistente
order as the master ○ Out of order across
domain ID ( require GTID) consistente per domain
● Preliminary benchmarks: almost 10x faster at 12 threads.
© 2014, MariaDB Corp.
Multi-Source Replication
● Collects data foranalytics usingbuilt-in replication.
● Aids in administrationexample: consolidatedbackups of multiple databases.
● Uses MariaDB 10’s improvedGlobal Transaction ID (GTID).
Online E-Commerce Application
Master S S S S
Content Management
System
Click-stream data
Data WarehouseSlaveETL
Master S S S S
Master S S S S
© 2014, MariaDB Corp.
Multi-DC Replication
© 2014, MariaDB Corp.
● Clustered MariaDB nodescooperate to remain in sync.
● With multiple master nodes,reads and updates both scale.
● Synchronous replication withoptimistic locking delivers highavailability with little overhead.
● Fast failover because all nodes remains synchronized.
● Integrated and tested binaries.
MariaDB Galera Cluster
Load Balancingand Failover
Application / App Server
© 2014, MariaDB Corp.
Sharding with Spider
● Contributed by a 3rd party developer.● Storage engine partitions large tables
across multiple DBs.● No changes to the client application.● Developers aren’t aware, can keep
applications simple.● Allows for more
parallelism, scale-out.● Transactional storage
engine.
.
Application
Customers
A-HI-PQ-Z
Spider
Customers
A-H
Shard 1
Customers
I-P
Shard 2
Customers
Q-Z
Shard 3
© 2014, MariaDB Corp.
MariaDB 10 Performance:TokuDB Storage Engine
● 3rd party partner - TokuTek.● Drop-in replacement for
InnoDB/XtraDB.● Advanced indexing and
compression algorithms.● Transactional.● Up to 20x performance gain
for inserts/updates.● Up to 90% less disk storage.● Online schema changes.● Reduces or eliminates slave lag.● Best when DB doesn’t fit in memory,
and SSD longevity is important.
© 2014, MariaDB Corp.
Optimizer Improvements
● Enhancements include:○ Disk access optimizations.○ JOIN optimizations.○ Subquery optimizations.○ Optimized derived tables and views.○ Execution control.○ Optimizer control.○ EXPLAIN improvements.○ Independent storage engine statistics
Less I/O, CPU, memory requirements. Faster execution.
© 2014, MariaDB Corp.
CPU Cache
DRAM
Disks, SANs
Nanoseconds Microseconds Milliseconds
6 orders of magnitude
How Does Fusion-io Flash Storage Accelerate Databases?
© 2014, MariaDB Corp.
How Much Faster Is MariaDB 10 With Fusion-io?
About 24 times faster
© 2014, MariaDB Corp.
MariaDB 10 Interoperability:CONNECT Storage Engine
● From 3rd party developer.● Maps diverse data
to tables.● JOIN mapped data
to DB tables.● Flat files including CSV.● Tables in external DBs.● Generated tables
(PIVOT etc.)● Plug-in API for your own mappings.
Powerful tool for data integration, federation.
Application
Spider
MariaDB Parser/Optimizer/Connection Pool
CONNECTEngine
OtherEngines
DatabaseTables
.log
XMLCSV
© 2014, MariaDB Corp.
MariaDB 10 Interoperability:Cassandra Storage Engine
● Window into a Cassandra ring:read/write like a table in MariaDB.
● Use standard SQL queries.● JOIN Cassandra data
to MariaDB tables.● Use a MariaDB cluster
for high-availabilityaccess.
● Bring data fromCassandra into OLTPapplications.
Application
Spider
DatabaseTables
MariaDB Parser/Optimizer/Connection Pool
CassandraEngine
OtherEngines
© 2014, MariaDB Corp.
Dynamic (& Virtual) Columns
● Store unstructured data in MariaDB tables with a simple API.● Use MariaDB’s indexing and transactions to manipulate
“document” style data fast and consistently.● Nest sets of dynamic columns inside of other dynamic columns -
hierarchical structuring.● Include multiple rows with dynamic columns in transactions.● Virtual Columns allows to create function based columns
Cust ID Account Balance Dyn_Col_BLOBs
2035 $154.04 NAME: John Smith|LOC: 45.35243, -74.98348|IMAGE: x27A8B8C ...
2036 $929.10 NAME: Jane Doe|LOC: 45.35243, -74.98348|AGE: 32| GENDER: F...
2037 $377.53 NAME: Carol Jones|AGE: 43|GENDER: F||IMAGE: xA9674DE678 ...
© MariaDB. Company Confidential.
MariaDB [test]> create table t1 (id int auto_increment primary key, -> name varchar(40), -> type enum ("shirt", "phone", "computer"), -> price decimal(10,2), -> dynstr mediumblob);Query OK, 0 rows affected (0.11 sec)
MariaDB [test]> insert into t1 (name, type, price, dynstr) values -> ("Funny shirt", "shirt", 10.0, COLUMN_CREATE(1, "blue", 10, "XL")), -> ("nokia", "phone", 649, COLUMN_CREATE(1, "black", 2, "touchscreen")), -> ("htc Desire hd", "phone", 579, COLUMN_CREATE(1, "black", 3, "Android")), -> ("BM/Lenovo Thinkpad X60s", "computer", 419, COLUMN_CREATE(1, "black", 3, "Linux"));Query OK, 4 rows affected (0.04 sec)Records: 4 Duplicates: 0 Warnings: 0
MariaDB [test]> select id, name, type, price, length(dynstr) as len, column_list(dynstr) as list from t1;+----+-------------------------+----------+--------+------+------+| id | name | type | price | len | list |+----+-------------------------+----------+--------+------+------+| 1 | Funny shirt | shirt | 10.00 | 17 | 1,10 || 2 | nokia | phone | 649.00 | 27 | 1,2 || 3 | htc Desire hd | phone | 579.00 | 23 | 1,3 || 4 | BM/Lenovo Thinkpad X60s | computer | 419.00 | 21 | 1,3 |+----+-------------------------+----------+--------+------+------+4 rows in set (0.03 sec)
© MariaDB. Company Confidential.
MariaDB [test]> select name from t1 where COLUMN_GET(dynstr, 1 as char(10)) = "black";+-------------------------+| name |+-------------------------+| nokia || htc Desire hd || BM/Lenovo Thinkpad X60s |+-------------------------+3 rows in set (0.01 sec)
MariaDB [test]> MariaDB [test]> select name, COLUMN_GET(dynstr, 1 as char(10)) from t1 where COLUMN_EXISTS(dynstr, 1);+-------------------------+-----------------------------------+| name | COLUMN_GET(dynstr, 1 as char(10)) |+-------------------------+-----------------------------------+| Funny shirt | blue || nokia | black || htc Desire hd | black || BM/Lenovo Thinkpad X60s | black |+-------------------------+-----------------------------------+4 rows in set (0.00 sec)
© 2014, MariaDB Corp.
Security
© 2014, MariaDB Corp.
Operations
● Improved Table Discovery - less work for DBAs;integrates storage engines. ✔ exclusive to MariaDB 10
● SHOW PLUGINS SONAME - dynamic report oninstalled components. ✔ exclusive to MariaDB 10
● SHUTDOWN Command - apps can shutdown the databaseprogrammatically. ✔ exclusive to MariaDB 10
● Kill Query by Query ID - fix performance blocks .● Online ALTER TABLE - schema changes without downtime.● Per-thread Memory Statistics - easier tuning and problem
diagnosis. ✔ exclusive to MariaDB 10
● Improved Error Messages - faster problem solving.✔ exclusive to MariaDB 10
© 2014, MariaDB Corp.
● Audit Plugin 1.1 includes table event logging such as triggers and stored procedure calls, unique to its integration with MariaDB.
● Building on these capabilities, V2.0 of the Audit Plugin is planned to include:○ Optional field substitution of placeholders in query logs to improve
privacy and security.○ Integration with MariaDB 10 Role-based Access Control - filtering
audit logs by role.○ Recording of privilege changes as auditable events.○ Log rotation.
● Projected availability: November, 2014.
MariaDB Audit Plugin 2.0 Roadmap
© 2014, MariaDB Corp.
Introducing MariaDB MaxScale, Web Scale Database Proxy
MaxScale hides complexity, making clusters of systems look like a single
server to a client.
● Simplifies complex replication schemesfor massive scale, high availability.
● Manages performance with logging.● Safeguards data through firewall filtering.● Connects diverse clients and databases
with multiple protocols, query transformations.
Client Simple
Requests
MaxScale
© 2014, MariaDB Corp.
MariaDB MaxScale: Lightweight, Modular, Intelligent, Transparent
ProtocolAuthenticatio
n
Mon
itorFilter &
LoggingRouting
MaxScaleCore
Routing
● MaxScale core is a high-performance message switch.
● Module APIs allow for multiple modules ofeach type.
● Modules can be client-facing, back-end facing, internal, or connect to other DBs.
19.06.2014
MaxScae
Each application server uses 2 connections: 1 R/W, 1 R
MaxScale connects the R/W client connections to the master and the R
connections are load-balanced to all slaves
Each application server uses only 1 connection
MaxScale creates 2 connections, one for R/W on the master node and one for R/O
load balanced on the slave nodes
MaxScale
R/WSplitting
MaxScale monitors the state of each node and only applies
operations on available slaves
Each application server uses only 1 connection
Only fully synchronised nodes are included
MaxScale monitors the state of the nodes and load balances all
operations
MaxScale
Each application server uses only 1 connection
If the “master” node fails a new one can be elected immediately
MaxScale selects one node as ”master” and the other nodes as ”
slaves”
MaxScale
© 2014, MariaDB Corp.
MariaDB 10.0 Roadmap
MariaDB 10.0 10.0.X Releases through 2014
Performance● Thread pool improvements including Percona enhancements.● SHOW EXPLAIN improvements (PARTITIONS, EXTENDED attributes).
Scalability ● Supportability matrix and testing for combining MariaDB 10 and MySQL 5.6 replication.
Interoperability
● Port and optimize for IBM’s Power 8 architecture (in conjunction with IBM).
● Improvements to CONNECT engine.● Experimental Mroonga Asian full-text search engine.
© 2014, MariaDB Corp.
MariaDB 10.1 RoadmapMariaDB 10.1
10.1 Beta Tentative Q4 CY201410.1 GA Tentative Q1 CY2015
Performance
● Per-query variables.● userstat improvements.● NUMA enhancements.● Improved thread management● Remote and portable tablespaces● Query timeouts.● InnoDB:
○ >16K InnoDB buffers.○ InnoDB defragmentation.○ InnoDB persistent auto-increment.○ InnoDB forced primary key.○ SHOW ENGINE INNODB STATUS.
● Memory engine varchar, BLOB types, default_tmp_storage_engine.● Merge WebscaleSQL improvements, integrate stress test.● UNION ALL without temporary tables.● Optimizer enhancements:
○ Semi-joins + outer-joins.○ Optimizer trace.○ EXPLAIN JSON.○ EXPLAIN ANALYZE.
© 2014, MariaDB Corp.
MariaDB 10.1 Roadmap
MariaDB 10.110.1 Beta Tentative Q4 CY201410.1 GA Tentative Q1 CY2015
Scalability
● Performance tuning Parallel Replication and GTID, benchmarks.● Row-level parallel replication enhancements.● Slave execution of triggers.● Selected WebscaleSQL scalability enhancements.
Availability ● Full integration of Galera Cluster into MariaDB.
Interoperability ● Open Query Graph (OQGRAPH) engine improvements.● GIS improvements: full OGC compliance (map data).
Operations● New Performance Schema instruments.● Progress report on mysqldump.● CREATE OR REPLACE for most database objects (GSOC).
© 2014, MariaDB Corp.
MariaDB 10.1 Roadmap
MariaDB 10.110.1 Beta Tentative Q4 CY201410.1 GA Tentative Q1 CY2015
Security
● Kerberos authentication support through plug-in.● Password Validation Plugin.● Password improvements in the server:
○ Set password expiry dates.○ Lock passwords after failed attempts.
● Audit Plugin addition: Track password changes.● Role-based access control improvements and performance tuning.
Other Features
● Version control in GitHub.● Additional character set and collation options.● Multiple triggers per table.● MDL improvements.● IPv6/IPv4 datatype.
© 2014, MariaDB Corp.
Resources
● bugs: mariadb.org/jira● mailing lists:
○ [email protected]○ [email protected]
● fb.com/MariaDB.dbms● twitter: @mariadb● #maria on irc.freenode.net● https://mariadb.com/kb/en● downloads: https://downloads.mariadb.org
○ apt,yum repositoriies available● Default in RHEL7, SuSE 12, Fedora, Slackware, Archlinux etc etc
© 2014, MariaDB Corp.
Beyond MariaDB 10.1
● Transactional DDL.
● Indexes on expressions.
● Field/Column type plug-in.
○ JSON/BSON.○ YAML.○ IPv6/IPv4.○ UUID.○ Own datatype for Dynamic
Columns.
● More storage engines and interoperability with other DBs - potentially:
○ InfiniDB.○ MongoDB.○ Hadoop/HDFS.○ HBase/Honeycomb.○ RocksDB/LevelDB.
● Message queues.
● Global DDL lock.
● Windowing functions.
© 2014, MariaDB Corp.
DatabaseTables
MariaDB 10
Role: DBA
Permissions:● Update Schema● View Statistics● Create Database
DBA
Developer
Sysadmin
MariaDB 10: Security:Role-Based Access Control
MySQL MariaDB
✘ ✔in 10.0.
© MariaDB. Company Confidential.
© 2014, MariaDB Corp.
Thread Time
27 .003761
28 34.2529
29 .000713
Why is this so slow? Lets find out… SHOW EXPLAIN FOR 28;
ID select_type table possible_keys
rows
1 SIMPLE tbl NULL 1855174
Oh! Sequential scan of a 18M row table… no wonder! I’ll let the team know...
MariaDB 10.0 Operations:SHOW EXPLAIN FOR Command
To: DevOps Team
From: DBA Debby
Subj. Slow Queries
Hey guys, figured out those blocking queries. The new reservation app is looking for the shortest route but that column has no index - we need one! I’ll add it now. Debby
And it works in the slow query log too!Also for UPDATE and DELETE.
© 2014, MariaDB Corp.
MariaDB MaxScale
Application
MariaDB
Partner Ecosystem
MaxScaleProxy
Technology Preview
© 2014, MariaDB Corp.
MariaDB MaxScale: Implementing Advanced Database Services
● MaxScale services interface to clients and servers through protocols.
● Router modules implement policy to determine which servers are best able to handle requests.
● Filters and logging modules implement a pipeline that can block, split, or transform requests.
Simple clients, complex architectures.
Technology Preview
Routing
Filte
r/Lo
g
Client Protocol
Server Protocol
Message Core&
StateMachine .log
© 2014, MariaDB Corp.
Client and Back-End Protocols
● Client protocols let applicationsconnect to MaxScale with differentdatabase languages and connections.○ Initial support for MySQL connectors.○ Future support planned for JSON/BSON, others.
● Back-end protocols allow MaxScale to connect to different database and cluster technologies.○ Initial support for MySQL Replication, MariaDB
Replication, Galera Cluster, and MySQL Server with NDB storage engine.
Protocol
Technology Preview
© 2014, MariaDB Corp.
Routing and Load Balancing
● Routing modules implementfundamental proxy algorithms.○ Connection based routing: establishes route upon
connection, doesn’t examine requests.○ Statement based routing: examines requests to
dynamically route requests to the appropriate back-end.
● Modular design allows for load balancing, HA failover, sharding, and other routing policies.
Routing
Technology Preview
© 2014, MariaDB Corp.
Filtering, Logging, and Auditing
● Can log requests that meet filtercriteria such as time-out, accessspecific row, etc.
● Configurable firewall blocks or allows requests based on policy.
● External filtering and transformation pipeline processes requests and results as they pass through MaxScale.○ Allows for creation of parallel streams.○ Transforms can alter both requests and returned
data.
Filter/Log
Technology Preview
© 2014, MariaDB Corp.
Authentication and Monitoring
● Authentication modules letMaxScale proxy user credentials.○ In the future, they may deliver alternative
authentication such as Kerberos, OpenStack Keystone, or PAMs.
● Monitoring modules provide thedetailed back-end stateinformation MaxScale needs to make the right routing decisions.
Authentication
Monitor
Technology Preview
© 2014, MariaDB Corp.
Read Scalability: MySQL Replication + Connection Load Balancing
● Applications are “cluster aware”, split reads and writes on separate connections.
● Using connection load balancing, MaxScale routes the read/write connection to the master, and uses round-robin load balancing to route read-only connections to the slaves.
● MaxScale monitors the cluster, adjusting resource pools should a slave fail, or should MHA failover promote a slave to master.
MaxScale
Technology Preview
© 2014, MariaDB Corp.
RW Scalability, HA Failover: Galera Cluster + Connection Load Balancing
● Applications can be simple, use one connection for reads and updates.
● Using connection load balancing, MaxScale load balances connections to each node in the Galera Cluster, round-robin.
● MaxScale monitors the cluster, routing requests only to fully synchronized nodes.
MaxScale
Technology Preview
© 2014, MariaDB Corp.
RW Splitting, Read Scalability: MySQL Replication + Statement Load Balancing
● Applications can be simple, use one connection for reads and updates.
● Using statement load balancing, MaxScale dynamically routes writes to the master, and round-robin load balances read requests to the slaves.
● MaxScale monitors the cluster, routing requests only to fully operational nodes.
MaxScale
R/WSplitting
Technology Preview
© 2014, MariaDB Corp.
Query Logging For Performance Diagnostics
DBAs or DevOps engineers can capture performance problems:
1. MaxScale accepts a query from a client application,
2. Forwards the query to the back-end, logs it into the “All Queries” and “Long Running Queries” log files.
3. Receives the result from the back-end,
4. Forwards the result to the client, logs the result into the “All Queries” log, and if it is one of the N longest-running queries, logs the result into the “Long Running Queries” log file.
MaxScale is flexible. Rather than logging, it could “tee” queries to another service such as an analytics warehouse using the same technique.
Technology Preview
MaxScale
Filte
r &
Logg
ing
.log
Long-Running Queries
.log
All Queries
1 4
2
4
2
42 3
© 2014, MariaDB Corp.
Query Transformation For Legacy Application Compatibility
Modify queries from legacy applications on the fly - for example a MySQL 5.1 app:
1. MaxScale accepts a query from a MySQL 5.1 compatible client,
2. If the query matches the regular expression “/CREATE TABLE/” then MaxScale substitutes “ENGINE” for “TYPE” in that statement, else it passes the statement through the filter unchanged.
3. Forwards the transformed statement to MariaDB 10 or MySQL 5.6.
4. Receives the result from the back-end.
5. Forwards the result to the client.
Complex pipelines with multiple stages that split, filter, and transform queries are possible.
Technology Preview
MaxScale
Filte
r &
Logg
ing
1 5
3 4
/CREATE TABLE/
s/TYPE/ENGINE/
2
© 2014, MariaDB Corp.
MaxScale Flexibility In Action: Experimental Binlog Router
Handle wide replication schemes using MaxScale as a binlog relay:
1. MaxScale registers as a slave with the master database,
2. The master streams binlog records to MaxScale as it would any slave.
3. MaxScale stores the binlog records.4. The slave servers register with
MaxScale, which behaves exactly as the master database.
5. MaxScale streams stored binlog records to the slaves to catch them up to the master, or to maintain their consistency, offloading the master database.
Prototype delivered to customer demonstrates how MaxScale can simplify web-scale challenges.
How can MaxScale help you?
Technology Preview
Binlog Records
Binlog Router
MaxScale
4 5
1 2
3
© 2014, MariaDB Corp.
MariaDB MaxScale Today
Beta Today, GA Q4 2014:● Platforms supported - all 64-bit only:
○ CentOS 5,6,7, RHEL 5,6, Fedora 19,20○ Debian 6,7, Ubuntu 12.04 LTS, 13.10, 14.04○ OpenSUSE 13.1
● Open source (GPL v2) on GitHub.● CLI and text-based administration, tooling.● Supports Pacemaker and Heartbeat HA.● Protocols, Authentication: MySQL/MariaDB connectors, authentication.● Monitors: MySQL/MariaDB replication + MHA, Galera cluster.● Routers: Connection and statement based load balancing, R/W splitting:
○ Complex replication hierarchies.○ Server and service weighting.○ Slave fault tolerance, slave consistency (lag) as routing decision parameter.○ Server maintenance mode.
● Logging/Filters: “Tee” splitter, everything and time-based logging, regex-based filtering/firewall, regex query transformation.
Technology Preview
© 2014, MariaDB Corp.
MaxScale Roadmap
● MaxScale V1.1 and beyond:● Integration with MariaDB Enterprise.● Hinting added to Statement Load Balancing: execute only on a slave,
on a specific node, on the master, on a slave with < specified lag etc.● Binlog relay: improved HA for master/slave replication.● MySQL Cluster (NDB) connection load balancing.● HTTPD/NoSQL protocols, including JSON/BSON, Cassandra, HBase.● Hadoop cluster replication applier: feed analytics/Big Data backends.● Logging integrated with RabbitMQ.● Authentication modules for PAM, OpenStack (Keystone), AWS-IAM.● More vertical market and migration focused filters.● Module APIs, DevOps-focused management API.● Dynamic configuration: online config and module updates.
Confidential
Technology Preview
© 2014, MariaDB Corp.
MaxScale: AdditionalCapabilities Under Investigation
● Protocols: more NoSQL integrations.● Protocols: mapping from legacy database statements to
facilitate migrations.● Routers: Advanced sharding support.● Routers: Geographic replication for disaster resilience.● Routers: Multivariate (A/B) testing.● Filters: firewalls designed for HIPPA, PCI use cases.● Filters: data obfuscation tools.● Filters: intelligent feed of operational data to analytics.● Performance and availability enhancements: caching,
deployment co-location, clustering, and failover improvements.
● Integration with third-party management tools.Technology Preview
Confidential
21.06.2014
19.06.2014
MaxScale
Clients
Proxy
Databases
19.06.2014
ProtocolAuthentic
ation
Mon
itor
Filter & Logging
Routing
MaxScaleCore
19.06.2014
Protocol
19.06.2014
Routing
19.06.2014
Filter & Log
21.06.2014
Filter & Log
19.06.2014
Authentication
19.06.2014
Monitor
21.06.2014
21.06.2014
21.06.2014
21.06.2014
21.06.2014
21.06.2014
19.06.2014
● Check on GitHub https://github.com/skysql/MaxScale● MariaDB Source https://downloads.mariadb.org/● Bugs report http://bugs.skysql.com/buglist.cgi?product=Maxscale● Google groups https://groups.google.com/forum/#!forum/maxscale● Binary Tarball http://downloads.skysql.com/files/SkySQL/MaxScale● MariaDB website http://www.mariadb.com