MariaDB 10.2 & MariaDB 10.1 by Michael Monty Widenius at Database Camp 2016 @ UN

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

MariaDB 10.1 & MariaDB 10.2

New features in MariaDB 10.1 and 10.2

Database camp, 2016, New YorkMichael “Monty” WideniusEntrepreneur, MariaDB [email protected]://mariadb.org/ http://mariadb.com/

http://mariadb.org/


The origin of Maria (DB)

Introducing Maria


Why MariaDB was created

“Save the People, Save the Product”

● To keep the MySQL talent together● To ensure that a free version of MySQL always exists● To get one community developed and maintained branch● Work with other MySQL forks/branches to share

knowhow and code

After Oracle announced it wanting to buy Sun & MySQL this got to be even more important.


MariaDB Foundation Overview

The Foundation is the owner of the main MariaDB server project

Custodian of the code, Guardian of the community

Foundation can never to be controlled by a single entity or person

Note that the Foundation is not about the MariaDB trademark or to decide upon the MariaDB roadmap!

The Foundation was created to ensure that anyone can become a contributor to the MariaDB project on equal terms!


MariaDB Foundation Goals

That MariaDB be actively developed in the community and to:● Ensure that community patches are reviewed and adopted● Increase adoption of MariaDB● Ensure sustainable high-quality efforts to build, test and

distribute MariaDB● Guarantee a community voice and define development rules● Keep MariaDB compatible with MySQL● Maintain mariadb.org.

The MariaDB foundation now employes:● Management team: Michael Widenius as CTO

● Otto Kekäläinen as CEO● 4 full time MariaDB developers (including the CTO)

● 1 documentation writer


MariaDB Foundation members

Many of the MariaDB 10.0+ features has been developed together with the MariaDB foundation!The foundation are very grateful to it's 2013-2016 members, who have made MariaDB possible:● Booking.com (3 years)● MariaDB Corporation (4 years)● Visma (2 years)● Development bank of Singapore (DBS)● Parallels/Odin (4 years)● Automattic● Zenimax● Additional sponsors can be found on mariadb.org


Where to get MariaDB

● In Distributions:● Red Hat Enterprise Linux, Fedora, Debian, Ubuntu, Mageia, Suse Enterprise, openSuse, Gentoo, Slackware, Arch, ALTLinux, TurboLinux, Chacra Project, Kdu, Unbreakable Linux ….

● FreeBSD, OpenBSD● Mac OS X with MacPorts or Homebrew

● From MariaDB.org● Sources, binaries in tar.gz or .zip (Windows)● Windows MSI installer● MariaDB apt and yum repositories


MariaDB server is a branch of MySQL

● User level (data, API, replication, configuration files..) compatible with MySQL

● Drop in replacement for MySQL● Easy upgrades (both from MySQL and earlier MariaDB

versions); We don't remove features!● More plugins, more storage engines, more features,

faster, better code quality.● GPL-only server license.● LGPL C, ODBC and Java connectors.● More open development

● Source in public repository on launchpad● Active external contributors● All development plans public on mariadb.com


MariaDB server releases

● MariaDB 5.1 was released in February 2010● MariaDB 5.2 was released in November 2010● MariaDB 5.3 was released in April 2012● MariaDB 5.5 was released in April 2013● MariaDB 10.0 was released in March 2014● MariaDB 10.1 was released in October 2015● MariaDB 10.2.1 Alpha released July 2016.

● Ga planned for December● Galera (multi-master, mesh network) is included in

MariaDB 10.1● Separate binaries available for 5.5 and 10.0


The MariaDB releases

● MariaDB 5.1 (based on MySQL 5.1)● Better build & test system, code cleanups, community

patches, new storage engines, table elimination.● MariaDB 5.2 (based on MariaDB 5.1)

● Community features that did not go into 5.1:● Virtual columns● Extended User Statistics● Segmented MyISAM key cache (faster multi user!)

● MariaDB 5.3 (based on MariaDB 5.2)● Optimizer features (faster subquerier, joins etc)● Microsecond, dynamic columns, faster HANDLER etc.● Better replication (group commit, more options)

● MariaDB 5.5 (based on MariaDB 5.3 and MySQL 5.5)


The MariaDB releases

● MariaDB 10.0● Most of MySQL 5.6 features● Multi-source and Parallel replication● Working GTID (Global transaction ID)● SQL standard ROLES

● MariaDB 10.1● Galera integrated● Even better parallel replication● Strong encryption● Speed improvements in a lot of areas

● MariaDB 10.2● Window functions● Common table expression: WITH statement● Full DEFAULT and CHECK CONSTRAINT handling


MariaDB Columnstore

● Massively parallel distributed storage engine● Optimized for OLAP with up to petabytes of data● Originally based on MySQL 5.1 + InfiniDB, now

based on MariaDB 10.0● SQL interface, which maks it much easier to use

than Hadoop or any NO SQL database● Superior performance for any aggregate or

statistical queries compared to NOSQL solutions● Compressed data (up to 80% smaller) makes

storage easier● MariaDB 1.0.1 Alpha columnstore released 14

June 2016 under GPL


Spider storage engine

● Distributed storage engine, since MariaDB 5.3● Uses storage nodes based on InnoDB● Uses the MariaDB partition engine to distribute

data.● MariaDB corporation has used Spider to replace

100 node Cassandra clusters with a 7 node Spider setup

● Lots of development is still happening on Spider in MariaDB 10.2


Feedback plugin

● All recent MariaDB versions has the feedback plugin● Enable by adding “plugin-load=feedback.so” and "enable-

feedback” to the [mysqld] section in my.cnf.● Feedback plugin will automatically send a report (basicly

SHOW STATUS) at startup and once a week● This information is used to decide what features should

be developed/expanded upon● For more information see

https://mariadb.com/kb/en/feedback-plugin/ ● For statistics see

http://mariadb.org/feedback_plugin/

http://mariadb.org/feedback_plugin/


Feedback pluginSome statistics

Major versions of MariaDB in use with feedback enabled


Parallel slave benchmark in 10.010x speedup with 12 threads

From: http://kristiannielsen.livejournal.com/18435.htmlCrash safe: --log-slave-updates --sync-binlog=1 --innodb-flush-log-at-trx-commit=1

http://kristiannielsen.livejournal.com/18435.html


Optimizer enhancements

● ORDER BY … LIMIT optimizer● UNION ALL optimization (no temporary tables

created)● Analyze <statement>

● Like EXPLAIN but exact information● ANALYZE and EXPLAIN supports

FORMAT=JSON● Engine condition pushdown always on

● Required by many storage engines● Affected rows in the slow log● Allowing storage engine to shortcut group by

queries (Used by scaledb and sequence)


Performance and Scalability

● Removing locks — using lock-free data structures● In meta-data locking● In Table definition cache● In XA (XID hash)

● Numerous scalability improvements● 60% speedup in sysbench on Power8

● No .frm files for temporary tables● Speeds up creation of temporary memory tables

● Removing calls to current_thd() and malloc()● Disable performance schema by default (gives 1%

→ 20% speedup)


Sysbench OLTP RO on Power8

http://svoj-db.blogspot.ru/2014/12/mariadb-on-power8-2014-wrap-up.html


Replication

● Galera (multi-master) is now part of 10.1● WSREP_MEMBERSHIP and WSREP_STATUS I_S.

● Optimistic parallel replication● Makes slave in some cases faster than the master

● Row based replication (RBR) triggers on slave● Slave can always handle RBR!● Enhanced semisync replication

● Wait for at least one slave before committing● Binary log dump thread enhancements (Google)

● Makes multiple slave setup's faster● domain_id based replication filters● RESET MASTER TO #


GIS

Better OpenGIS standard compliance:● INFORMATION_SCHEMA (IS) tables

● GEOMETRY_COLUMN SPATIAL_REF_SYS● Support for Spatial Reference systems

● REF_SYSTEM_ID per GEOMETRY column● Missing OGC standard spatial functions added:

● ST_Boundary● ST_ConvexHull● ST_IsRing● ST_PointOnSurface● ST_Relate


Consistent support ofIF [NOT] EXIST and OR REPLACE

In 10.0 only TABLES was supported. In 10.1 we also support:● CREATE DATABASE (MDEV-7280)● CREATE FUNCTION UDF (MDEV-7283)● CREATE ROLE (MDEV-7288)● CREATE SERVER (MDEV-7285)● CREATE USER (MDEV-7288)● CREATE VIEW (MDEV-7283)● DROP ROLE (MDEV-7288)● DROP USER (MDEV-7288)● CREATE EVENT and DROP EVENT (MDEV-7281)● CREATE INDEX and DROP INDEX (MDEV-7284)● CREATE TRIGGER and DROP TRIGGER (MDEV-7286)


Information schema

● INFORMATION_SCHEMA.SYSTEM_VARIABLES● Shows help, limits and where a variable was set

● IS plugins can support SHOW and FLUSH statements

● SHOW QUERY_RESPONSE_TIME● FLUSH QUERY_RESPONSE_TIME● SHOW LOCALES

● New columns in● INFORMATION_SCHEMA.APPLICABLE_ROLES● INFORMATION_SCHEMA.VIEWS


Miscellaneous

● SET STATEMENT● Set variables for duration of query

● Compound statements● IF (…) THEN CREATE TABLE … ; END IF

● Sequence engine enabled by default● Microseconds in GET_LOCK()● Assisted discovery in OQGraph● SQL standards-compliant behavior when dealing

with Primary Keys with Nullable Columns● CREATE TABLE (a int primary key) and CREATE TABLE (a int, primary key(a)) are now identical.


Admin/Maintainability

● SET DEFAULT ROLE

● Statement timeouts● SET STATEMENT max_statement_time= 100 SELECT ….

● PERFORMANCE_SCHEMA without .frm files● Enabling slow log per session● enforce_storage_engine and

default_tmp_storage_engine variables● Better help for SET/ENUM command-line options● --getopt-prefix-matching

● For testing configuration files or MySQL 5.7 compatiblity

● Many new status variables


Security

● Encryption, contributed by Eperi and Google● Encryption of InnoDB/XtraDB tables, table spaces and logs, Aria temporary tables and the binary log files

● Encryption keys in file or trough key manager● Scrubbing (Get back lost space)● Password validation● Security-enhanced binaries


Compilation and Packaging

● Link with dynamic system libjemalloc.so● GUI-friendly cmake options to enable/disable

plugins● Special compilation options for security-enhanced

binaries● Support for pkg-config, a mariadb.pc file added


XtraDB/InnoDB

● Encryption● Tablespaces and binary logs

● Per-table or all tables● Background key rotation● Key identifiers specified per table

● Scrubbing (Ensure that data is really deleted)● Tablespaces and logs

● Defragmentation trough OPTIMIZE TABLE● --innodb-defragment=1

● Support for 4K and 64K page size● Multi-threaded flush


XtraDB/InnoDB

● FusionIO related● CREATE TABLE ... ATOMIC_WRITES=...● Real time compression (about 2x space saving):

● CREATE TABLE ... PAGE_COMPRESSION=...

● – zlib, lz4, lzo, lzma, snappy● TRIM

● Used to compress tables


XtraDB/InnoDB

● I_S.INNODB_SEMAPHORE_WAITS, I_S.INNODB_MUTEXES

● Monitor progress of online ALTER TABLE● Enforce that tables has a PRIMARY KEY● Prefix index queries optimization● Lazy flushing


Plugin API

● Two new plugin types● Password verification

● simple_password_check● cracklib_password_check

● Encryption● file_key_management (For table encryption)

● example_key_management● password_key_management


Services for Plugins

● Five new services● md5● sha1● encryption and key management● thd_specifics● wsrep (write set replication)


Connect storage engine

MariaDB 10.0 includes the Connect storage engine by Olivier Bertrand.With the connect storage engine you can read, write and update files in a lot of different storage formats:● Various fixed and dynamic text formats● .DBF (dBASE format)● .CSV● .INI● .XML● ODBC ; Table extracted from an application accessible with

ODBC; You can for example connect MariaDB to an Oracle database with this.

● JSON + BSON


MariaDB 10.2Existing features

● Window functionsSELECT name, test, score, AVG(score) OVER (PARTITION BY test)

AS average_by_test FROM student;

+---------+--------+-------+-----------------+

| name | test | score | average_by_test |

+---------+--------+-------+-----------------+

| Chun | SQL | 75 | 65.2500 |

| Chun | Tuning | 73 | 68.7500 |

| Esben | SQL | 43 | 65.2500 |

| Esben | Tuning | 31 | 68.7500 |

| Kaolin | SQL | 56 | 65.2500 |

| Kaolin | Tuning | 88 | 68.7500 |

| Tatiana | SQL | 87 | 65.2500 |

| Tatiana | Tuning | 83 | 68.7500 |

+---------+--------+-------+-----------------+

Valid window functions include:

ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, COUNT, SUM, AVG, BIT_OR, BIT_AND and BIT_XOR



● Common table expressions (WITH), allows you to refer to a sub query as a table:

WITH t as (select a from t1 where b >= 'c')

select * from t2,t where t2.c=t.a;

select t1.a,t1.b from t1,t2

where t1.a>t2.c and t2.c in

(WITH t as (select * from t1 where t1.a<5)

select t2.c from t2,t where t2.c=t.a);

We are working on recursive CTEs. MDEV-9864



● Support for CHECK constraint:create table t1 (a int check(a>0),b int check (b> 0), constraint abc check (a>b));

● Support for DEFAULT with expressions (including for TEXT and BLOB):

create table t1 (a bigint default uuid_short(), b int default 0, c int default 0, d int default (a+b));



● Faster client connecting to MariaDB:● 85 % speedup to default MariaDB 10.1● 13 % speedup to 10.1 with thread cache enabled

● SHOW CREATE USER● ALTER USER● New CREATE USER options for limiting

resource usage and tls/ssl● Lots of restrictions lifted for Virtual columns● DECIMAL decmals has increased from 30 to 38● More information from EXPLAIN FORMAT=JSON


MariaDB 10.2Upcoming features

● JSON (like in MySQL 5.7)● Virtual columns with functional indexes● Galera 4.0● Google patches (GTID indexing, "no slave left

behind") ● “Time machine”

● Rollback binlog to certain point in history

For full list of things considered see: https://mariadb.com/kb/en/mariadb/plans-for-10x/


Free MariaDB/MySQL/SQL99 documentation

The knowledgebase allows you to:● Find answers to your problems● Ask questions and get answers from others● Add your own documentation or help with translations


There are a lot of others involved

● MariaDB corporation has 80+ employees● Many external contributors; Most features in MariaDB 5.2 and 10.1

were contributed by the community!● Many of the advanced features in MariaDB 5.3 and in MariaDB 10.2

are sponsored features● In the mariadb.com/kb knowledge base (free MariaDB and MySQL

documentation) we have now 4323 (mostly English) articlesStatistics from the June 2016:● Added/Changed Articles: 287● On Freenode #maria, 669 people wrote 9599 lines● Launchpad & Github Activity

● 12 active branches / Git: 149 forks, 569 stars, 117 watching● 395 commits / Git total: 172611 commits, 71 branches

● More than 4M downloads of MariaDB. Probably >> 8M users


MariaDB popularity is increasing

● In December 2012● Wikipedia announced they are moving to MariaDB.

● In January-March 2013● DB at Mozilla blogged they have moved to MariaDB● Fedora voted 7-0 to make MariaDB the default MySQL database● OpenSuse 12.3 included MariaDB as default.● Slackware, Chakra Linux and Arch Linux has MariaDB as default.

● In April 2013● Google is basing their new SQL offerings on MariaDB● FusionIO is showing benchmarks with MariaDB.

● June 2013● RedHat announced it will include MariaDB in RedHat Enterprise.

● December 2013● MariaDB was added to Debian and later included in Ubuntu

● 2014● RedHat Enterprise Linux 7, Suse Enterprise and Oracle unbreakable

Linux has MariaDB as default


Reasons to switch to MariaDB today

● MariaDB has 20 man years of more development than MySQL (and the gap will continue growing).

● MariaDB is maintained by the people that originally created MySQL and has the best knowledge of the MySQL code.

● MariaDB is binary compatible (data and API) with MySQL, so its trivial to replace MySQL with MariaDB (minutes).

● Reasons to switch to MariaDB● Faster queries thanks to XtraDB and TokuDB (both from Percona), a

much better optimizer, better replication and better code.● Open source development: Anyone can be part of the development at all

stages. Developer meetings are public!● More features, including critical ones like true parallel replication, better

statistics, dynamic columns and Spider and other storage engines.● Less risk as MariaDB will not remove features like MySQL is doing

(thread pool, storage engines, safemalloc (developer feature), older OS)

● Supports new architectures like Power8


MariaDB 10.1 was made possible thanks to

● MariaDB corporation● Replication, optimizer, security, speed enhancements, bug fixing, etc

● MariaDB foundation● Power8 optimization● Documentation● Query timeouts● Character set enhancements and speedups● Upgraded regexp library (PCRE)● Reviews for replication, encryption, compression, Galera, Connect storage engine, Moronga storage engine, Spider, OR REPLACE etc.



● Google● Encryption, scrubbing, enhanced semisync, dump thread enhancements, thd_specifics plugin service

● Eperi GmbH● Table level encryption, plugin for secure encryption

● DaumKakao Inc● Defragmentation, online alter progress monitoring

● Codership● Galera (wsrep patches)

● Antony Curtis● Compound statements

● Sriram Patil● CREATE OR REPLACE/IF NOT EXISTS



● Daniel Black● New status variables for replication & other patches

● Stewart Smith● Optimizations for Power8

● Daniël van Eeden● RESET MASTER TO #

In cooperation with:● IBM

● Power8 performance and scalability optimizations● FusionIO

● Atomic writes, page compression, trim, multi-threaded flush for XtraDB/InnoDB



Also used code snippets by:

● Facebook / Webscale● Defragmentation, prefix index queries optimization, lazy flushing, buffer pool list scan optimization, configurable long semaphore wait timeout

● Percona● SET STATEMENT, enforce_storage_engine

● Oracle● UNION ALL optimization default_tmp_storage_engine

● InnoDB storage engine updates


How you can help

● Help develop features in MariaDB 10.x● There is a lot of easy thing 'any' C/C++ developer could do

● Help with writing, answering questions or translating documentation in the knowledge base at https://mariadb.com/kb/en/

● Be active on the MariaDB email lists ([email protected])

● Spread the word about MariaDB● Talk about MariaDB in your blogs or at conferences.

● Convince your company to support, become a member or hire a developer for the MariaDB foundation.

● Become a support partner to MariaDB Corporation.You can find information of how to be part of the MariaDB development team at: https://mariadb.com/kb/en/community/

https://mariadb.com/kb/en/


Questions ?

For questions later, use the public MariaDB email list [email protected] or #maria on Freenode.

For questions regarding the MariaDB foundation:[email protected]

mailto:[email protected]

MariaDB 10.2 & MariaDB 10.1 by Michael Monty Widenius at Database Camp 2016 @ UN

Technology

Transcript of MariaDB 10.2 & MariaDB 10.1 by Michael Monty Widenius at Database Camp 2016 @ UN