MySQL 8 Michael Messina

Senior Managing Consultant, Rolta-AdvizeX

mmessina@advizex.com

Introduction

• Michael Messina

• Senior Managing Consultant Rolta-AdvizeX,

• Working with Oracle Approximately 25 years, MySQL about 12 years.

• Background includes Performance Tuning, High Availability and Disaster Recovery

• MySQL Cloud Service 2018 Implementation Specialist

• Oracle Database OCP

• Oracle RAC Certified Expert

• Oracle Exadata Implementation Specialist

• Oracle ACE

• MMESSINA@ADVIZEX.COM / MIKE.MESSINA@ROLTA.COM

• www.advizex.com

Who’s Using MySQL

3

Cloud

Web & Enterprise

OEM & ISVs

Still a Community Driven Open Source Database

• MySQL 8 Update Contributors based on conversions at Open World

Facebook

Linked-in

Percona

These are just the main recognizable contributors

Agenda

• MySQL History

• Why MySQL

• MySQL 8 So far……

• MySQL Changes

• MySQL Deprecated/Removed

• Data Dictionary

• Atomic Data Definition Statement Support

• Security and User Management

• Persistent Runtime Configuration Changes

• Invisible Indexes

• WITH

• SQL Windowing Functions

Agenda

• TmpTable storage engine

• Optimizer Hints

• JSON (Document Store)

• Performance

• Group Replication

MySQL History

• MySQL was created by a Swedish company, MySQL AB

• First release on 05/23/1995, making a 30 year History for MySQL

• Version 3.20 released January 1997

• Windows Version of MySQL Release January 1998

• Version 3.21 production release in 1998

• Version 4.0 Beta release August 2002

• Version 4.0 Production release March 2003

Unions

• Version 5.0 Beta release March 2005

• Version 5.0 Production release October 2005

cursors, stored procedures, triggers, views, XA transactions

MySQL History

• Sun Acquires MySQL AB in 2008

• Version 5.1 Production Release November 2008

event scheduler, partitioning, row-based replication, server log to tables

• Oracle Acquires Sun Microsystems January 2010

• MySQL Server 5.5 December 2010

Unicode character sets utf16, utf32, and utf8mb4

New Partitioning options

• Version 5.6 February 2013

full-text search for innodb

NoSQL memcache

• Version 5.7 October 2015

Group Replication

• MySQL 8

Development Milestone 2016

Release Candidates 2017

Why MySQL

• Long History of a Solid Scalable Database (Proven over 30 year)

• Great Proven Performance

• Worlds Most Popular Open Source Database

• Open Source and run Community Editions for Free

• Large Database Company Oracle more then 20,000 developers

• Enterprise Options for Support by Oracle at very low cost

• Rich tool set to manage and monitor available

• Enterprise Version has Database Firewall a security option from Oracle that only the Oracle Database can complete with no other open source database has this.

• Large Support Community

Why MySQL

• Run in Cloud / Virtualized / Dedicated Hosts

Cloud Infrastructure as a Service - IaaS

Cloud Database as a Service - DBaaS

• Full and Incremental Database Backup Capability

• Point in Time Recovery

• Highly Scalable

Many scalability options

NDB Cluster

Sharding

• Can Be Made Highly Available

Many Options to Provide Redundancy for availability based on needs

NDB Cluster

Group Replication

Async Replication

MySQL 8 So far…….

• 09/2016 MySQL 8.0.0 Development Milestone

• 04/2017 MySQL 8.0.1 Development Milestone

• 07/2017 MySQL 8.0.2 Development Milestone

• 09/2017 MySQL 8.0.3 Release Candidate 1

• 01/2018 MySQL 8.0.4 Release Candidate

• 04/2018 MySQL 8.0.11 GA

MySQL 8 Upgrade Hints

• Move all user tables to innodb before doing upgrade, not required, but recommended for sure

• errors on older table types have been known to happen due to conflict with default SQL_MODE.

When you have queries on older type tables, you have to do

mysql> set global sql_mode = ‘’;

otherwise you’ll get mysql errors for stuff like (select * from table where id = <id>)

This appears to be a bug, but still RC releases, but thought we would share

MySQL 8 Changes

• tx_isolation and tx_read_only system removed, use transaction_isolation and transaction_read_only

• log_warnings system variable and --log-warnings server parameter removed, use log_error_verbosity system variable instead

log_error_verbosity = (1- Errors, 2- Errors and Warnings, 3-Errors, Warnings and Notes)

• sql_log_bin server parm removed, sql_log_bin has session scope only

applications that rely on accessing @@global.sql_log_bin should be adjusted.

• encryption-related items have been removed

The ENCODE() and DECODE() functions.

The ENCRYPT() function.

The DES_ENCRYPT(), and DES_DECRYPT() functions, the --des-key-file option, the have_crypt system variable, the DES_KEY_FILE option for the FLUSH statement, and the HAVE_CRYPT CMake option.

Use ENCRYPT(), consider using SHA2() instead for one-way hashing or others use AES_ENCRYPT() and AES_DECRYPT() instead.

MySQL Changes

• The client-side --ssl and --ssl-verify-server-cert removed.

Use --ssl-mode=REQUIRED instead of --ssl=1 or --enable-ssl.

Use --ssl-mode=DISABLED instead of --ssl=0, --skip-ssl, or --disable-ssl.

Use --ssl-mode=VERIFY_IDENTITY instead of --ssl-verify-server-cert options. (The server-side --ssl option remains unchanged.)

• mysql_install_db program has been removed, use myqld with the --initialize or --initialize-secure for new database creation.

• No Direct upgrade for non-innodb partitioned tables, change all partitioned tables to innodb prior to upgrade

• InnoDB no longer supports compressed temporary tables.

MySQL 8 Deprecated/Removed

• Deprecated

innodb_undo_tablespaces which controls the number of tablespaces to divide rollback segments into is deprecated

log_syslog for write error log to syslog deprecated

• Removed

sync_frm as .frm files are obsolete

group_replication_allow_local_disjoint_gtids_join

Group Replication will no longer allow any disjointed GTIDs

InnoDB file format configuration parameters were removed:

innodb_file_format

innodb_file_format_check

innodb_file_format_max

innodb_large_prefix

innodb_checksums

show_compatibility_56

MySQL Deprecated/Removed

• The MySQL Query Cache has been Removed in MySQL 8

No more FLUSH QUERY CACHE and RESET QUERY CACHE statements

System variables around query cache removed

query_cache_limit

query_cache_min_res_unit

query_cache_size

query_cache_type

Status Variables removed

qcache_free_blocks

qcache_free_memory

qcache_hits

qcache_inserts

qcache_lowmem_prunes

qcache_not_cached

qcache_queries_in_cache

qcache_total_blocks

Data Dictionary

• MySQL Now has an actual Data Dictionary.

• All privilege tables are now Innodb not MyISAM

• Renamed InnoDB Information Schema Views

Old Name New Name

INNODB_SYS_COLUMNS INNODB_COLUMNS

INNODB_SYS_DATAFILES INNODB_DATAFILES

INNODB_SYS_FIELDS INNODB_FIELDS

INNODB_SYS_FOREIGN INNODB_FOREIGN

INNODB_SYS_FOREIGN_COLS INNODB_FOREIGN_COLS

INNODB_SYS_INDEXES INNODB_INDEXES

INNODB_SYS_TABLES INNODB_TABLES

INNODB_SYS_TABLESPACES INNODB_TABLESPACES

INNODB_SYS_TABLESTATS INNODB_TABLESTATS

INNODB_SYS_VIRTUAL INNODB_VIRTUAL

Data Dictionary

• Uniformly stores database dictionary data (metadata)

• Removes file based metadata storage

• Makes data dictionary transactional, making data dictionary crash safe

Commit, rollback

• Central caching of data dictionary metadata.

Dictionary Object Cache

shared global cache that stores previously accessed data dictionary objects in memory

LRU-based

• Access via the information_schema and “show” statements

• Data dictionary metadata all in innodb tables

• Contained in a single innodb tablespace (mysql.ibd)

Atomic Data Definition Statement Support

• With Data Dictionary we get Atomic DDL

• Combines the data dictionary updates, storage engine operations, and binary log writes associated with a DDL operation into a single, atomic transaction

• Makes so any DDL operation and associated operations are all committed or all rolled back this creates more consistent and stable DDL operations.

• Innodb DDL only

• Partial execution of account management statement will no longer be permitted.

• Uses mysql.innodb_ddl_log table to record the operations for the DDL statement allows statement recovery.

• enable innodb_print_ddl_logs to have MySQL write the DDL logs to stderr, for most systems this will be the error log file for the database.

Resource Management

• Resource Groups

permits assigning threads to particular groups

threads execute according to the resources available to the group

modify these attributes as appropriate for workloads

• Types – Once Group created, Type for Group can not be changed

System

User

• Resources

CPU Affinity – no affinity = All CPUs otherwise the CPU Ids Allowed to be used

OS Thread priority – execution priority range from -20 highest priority to lowest priority of 19

• 2 Groups by default SYS_Default

USR_Default

Priority 0 and No CPU Affinity for default groups

Can not be altered or changed

New User Threads go to USR_Default and SYS Threads to SYS_Default

Resource Groups

Example: RESOURCE_GROUP_NAME: USR_default

RESOURCE_GROUP_TYPE: USER

RESOURCE_GROUP_ENABLED: 1

VCPU_IDS: 0-3

THREAD_PRIORITY: 0

CREATE RESOURCE GROUP 'name'

TYPE=SYSTEM|USER [VCPU=num|start-end[,num|start-end]*]

[THREAD_PRIORITY=num] [ENABLE|DISABLE] ;

ALTER RESOURCE GROUP 'name'

[VCPU=num|start-end[,num|start-end]*]

[THREAD_PRIORITY=num] [ENABLE|DISABLE] [FORCE] ;

DROP RESOURCE GROUP 'name' [FORCE];

SELECT * from INFORMATION_SCHEMA.RESOURCE_GROUPS;

Security and User Management

• Roles

Implements roles to group privileges

Grant roles to accounts to standardize privileges

Control privileges to many uses through the privileges of roles

• Password History

Keeps password history

Allows for password reuse rules and restrictions on pervious passwords

Can apply globally or per account

• caching_sha2_password plugin

Improvement on the sha2_password plugin

Implements sha2 password hashing like the sha2_password plugin

Improves on sha2_password plugin by using caching to address latency issues at connect time

Change mysql_native_password to caching_sha2_password is recommended

Persistent Runtime Configuration Changes

• SET PERSIST allows runtime configuration to be kept and survive a restart. In previous versions changes made using SET GLOBAL would be discarded upon restart.

• Adds setting for the parameter variable to mysqld-auto.cnf

• PERSIST – Set variable for current runtime and through restart

• PERSIST_ONLY – Set variable for persist to next restart does not change current runtime.

• RESET_PERSIST – Reset and remove the persist value setting.

• Be Careful this will take priority over you my.cnf parameters

• Can now change things and take affect at runtime

SET PERSIST innodb_buffer_pool_size=8G ;

SET PERSIST max_connections = 1000 ;

Invisible and Descending Indexes

• MySQL now supports invisible indexes.

not used by the optimizer at all

maintained normally with DML operations.

Indexes visible by default.

Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required.

• MySQL now supports descending indexes

DESC in an index definition is no longer ignored but causes storage of key values in descending order.

Prior to this indexes could be scanned in reverse order but it took a performance hit”.

WITH Now Exists in MySQL

• The with clause has been in use for quite some time to offer both simple query builds and better performance for some queries. MySQL now has the capability for using WITH.

WITH tab1 (select * from table1 where columna = ‘My Value’),

tab2 (select * from table2 where mycol = ‘My Value too’)

select * from tab1 a, tab2 b where a.keycol = b.keycol ;

SQL Windowing Functions

• SQL aggregate functions support windowing

COUNT, SUM, AVG, MIN, MAX, BIT_OR, BIT_AND, BIT_XOR,

STDDEV_POP (and its synonyms STD, STDDEV), STDDEV_SAMP,

VAR_POP (VARIANCE synonym for VAR_POP), VAR_SAMP

• Specialized window functions

RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE,

ROW_NUMBER, FIRST_VALUE, LAST_VALUE, NTH_VALUE, LEAD, LAG

SQL Windowing Functions

Simple Example using and emp_sales table:

SELECT empid, date, sale, SUM(sale) OVER (PARTITION BY

employee) AS sum FROM emp_sales;

+-------+------------+------+------+

| empid | date | sale | sum |

+-------+------------+------+------+

| 1 | 2018-03-01 | 200 | 900 |

| 1 | 2018-04-01 | 300 | 900 |

| 1 | 2018-05-01 | 400 | 900 |

| 2 | 2018-03-01 | 400 | 1200 |

| 2 | 2018-04-01 | 300 | 1200 |

| 2 | 2018-05-01 | 500 | 1200 |

+-------+------------+------+------+

NOWAIT and SKIP LOCKED

• Default behavior is for a transaction to wait for a lock.

• NOWAIT

Forces transaction to fail if lock can not be obtained immediately.

select * from my_table where my_key BETWEEN 2 AND 3 AND reserved = 'NO’ FOR UPDATE NOWAIT ;

• SKIP LOCKED

Transactions will skip rows where a lock can not be obtained.

Example Using a SELECT… FOR UPDATE

START TRANSACTION;

select * from my_table where my_key BETWEEN 2 AND 3 AND reserved = 'NO’ FOR UPDATE SKIP LOCKED;

Execute the actual update

update my_table set reserved = ‘YES’ where my_key between 2 and 3;

commit ;

TmpTable Storage Engine

• Efficient Storage engine for Internal Temporary Tables.

• Allows larger sort operations that flow over to disk to be faster

• Allows more efficient index builds

• No Blob/Text Column Support

Optimizer Hints

• Introduced the hint syntax /*+ hint */ in 5.7

• Look Familiar?????

• New Hints

INDEX_MERGE and NO_INDEX_MERGE

control index merge

JOIN_FIXED_ORDER, JOIN_ORDER, JOIN_PREFIX, and JOIN_SUFFIX

control table order for joins

SET_VAR

Sets variable to be used at that value for the statement only after statement completes variable reverts to prior value.

JSON

• Native JSON Document Storage

JSON data type

Ability to create virtual columns that can be indexed

• Extended Syntax for Ranges in JSON path expressions

• JSON Table Functions

JSON_TABLE() - creates a relational view of JSON data

• JSON Aggregation Functions

JSON_ARRAYAGG() - generate JSON arrays

JSON_OBJECTAGG() - generate JSON objects

JSON

• JSON Merge Functions

JSON_MERGE_PATCH() - Merger multiple JSON documents by replacing any match in first document with second document

JSON_MERGE_PRESERVE() - Merge multiple JSON documents

• JSON Update

JSON_REMOVE() - Removes values from a JSON document

JSON_SET() - can replace existing values and add new values for JSON document

JSON_REPLACE() - Replace existing values in a JSON document

JSON Improved Sorting

1.2 to 1.8 times performance improvement

MySQL Document Store (High Level Summary)

• Work with SQL Relational Tables and schema less JSON collections

• Eliminates need for a separate NoSQL solution

• Innodb Based

• Highly reliable

• Fully consistent

• Works in Group Replication for Availability and redundancy

• Online backup with the mysql database

• With Enterprise Transparent Data Encryption for higher security

• Works with MySQL Auditing

Performance

• Improved the prefetching by taking the query optimizer’s estimates into account when allocating the buffer

• Makes scans faster (index scan, range scan, table scan)

• For replication Reduce contention between IO and SQL threads

• Performance improvement for large LOBs

Up to 14x in internal tests at Oracle

• Optimizer cost model

Take into account whether data is already in memory or need to be read from disk

• Optimizer Histograms on (detailed stats on columns)

Better Join Order Compare and Index Selection.

• Scalable Redo log with dedicated redo log threads

log_writer writes from log buffer to file

log_flusher executes fsync()

log_notifier notifies user threads finish fsync()

log_checkpointer writes checkpoints

Performance

• Innodb Performance improves around 30% of Mysql 5.7 and 50% over MySQL 5.6, Sysbench OLTP-RW 48 cores trx_commit=1

** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle

Performance with SSD

• Sysbench OLTP_RO 48 cores 2 SSD Flash Drives

** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle

Performance with SSD

• OLTP_RW Update-NoKey Test, 48 cores 2 SSD Flash Drive

** Chart from Dimitri KRAVTCHUK, MySQL Performance Architect Oracle

Group Replication

• Better event recording for mutex and condition synchronization objects in GCS are recorded

can been seen through Performance Schema tables.

Give MySQL GCS developers a look for abnormal wait events like bottlenecks or deadlocks. MySQL XCom is currently single threaded and not affected by this work.

• Transaction SAVEPOINT support in Group Replication. Transaction savepoints are now supported in Group Replication

• No More disjointed GTIDs, all must match or database can not join group.

• More date for the replication applier

New fields were added to the existing performance schema tables.

Example: introduced immediate and original commit timestamps are exposed. This feature will greatly help users to profile the applier execution and conduct analysis when they find that for some reason there is replication lag.

Now see at which time the last transaction in the applier was queued in the relay log, when it was processed and when it was applied.

Group Replication

• Crash Recovery DDL in replication

40

Any Further Questions?