Wednesday, May 15, 13 - Khan Kennels · 2013-07-05 · InnoDB: Tablespace Management Wednesday, May...

1

Wednesday, May 15, 13

<Insert Picture Here>

Whats New With MySQL 5.6Ligaya Turmelle Principle Technical Support Engineer - MySQL https://joind.in/8176


https://joind.in/8017

https://joind.in/8017

3


[email protected] Support Engineer

MySQL ACE~10 years

@lig

About Me


4

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Wednesday, May 15, 13Safe harbor statement

MySQL

• world's most popular open source database software

• a key part of LAMP (Linux, Apache, MySQL, PHP / Perl / Python)

• Site: http://www.mysql.com/

• Developer Zone: mysql.org or dev.mysql.com

• Download: http://dev.mysql.com/downloads/

• Online Manual: http://dev.mysql.com/doc/refman/5.6/en/index.html


http://dev.mysql.com/downloads/

http://dev.mysql.com/downloads/

http://dev.mysql.com/doc/refman/5.1/en/index.html




6

Agenda• Security• InnoDB• Optimizer• Replication• Performance Schema

Wednesday, May 15, 13Newest GA version. 5.6.11

In each of the various areas I am only going to be going over a few of the (in my opinion) choicer improvements or enhancements. To learn more you can check out the manual page “What Is New in MySQL 5.6”

7

Security


8

User Account Security


• Optional SHA-256 password hashing

9


Wednesday, May 15, 13Plugin

supports stronger encryption for user account passwords via built in sha256_password plugin.

See http://dev.mysql.com/doc/refman/5.6/en/sha256-authentication-plugin.html

• Optional SHA-256 password hashing• Example:

10


mysql> CREATE USER 'sha256user'@'localhost' IDENTIFIED WITH sha256_password;Query OK, 0 rows affected (0.00 sec)

mysql> SET old_passwords = 2;Query OK, 0 rows affected (0.00 sec)

mysql> SET PASSWORD FOR 'sha256user'@'localhost' = PASSWORD('sha256P@ss');Query OK, 0 rows affected (0.09 sec)

Wednesday, May 15, 13old_passwords value sets the password hashing format

See http://dev.mysql.com/doc/refman/5.6/en/sha256-authentication-plugin.html

mysql> SELECT plugin FROM mysql.user WHERE User = 'sha256user';+-----------------+| plugin |+-----------------+| sha256_password |+-----------------+1 row in set (0.00 sec)

• mysql.user.password_expired

11


Wednesday, May 15, 13default value is 'N', but can be set to 'Y' with ALTER USER statement. (Think cron job to set it)

Built in. NOT for an application password.

After expired, all operations performed in *subsequent* connections to the server using the account - result in an error until the user issues a SET PASSWORD statement.

ALTER USER man page - http://dev.mysql.com/doc/refman/5.6/en/alter-user.html

• mysql.user.password_expired• Ex:

Expire a users password

12


mysql> CREATE USER 'test'@'localhost';Query OK, 0 rows affected (0.00 sec)

mysql> ALTER USER 'test'@'localhost' PASSWORD EXPIRE;Query OK, 0 rows affected (0.00 sec)

Wednesday, May 15, 13default value is 'N', but can be set to 'Y' with ALTER USER statement. (Think cron job to set it)

After an account's password has been expired, all operations performed in subsequent connections to the server using the account result in an error until the user issues a SET PASSWORD statement to establish a new account password.


• mysql.user.password_expired• Ex:

Connecting as that user

13


mysql> SELECT 1;ERROR 1820 (HY000): You must SET PASSWORD before executing this statementmysql> SET PASSWORD = PASSWORD('new_password');Query OK, 0 rows affected (0.00 sec)

mysql> SELECT 1;+---+| 1 |+---+| 1 |+---+1 row in set (0.00 sec)

Wednesday, May 15, 13After an account's password has been expired, all operations performed in subsequent connections to the server using the account result in an error until the user issues a SET PASSWORD statement to establish a new account password.


• Password strength• Validate_password plugin• 3 Policies

• Low, Medium, Strong• 7 server settings available to control options

14


Wednesday, May 15, 13Plugin

Only checks passwords supplied as a cleartext value. This affects the CREATE USER, GRANT, and SET PASSWORD statements. Passwords given as arguments to the PASSWORD() and OLD_PASSWORD() functions are checked as well.

0 or LOW Length (min 8)1 or MEDIUM Length; numeric (1), (1)lowercase/(1)uppercase, and (1)special characters2 or STRONG Length; numeric (1), (1)lowercase/(1)uppercase, and (1)special characters; password substrings of length 4 or longer must not match dictionary file

--validate_password: how the plugin is loaded--validate_password_dictionary_file: path name of the dictionary file --validate_password_length: minimum number of characters-- validate_password_mixed_case_count: minimum number of lowercase and uppercase characters--validate_password_number_count: minimum number of numeric (digit) characters--validate_password_policy: password policy enforced

See http://dev.mysql.com/doc/refman/5.6/en/validate-password-plugin.html

15

InnoDB


• Fulltext• Online DDL• Tablespace Management• Dump, Restore/Warm Buffer Pool• NoSQL interface• Improved Performance & Scalability

16

InnoDB Enhancements

Wednesday, May 15, 13Data Definition Language

• Not just for MyISAM any more• Physically represented as a InnoDB table• Uses MATCH() ... AGAINST syntax in SELECT• Fully transactional, fast look up• Natural language/Boolean modes

17

InnoDB: Fulltext

Wednesday, May 15, 13It can do a lot more then what I mention, but I am not strong on Fulltext matching to explain it all. See the manual.

• Performing the operation “in-place”• Improves responsiveness and availability• Concurrency vs. Responsiveness

• LOCK=EXCLUSIVE• LOCK=SHARED• LOCK=NONE

• Avoids temp increases in disk space usage/IO

18

InnoDB: Online DDL

Wednesday, May 15, 13DML operations (INSERT, UPDATE, DELETE, etc) and queries run on the table while the DDL (Ex: ALTER) is in progress - for example if you were modify its indexes or column definitions - you don’t have to wait for the table to be rebuilt.

- You control these (C&L) with the LOCK clause. - omit the LOCK clause or specify LOCK=DEFAULT, MySQL allows as much concurrency as possible depending on the *type of operation*. - If the specified LOCK clause for an operation is not available - it acts as an assertion and errors out. - block access to the table entirely (LOCK=EXCLUSIVE clause), - allow SELECT but not DML (LOCK=SHARED clause), - allow full query and DML access to the table (LOCK=NONE clause). - By doing the changes in-place where possible, rather than creating a new copy of the table, it avoids temporary increases in disk space usage and the I/O overhead of copying the table and reconstructing all the secondary indexes.

• Works with innodb_file_per_table• Easier to reclaim space• Specific tables on separate devices• Transportable tablespaces• Compression for tables and indexes• Dynamic Row Format

19

InnoDB: Tablespace Management

Wednesday, May 15, 13Historically, all InnoDB tables and indexes were stored in the system tablespace. InnoDB's file-per-table mode is a more flexible alternative, where you store each InnoDB table and its indexes in a separate file. Each such .ibd file represents a separate tablespace.

Advantages: - can reclaim disk space when truncating or dropping a table. - store specific tables on separate storage devices (Ex: SSD), for I/O optimization, space management, or backup purposes. - copy individual InnoDB tables from one MySQL instance to another. Todd Farmer has a blog post on how to copy over a table working only with the tablespace. FLUSH TABLE ... FOR EXPORT, UNLOCK TABLE, ALTER TABLE DISCARD TABLESPACE, ALTER TABLE IMPORT TABLESPACESee http://mysqlblog.fivefarmers.com/2012/11/07/smarter-innodb-transportable-tablespace-management-operations/ - Compress to save space and IO - used with SSD, compressed OLTP workloads are now possible - If it fits, the entire row is stored in the index node. Otherwise long data values are stored off-page.

• Shortened Buffer Warmup• At shutdown/startup or manually

• Examples:

20

- Specify that a dump should be taken at shutdown: SET innodb_buffer_pool_dump_at_shutdown=ON;

- Specify that a dump should be loaded at startup:SET innodb_buffer_pool_load_at_startup=ON;

- Trigger a dump of the buffer pool manually:SET innodb_buffer_pool_dump_now=ON;

- Trigger a load of the buffer pool manually:SET innodb_buffer_pool_load_now=ON;

InnoDB: Dump, Restore/Warm Buffer

• Small footprint on disk

Wednesday, May 15, 13Bullet 1: “After you restart a busy server, there is typically a warmup period with steadily increasing throughput, as disk pages that were in the InnoDB buffer pool are brought back into memory as the same data is queried, updated, and so on. Once the buffer pool holds a similar set of pages as before the restart, many operations are performed in memory rather than involving disk I/O, and throughput stabilizes at a high level.

This feature shortens the warmup period by immediately reloading disk pages that were in the buffer pool before the restart, rather than waiting for DML operations to access the corresponding rows. The I/O requests can be performed in large batches, making the overall I/O faster. The page loading happens in the background, and does not delay the database startup.”

Especially important with large buffer pools - Warmup can go from hours to minutes

Bullet 2: - Control of when to dump and load the buffer pool is controlled by server options.

Bullet 3: - Although the buffer pool itself could be many gigabytes in size, the data that InnoDB saves on disk to restore the buffer pool is tiny by comparison: just the tablespace and page IDs necessary to locate the appropriate pages on disk.

• Direct access to InnoDB tables

• Key-value access via the memcached API

• Data Accessible via SQL

• mysqlnd_memcache PHP extension

21

InnoDB: NoSQL Interface

Wednesday, May 15, 13 - bypassing the SQL parser, the optimizer, and even the Handler API layer. - you use the memcached protocol to read and write data directly to InnoDB, and let MySQL manage the in-memory caching through the InnoDB buffer pool. - You can still access the underlying table through SQL, for reporting, analysis, ad hoc queries, bulk loading, set operations such as union and intersection, and other operations well suited to the expressiveness and flexibility of SQL.- mysqlnd_memcache PHP extension transparently translating SQL into requests for the MySQL InnoDB Memcached Daemon Plugin (server plugin). Still early so you are warned.

• Internal Improvements• Persistent Optimizer Statistics

• Increased plan stability, accurate statistics• Better user control, automatic/manual

• SSD Optimizations• 4/8K page sizes• .ibd files can be outside the datadir• separate tablespace for undo logs

22

InnoDB: Performance & Scalability

Wednesday, May 15, 13 Bullet 1: reduced contention by splitting the kernel mutex, moving flushing operations from the main thread to a separate thread, enabling multiple purge threads, and reducing contention for the buffer pool on large-memory systems.Bullet 2: sudden changes in query execution plans can cause huge performance issues with a system. You now can control how much sampling is done and when it will be done. - On by default, enabled by the configuration option innodb_stats_persistent. - How much done: innodb_stats_persistent_sample_pages and innodb_stats_transient_sample_pages - When it is done: ANALYZE TABLE and innodb_stats_auto_recalc - determines whether the statistics are calculated automatically whenever a table undergoes substantial changes (to more than 10% of the rows) - To revert to the previous method of collecting statistics that are periodically erased, run the command ALTER TABLE tbl_name STATS_PERSISTENT=0.Bullet 3: SSDs are finding there way into more and more systems - Innodb page size default is 16K. The 4/8K sizes can work better with SSD. Set with innodb_page_size config - Tables with high levels of random read/writes may have improved performance on SSDs - the IO patterns for the undo log (rollback segments) work well with SSD


23

More Information on InnoDB Improvements

http://dev.mysql.com/doc/refman/5.6/en/innodb-performance.html

Wednesday, May 15, 13A lot more then what I covered was done. To learn more, be sure to check out the web site.

24

Optimizer


• Now also works with • DELETE• INSERT• UPDATE• REPLACE

• Optional JSON output

25

EXPLAIN Plan

Wednesday, May 15, 13EXPLAIN EXTENDED FORMAT=JSON UPDATE...

• Stop sorting when X first rows of sorted result found• How it works

• If ordering done by index - fast• If filesort

• All rows that match the query without the LIMIT clause are selected

• Most or all of them are sorted, until the first X are found.• Sort stops

• If able to fit - will do it all in the sort_buffer_size

26

ORDER BY.. Limit Optimization

Wednesday, May 15, 13Avoids merge file and does it all in memory.

• Reduce random disk accesses • How it works

• Range scan of secondary index• Collect PK values of chosen rows• Order the PK values• retrieve base table data by ordered PK

• Extra shows “Using MRR”

27

Multi-Range Read

Wednesday, May 15, 13leads to a more sequential scan of the base table

28

Index Condition Pushdown

Wednesday, May 15, 13To understand the index condition pushdown optimization you have to have a basic understanding of MySQL’s API architecture.

A query comes in (SQL interface), - is parsed and the syntax is checked (Parser), - an execution plan is determined (optimizer), - execution plan sent to the query processor. (Caches & Buffers would be something like the query cache.). * query processor then ** takes the query plan and generates a queue of method calls (Handler calls to the API interface of the various storage engines). - The storage engine handles the actual data/file/disk access.

Different storage engines handle these requirements differently. For example the MyISAM storage engine caches (key buffer), the indexes of its tables but leaves the caching of the actual data to the operating system. InnoDB on the other hand tries to cache everything within the buffer pool. If it can’t it uses various techniques to be ‘smart’ about your needs.

• Works with your indexes• Reduce accesses to base table• Potentially reduce rows returned and handled in the

server• How does it work

• Example without • Example with

29

Index Condition Pushdown

Wednesday, May 15, 13Without: 1. In the storage engine, get the next row, first by reading the index tuple, and then by using the index tuple to locate and read the full table row. 2. Pass the full table row out of the storage engine back to the MySQL Server and test the part of the WHERE condition that applies to this table. Accept or reject the row based on the test result.

With:1. In the storage engine, get the next row's index tuple (but not the full table row). 2. In the storage engine, test the part of the WHERE condition that applies to this table *and* can

be checked using only index columns. - If the condition is not satisfied, proceed to the index tuple for the next row. - If the condition is satisfied, use the index tuple to locate and read the full table row.

3. Pass this out of the storage engine back to the MySQL Server and test the remaining part of the WHERE condition that applies to this table. Accept or reject the row based on the test result.

• Primarily for developers• See how the optimizer chooses the execution plan• Works for

• SELECT• INSERT / REPLACE• UPDATE• DELETE

30

Tracing the Optimizer

Wednesday, May 15, 13SET OPTIMIZER_TRACE="enabled=on",END_MARKERS_IN_JSON=on;

http://dev.mysql.com/doc/internals/en/optimizer-tracing.html

31

Replication


• Unique Identifier across all servers• pair of coordinates

Ex: source_id:transaction_id• Transaction based• No need for log files or positions anymore• Preserved between master and slave• Easier failover

32

Global Transaction ID

Wednesday, May 15, 13* source_id = originating server - typically server_uuid* transaction_id = sequence number determined by the order in which the transaction was committed on the source_id server - can not be 0 - can be given as a range. Ex: 1-5* All data for synchronizing replication is taken from the replication stream. No more MASTER_LOG_FILE and MASTER_LOG_POS in CHANGE MASTER. Just use MASTER_AUTO_POSITION* Can always determine the source for any transaction applied on any slave. Also once a transaction with a given GTID is committed on a given server, any subsequent transaction having the same GTID is ignored by that server.* the mysqlfailover utility

• Key Steps• Synchronize the servers & read only

33

Setup Replication with GTID

Wednesday, May 15, 13Cold start - replication for the first time or able to stop the servers - Step 1: SET @@global.read_only = ON;

• Key Steps• Synchronize the servers & read only• Stop the servers

34


Wednesday, May 15, 13 - Step 2: mysqladmin -u user -p shutdown

Cold start - replication for the first time or able to stop the servers - Step 1: SET @@global.read_only = ON;

• Key Steps• Synchronize the servers & read only• Stop the servers• Configure & restart both servers

• grid_mode=ON• log-bin• log-slave-updates• enforce-gtid-consistancy• read-only• skip-slave-start

35


Wednesday, May 15, 13 - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=ON --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet

Cold start - replication for the first time or able to stop the servers - Step 1: SET @@global.read_only = ON; - Step 2: mysqladmin -u user -p shutdown



• CHANGE MASTER & restart slave

36


Wednesday, May 15, 13 - Step 4: mysql> CHANGE MASTER TO > MASTER_HOST = host, > MASTER_PORT = port, > MASTER_USER = user, > MASTER_PASSWORD = password, > MASTER_AUTO_POSITION = 1; mysql> START SLAVE;

Cold start - replication for the first time or able to stop the servers - Step 1: SET @@global.read_only = ON; - Step 2: mysqladmin -u user -p shutdown - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=ON --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet



• CHANGE MASTER & restart slave• Disable read-only

37


Wednesday, May 15, 13 - Step 5: mysql> SET @@global.read_only = OFF;

Cold start - replication for the first time or able to stop the servers - Step 1: SET @@global.read_only = ON; - Step 2: mysqladmin -u user -p shutdown - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=ON --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet - Step 4: mysql> CHANGE MASTER TO > MASTER_HOST = host, > MASTER_PORT = port, > MASTER_USER = user, > MASTER_PASSWORD = password, > MASTER_AUTO_POSITION = 1; mysql> START SLAVE;

• Python• Replication health monitoring• Handles Failover

• auto• elect• fail

• Automatic !== Instantaneous

38

mysqlfailover Utility

Wednesday, May 15, 13 - Only works with GTIDs - 3 failover modes - auto = failover to the list of slave candidates. Can’t find any viable candidate, look at all slaves attached to master for a viable candidate, still none - error out. - elect = failover to the list of slave candidates. Can’t find any viable candidate - error out - fail = error outStupid easy

• Increases slave throughput and reduces lag • Break up concurrent SQL threads by database

• Assumes• Data and updates partitioned by database• Updates occur in same relative order as on master• not necessary to coordinate transactions between DBs

• # threads controlled by slave_parrallel_workers

39

Multi-threaded Slaves

Wednesday, May 15, 13Since transactions on different databases can occur in a different order on the slave than on the master, simply checking for the most recently executed transaction is not a guarantee that all previous transactions on the master have been executed on the slave. This has implications for logging and recovery when using a multi-threaded slave.

• Works with RBR• Save disk space, network & memory• Control over what is placed in binlog

• binlog_row_image• minimal• full• noblob

40

Row Image Control

PK ChangedColumns

Wednesday, May 15, 13MySQL RBR originally always logged the entire row that was changed. This lead to the potential of RBR binary logs being much larger then SBR binary logs.

- minimal = log only required columns - full = log all columns - noblob = log all columns except unneeded blob/text columns

• Ensures replicated data is correct, consistent and accessible

• Detects corrupt events before they’re applied• Returns an error

• Protects entire replication path• Memory• Disk• Network

41

Event Checksums

Wednesday, May 15, 13to write checksums for the events - binlog_checksum=crc32to read checksums from the binary log - master_verify_checksum =ONSQL thread to read checksums from relay log -slave-sql-verify-checksum=1

42

PerformanceSchema


• Monitor MySQL Server execution at a low level• Can be configured dynamically

• Data held in performance_schema DB• Can be queried with SELECT• Views/Temp Tables - no persistance

• Activation does not alter server behavior• Some loss of performance

43

General Information

Wednesday, May 15, 13 - configure dynamically by updating tables in the P_S DB with SQL. Affects collection immediately - There are various blog posts/benchmarks and bug reports on it. I have heard of values as low as 5% and as high as 28%. You can disable if you really need it.

• Table and Index IO• Statements and stages within statements• Table Locks• Users/Hosts/Accounts• Network IO

44

New Instrumentation


• Controlling it• What can be watched:mysql> SELECT * FROM performance_schema.setup_instruments;

• Turn on watchersmysql> UPDATE setup_instruments SET enabled = ‘YES’ WHERE NAME LIKE 'wait/io/file/innodb/%';

• Turn off watchersmysql> UPDATE setup_instruments SET enabled = ‘NO’ WHERE NAME LIKE 'wait/io/file/innodb/%';

45

Using Performance Schema


• ps_helper• Created by Mark Leith• Collection of views and procedures• Examples:

• View: top_io_by_thread• View: statements_with_full_table_scans• View: schema_tables_with_full_table_scans• View: schema_unused_indexes

46

Using Performance Schema

Wednesday, May 15, 13* top_io_by_thread: Show the top IO consumers by thread, ordered by total latency. (5.5)* statements_with_full_table_scans: Lists all normalized statements that use have done a full table scan ordered by the percentage of times a full scan was done, then by the number of times the statement executed (5.6)* schema_tables_with_full_table_scans: Find tables that are being accessed by full table scans, ordering by the number of rows scanned descending* schema_unused_indexes: Find indexes that have had no events against them (and hence, no usage) for the the lifetime of the server or since statistics were last truncated (5.6)

47

Conclusion• Lot of changes

• Security• InnoDB• Optimizer• Replication• Performance Schema

Wednesday, May 15, 13In each of the various areas I am only going to be going over a few of the (in my opinion) choicer improvements or enhancements. To learn more you can check out the manual page “What Is New in MySQL 5.6”

48

Questions?


San Francisco, September 21-23Additional Day of TutorialsOracle.com/mysqlconnect

Keynotes Conferences Sessions Birds-of-a-feather sessions Tutorials Hands-on Labs Exhibition Hall Demo Pods Receptions

Early Bird Discount: Register Now and Save US$500!


50


Wednesday, May 15, 13 - Khan Kennels · 2013-07-05 · InnoDB: Tablespace Management Wednesday, May...

Documents

Transcript of Wednesday, May 15, 13 - Khan Kennels · 2013-07-05 · InnoDB: Tablespace Management Wednesday, May...