What you wanted to know about MySQL, but could not find using inernal instrumentation only

What you wanted to know about MySQLbut could not find using internal instrumentation only

February, 3, 2017

Sveta Smirnova

∙ MySQL Support engineer∙ Author of

∙ MySQL Troubleshooting∙ JSON UDF functions∙ FILTER clause for MySQL

∙ Speaker∙ Percona Live, OOW, Fosdem,

DevConf, HighLoad...

Sveta Smirnova

2

http://shop.oreilly.com/product/0636920021964.do

Year 2009

3

∙ In modern versions we have a lot of onlineinformation

∙ However users usually notice error from logfiles, when context is already gone

∙ Partially this is solved by modern monitoringtools (PMM) which can save historicalstatistics

∙ But not about everything

Historical Data

4

∙ It is easy to find in the Audit log records querywhich failed with this error<AUDIT_RECORD

NAME="Query"RECORD="2_2017-01-12T20:40:36"

TIMESTAMP="2017-01-12T20:41:32 UTC"COMMAND_CLASS="update"CONNECTION_ID="3"

STATUS=" 1205"SQLTEXT="update t1 set f1=f1-1"

USER="root[root] @ localhost [127.0.0.1]"HOST="localhost"

OS_USER=IP="127.0.0.1"DB="test"

/>

∙ But there is the query which holds the lock?∙ Even hard to find online∙ Multiple statement transactions make it worse∙ However server has all information to print all

queries of locking transaction∙ MySQL Bug #84563

Lock wait timeout

5

http://bugs.mysql.com/bug.php?id=84563

∙ It is easy to find in the Audit log records querywhich failed with this error

∙ But there is the query which holds the lock?

∙ Even hard to find online∙ Multiple statement transactions make it worse∙ However server has all information to print all


Lock wait timeout

5



∙ But there is the query which holds the lock?∙ Even hard to find online∙ Especially if you have thousands of running

threads!

∙ Multiple statement transactions make it worse∙ However server has all information to print all


Lock wait timeout

5



∙ But there is the query which holds the lock?∙ Even hard to find online∙ Multiple statement transactions make it worse

∙ However server has all information to print allqueries of locking transaction

∙ MySQL Bug #84563

Lock wait timeout

5




queries of locking transaction

∙ MySQL Bug #84563

Lock wait timeout

5





Lock wait timeout

5


∙ First transaction––––––––––––LATEST DETECTED DEADLOCK––––––––––––2017-01-19 13:03:42 7f37fc636700*** (1) TRANSACTION:TRANSACTION 1298, ACTIVE 3 sec starting index read...DELETE FROM t WHERE i = 1*** (1) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 314 n bits 72 index ‘GEN_CLUST_INDEX‘of table ‘test‘.‘t‘ trx id 1298 lock_mode X waiting...

∙ Second transaction∙ Which query held the lock?∙ SELECT * FROM t WHERE i = 1 LOCK IN SHARE MODE;

∙ How would we know?∙ refman/.../innodb-deadlock-example.html∙ Bug #84607

What exactly caused the deadlock?

6

http://dev.mysql.com/doc/refman/5.7/en/innodb-deadlock-example.html


∙ First transaction∙ Second transaction

*** (2) TRANSACTION:TRANSACTION 1297, ACTIVE 7 sec starting index read...DELETE FROM t WHERE i = 1*** (2) HOLDS THE LOCK(S):RECORD LOCKS space id 0 page no 314 n bits 72 index ‘GEN_CLUST_INDEX‘...*** (2) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 314 n bits 72 index ‘GEN_CLUST_INDEX‘...*** WE ROLL BACK TRANSACTION (1)

∙ Which query held the lock?∙ SELECT * FROM t WHERE i = 1 LOCK IN SHARE MODE;



6



∙ First transaction∙ Second transaction∙ Which query held the lock?

∙ SELECT * FROM t WHERE i = 1 LOCK IN SHARE MODE;



6



∙ First transaction∙ Second transaction∙ Which query held the lock?∙ SELECT * FROM t WHERE i = 1 LOCK IN SHARE MODE;



6




∙ How would we know?

∙ refman/.../innodb-deadlock-example.html∙ Bug #84607


6




∙ How would we know?∙ refman/.../innodb-deadlock-example.html

∙ Bug #84607


6






6



∙ Performance Schema∙ Bug #71364 Please provide warning text

information into P_S∙ Bug #61030 Make an I_S table of client error

codes∙ Bug #58058 please add instrumentation to track

error counts on a server

∙ General logging

Some past requests

7







∙ Performance Schema∙ General logging

∙ Bug #70796 Error messages and warnings forsql-mode behaviours need more verbosity

∙ Bug #64190 Log failed queries in a separate log∙ Bug #60884 Enable logging of all errors to the

error log∙ Bug #34137 Additional logging of the server

shutdown process

Some past requests

7








∙ Which kind of query can produce this output?∙ t is InnoDB table

mysql> select * from table_handles where object_name=’t’\G*************************** 1. row ***************************

OBJECT_TYPE: TABLEOBJECT_SCHEMA: test

OBJECT_NAME: tOBJECT_INSTANCE_BEGIN: 140108477034256

OWNER_THREAD_ID: 23OWNER_EVENT_ID: 3788INTERNAL_LOCK: NULLEXTERNAL_LOCK: READ EXTERNAL

1 row in set (0,00 sec)

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t read;

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t read;∙ select * from t [lock in share mode];

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t read;∙ select * from t [lock in share mode];∙ select * from t where i [=,in,<,>] ...

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t read;∙ select * from t [lock in share mode];∙ select * from t where i [=,in,<,>] ...∙ But not select * from t where unique_key = ... !

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ t is InnoDB table

mysql> select * from table_handles where object_name=’t’\G*************************** 1. row ***************************

OBJECT_TYPE: TABLEOBJECT_SCHEMA: test

OBJECT_NAME: tOBJECT_INSTANCE_BEGIN: 140108477034256

OWNER_THREAD_ID: 23OWNER_EVENT_ID: 4379INTERNAL_LOCK: NULLEXTERNAL_LOCK: WRITE EXTERNAL

1 row in set (0,00 sec)

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t write;

∙ Manual says: "The table lock used at the storageengine level. The value is one of READEXTERNAL or WRITE EXTERNAL."

∙ Is this storage engine level operation?

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t write;∙ select * from t for update;

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ lock table t write;∙ select * from t for update;∙ update t set i=i+sleep(i) where i [=,in,<,>] ...

∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ Bug #84609

∙ Bug #84610

Table_handles

8



∙ Which kind of query can produce this output?∙ Bug #84609∙ Bug #84610

Table_handles

8



∙ In past we had only one troubleshooting tool∙ SHOW SLAVE STATUS

∙ Today Performance Schema supportsreplication

∙ But it still misses

Replication

9

∙ In past we had only one troubleshooting tool∙ Today Performance Schema supports

replication

∙ But it still misses

Replication

9


replication∙ But it still misses

∙ Bug #81249 SLAVE_NET_TIMEOUT TOP_S FOR SLAVE THREAD VARIABLES

∙ Bug #78918 Metric for succesful slave reconnects∙ Bug #77605 Add more information to SQL

thread-related P_S tables

Replication

9







replication∙ But it still misses

∙ Bug #76828 Slave details on a master∙ Bug #74809 Stats per binlog event type∙ Bug #72826 Support for joining

replication_execute_status_by_%∙ Bug #70951 Threads shutdown info

Replication

9






∙ What does this output mean?2017-01-20T21:44:52.301177Z 5 [Note] Aborted connection 5 to db: ’test’ user: ’root’host: ’localhost’ (Got timeout reading communication packets)

∙ Timeout while connection was establishing?∙ Connection was aborted, because

interactive_timeout/wait_timeout passed?∙ Something else?∙ Bug #51219, Bug #28836, Bug #78843, Bug

#84612

Connection errors

10






∙ What does this output mean?∙ Timeout while connection was establishing?

∙ Connection was aborted, becauseinteractive_timeout/wait_timeout passed?

∙ Something else?∙ Bug #51219, Bug #28836, Bug #78843, Bug

#84612

Connection errors

10






∙ What does this output mean?∙ Timeout while connection was establishing?∙ Connection was aborted, because

interactive_timeout/wait_timeout passed?

∙ Something else?∙ Bug #51219, Bug #28836, Bug #78843, Bug

#84612

Connection errors

10







interactive_timeout/wait_timeout passed?∙ Something else?

∙ Bug #51219, Bug #28836, Bug #78843, Bug#84612

Connection errors

10







interactive_timeout/wait_timeout passed?∙ Something else?∙ Bug #51219, Bug #28836, Bug #78843, Bug

#84612

Connection errors

10






∙ Bug #77888 max_used_connection peruser/account missing in P_S/sys

∙ Bug #77581 Collect DNS timing informationinto Performance_Schema

∙ Bug #76403COUNT_ABORTED_CLIENT_ERRORS toP_S.host_cache

∙ Bug #72219 First and last connectiontimestamps to P_S.users table

Other connection requests

11










∙ Bug #71305PERFORMANCE_SCHEMA.THREADStable, add a PORT column

∙ Bug #71186 P_S.host_cache does not collectconnections aborted entries

∙ Bug #69880 Track and expose connectioncreation timestamp


11








∙ Bug #69725 P_S.socket_instances doesn’tinclude named pipe or shared memoryconnections

∙ Bug #45817 Please add SHOW command forinc_host_errors(max_connect_errors)

∙ Bug #21565 More verbose connection log


11







∙ One more outputmysql> flush status;Query OK, 0 rows affected (0,00 sec)

mysql> select ...600048 rows in set (1 min 17,26 sec)

mysql> show status like ’Created_tmp%’;+-------------------------+-------+| Variable_name | Value |+-------------------------+-------+| Created_tmp_disk_tables | 2 || Created_tmp_files | 6 || Created_tmp_tables | 3 |+-------------------------+-------+3 rows in set (0,00 sec)

∙ Were tables created in simultaneously?∙ What is their size?∙ Solution: watch lsof∙ Bug #74484∙ Bug #84613

Temporary tables

12



∙ One more output∙ Were tables created in simultaneously?

∙ What is their size?∙ Solution: watch lsof∙ Bug #74484∙ Bug #84613

Temporary tables

12



∙ One more output∙ Were tables created in simultaneously?∙ What is their size?

∙ Solution: watch lsof∙ Bug #74484∙ Bug #84613

Temporary tables

12



∙ One more output∙ Were tables created in simultaneously?∙ What is their size?∙ Solution: watch lsof

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEmysqld 8697 sveta 70u REG 0,43 11765657 43001204 /tmp/mysqld.1/MYSeEOHe (deleted)mysqld 8697 sveta 71u REG 0,43 11765657 43001205 /tmp/mysqld.1/MYVwF8Od (deleted)

∙ Bug #74484∙ Bug #84613

Temporary tables

12



∙ One more output∙ Were tables created in simultaneously?∙ What is their size?∙ Solution: watch lsof∙ Bug #74484

∙ Bug #84613

Temporary tables

12



∙ One more output∙ Were tables created in simultaneously?∙ What is their size?∙ Solution: watch lsof∙ Bug #74484∙ Bug #84613

Temporary tables

12



∙ I_S.OPTIMIZER_TRACE is good addition forOptimizer

∙ But what about other parts of the server?

∙ Runtime∙ Parser∙ Binary logging∙ InnoDB

∙ Bug #84620

Trace

13



∙ But what about other parts of the server?∙ Runtime

∙ Parser∙ Binary logging∙ InnoDB

∙ Bug #84620

Trace

13



∙ But what about other parts of the server?∙ Runtime∙ Parser

∙ Binary logging∙ InnoDB

∙ Bug #84620

Trace

13



∙ But what about other parts of the server?∙ Runtime∙ Parser∙ Binary logging

∙ InnoDB∙ Bug #84620

Trace

13



∙ But what about other parts of the server?∙ Runtime∙ Parser∙ Binary logging∙ InnoDB

∙ Bug #84620

Trace

13


∙ General∙ Bug #83626 Collect per column usage data in

performance_schema∙ Bug #71755 Provide per partition summary

information in PERFORMANCE_SCHEMA∙ Bug #81020 performance_schema: Please add

optimizer usage statistics∙ Bug #55171 How much sort_buffer_size are

actually used?

∙ InnoDB

More tracing requests

14









∙ General∙ InnoDB

∙ Bug #81611 Add P_S metrics to collectcompressed page bytes vs other types written torelog

∙ Bug #78448 Provide better metrics oninnodb_sort_buffer_size usage

∙ Bug #71698 Add instrumentation for thedoublewrite buffer and undo segments

More tracing requests

14








∙ SHOW PROCESSLIST has multiple states

∙ Some of them are clear∙ But what do these mean?∙ Bug #57544∙ Bug#72083∙ Bug #84615

Vague stages

15




∙ SHOW PROCESSLIST has multiple states∙ Some of them are clear

∙ But what do these mean?∙ Bug #57544∙ Bug#72083∙ Bug #84615

Vague stages

15




∙ SHOW PROCESSLIST has multiple states∙ Some of them are clear∙ But what do these mean?

∙ System lock∙ statistics∙ freeing items∙ Sending data∙ cleaning up∙ closing tables∙ end

∙ Bug #57544∙ Bug#72083∙ Bug #84615

Vague stages

15




∙ SHOW PROCESSLIST has multiple states∙ Some of them are clear∙ But what do these mean?∙ Bug #57544

∙ Bug#72083∙ Bug #84615

Vague stages

15




∙ SHOW PROCESSLIST has multiple states∙ Some of them are clear∙ But what do these mean?∙ Bug #57544∙ Bug#72083

∙ Bug #84615

Vague stages

15




∙ SHOW PROCESSLIST has multiple states∙ Some of them are clear∙ But what do these mean?∙ Bug #57544∙ Bug#72083∙ Bug #84615

Vague stages

15




Summary

∙ Feature requests∙ Comments∙ Fixes

To better MySQL!

17

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

https://github.com/svetasmirnova

Thank you!

18

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

https://github.com/svetasmirnova

DATABASE PERFORMANCEMATTERS

What you wanted to know about MySQL, but could not find using inernal instrumentation only

Software

Transcript of What you wanted to know about MySQL, but could not find using inernal instrumentation only