15 protips for mysql users pfz

1

15 Pro-tips for MySQL UsersPFCongress - 17 september 2011Utrecht - Netherlands

whoami

2

Joshua ThijssenSenior Software Engineer @ EnriseDevelopment in PHP, Python, Perl, C, Java....

Blogs:http://www.adayinthelifeof.nlhttp://www.enrise.com/blogEmail: [email protected]

Twitter: @jaytaph

3

What are we going to discuss?

‣ QUESTIONS? RAISE YOUR HAND OR YELL LOUD

‣ 15 MySQL Pro-tips

‣ No “theoretical tips”, all taken from the field.

‣ Starting simple - ending “complex”

4

Tip 0

‣ MySQL 5.0 != 5.5

0) Use the correct MySQL version

5

Tip 0: Use the correct MySQL version

‣ RHEL 5-7: 5.0.77

‣ RHEL 6-1: 5.1.52

‣ Debian (lenny) 5.0.51a

‣ Debian (squeeze): 5.1.49

‣ Debian (sid): 5.1.58

http://distrowatch.com/table.php?distribution=redhathttp://distrowatch.com/table.php?distribution=debian

6

Tip 1

‣ EXPLAIN IS YOUR BESTEST FRIEND

1) Know how to use explain (and profiler)

7

Tip 1: Know your EXPLAIN (1)

‣ I will not show you how to use EXPLAIN.

‣ Use EXPLAIN and EXPLAIN EXTENDED/ SHOW WARNINGS;

8


mysql> desc varchartest;+----------+--------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+----------+--------------+------+-----+---------+-------+| id | int(11) | NO | PRI | 0 | || name | varchar(255) | NO | MUL | NULL | || utf8name | varchar(255) | NO | MUL | NULL | |+----------+--------------+------+-----+---------+-------+3 rows in set (0.01 sec)

mysql> desc varchartest;+----------+--------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+----------+--------------+------+-----+---------+-------+| id | int(11) | NO | PRI | 0 | || name | varchar(255) | NO | MUL | NULL | || utf8name | varchar(255) | NO | MUL | NULL | |+----------+--------------+------+-----+---------+-------+3 rows in set (0.01 sec)

10


mysql> SHOW WARNINGS\g+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Level | Code | Message |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Note | 1003 | select `pfz`.`varchartest`.ìd` AS ìd`,`pfz`.`varchartest`.`name` AS `name`,`pfz`.`varchartest`.ùtf8name` AS ùtf8name` from `pfz`.`varchartest` where (`pfz`.`varchartest`.`name` like 'joshua') |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> SHOW WARNINGS\g+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Level | Code | Message |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Note | 1003 | select `pfz`.`varchartest`.ìd` AS ìd`,`pfz`.`varchartest`.`name` AS `name`,`pfz`.`varchartest`.ùtf8name` AS ùtf8name` from `pfz`.`varchartest` where (`pfz`.`varchartest`.`name` like 'joshua') |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

11

Tip 1: Know your PROFILER (1)

mysql> SET profiling=1;mysql> SELECT * FROM table;mysql> SHOW PROFILE CPU, BLOCK IO;+--------------------------------+----------+----------+------------+--------------+---------------+| Status | Duration | CPU_user | CPU_system | Block_ops_in | Block_ops_out |+--------------------------------+----------+----------+------------+--------------+---------------+| starting | 0.000930 | 0.000000 | 0.000000 | 0 | 8 || checking query cache for query | 0.000547 | 0.000000 | 0.000000 | 0 | 0 || checking permissions | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || Opening tables | 0.000142 | 0.000000 | 0.000000 | 0 | 0 || System lock | 0.000030 | 0.000000 | 0.000000 | 0 | 0 || Table lock | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || init | 0.000207 | 0.000000 | 0.000000 | 0 | 0 || optimizing | 0.000007 | 0.000000 | 0.000000 | 0 | 0 || statistics | 0.000068 | 0.000000 | 0.000000 | 0 | 0 || preparing | 0.001393 | 0.004000 | 0.000000 | 0 | 0 || Creating tmp table | 0.001658 | 0.000000 | 0.000000 | 0 | 8 || executing | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || Copying to tmp table | 0.000834 | 0.000000 | 0.004001 | 0 | 8 || Sorting result | 0.000179 | 0.000000 | 0.000000 | 0 | 0 || Sending data | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000022 | 0.000000 | 0.000000 | 0 | 0 || removing tmp table | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || query end | 0.000004 | 0.000000 | 0.000000 | 0 | 0 || freeing items | 0.000844 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000023 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000060 | 0.000000 | 0.000000 | 0 | 8 || cleaning up | 0.000007 | 0.000000 | 0.000000 | 0 | 0 |+--------------------------------+----------+----------+------------+--------------+---------------+

mysql> SET profiling=1;mysql> SELECT * FROM table;mysql> SHOW PROFILE CPU, BLOCK IO;+--------------------------------+----------+----------+------------+--------------+---------------+| Status | Duration | CPU_user | CPU_system | Block_ops_in | Block_ops_out |+--------------------------------+----------+----------+------------+--------------+---------------+| starting | 0.000930 | 0.000000 | 0.000000 | 0 | 8 || checking query cache for query | 0.000547 | 0.000000 | 0.000000 | 0 | 0 || checking permissions | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || Opening tables | 0.000142 | 0.000000 | 0.000000 | 0 | 0 || System lock | 0.000030 | 0.000000 | 0.000000 | 0 | 0 || Table lock | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || init | 0.000207 | 0.000000 | 0.000000 | 0 | 0 || optimizing | 0.000007 | 0.000000 | 0.000000 | 0 | 0 || statistics | 0.000068 | 0.000000 | 0.000000 | 0 | 0 || preparing | 0.001393 | 0.004000 | 0.000000 | 0 | 0 || Creating tmp table | 0.001658 | 0.000000 | 0.000000 | 0 | 8 || executing | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || Copying to tmp table | 0.000834 | 0.000000 | 0.004001 | 0 | 8 || Sorting result | 0.000179 | 0.000000 | 0.000000 | 0 | 0 || Sending data | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000022 | 0.000000 | 0.000000 | 0 | 0 || removing tmp table | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || query end | 0.000004 | 0.000000 | 0.000000 | 0 | 0 || freeing items | 0.000844 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000023 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000060 | 0.000000 | 0.000000 | 0 | 8 || cleaning up | 0.000007 | 0.000000 | 0.000000 | 0 | 0 |+--------------------------------+----------+----------+------------+--------------+---------------+

12

Tip 2

‣ THERE ARE ONLY A FEW “BASIC” ONES.

2) Know the most basic my.cnf settings

13

Tip 2: My.cnf settings (1)

Know the most important ones:key_buffer_size, innodb_buffer_pool_size, sort_buffer_size, max_connections

14


‣ Some settings work on global level, some per connection!

‣ Know some quirks: (max_heap_table_size vs tmp_table_size, binlog-do-db, replicate-ignore-db etc)

15


‣ http://www.omh.cc/mycnf/

‣ http://rackerhacker.com/mysqltuner/

‣ http://www.day32.com/MySQL/

‣ phpmyadmin

16

Tip 3

‣ RESTORING JUST ONE TABLE CAN BE PAINFUL OTHERWISE

3) Backup on table level

17

Tip 3: Backup on table level (1)

‣ COULD YOU RESTORE TABLE x? YES! YES I CAN!

‣ mysqldump can dump per database OR by table.

‣ Simple scripts to scan/dump tables.

‣ Easy restore for single table (or part of table)

18

Tip 4

‣ DON’T ASK WHAT YOU DON’T NEED

4) Don’t use “SELECT *” when you only need one or two fields.

19

Tip 4: Select * (1)

‣ DON’T ASK WHAT YOU DON’T NEED

‣ Much more data to be read from disk

‣ Much more data will be send over, thus slower (blobs/texts)

‣ Cannot use covering indices

20

Tip 4: Select * (2)

mysql> SHOW FULL COLUMNS FROM `covering`;+--------------+------------------+------------------+------+-----+---------+----------------+| Field | Type | Collation | Null | Key | Default | Extra |+--------------+------------------+------------------+------+-----+---------+----------------+| id | int(10) unsigned | NULL | NO | PRI | NULL | auto_increment || email | varchar(255) | ascii_general_ci | NO | MUL | NULL | | | want_mailing | tinyint(1) | NULL | NO | MUL | NULL | || extra_info | varchar(255) | ascii_general_ci | NO | MUL | NULL | |+--------------+------------------+------------------+------+-----+---------+----------------+3 rows in set (0.00 sec)mysql> SHOW INDEXES FROM `covering`;+----------+------------+------------+--------------+--------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | +----------+------------+------------+--------------+--------------+-----------+-------------+| covering | 0 | PRIMARY | 1 | id | A | 3 || covering | 1 | idx_email | 1 | email | A | 3 || covering | 1 | idx_email | 2 | want_mailing | A | 3 || covering | 1 | idx_email2 | 1 | want_mailing | A | 1 || covering | 1 | idx_email2 | 2 | email | A | 3 |+----------+------------+------------+--------------+--------------+-----------+-------------+5 rows in set (0.01 sec)

mysql> SHOW FULL COLUMNS FROM `covering`;+--------------+------------------+------------------+------+-----+---------+----------------+| Field | Type | Collation | Null | Key | Default | Extra |+--------------+------------------+------------------+------+-----+---------+----------------+| id | int(10) unsigned | NULL | NO | PRI | NULL | auto_increment || email | varchar(255) | ascii_general_ci | NO | MUL | NULL | | | want_mailing | tinyint(1) | NULL | NO | MUL | NULL | || extra_info | varchar(255) | ascii_general_ci | NO | MUL | NULL | |+--------------+------------------+------------------+------+-----+---------+----------------+3 rows in set (0.00 sec)mysql> SHOW INDEXES FROM `covering`;+----------+------------+------------+--------------+--------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | +----------+------------+------------+--------------+--------------+-----------+-------------+| covering | 0 | PRIMARY | 1 | id | A | 3 || covering | 1 | idx_email | 1 | email | A | 3 || covering | 1 | idx_email | 2 | want_mailing | A | 3 || covering | 1 | idx_email2 | 1 | want_mailing | A | 1 || covering | 1 | idx_email2 | 2 | email | A | 3 |+----------+------------+------------+--------------+--------------+-----------+-------------+5 rows in set (0.01 sec)

22

Tip 5

‣ ENFORCE CONSISTENCY

5) Use triggers and stored procedures

23

Tip 5: Triggers and stored procedures (1)

‣ ENFORCE CONSISTENCY

‣ 6 triggers per table (insert, update, delete, before and after the mutation)

‣ 3rd party tools (phpmyadmin etc) can also use the database without loosing data consistency.

‣ Watch out with (phpmyadmin) table dumps!

24

Tip 6

‣ THERE ARE MUCH BETTER SOLUTIONS

6) Don’t use FULLTEXT searches

25

Tip 6: Don’t use FULLTEXT search (1)


‣ They only work for MyISAM tables.

‣ Not compatible with other DB’s.

‣ Slow (especially compared to Solr, Sphinx).

‣ No extra features (faceted search, spell checking etc).

26

Tip 7

‣ IT LOOKS LIKE YOU NEED MORE ADVANCED SEARCH?

7) Some wildcard searches (%item%) are bad for performance

27

Tip 7: Wildcard searches (1)


‣ MySQL cannot use indexing!

‣ Revert your data: search for ‘moc.esirne@%’ instead of ‘%@enrise.com’.

‣ Use a better solution (solr, sphinx). You probably want it (no, really)

28

Tip 8

‣ MAKE CACHING AND LOCKING HAPPY AGAIN

8) Shard your volatile and non-volatile data.

29

Tip 8: Sharding (1)

mysql> SHOW FULL COLUMNS FROM `pages`;+------------+------------------+-------------------+------+-----+-------------------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+------------+------------------+-------------------+------+-----+-------------------+-------+| page_id | int(10) unsigned | NULL | NO | PRI | NULL | || created_dt | timestamp | NULL | NO | | CURRENT_TIMESTAMP | || creator_id | int(11) | NULL | NO | | NULL | || title | varchar(100) | latin1_swedish_ci | NO | | NULL | || contents | text | latin1_swedish_ci | NO | | NULL | || hit_count | int(11) | NULL | NO | | 0 | |+------------+------------------+-------------------+------+-----+-------------------+-------+6 rows in set (0.01 sec)

mysql> SHOW FULL COLUMNS FROM `pages`;+------------+------------------+-------------------+------+-----+-------------------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+------------+------------------+-------------------+------+-----+-------------------+-------+| page_id | int(10) unsigned | NULL | NO | PRI | NULL | || created_dt | timestamp | NULL | NO | | CURRENT_TIMESTAMP | || creator_id | int(11) | NULL | NO | | NULL | || title | varchar(100) | latin1_swedish_ci | NO | | NULL | || contents | text | latin1_swedish_ci | NO | | NULL | || hit_count | int(11) | NULL | NO | | 0 | |+------------+------------------+-------------------+------+-----+-------------------+-------+6 rows in set (0.01 sec)

30

‣ Remember: an update on a table will invalidate ALL caches referring to that table.

‣ UPDATE pages SET hit_count = hit_count + 1;

‣ Thus: page table will NEVER be cached.

Tip 8: Sharding (2)

31

‣ Define hot data (volatile, changes often) and cold data (static, changes never or infrequently)

‣ move to different tables

‣ UPDATE page_stats SET hit_count = hit_count + 1;

‣ Query cache is happy again

Tip 8: Sharding (3)

32

‣ MySQL 5.1 and higher has got partitioning (but you should shard anyway).

Tip 8: Sharding (4)

33

Tip 9

‣ PK’S ARE ON EVERY INDEX

9) Don’t use a large primary key for InnoDB tables.

34

Tip 9: Large primary keys (1)

‣ InnoDB adds the primary key to EACH index.

‣ No primary key given? It uses an internal 6(!)-byte key.

35

Tip 10

‣ COUNT(*) => MYISAM = FAST‣ COUNT(*) => INNODB = SLOW

10) Don’t “SELECT COUNT(*) FROM TABLE” on InnoDB.

36

Tip 10: SELECT COUNT(*) (1)

‣ InnoDB implements MVCC (multi-version concurrency control).

‣ COUNT(*) must be counted and is not fetched from metadata.

37

‣ What do you want to COUNT(*)?

‣ Just for displaying purposes (there are X amount of pages): do you need the EXACT amount? (guess &| cache)

Tip 10: SELECT COUNT(*) (2)

38

Tip 11

‣ IT ISN’T THAT VARIABLE AS YOU MIGHT THINK

11) Don’t rely on the VARCHAR()

39

Tip 11: VARCHAR() (1)

‣ IT DOESN’T SOLVE YOUR PROBLEMS

40

‣ DON’T WORRY ABOUT THE UTF-8, I’LL BASH THAT LATER


mysql> SHOW FULL COLUMNS FROM `varchartest`;+----------+--------------+-------------------+------+-----+---------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+----------+--------------+-------------------+------+-----+---------+-------+| id | int(11) | NULL | NO | PRI | 0 | || name | varchar(255) | latin1_swedish_ci | NO | MUL | NULL | || utf8name | varchar(255) | utf8_general_ci | NO | MUL | NULL | |+----------+--------------+-------------------+------+-----+---------+-------+3 rows in set (0.01 sec)mysql> SHOW INDEXES FROM `varchartest`;+-------------+------------+--------------+--------------+-------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality |+-------------+------------+--------------+--------------+-------------+-----------+-------------+| varchartest | 0 | PRIMARY | 1 | id | A | 3 || varchartest | 1 | idx_name | 1 | name | A | 3 || varchartest | 1 | idx_utf8name | 1 | utf8name | A | 3 |+-------------+------------+--------------+--------------+-------------+-----------+-------------+3 rows in set (0.00 sec)


42

Tip 12

‣ DON’T EXCHANGE ONE PROBLEM FOR ANOTHER

12) UTF-8 is not the enemy, but it certainly isn’t your friend.

43

Tip 12: UTF-8 (1)

‣ Dr Jeckyl and Mr Hyde

‣ Solves all your multi-language problems!

‣ But gives back performance issues.

44

Tip 12: UTF-8 (1)



46

ALL temporary buffers are allocated for worst-case scenario’s.

This means a varchar(255) in UTF-8 uses 255*3 + 2 = 767 bytes PER row, even if you have only 1 single char inside.

Tip 12: UTF-8 (1)

47

Tip 13

13) Know your cardinality & selectivity

48

Tip 13: Cardinality & Selectivity (1)

‣ Cardinality: the number of unique entries inside the index.

‣ Selectivity: percentage of unique entries.

‣ S(I) = cardinality / count * 100%

49


50

‣ with 10 records: 5/10 * 100% = 50%

‣ with 1000 records: 75/1000 * 100 = 7.5%

‣ with 10.000 records: 200/10000 * 100% = 2%


country_id (max +-200, but effectively +- 50, maybe less)

51

‣ A selectivity < 30% ? Full table scan!

‣ ANALYZE TABLE frequently.


52

‣ Adding records changes your cardinality and thus selectivity.

‣ Develop against a “real” dataset.


53

Tip 14

‣ NOW(), RAND(), UUID(), CONNECTION_ID() ETC..

14) Non-deterministic functions do not go well with query caching

54

Tip 14: Query caching (1)

mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16768400 || Qcache_hits | 3860 || Qcache_inserts | 975 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 0 || Qcache_total_blocks | 1 |+-------------------------+----------+8 rows in set (0.00 sec)

mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16768400 || Qcache_hits | 3860 || Qcache_inserts | 975 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 0 || Qcache_total_blocks | 1 |+-------------------------+----------+8 rows in set (0.00 sec)

mysql> select * from varchartest;+----+------+----------+| id | name | utf8name |+----+------+----------+| 1 | j | joshua || 2 | j | jeroen || 3 | d | david |+----+------+----------+3 rows in set (0.00 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3860 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)


55






56




mysql> select * from varchartest ORDER BY RAND();+----+------+----------+| id | name | utf8name |+----+------+----------+| 2 | j | jeroen || 1 | j | joshua || 3 | d | david |+----+------+----------+3 rows in set (0.05 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3861 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 487 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)

mysql> select * from varchartest ORDER BY RAND();+----+------+----------+| id | name | utf8name |+----+------+----------+| 2 | j | jeroen || 1 | j | joshua || 3 | d | david |+----+------+----------+3 rows in set (0.05 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3861 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 487 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)

57

SELECT * FROM table WHERE YEAR(created_dt) < YEAR(NOW());

vs

SELECT * FROM table WHERE YEAR(created_dt) < ‘2010’;


58

Tip 15

‣ AND GET SOME NICE TITLES WHILE YOU’RE AT IT...

15) Certify yourself as a DBA and/or DBE.

59

Tip 15: Certify yourself (1)

‣ THEY ARE NOT VERY EASY EXAMS, BUT WELL WORTH IT

‣ Oracle Certified MySQL Associate

‣ Oracle Certified Professional MySQL 5.0 Developer

‣ Oracle Certified Professional MySQL 5.0 Database Administrator

‣ Oracle Certified Expert, MySQL 5.1 Cluster Database Administrator

‣ “Old”, but still, get them all!

60

Let’s summarize

(1) Know how to use explain.

(2) Know the most basic my.cnf settings.

(3) Backup on table level.

(4) Don’t use “SELECT *” when you only need 1 or 2 fields.

(5) Use triggers and stored procedures.

(6) Don’t use FULLTEXT searches.

(7) Wildcard searches (%item%) are bad for performance.

(8) Shard your volatile and non-volatile data.

(9) Don’t use a large Primary Key for InnoDB tables.Don’t “Select COUNT(*)” on InnoDB.

(10) Don’t rely on the VARCHAR().

(11) UTF-8 is not the enemy, but it certainly isn’t your friend.Know your cardinality & selectivity.Non-deterministic functions do not go well with query caching. Certify yourself as a DBA and/or DBE.

Any questions?

∂ QUESTIONS?

http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg

Daycamp for developers

http://www.enrise.comhttp://www.daycamp4developers.com/

63

‣ THANK YOU FOR YOUR ATTENTION‣ Please rate my talk: http://joind.in/3662

15 protips for mysql users pfz

Technology

Transcript of 15 protips for mysql users pfz