Linux and Mysql Tuning

Getting the most from your hardware

The Ubiquitous “About Me”

Ran datacenters for: National Geographic ExpertCity (now Citrix Online) Fastclick (now Valueclick) University of California (now broke);

Now CEO/founder of LogicMonitor.com Probably just like you. (Not a kernel committer,

or poring over MySQL source code).

Your Mileage May Vary

Nothing I am going to say will have any impact in your environment.

Unless it does.

So...test, measure, rinse, repeat.

What to measure?

Everything! Database (innodb, MyISAM, replication), OS,

disk, application – all in as much detail as possible.

Measure YOUR application Get monitoring that can query db directly, and

slice/dice (registrations/sec; customer updates received, etc).

Also instrument front ends (via JMX, WMI, web status, etc) so you can measure effect on app of db adjustments

Talk outline

Kernel IO scheduler tuning InnoDB thread tuning Stopping swapping

Swappiness NUMA factors Huge pages

Why?

Limits are always one of CPU, RAM, Disk, Network.

But with better <CPU|RAM|Disk|network> the limit will move to next object

Default configurations may be for: Your server:

IO Scheduler Tuning

Kernel decides how to order disk operations, in order to combine IO ops into one op, and also minimize disk seeks, or not.

Defaults work. No benefit in other policies, only harm. Don't bother testing.

But, it's trivial to test, so don't take my word for it. With battery backed RAID cache, others have reported good results with noop.

Kernel, hardware dependent. Test!echo noop > /sys/block/sda/queue/scheduler

VM on ESX with no-cache RAID1 Switching from CFQ to Noop Amount of non-merged writes overwhelmed

disk.

Switch to NoOP Switch to NoOP

VM on ESX no-cache RAID1

Switch from anticipatory to CFQ at 13:10 CFQ has slightly more reads merged, slightly

better performance time.

Switch to CFQ

Lots of IO where ordering doesn't matter

4000 disk operations to SSD Switching from CFQ to NoOP – slightly worse

IO completion time.

Switch to NoOPSwitch to NoOP

But it saves CPU, right?

Not significantly, if at all.

Policy switched

Innodb Thread Concurrency

If database is slow, but CPU not busy...this may be issue.

Defaults range from 4 (in some distro's) to infinite, depending on version. 8 seems most common default in real world.

“Infinite” changes between versions. Default probably too low for “real” DB server.

Test! Benchmark first, then real life Use a threaded benchmark. sysbench --test=oltp --mysql-table-

engine=innodb --mysql-socket=/var/lib/mysql/mysql.sock --max-time=300 --db-driver=mysql --num-threads=10 --max-requests=300000 run

Use same benchmark, vary server config.

Results will vary with # users

How many user threads to test? Use real data.

My results, not yours

This is true on some hardware, with some version of MySQL, with 10 client threads.

Your mileage WILL vary. Test. Higher settings not always better.

More InnoDB threads – more efficient use of CPU

8 Innodb threads

4 Innodb Threads

16 Innodb Threads

∞ Innodb Threads

Swapping

Use of swap not bad – swapping is bad. OK (but not ideal):

Performance Killer:

Disable swap?

Erm … probably not. “Because new pages are always being read in

and processes are always allocating new memory, the OS will have to make a decision of what pages to evict from physical memory. If a page is dirty, it can only be evicted if there's swap. So if you have dirty pages that are very rarely used, swap allows you to keep more hot, clean pages in memory.”

http://kerneltrap.org/node/3202

Easy things first

echo 0 > /proc/sys/vm/swappiness echo “vm.swappiness = 0” >> /etc/sysctl.conf; sysctl -p

This may be enough. If so, KISS.

NUMA factors

System tries to allocate memory local to the CPU a process runs on.

In dual processor system, if one process tries to grab > 50% of system's memory, it will use all free memory local to one processor.

other process on that CPU requesting memory may cause swap activity to get local memory – even though there is free non-local memory.

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

Numa, II I couldn't reproduce:

Run mysql with affinity to a physical CPU, with buffer pool that consumes all local memory.

Start Java process with affinity to same CPU. Mem was allocated from other core. But.. no swap used.

But just in case...easy to address: numactl --interleave all mysqld

If you are allocating more than 50% memory to innodb (or any other single process), you are guaranteed non-local memory access. So may as well be deterministic, and possibly benefit from this swap-fix.

HugePages

Currently huge pages are not swapped (may change)

Will have some (probably very minor) performance benefit (big in some cases)

But.. more complex (and thus risky.) Need to enable in kernel, configure

kernel,ulimits, security groups, configure mysql. See http://dev.mysql.com/doc/refman/5.0/en/large-page-support.html

HugePages

Set nr_hugepages too low – Mysql wont use, and you just wasted memory. Ditto too high.

Pins Innodb buffer pool cache – but will increase pressure on other memory if swapping is triggered

Will reduce TLB cache lookups/misses. Does that matter? Test!

Summary My app/kernel/hardware/DB version is not

yours. YMMV. Test IO scheduler, as its so easy. Test InnoDB thread concurrency. Address swapping in the simplest way you can. Trend everything. All the time. Compare pre-

and post releases.

That's it!

Questions? sfrancis@logicmonitor.com Www.logicmonitor.com Blog.logicmonitor.com