Best Practices for Deploying MySQL on the Solaris Platform

8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform

http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 1/49



Agenda

MySQL -Solaris IntegrationMySQL High Availability Data ServiceConsolidate MySQL Deployment Using SolarisContainersDTrace (Jenny Chen)



MySQL-Solaris Integration



Optimized MySQL on Open Solaris

MySQL 5.0.45 (32bit) integrated with OpenSolaris build 76

SXDE 01/08MySQL 5.0.45 (64bit) integrated with OpenSolaris build 87

SXCE



Layout of MySQL on Open Solaris

MySQL 5.0.45 packages Default data directory

/var/mysql/5.0/dataDefault configuration directory

/etc/mysql/5.0Installation directory/usr/mysql/5.0/

Latest version accessible from /usr/mysql/binSymbolic link from all directories involved in 5.0.45release to corresponding /usr/mysql



Optimization Of MySQL On Open Solaris

Compiled with optimal compiler options usingSun Studio compilersAdded SMF support for MySQL

Easily accommodates varying configurationsInitializes MySQL database thereby enhancing user

experienceMysql user can manage MySQL database via SMF

Support for MySQL cluster engine (ndb)



Compiler Optimization

Enable in-lining Change header file univ.i to add Sun Studio

If (!defined(SUNPRO_C)#define UNIV_MUST_NOT_INLINE

Enable pre-fetching-xprefetch=auto and -xprefetch_level=3

Set optimization level-xO4



Service Management Facility (SMF)

Makes Solaris Services Self-HealingServices automatically restart in dependencyorder

Misbehaving and mis-configured services areeasier to debug

Log files for each serviceAdministrators can securely delegate tasks tonon-root users



MySQL-Service Management Facility

Dynamically configured properties mysql/bin

Installation , default : /usr/mysql/5.0/bin mysql/data

Data Directory , Default: /var/mysql/5.0/data mysql/enable_64bit

Flag to select 32bit or 64bit , Default : false

First time installations Creates system tables (mysql_install_db.sh )



Starting MySQL on Open Solaris

Older Open Solaris Build (Build 76- Build-87) 32Bit MySQL integrated Create mysql user Enable MySQL SMF service

Latest Open Solaris Build (Build 87 onwards) 32Bit and 64bit MySQL integrated Default mode : 32bit serviceFor starting 64bit Set enable_64bit == true



Optimized MySQL on Solaris 10

CoolStack SAMP stack + more

Optimized MySQL downloadStandalone packageSAMP stack component

Available at Sun Download center: http://cooltools.sunsource.net/coolstack

Version MySQL 5.0.45Similar optimizations as in Open Solaris

http://cooltools.sunsource.net/coolstack

http://cooltools.sunsource.net/coolstack



MySQL-High Availability Data Service



Solaris Cluster OverviewProvides general purpose HA platform

Fifty percent of enterprises that lack arecovery plan go out of business withinone year of a significant

disaster Gartner Group

Availability is our customers'most critical requirement Sun Cluster VOC Survey



Solaris Cluster : Hardware ComponentsServers with local storage

Can have up to 16 nodesShared storage

Tolerates single-node failures

Centralizes configuration files

Cluster interconnectAt least two redundant networks

Public network interfaces Spreads outbound packets



Solaris Cluster AlgorithmsCluster membership monitor Ensures data integrity

Determines cluster membershipCluster configuration repository

Global repositoryEnsures consistent view

Disk FencingFences off non-cluster nodes

Prevents PartitionQuorum

Uses a majority voting schema



MySQL High Availability Data ServiceHA-MySQL is a failover data service

Node1

Node2

Node

3

Node4

DDBB



MySQL High Availability Data ServiceSupported configurations

Standalone MySQL server

MySQL replication server

Single/Multiple MySQL instances in master configuration

Single/Multiple MySQL instances in slave configuration

Solaris containers supportGlobal zoneNon-global failover zoneNon-global zone



HA-MySQL Service Deployment

Node1 Node2:Zone2 Node1:Zone1 Node2

MySQL

Stor Host

MySQL

Stor Host



MySQL-Solaris Cluster BenefitsEnhanced end-to-end infrastructure availabilityContinuous MySQL Availability

Automatic failover if master node fails

Low cost solution Software is free and open sourced

Efficient Resource Utilization Multiple applications can be consolidated

Ease of operations SC enables clustered systems to be managed as if

they were on a single system



Additional information

Step by step deployment guide http://docs.sun.com/app/docs/doc/819-3059

Failover study of HA-MySQL http://blogs.sun.com/krishs/date/200804

Solaris Cluster http://www.sun.com/software/solaris/cluster/index.xmlhttp://opensolaris.org/os/community/ha-clusters/

http://docs.sun.com/app/docs/doc/819-3059

http://blogs.sun.com/krishs/date/200804

http://www.sun.com/software/solaris/cluster/index.xml

http://www.sun.com/software/solaris/cluster/index.xml

http://blogs.sun.com/krishs/date/200804

http://docs.sun.com/app/docs/doc/819-3059



Consolidate MySQL installations using SolarisContainers



Solaris ContainersContainers : Zones + Resource Management

Zones: isolated virtual application environmentsResource management – resource control (CPU,

Memory)

Achieving Consolidation Goals Reduce Hardware

Combine low utilization systems

Isolate applications from faults Maintain Service Levels

Fine tune response times



Dtrace (Jenny Chen)



DTrace

Use DTrace with MySQL to drill downMonitoring MySQL Performance

Examples & Solutions

Easy Steps To add DTrace probes intoMySQL core server and storage Engines

Easy Steps to display MySQL DTraceprobes into Chime visualization Tool for DTrace



Why DTrace

Solaris 10 Dynamic Tracing Facility to providecomprehensive view of operating system andapplication behaviour

> DTrace to examine particular system areas: disk I/O, CPU,

Memory> Process Tracing and Debugging

USDT (User-level statically defined tracing) place custom probes inapplication codeAdd USDT into MySQL source to monitor MySQL and gather the useful data missing by the current MySQL monitor tools:



DTrace: Monitor I/Os

Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)#!/usr/sbin/dtrace –s

#pragma D option quiet

io:::wait-start

/ execname == “mysqld” /

{ self->start = timestamp; }

io:::wait-done

/ execname == “mysqld” && self->start /

{ this->elapsed = timestamp - self->start;

@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);

self->start = 0;

}

profile:::tick-5s

{ printf(“-------------------------------------------------\n”);

printf(“%6s %8s %20s %50s\n”, “PID”, “TIME”, “DEVICE”, “FILE”);

printa(“%6d %@8d %20s %8s \n”, @files);

printf (“------------------------------------------------\n”); }

#./mysqliowait.d

---------------------------------------------------

PID TIME DEVICE FILE

113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1



DTrace: Monitor CPUclassic performance problem#!/usr/sbin/dtrace -s#pragma D option quietsyscall:::entry{

self->ts=vtimestamp;}syscall:::return

/self->ts/{

@a[execname, probefunc] = count();@b[execname, probefunc] = sum (vtimestamp - self->ts);self->ts=0;

}END{

printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","COUNT");printa("%-16s %-16s %-@8d\n",@a);printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","TIME");printa("%-16s %-16s %-@8d\n",@b);

}



# ./syscall.dEXEC SYSCALL COUNTdevfsadm lwp_park 1dtrace fstat 1...

mysqld read 106542mysqld gtime 109613mysqld pread 1181669 ---------------------------------------------------EXEC SYSCALL TIME

dtrace lwp_sigmask 218pkill getpid 302...mysqld read 259284183mysqld write 267556239

mysqld pread 4650457224



Agrregated user stack backtrace to understand of the nature of pread() inMySQL source code – useful for mysql developers

# dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() }

dtrace: description 'syscall::pread:entry ' matched 1 probe...

libc.so.1`_pread+0xamysqld`my_pread+0x54mysqld`_mi_read_static_record+0x67mysqld`mi_rnext+0x1fe

...mysqld`handle_one_connection+0x855libc.so.1`_thr_setup+0x67

1564811....

Replace high cost pread with

“--myisam_use_mmap=1”- useful for mysql DBA

A 94% performance improvement!



Dtrace: Memor y AnalysisCheck which process causes anonymous page in# dtrace -n anonpgin '{@[execname] = count()}'dtrace: description anonpgin matched 1 probesshd 2

vmstat 23mysqld 673

Use Dtrace to measure waiting for paging in

# ./ whospaging.d – available at Solaris Internals( http://www.solarisinternals.com/si/dtrace/ )Who's on cpu (milliseconds):

sshd 1vmstat 3mysqld 120sched 43210

Who's waiting for pagin (milliseconds):mysqld 239082

http://www.solarisinternals.com/si/dtrace/

http://www.solarisinternals.com/si/dtrace/



Dtrace Probes In MySQL

Provide deep view of internal MySQL core server and storage engines'operation & behaviour >Database information>Query execution latency>Index & table scan cost>Wait events inside MyISAM & Innodb>Deadlock information>Query cache hit/miss>And many more...

Speed resolution of performance bottlenecks with in database designand MySQL server configurations

Negligible performance overhead

Easy steps to create & insert your own Dtrace probes into MySQL

MySQL DTrace GUI Monitor Tool - Chime



Quer y execution Time

Enable Slow query log with “–log-slow-queries” requires re-startMySQL server

SQL statements with query execution time longer than“long_query_time” second in the log file

SQL statements generating most loads on the application maynot in slow query log

Replication query statements are not available in slow query log

Time spending by the query optimizer to generate query plan isnot available in slow query log

Using DTrace can get mising query execution information online



Inser t DTrace Probes Into MySQL

Step 1: Figure out what probes are needed to insert into thesource code

Step 2: Define MySQL Provider and probes# cat mysql_dtrace.d

provider mysql

{

probe query__execute__start(void *, char *, char *, const char *, char *);

probe query__execute__finish(void *, char *, char *, const char *, char *,int);

}

>Two Probes defined in the mysql provider >Note to use two underscore(__) translated to hypen automatically



Step 3: Define a header file “mysql_dtrace.h” with definitions for probes

dtrace -h -s mysql_dtrace.d

#ifndef _MYSQL_H

#define _MYSQL_H

#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) \

__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)

#define DTRACE_QUERY_EXECUTE_START_ENABLED() \

__dtraceenabled_mysql___query__execute__start()

extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char *, char *);

extern int __dtraceenabled_mysql___query__execute__start(void);

#endif



Step 4: Insert the probes into source code #include <mysql_dtrace.h>

... bool mysql_execute_command(THD *thd)

{

DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,

thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);

...

DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,

thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,

thd->query, res ==0 ? 0: -1);

go to end;...

}



Step 5: Build MySQL with DTrace In the Makefile.in, compile 64-bit MySQL with Dtrace

mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)

mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)

dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)

Inserting DTrace probes comleted , DTrace probes are ready to use!



Step 6: Use inserted DTrace probes to measure query executiontime with other database information

#!/usr/sbin/dtrace -s

…mysql*:::query-execute-start

{ self->start = timestamp; }

mysql*:::query-execute-finish

/self->start/

{ this->query = copyinstr(arg4); }

mysql*:::query-execute-finish

/self->start/

{ this->elapsed = (timestamp - self->start) / 1000000;

this->who = strjoin(copyinstr(arg2), strjoin("@", copyinstr(arg3)));

printf(" %-16.16s %-18.18s %5d %3d %-32.32s\n", arg1 ? copyinstr(arg1)

: ".", this->who, this->elapsed, (int)arg5, this->query); self->start = 0;

}



# ./mysqld_qestat.d

DATABASE USER@HOST ms RET QUERY

sbtest root@localhost 0 0 show tablessbtest root@localhost 0 0 show databasessbtest root@localhost 178 0 select * from sbtest

Use the same steps to insert Dtrace probes to<source_tree>/sql/sql_select.cc at the start and end of choose_plan()function to measure the time spent in query optimization

optimizer_prune_level=1 reduce query compilation time

Reduce optimizer_search_depth or optimizer_search_depth=0



Index & Table scan cost

Identify the places in MySQL source to handle scanning index, andtable>Index-scan functions: index_next, index_next_same, index_prev,index_first, index_last>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos>Insert DTrace Probes at the start and before return from the functions

can measure the time spending on scanning table or index.mysql*:::innodb-index-next-start{ @indexnext[args[0]] = count();

self->inext = timestamp; }mysql*:::innodb-index-next-finish

/self->inext/{ @indexnexttime[args[0]] = sum(timestamp - self->inext);self->inext = 0; }

Expensive index-scan/table-scan report from Dtrace requires tooptimize schema accordingly



Buffer wait in Innodb Innodb buffer wait is common in I/O-bound MySQL system while reading

page synchronous from disk Insert Dtrace probes at: innobase/buf/buf0rea.c,

ulint buf_read_page( ulint space, ulint offset){ ...

DTRACE_INNODB_BUFFER_WAIT_START();

/* We do the i/o in the synchronous aio mode to save threadswitches: hence TRUE */count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space,

tablespace_version, offset);DTRACE_INNODB_BUFFER_WAIT_FINISH();

...}

Increase innodb_buffer_pool size Tune SQL to access rows with fewer block reads(i.e. By adding indexes)



Query cache probes

Use DTrace to measure query cache hit and query cache miss todetermine how well the query cache is performing > Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client

> DTrace Test script to report query cache hit and miss counts by query string, andtotals

mysql*:::query-cache-hit,

mysql*:::query-cache-miss

{ this->query = copyinstr(arg4); }

mysql*:::query-cache-hit { @elapsed[this->query, "hit"] = count();

hits++; }

mysql*:::query-cache- miss{ @elapsed[this->query, "miss"] = count();

misses++; }



DTrace Performance Impact

Inserting DTrace Probes into MySQL source code are useful for MySQL DBA, MySQL & application developers. Theperformance impact of adding DTrace probes is critical for enterprise environment.

Cost of inserting USDT probe can be basically negligible:

Each probe inserted into the source code can be enabled by adding thecode like:

if (PROVIDER_PROBE_ENABLED(){

PROVIDER_PROBE(arg0,...);}



DTrace Probes In MySQL 6.0

probe insert_row_start(); probe insert_row_end(); probe filesort_start(); probe filesort_end(); probe delete_start(); probe delete_end(); probe select_start() probe select_end();

probe update_start(); probe update_end();

compile with “--enable-dtrace” configure option to usethe Dtrace probes



Integrate with Chime Tool Chime is graphical tool for visualizing DTrace aggregations. It

provides alternative CLI-based tool output with more visuallyapprealing and more useful to display data over time

Available to download:http://www.opensolaris.org/os/project/dtrace-chime . One-stepinstallation: > Run pkgadd -d osol0chime-<arch>-1.4.pkg

chime requires Solaris Nevada build 35 or later > Run local: /opt/OSOL0chime/bin/chime > Run remotely: /opt/OSOL0chime/bin/chime <hostname>

chime provides wizard to automatically generate new display for DTrace script

http://www.opensolaris.org/os/project/dtrace-chime

http://www.opensolaris.org/os/project/dtrace-chime



Resources> http:// www.opensolaris.org / os/community/dtrace

- OpenSolaris Community: Dtrace

> http://developers.sun.com/solaris/articles/solaris_perftools.html - Solaris Performance and Tools > http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view

- Statically Defined Tracing for User Applications chapter of DTraceManual

> http://www.brendangregg.com- DTrace toolkits

>http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases

- DTrace Topics Databases

Acknowledgements Brandan Gregg – Sun Microsystems Engineer in Advanced ProductsGroup

http://www.opensolaris.org/


http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view

http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view


Best Practices for Deploying MySQL on the Solaris Platform

Documents

Transcript of Best Practices for Deploying MySQL on the Solaris Platform