Tuning Web 2.0 Applications using Dtrace

1

Dtrace Web2.0Java, AJAX, PHPand the rest

Peter KarlssonSolaris Technology EvangelistSun Microsystems

1

2

• Why should you care??• Introduction to Dtrace• DTrace and AJAX

> DTrace and JavaScript> Dtrace on the BackEnd

– Java, PHP, Phyton, Ruby etc.> DTrace and Postgres

• Putting it all together• Summary and Resources

Agenda

3

Why should you care

• Isn't DTrace a Solaris thing, kernel nerds and such

• Nope, Dtrace now have providers for a large number of languages used in AJAX, such as:> JavaScript> Java> PHP> Phyton> Ruby

• As well as for some databases such as Postgres

4

Introduction to DTrace

• DTrace was introduced in Solaris 10• Allows for dynamic instrumentation of the

OS and applications> A typical Solaris 10 system has around 50K probe points plus any

probes that can be dynamically inserted in to ANY application running on the system

• Available on any Solaris 10 system• Comes with a new dynamic language, D

> D is used to scrip instrumentation wether it's for your application or you are looking at the system it self

5

Introduction to DTrace, (Cont.)

• Designed explicitly for use on production systems• Zero performance impact when not in use• Desgined with safty in mind, includes safegardes against

panics, crashes, data corruption or pathological performance degradation

• Powerful data management primitives eliminate need for most postprocessing -> aggregations

• Unwanted data is pruned as close to the source as possible

6

The DTrace Revolution

• DTrace tightens the diagnosis loop:

hypothesis → instrumentation → data gathering → analysis → hypothesis ........

• Tightened loop effects a revolution in the way we diagnose transient failure

• Focus can shift from instrumentation stage to hypothesis stage:> Much less labor intensive, less error prone> Much more brain intensive> Much more effective! (And a lot more fun)

7

In a nutshell : DTrace architecture

libDTrace(3LIB)

DTrace(7D)

DTrace

userlandkernel

DTrace(1M)lockstat(1M)

plockstat(1M)

script.d

DTraceconsumers

sysinfo vminfo fasttrapsdtsyscall fbtproc

DTraceproviders

8

The D Language

• D is a C-like language specific to DTrace with some constructs similar to awk(1)

• Global, thread-local and probe-local variables• Built-in variables like execname and timestamp• Predicates can use arbitrary expressions to select which data

is traced and which is discarded• Actions to trace data, record stack backtraces, stop

processes at points of interest, etc.

9

Providers

• A provider allows for instrumentation of a particular area of the system

• Providers make probes available to the framework• Providers transfer control to the DTrace framework when an

enabled probe is hit• DTrace has several providers, e.g.:

> The pid provider for C and C++ applications> The hotspot and dvm provider for Java applications> The syscall provider for system calls> The io provider for system I/O> The profile provider for cyclical events

10

D Language - Format.

probe description/ predicate /{

action statements}

• When a probe fires then action is executed if predicate evaluates true

• Example, “Print all the system calls executed by bash”

#!/usr/sbin/dtrace -ssyscall:::entry/execname==”bash”/{

printf(“%s called\n”,probefunc);}

11

Aggregations

• Often the patterns are more interesting than each individual datum

• Want to aggregate data to look for larger trends• DTrace supports the aggregation of data as a first class

operation• An aggregation is the result of an aggregating function

> count(), min(), max(), avg(), quantize()• May be keyed by an arbitrary tuple

12

Aggregation - Format

• @name[keys] = aggfunc(args);• '@' - key to show that name is an aggregation.• keys – comma separated list of D expressions.• aggfunc could be one of...

> sum(expr) – total value of specified expression > count() – number of times called.> avg(expr) – average of expression> min(expr)/max(expr) – min and max of expressions> quantize()/lquantize() - power of two & linear distribution

13

Aggregation Example#!/usr/sbin/dtrace -spid$target:libc:malloc:entry{ @["Malloc Distribution"]=quantize(arg0);}$ aggr_malloc.d -c whodtrace: script './aggr2.d' matched 1 probe...dtrace: pid 1401 has exited

Malloc Distribution value ------------- Distribution -------- ------------------------------------------count 1 | 0 2 |@@@@@@@@@@@@@@@@@ 3 4 | 0 8 |@@@@@@ 1 16 |@@@@@@ 1 32 | 0 64 | 0 128 | 0 256 | 0 512 | 0 1024 | 0 2048 | 0 4096 | 0 8192 |@@@@@@@@@@@ 2 16384 | 0

14

Calculating time spent

• One of the most common request is to find time spent in a given function

• Here is how this can be done#!/usr/sbin/dtrace -ssyscall::open*:entry,syscall::close*:entry{ ts=timestamp;}

syscall::open*:return,syscall::close*:return{ timespent = timestamp - ts; printf("ThreadID %d spent %d nsecs in %s", tid, timespent, probefunc); ts=0; /*allow DTrace to reclaim the storage */ timespent = 0;}

Whats wrong with this??

15

Thread Local Variable

• self->variable = expression;> self – keyword to indicate that the variable is thread local> A boon to multi-threaded debugging> As name indicates this is specific to the thread.

– See code re-written#!/usr/sbin/dtrace -ssyscall::open*:entry,syscall::close*:entry{ self->ts=timestamp;}

syscall::open*:return,syscall::close*:return{ timespent = timestamp - self->ts; printf("ThreadID %d spent %d nsecs in %s", tid, timespent, probefunc); self->ts=0; /*allow DTrace to reclaim the storage */ timespent = 0;}

16

Built-in Variable

• Here are a few built-in variables.arg0 ... arg9 – Arguments represented in int64_t formatargs[ ] - Arguments represented in correct type based on function.cpu – current cpu id.cwd – current working directoryerrno – error code from last system callgid, uid – real group id, user idpid, ppid, tid – process id, parent proc id & thread id probeprov, probemod, probefunc, probename - probe info.timestamp, walltimestamp, vtimestamp – time stamp nano sec from an arbitary point and nano sec from epoc.

17

DTrace for scripting languages

Dynamic Tracing for Dynamic development

18

DTrace and Scripting Language

• DTrace has been integrated into many scripting languages. • We will see two samples.

> JavaScript– Using Mozilla Firefox 3.0b3

> PHP– For this exercise we will use Coolstack version of PHP.– Get coolstack from: http://cooltools.sunsource.net/coolstack/

• Please note the DTrace providers are experimental and not official

19

DTrace for Javascript

Get look in to the fire, Firefox that is.......

20

DTrace and Javascript

• DTrace probes have been added to Mozilla to help observe Javascript application.

• Here is the list of the probes> execute-start and execute-done> function-entry & function-return> object-create, object-create-start, object-create-done &

object-finalize> layout-start & layout-end

21

DTrace and Javascript

• Here is an example that prints java script function flow.

#!/usr/sbin/dtrace -ZFsjavascript*:::_function-entry{ trace(copyinstr(arg2));}

javascript*:::function-return{ trace(copyinstr(arg2));}

22

Lets look at some JavaScript with DTrace

23

DTrace for PHP

D tracing for P Hyper Processor

24

DTrace and PHP

• DTrace provider for PHP is a PECL/PEAR module• To get the provider

> # pear install dtrace– or

> # pecl install dtrace– then add

> “extension=dtrace.so” to the php.ini file– then

> restart PHP

25

DTrace and PHP

• Couple links for more information• Wez Furlong who created the provider has details in

http://netevil.org/node.php?uuid=42f82a8b-09b7-5013-1667-2f82a8b24ead• Bryan Cantrill has a nice demo script in his blog.

http://blogs.sun.com/bmc/entry/dtrace_and_php_demonstrated

26

DTrace and PHP

• There are two probes in the provider.> function-return> function-entry

• You can use the 5 args in the action.> arg0 = the function name> arg1 = the filename> arg2 = the line number> arg3 = classname (or an empty string)> arg4 = object/class operator (::, ->, or an empty string)

27

DTrace and PHP

• Here is a PHP example

<?php$numprimes=10000; $pArray[0]=3; $test=5; $num=0;function doesDevide($x,$y) { return($x % $y); }function isprime($x){ global $pArray; global $num; $index=0; $check=1; while($check==1 && $index <= $num && $x >= ($pArray[$index] * $pArray[$index]) ) { if( doesDevide($x , $pArray[$index]) == 0) { $check=0; } else $index++; } return($check);}

prime.php

28

DTrace and PHP

• Here is a PHP example(cont)

while($num<$numprimes){ if(isPrime($test)==1){ $num++; $pArray[$num]=$test; if($num%1000==0){ printf("Progress done ...%d\n",$num); } } $test+=2;}

?>

29

DTrace and PHP

• Here is a simple D-Script

#!/usr/sbin/dtrace -Zqs

php*:::function-entry{ @[copyinstr(arg0)]=count();}

• Note: -Z will allow probes that have zero match. • Run this script in one window while you run the php script in

another window.

30

DTrace and PHP

• Example run:

./php.d ^C printf 10 isprime 52378 doesDevide 684216

31

DTrace & Postgres

Observe post deploy of PostgreSQL

32

DTrace probes in Postgres

• Postgres 8.2 and later has following embedded D probes.> probe transaction__start(int);> probe transaction__commit(int);> probe transaction__abort(int);> probe lwlock__acquire(int, int);> probe lwlock__release(int); > probe lwlock__startwait(int, int);> probe lwlock__endwait(int, int);> probe lwlock__condacquire(int, int);> probe lwlock__condacquire__fail(int, int);> probe lock__startwait(int, int);> probe lock__endwait(int, int);

33

How to get Postgres with DTrace

• Get Solaris Express Developer Edition 9/07• OR • Download the source

> For 32-bit version– ./configure –enable-dtrace

> For 64 bit:– $ configure CC='gcc -m64' –enable-dtrace \ DTRACEFLAGS='-64'– $ configure CC='/opt/SUNWspro/bin/cc -xtarget=native64'\ --enable-dtrace

DTRACEFLAGS='-64'> Run make or gmake.

34

Get started with Postgres

• Here's how you'd run Postgres 8.2:• 1) As root, su to postgres

# su – postgres• 2) Create Postgres DB cluster

$ /usr/postgres/8.2/bin/initdb -D /var/postgres/8.2/data• 3) As root, use the SMF's svadm command to start Postgres

# /usr/sbin/svcadm enable postgresql:version_82

35

DTrace and PostgreSQL

• Create the bench db$ createdb bench• Populate the data in db $ pgbench -i -s 5 bench• Run the benchmark$ pgbench -c 2 -t 400000 bench

• Find the pids of the postgres process.$ pgrep -l postgres

36


• Here are a few example D-scripts. #!/usr/sbin/dtrace -Zqspostgresql*:::transaction-start{ self->ts=timestamp; @cnt[pid]=count();}

postgresql*:::transaction-commit{ @avg[pid]=avg(timestamp - self->ts);}tick-5sec{ normalize(@avg, 1000000); printf("%15s %30s %30s\n","PID","Total queries","Avegrage time (ms)"); printf("\t======================================================================\n"); printa("%15d %@30d %@30d\n",@cnt,@avg); printf("\t======================================================================\n\n"); clear(@cnt); clear(@avg);}

postgres_avg_query_time.d

37


• Example output

# ./postgres_avg_query_time.d PID Total queries Avegrage time (ms) ================================================================== 23814 46 57 23817 58 34 23816 59 32 23815 59 33 23818 75 26 ==================================================================

38


• Here is a simple script to print all the SQL statements executed.

#!/usr/sbin/dtrace -ZwqsBEGIN{

freopen(“sql.output”);}pid$1::pg_parse_query:entry{ printf("%s\n",copyinstr(arg0));}• This script will send its output to a file. sql.output

postgres_queries.d

39


• Sample output:

BEGIN;UPDATE accounts SET abalance = abalance + 344 WHERE aid = 212898;SELECT abalance FROM accounts WHERE aid = 212898;UPDATE tellers SET tbalance = tbalance + 344 WHERE tid = 22;UPDATE branches SET bbalance = bbalance + 344 WHERE bid = 3;INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (22, 3, 212898, 344, CURRENT_TIMESTAMP);END;

BEGIN;UPDATE accounts SET abalance = abalance + 15549 WHERE aid = 474266;SELECT abalance FROM accounts WHERE aid = 474266;UPDATE tellers SET tbalance = tbalance + 15549 WHERE tid = 19;UPDATE branches SET bbalance = bbalance + 15549 WHERE bid = 5;INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (19, 5, 474266, 15549, CURRENT_TIMESTAMP);END;

40


• More info on PostgreSQL and DTrace• Look at Robert Lor's Blog

http://blogs.sun.com/robertlor/• For more examples see

http://pgfoundry.org/projects/dtrace/• Documentation

http://www.postgresql.org/docs/8.2/interactive/dynamic-trace.html

41

DTrace & mySQL

● More on mySQL to come in 6.0........

42

DTrace and MySQL

• mySQL 6.0 will have embedded DTrace probes.> grab the latest src from mysql.bkbits.net

• Until then here is a simple script to capture all the SQL statements executed in a live running MySQL process

• No extra recompiles needed.

43

DTrace and MySQL

• First find the demangled name for the function dispatch_command()> dem `nm mysqld | cut -f 8 -d "|" ` | grep dispatch_command:__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b _ ==

bool dispatch_command(enum_server_command,THD*,char*,unsigned)

44

DTrace and MySQL

• Now for the D script#!/usr/sbin/dtrace -qws/* * __1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ == bool dispatch_command(enum_server_command,THD*,char*,unsigned) * */

BEGIN{ freopen("sql_commands_out");}

pid$target::__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_:entry{ printf("%d:: %s\n",tid,copyinstr(arg2));}

45

DTrace and MySQL

• Output looks like6590:: BEGIN6590:: SELECT c from sbtest where id=5035256590:: SELECT c from sbtest where id=4964716590:: SELECT c from sbtest where id=5000736590:: SELECT c from sbtest where id=5049216590:: SELECT SUM(K) from sbtest where id between 497386 and 4974856590:: SELECT c from sbtest where id between 537144 and 537243 order by c6590:: SELECT c from sbtest where id=3990196590:: SELECT c from sbtest where id=4978996590:: SELECT c from sbtest where id=4967746590:: SELECT c from sbtest where id=5000696590:: SELECT c from sbtest where id between 499439 and 4995386590:: SELECT DISTINCT c from sbtest where id between 502890 and 502990 order by c6590:: UPDATE sbtest set k=k+1 where id=5000856590:: UPDATE sbtest set c='183967968-592575299-975263686-423718108-495026220-901629681-234741050-54888582-190117389-133959759' where id=4953516590:: SELECT c from sbtest where id=5176156590:: SELECT c from sbtest where id=5036736590:: DELETE from sbtest where id=5048046590:: INSERT INTO sbtest values(504804,0,' ','aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeeyyyyyyyyyy')6590:: COMMIT

46

The END

Peter KarlssonTechnology EvangelistSun Microsystems

46

Tuning Web 2.0 Applications using Dtrace

Documents

Transcript of Tuning Web 2.0 Applications using Dtrace