Course Notes: Application/OS performance: What does it depend on? Hands-on Lab

Hewlett-Packard Company Tactical Methods to Performance Analysis HOL2996 – Application and OS Performance LAB

Prepared by: Chris & Greg Tinker

[email protected]

[email protected]

Document Revision 1.0

Date Prepared: 04/2013

mailto:[email protected]

mailto:[email protected]

../hp.com/go/discover

HOL2998 HOL2998 – Application and OS Performance LAB

LABS Solution Performance -- Tactical Methods to Performance Analysis

HP GSSE TEAM HP Restricted of 17

Tinker (Version 1.0/Jan-2013) © Copyright 2013 Hewlett-Packard Development Company, L.P.

HOL2998_Tinker_AppOS_Performance_LAB.docx Last changed: 08 July 2013 at 12:41

Document Revision History

Ver. No. Ver. Date Prepared By Reviewed By

Approved By

Affected Section & Summary of Change

1.0 04/01/2013 Chris & Greg Initial Version

Contents

1 Setup network communication between hosts ................................................................................................ 3

1.1 Lay the groundwork ............................................................................................................................. 3

1.1.1 HOL2998-APP-111– Laying the groundwork ............................................................................................ 3

2 Backup application performance issue ............................................................................................................ 5

2.1 Application installation .......................................................................................................................... 5

2.1.1 HOL2998-APP-211– Application installation ........................................................................................... 5

3 Network performance with Netperf ............................................................................................................... 13

3.1 Expectations achieve wire performance ........................................................................................... 13

3.1.1 HOL2998-APP-311– Application installation ......................................................................................... 13






1 Setup network communication between hosts

The backup application requires ssh to communicate

1.1 Lay the groundwork

Two OS instances – high speed network connecting the two

1.1.1 HOL2998-APP-111– Laying the groundwork

Test Case ID Acceptance Criteria

HOL2998-APP-111 Pass Fail Partial N/A

Lab Time:

15 minutes

Description:

Setup ssh key authentication communication bi-directional between two systems

Prerequisites/Pre-conditions:

1. Guest operating system – Linux (Two or more systems) 2. ssh client and server installed

Procedures:

1. Boot one of the Linux systems, login as root. 2. create keys

a. as any user issue (ssh-keygen –t dsa –b 1024)and press return (just press return when prompted with a pass phrase) until you get a key

3. Boot second Linux machine – login as root user and repeat step two 4. place pub key of SOURCE into the authorized_keys file of the target

a. (ssh-copy-id –i ~/.ssh/id_dsa.pub root@remote_IP)

Expected Results:

1. ( ssh –l user IP/HOSTNAME ) does not prompt for password






Comments:

Tools/commands that may be useful

ifconfig (ipconfig windows) ethtool (see man page for examples) Mii-tool (see man page for options)

Student Notes:






2 Backup application performance issue

Background: After an upgrade to OS and hardware, backup performance is suffering resulting in missing

backup SLAs. o Though “everything” is essentially the same with newer/faster hardware, the backup

program is substantially slower than expected. The backup traverses the network which is now 1000Bast-T instead of the previous 10Base-

T and the hardware has 64 cores instead of 16.

2.1 Application installation

Determine what factors are causing the slower performance

2.1.1 HOL2998-APP-211– Application installation



Lab Time:

45 minutes

Description:

Install application and execute; confirm it executes


1. Guest operating system – Linux (Two systems required) 2. ssh with key – no password required 3. The attached backup shell script.

backup.sh.gz

Procedures:

1. Login to the HOST then sudo to root. (sudo su -). 2. Use the above backup script (gzip) file, on your Linux HOST (it does not matter which

one) 3. extract the archive – ( gunzip bacup.sh.gz) 4. confirm ssh keys are valid between the two guest






# ssh –l root IP_hostB

(if this fails, or requires password, setup ssh keys!!!

This MUST WORK… 5. execute Backup script.

THIS IS AN EXAMPLE – the IP address on your destination machine will be DIFFERENT.. ./backup root [TARGETIP or hostname]

./backup.sh root 192.168.1.183

Backup to TARGET 192.168.1.183 START Sat Jan 19 10:49:46 EST 2013

The backup administrator has reported that now that we are running on the Newer Technology the backups are taking LONGER periods of time. They cannot determine why the backups are slower as the CPU usage is low, latency is low, network seems to be the problem?

Determine why the backup is so slow.

1. Are there any errors on transport?

# ifconfig –a

# ethtool eth# [look for speed, and other stuff] ethtool –S eth[x] for errot counters. # mii-tool – quick way to see interface. (Where X is #) (may wish to use with watch command watch ethtool –S eth# | grep –I rx_bytes

2. What is the overall performance

# dstat

What is the system doing? CPU consumption, network

consumption, disk latency, throughput of disk and network.

Does anything stand out?

# collectl –sncd (N = network, C – CPU, D = Disk)

User upper case flags for details!!!

What are your observations?






CPU utilization, network saturation, disk saturation. 3. Confirm speeds and feeds – disk, network, cpu, application.. does all seem ok?

4. What is the application WAITING on?

# watch ps --sort -pcpu -eo

pid,ppid,s,tid,nlwp,sched,rtprio,ni,pri,psr,sgi_p,pcpu,stat,eip,wchan:25,f,args:80

Example output of above command

Every 2.0s: ps --sort -pcpu -eo pid,ppid,s,tid,nlwp,sched,rtprio,ni,pri,psr,sgi_p,pcpu,stat,eip,wchan:25,f,args:80

Sat Jan 26 12:04:41 2013

PID PPID S TID NLWP SCH RTPRIO NI PRI PSR P %CPU STAT EIP WCHAN F COMMAND

6627 6620 R 6627 1 0 - 0 19 1 1 97.1 R+ 0804a2ae - 0 mybackup

6628 6620 S 6628 1 0 - 0 19 0 * 5.4 S+ b76df424 poll_schedule_timeout 4 ssh -o

PasswordAuthentication=no -l root 192.168.1

6626 6620 S 6626 1 0 - 0 19 0 * 2.3 S+ b7772424 pipe_wait 0 tar -cf - /usr

WCHAN = Wait Channel Can a PROCESS consume more than 100%? Trace the process consuming all of “one” CORE.. What is it doing?

# strace -f -F -i -r -T -o /tmp/trace.out -p 6627

NOTE: ONLY RUN for a few seconds then ctrl^c .. this will be lots of data..

Take a look at the trace.. what it is doing?

From this trace we see that the reads and writes are very

fast:

0.002195 [b76e3424] read(0,

"\265s\327bKxo}\263pm\375\6JK\3\307\1\315\305\36\3108`\347\217U\273\34~oK"..., 32768) = 32768

<0.000149>

0.001113 [b76e3424] write(1,

"U\307\215P\324\317O)s\312\330O\250QAO\23Y\247\2656\\u\222W\361?9\356\324\n\210"..., 16384) = 16384

<0.000140>

0.001065 [b76e3424] write(1,

"?*r\37\363'\315\310j\255\226\351\1g\270\v\225\221\272o\251<\236\361\222\32\340\2&\307\2505"...,

16384) = 16384 <0.000169>

0.001870 [b76e3424] read(0, "\0\0B\0\0\0reference/api/org/eclipse/"..., 32768) = 32768

<0.000113>

0.001324 [b76e3424] write(1,

"/\263\237\321\312\3331+\333\330\225\242\220\371\341\366\35\373.\247<[W\371\360\231\367\\r\37x["...,

16384) = 16384 <0.000079>

0.000279 [b76e3424] write(1,

"\3072\332)\253\355\317^\327kR0j\216/\333\377C0\6\235\0377\260\263|\241\246\\l0\377"..., 16384) =

16384 <0.000526>

0.002239 [b76e3424] read(0,

"#md\v\370\205\345\357xr\255\223G?FHHi\310\17\377\205O2\204!D\2000I\305\236"..., 32768) = 32768

<0.000288>

0.001540 [b76e3424] write(1,

"\363\222\310\274\310\371\201\255\30kDTf)\344\247&\205\346\236\0\v9O\371\240\221gU\237\327\313"...,

16384) = 16384 <0.000186>

0.000590 [b76e3424] write(1,

"m\333\266m\333\266m\333\266m\333\266\355w\376S\365\252n\274F5\356\2737b6f\314\316\212X"..., 16384)

= 16384 <0.000119>

0.000455 [b76e3424] write(1,

"\325\377\33\252M\306\300\321\352\177\2516/\247C\366\326\277<\271\2\23K;2\27`\342\235&\267\263"...,

16384) = 16384 <0.000086>

0.001313 [b76e3424] read(0,






"\30G\340\364\353\21gv\370Z\213>\303\27;\330\3651\247\37k\366.\36X>\201\v:q}*"..., 32768) = 32768

<0.000067>

0.001431 [b76e3424] write(1, "\207\267\237}a\v\226/\206\252\236\3\210

\10\373\177s\240\332\225F\204T\34\220\0343\217TCy"..., 16384) = 16384 <0.000168>

0.000704 [b76e3424] write(1,

"\377\327?5T\245\312\376\336:\364N\\\374\255|^\246;C\36\267\333\244\222\r\f\3679\243}\\"..., 16384)

So we can see that the system call is not taking long at all.. strace as the ability to count the calls and the time spent in them!

# time –p strace -c -o /tmp/trace.out -p 6627

ctrl^c

^CProcess 6627 detached

real 7.40

user 0.00

sys 0.5

NOTE: Some OS’s allow for this to be used in combination with the others and prints the summary at the bottom.. not Linux strace though.

#cat /tmp/trace.out

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------

100.00 0.000585 0 1661 write

0.00 0.000000 0 2700 read

------ ----------- ----------- --------- --------- ----------------

100.00 0.000585 4361 total

So we can see that in 7.4 seconds, the application made 2700 read system calls from the application level but spent virtually no time waiting on them. We are blocked on CPU –user code execution.

5. STOP the backup ctrl ^C if it has not finished. 6. execute Backup script again with new OPTION!

THIS IS AN EXAMPLE – the IP address on your destination machine will be DIFFERENT.. ./backup root [TARGETIP or hostname]

./backup.sh root 192.168.1.183 FAST

Backup to TARGET 192.168.1.183 START Sat Jan 19 10:49:46 EST 2013






The backup administrator is now reporting that the backup is running MUCH faster!

Determine why the backup is running FASTER. Note: most command require root.

7. Are there any errors on transport?

# ifconfig –a

# ethtool –S eth# # mii-tool (Where X is #) (may wish to use with watch command watch ethtool –S eth# | grep –I rx_bytes

8. What is the overall performance

# dstat

What is the system doing? CPU consumption, network

consumption, disk latency, throughput of disk and network.

Does anything stand out?

# collectl –sCND (N = network, C – CPU, D = Disk)

What are your observations? Do you notice something about the CPU usage? CPU utilization, network saturation, disk saturation.

9. Confirm speeds and feeds – disk, network, cpu, application.. does all seem ok?

10. What is the application WAITING on?

# watch ps --sort -pcpu -eo


Example output of above command

Every 2.0s: ps --sort -pcpu -eo

pid,ppid,s,tid,nlwp,sched,rtprio,ni,pri,psr,sgi_p,pcpu,stat,eip,wchan:25,f,args:80 Sat Jan 26

12:32:58 2013

PID PPID S TID NLWP SCH RTPRIO NI PRI PSR P %CPU STAT EIP WCHAN F COMMAND

9332 9322 R 9332 1 0 - 0 19 1 1 93.0 R+ b7726424 - 0 mybackup

9333 9322 S 9333 1 0 - 0 19 2 * 36.8 S+ b76ab424 - 4 ssh -o

PasswordAuthentication=no -l root 192.168.1

9331 9322 D 9331 1 0 - 0 19 3 * 10.4 D+ b7762424 - 0 tar -cf -

/usr

WCHAN = Wait Channel Can a PROCESS consume more than 100%? (i.e. multiple threads)






11. Trace the process consuming all of “one” CORE.. What is it doing?

# strace -f -F -i -r -T -o /tmp/trace.out -p 6627

NOTE: ONLY RUN for a few seconds then ctrl^c .. this will be lots of data..

Take a look at the trace.. what it is doing?

So we can see that the system call is not taking long at all.. strace as the ability to count the calls and the time spent in them!

# time –p strace -c -o /tmp/trace.out -p 9332

ctrl^c

^CProcess 9332 detached

real 5.20

user 0.00

sys 0.2

NOTE: Some OS’s allow for this to be used in combination with the others and prints the summary at the bottom.. not Linux strace though.

#cat /tmp/trace.out

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------

51.99 0.002249 0 6274 read

48.01 0.002077 1 3266 write

------ ----------- ----------- --------- --------- ----------------

100.00 0.004326 9540 total

So we can see that in 5.2 seconds, the application made 6274 reads compared to the 2700 reads in 7.4 seconds previously.

So the APPLICATION change significantly increased performance.






Expected Results:

This is a crude example of a recent escalation after changes were made to the OS platform and physical hardware. The application was not designed for the newer platform chipset – a slight modification to the application and performance was better than the previous solution.

Comments:

Commands to use to help narrow your search:

Ifconfig (ipconfig windows) ethtool (see man page for examples) Mii-tool (see man page for options) Collectl (collectl –sNDC dstat –a netstat –rn iostat –xd 1 30 top

Performance tools:

tfgen ttcp iperf netperf/netserver

In more detail – mentioned in presentation, identify what the app is waiting on. One such method is to trace the app. In this case, you can use “watch” to loop ps commands to see the wait channel and other issues with the most busies processes on the system.

watch ps --sort -pcpu -eo


Student Notes:






3 Network performance with Netperf

Background: Network performance testing – working direct on the TCP network stack

3.1 Expectations achieve wire performance

Using a TCP network generator tool to load network




Lab Time:

45 minutes

Description:

Install netperf and run netperf stats to load network using TCP_STREAM test.


1. VirtualBox 4.2 or > 2. Guest operating system – Linux. 3. netperf

Procedures: 1. TWO Operating (Linux, windows, HPUX, Solaris, etc) systems

You do not need two if you plan to use UDB and throw to port 9 2. Select one system to be the SERVER 3. Open terminal session 4. Capture IP of SERVER on both, and capture IP of both machines.

ifconfig –a 5. Start the packet receive engine on one of your systems.

Note that when you install netserver, it starts automatically in Ubuntu netserver

6. Goto Client system (other guest) 7. Start a network benchmark, perform each of these one at a time and capture

performance metrics and explain. Capture IP of CLIENT ifconfig –a

Start a Packet generator throwing data from the CLIENT to the server over the specified interfaces

netperf -f M -l 200 -L CLIENT_IP -H SERVER_IP -t TCP_STREAM -p 12865 -- -s 16k –S 16k

EXAMPLE:






# netperf -f M -l 200 -L 192.168.0.187 -H 192.168.0.197 -t TCP_STREAM -p 12865 -- -s 16k

-S 16k

MIGRATED TCP STREAM TEST from 192.168.0.187 (192.168.0.187) port 0 AF_INET to

192.168.0.197 (192.168.0.197) port 0 AF_INET : demo

Recv Send Send

Socket Socket Message Elapsed

Size Size Size Time Throughput

bytes bytes bytes secs. MBytes/sec

32000 32000 16000 200.00 14.97

Expected Results:

1. Using iostat, dstat, and other tools – isolate what is causing MAX performance issue.

2. Using strace – determine if the issue is in USER or SYSTEM space.

Comments:

Student Notes:






4 Application Performance accessing file

Background: Performance degradation over time when accessing Files

4.1 Users complaining about slow response from server

Identify root cause of application degradation




Lab Time:

30 minutes

Description:

There are two DATABASE files located on the machine – a recreation of a production issue. Identify performance issue.


1. Linux OS 2. collectl 3. DD 4. Other native commands

Procedures: 1. The proctor has created the condition on the systems 2. Mount the filesystem

a. back up (explain why we umount the filesystem)

b. (mount /dev/lab/ext4-2 /ext4-2) 3. You will need to measure the applications performance

a. Measure the IO Size, latency, KB/sec, average request size etc…

b. You will need at least 2 getty (terminal windows)…

c. In one window – make sure that you have iostat –xd 1 1000 always running…

4. Read the file with a backup tool a. ( iostat –xd 1 1000 ) b. time –p tar –cvf - /ext4-2/exchangeDB | cat - >

/dev/null c. What is the Time to perform the backup d. What was the physical IO size, latecy, achieved throughput, etc






5. time –p tar –cvf - /ext4-2/exchangeDB | cat - > /ext4-1/exchangeDB_tar

a. Make sure IOstat is still running ( iostat –xd 1 1000)

6. Umount the filesystem (umount /ext4-1) 7. Mount the filesystem back up

a. Use the following command (mount /dev/lab/ext4-1 /ext4-1)

8. Use TAR to read the new file a. iostat –xd 1 1000 b. time –p tar –cvf - /ext4-1/exchangeDB_tar | cat - > /dev/null

c. What is the Time to perform the backup d. Why the difference?

9. e4defrag –c /ext4-2/exchangeDB

<File> now/best size/ext

/ext4-2/exchangeDB 103140/1 7 KB

Total/best extents 103140/1

Average size per extent 7 KB

Fragmentation score 91

[0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]

This file (/ext4-2/exchangeDB) needs defragmentation.

Expected Results:

Application performance, with respect to data access, is highly dependent on the IO boundaries introduced by the lower layers of the IT stack. Though file systems are optimized for particular workloads, disk arrays may be better suited for alternative workloads and thus the discrepancy in interoperability can severally limit achievable performance.

Comments:






Student Notes:

Course Notes: Application/OS performance: What does it depend on? Hands-on Lab

Technology

Transcript of Course Notes: Application/OS performance: What does it depend on? Hands-on Lab