7/31/2019 Petty W 1130 Daily Dose of Systemtap

1/20

7/31/2019 Petty W 1130 Daily Dose of Systemtap

2/20

SYSTEMTAP FOR SYSTEMADMINISTRATORS

Heath PettySr. Technical Account Manager, Red HatMay 4th 2011

7/31/2019 Petty W 1130 Daily Dose of Systemtap

3/20

7/31/2019 Petty W 1130 Daily Dose of Systemtap

4/20

7/31/2019 Petty W 1130 Daily Dose of Systemtap

5/20

Read the Systemtap Beginners Guide for details

As a quick start, Install the following packages(use yum to grab all the necessary

dependancies):

systemtapsystemtap-runtimekernel-devel-$(uname -r)

kernel-debuginfo-$(uname -r)kernel-debuginfo-common-[arch]-$(uname -r)

7/31/2019 Petty W 1130 Daily Dose of Systemtap

6/20

But I don't want to install kernel-debuginfo on allmy production systems

(we'll get to that later)

7/31/2019 Petty W 1130 Daily Dose of Systemtap

7/20

Hello world is nice for testing, but lets do a moreinteresting test:

stap -v -e 'probe vfs.read {printf("somethingread something from somewhere\n"); exit()}'

7/31/2019 Petty W 1130 Daily Dose of Systemtap

8/20

Now that we know it works, lets fill in somedetails, but how?

Documentation time !

Man pagesRed Hat Systemtap documentation

Using stap -LSystemtap Wiki:

http://sourceware.org/systemtap/wiki

7/31/2019 Petty W 1130 Daily Dose of Systemtap

9/20

7/31/2019 Petty W 1130 Daily Dose of Systemtap

10/20

Now that we know systemtap is working, how canwe get around installing kernel-debuginfo,compilers and other packages on your production

machines?

To accomplish this we can compile the systemtapscript on a dev machine and then load that

module on a production machine

7/31/2019 Petty W 1130 Daily Dose of Systemtap

11/20

Requirements:Dev machine must be same arch as the targetsystem.The kernel-devel and kernel-debuginfo packageversions must match the target system.Systemtap-runtime must be installed on thetarget system.

7/31/2019 Petty W 1130 Daily Dose of Systemtap

12/20

Example:stap -e 'probe vfs.read {exit()}' -m simple -p4

This will create the simple.ko file in the current

directory. Copy this file to the target system andrun:

staprun simple.ko

7/31/2019 Petty W 1130 Daily Dose of Systemtap

13/20

Awesome, but how do I gather information frommy script? I can't just sit there and watch it.

Either direct output from a system script to a file,or use the handy flight recording functionality

(the -F stap flag).Writing to a rotating log file:

stap -F -o /tmp/name.log -S 1,2 stap -F

I'll show how to use these options with ourexamples.

7/31/2019 Petty W 1130 Daily Dose of Systemtap

14/20

Where to next? Should I write my own systemtapscripts? How do I know what to do? This is the

point where I give up on systemtap, because itseems really complicated Like learning a wholenew programming language.

There is a solution! There are numerous examplescripts provided with systemtap that will address alarge number of problems. There are even moreonline. Lets review a few of the examples and

have some fun.

7/31/2019 Petty W 1130 Daily Dose of Systemtap

15/20

Sample 1. I really don't like spaces in filenames I'm old skool.

/usr/share/doc/systemtap-[ver]/examples/general/badnames.stp

7/31/2019 Petty W 1130 Daily Dose of Systemtap

16/20

7/31/2019 Petty W 1130 Daily Dose of Systemtap

17/20

Sample 3: Lets find who is leaking memory

/usr/share/doc/systemtap-[ver]/examples/memory/mmanonpage.stp

7/31/2019 Petty W 1130 Daily Dose of Systemtap

18/20

Sample 4: See if network buffers get full.

/usr/share/doc/systemtap-1.2/examples/network/sk_stream_wait_memory.stp

7/31/2019 Petty W 1130 Daily Dose of Systemtap

19/20

I have more examples if time permits, if not, then

we can take a look at them in Support Central

7/31/2019 Petty W 1130 Daily Dose of Systemtap

20/20