Instrumentation

Instrumentation Tools Demonstrations Questions

Linux Instrumentation

Ian Munsie

June 24, 2010

Ian Munsie Linux Instrumentation


Outline

1 Overview of available Instrumentation ToolsCompetitions Instrumentation ToolsOverview of Linux Instrumentation ToolsInteractions Between Instrumentation ToolsInteresting Instrumentation Tools

2 DemonstrationsFinding cache misses with perfLocate sources of block I/O with tracepointsUsing kprobes to analyse a running kernel


Instrumentation Tools Demonstrations Questions Non-Linux Linux Interesting

Outline

1 Overview of available Instrumentation ToolsCompetitions Instrumentation Tools

DTraceOverview of Linux Instrumentation Tools

tracepointskprobesuprobesgprofptraceutraceftraceLTTngoProfileSystemTapperformance events

Interactions Between Instrumentation ToolsInteresting Instrumentation Tools



What major players are in this area? DTrace



DTrace

All purpose kernel and userspace tracingBetter than LinuxOriginated from SolarisMainstream ports for FreeBSD, NetBSD, Mac OS XLicencing Issues with LinuxNo performance impact when probes disabledMinimal probe effect

DefinitionProbe Effect: The phenomena where observing a system willchange the behaviour of that system



DTrace Implementation

Well documented probes

DefinitionProbe: A location or action which can be hooked into in order toperform some arbitrary action

Probes placed statically in source codeProbe location well consideredProbes in Solaris kernel, postgreSQL, x.org, . . .Can create new probes dynamicallyDynamic tracing frameworkDTrace D programming language



DTrace on Linux?

DTrace approach generally unsuitable for Linux kernelLinux kernel evolves rapidlyResistance to placement of static probesUnofficial Linux port does existSystemTap can use DTrace markers



Enough DTrace envy, what about Linux?



Userspace

Non-Mainline KernelMainline Kernel

ftrace

ptrace

utrace

stracegdb

SystemTap

oProfile

perf events

kprobes

uprobes

tracepoints LTTng

perfDTrace Markers

HardwareCounters

KernelModules

gprof



Static probes? tracepoints



Tracepoints

Static probe points in the kernel sourceReplace old Kernel MarkersInfrastructure mainlineLow performance impactTRACE_EVENT() macro



Kernel dynamic probes? kprobes



Kprobes

Insert a new probe into kernel at runtimeDoes not do userspace probesHeavily architecture specificCan be used by loadable kernel modulesUsed by other instrumentation toolsProvides three types of probes

kprobesProbe instructionjprobesProbe function callkretprobesProbe function return

Mainline



Userspace dynamic probes? uprobesNot mainline yet.



Uprobes

Kprobes for userspaceNot mainline yetpredict that may soon changeUsed by SystemTapHandlers run in task contextBackground page replacement mechanismProbed instruction single stepped in XOL areaNo longer depends on utraceHandlers implemented as kernel moduleperf interface



Userspace profiling? gprof



gprof

Legacy, but deserves a mentionCounts and profiles calls to functionsFunction granualityCompile executable with -pg flagProduce function call graphDoes NOT profile libraries or kernelKprof provides GUI to visualise call graph



gprof Flat Profile



gprof Call Graph



kprof Profile Visualisation Tool



Userspace tracing? ptrace . . .A bit ugly.



ptrace

Means of one process observing/controlling anotherProminently used by gdb, strace, ltraceUgly, limited interfaceProcesses cannot be traced by multiple processesSignal oriented architectureLarge overhead



ptraceltrace



ptracestrace



Improved userspace tracing? utraceNot mainline yet.



utrace

Improved userspace tracingNot mainlineProminently used by SystemTapUtrace clients run in kernel, typically in modulePtrace reimplemented as a utrace clientInfrastructure to monitor threadsEstablish engine for each monitored thread

Event reportingThread controlThread machine state access



Kernel tracing? ftrace



ftrace

MainlineKernel tracing infrastructureOriginated from the realtime effortsHijacks mcount (from gprof) for own purposesLow performance impactTracing Plugins

FunctionFuntion graphContext switchesTime interrupts disabledTime preemtion disabledDelay for high priority tasksPower state transitionsBranch prediction. . .

Exposed through debugfsManipulate with echo & catGained event supportperf events favoured for this



ftraceFunction Tracer



ftraceTask scheduling



ftraceMeasuring task wakeup latency



Kernel and Userspace tracing? LTTngNot mainline.



LTTng

Kernel & Userspace tracing packageNot mainlineAdds instrumentation points into kernelUses tracepoints, ftrace, kprobesCan plot tracesCompletely failed to work for me in every way



LTTng Screenshot

LTTng did not work for me, but heres what its supposed to do:



Hardware performance counters? oProfile



oProfile

System wide sampling profilerMainlineInterface to Performance Measurement UnitGUI eases selection of performance countersCan generate gprof style call graph



oProfile GUI



oProfile Report



oProfile Annotate



oProfile Call Graph



Something to pull everything together? SystemTapNot mainline.



SystemTap

Powerful kernel & userspace analysis suiteScripting languageGenerates, Compiles and loads kernel modulesPrimarily uses kprobesUtrace for userspace tracingUprobes supportMany userspace dependenciesHard to useMore of interest to enterprise distrosVersion 1.2 recently released

Prototypical support for perf eventsHardware breakpoint probe support

Not mainline and . . .



SystemTap

Lets face it, system tap isnt going to be merged,so why even bring it up? Every kernel developer Ihave _ever_ seen agrees that all the new tracing is amillion times superior. Im sure there are system tappeople who disagree, but quite frankly, I dont see itbeing merged considering how little the system tappeople ever did for the kernel.

So if things like system tap and "security modelsthat go behind the kernel by tying into utrace" are thereasons for utrace, color me utterly uninterested. Infact, color me actively hostile. I think thats the worstpossible situation that wed ever be in as kernel people(namely exactly the "do things in kernel space byhiding behind utrace without having kernel peopleinvolved")

LinusIan Munsie Linux Instrumentation


Something else to pull everything together? perf eventsThe future of Linux Instrumentation.



perf events

All in one Performance toolNewcomer, but already mainlineInterface to Performance Measurement UnitVirtual countersIntegrates with tracepointsKprobe supportScripting supportSLAB subsystem profilingLock profilingScheduler analyserTimechart



Sample perf report



Sample perf call graph



perfAvailable events



perf top



perf timechart



How does it all fit together?



Userspace


ftrace

ptrace

utrace

stracegdb

SystemTap

oProfile

perf events

kprobes

uprobes

tracepoints LTTng

perfDTrace Markers

HardwareCounters

KernelModules

gprof



Interesting Technologies for the Future

DTraceSystemTapLTTnggprofptraceutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapLTTnggprofptraceutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTnggprofptraceutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofptraceutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobeskprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobesNot mainline yet, likely soon. . .kprobestracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobesNot mainline yet, likely soon. . .kprobesMainline, Used by otherstracepointsftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobesNot mainline yet, likely soon. . .kprobesMainline, Used by otherstracepointsMainline, Used by othersftraceperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobesNot mainline yet, likely soon. . .kprobesMainline, Used by otherstracepointsMainline, Used by othersftraceMainline, Activeperf events




DTraceLicence issues, not suited to LinuxSystemTapNot mainline, enterprise focusLTTngNot mainline, enterprise focusgprofLimitations, not going anywhereptraceUgly, set in stoneutraceTrouble going mainlineoProfileNot going anywhereuprobesNot mainline yet, likely soon. . .kprobesMainline, Used by otherstracepointsMainline, Used by othersftraceMainline, Activeperf eventsMainline, Very active



Userspace


ftrace

ptrace

utrace

stracegdb

SystemTap

oProfile

perf events

kprobes

uprobes

tracepoints LTTng

perfDTrace Markers

HardwareCounters

KernelModules

gprof



Userspace


ftraceperf events

kprobes

tracepoints

perf

HardwareCounters


Instrumentation Tools Demonstrations Questions Cache Misses Block I/O Perf Kprobes

Outline

2 DemonstrationsFinding cache misses with perfLocate sources of block I/O with tracepointsUsing kprobes to analyse a running kernel



Finding cache misses with perf




Fairly common scenarioHigh cache misses is bad for performanceUse a very simple C program as an exampleTask: sum every element in a large arrayComplication: array & program too big for L2 cacheTwo approaches with different array access orderUse perf to observe cache misses




$ perf record -e cache-misses ./cachetest$ perf report



Locate sources of block I/O with perf and tracepoints



perfLocate sources of block I/O

System is under heavy disk I/O causing slowdownsTraditional top does not reveal offending processesEven atop only shows I/O at process granularityblock:block_rq_issue tracepoint availableKernel must be compiled with block I/O tracingperf can produce call graphs to track exact origin



perfLocate sources of block I/O

# perf record -g -a -e block:block_rq_issue sleep 10# perf report



Analyse a running kernel with perf and kprobes



Analyse a running kernel with perf and kprobes

Want to pull information out of a running kernelInformation is not already revealed to userspaceNo tracepoints provide the informationLive systemcannot afford downtimeKernel debugging information and source is availableLets probe the scheduler as an example. . .



Analysing a running kernel with perf and kprobes

# perf probe line schedule




# perf report



Thankyou all for listening

perf can be found in the Linux kernel sources under/tools/perfperf is being actively developed in the tip kernel treeLinux Weekly News has some good further reading onftracegoogle site:lwn.net ftraceperf articles not as easy to find yet, here are some:

http://lwn.net/Articles/339361Perfcounters added to the mainline

http://lwn.net/Articles/373842Scripting support for perf

http://lwn.net/Articles/382554A live mode for perf

http://lwn.net/Articles/346470Fun with tracepoints

http://lkml.org/lkml/2010/3/17/91Ingo Molnar on database I/O latency


Overview of available Instrumentation ToolsCompetition's Instrumentation ToolsOverview of Linux Instrumentation ToolsInteractions Between Instrumentation ToolsInteresting Instrumentation Tools

DemonstrationsFinding cache misses with perfLocate sources of block I/O with tracepointsUsing kprobes to analyse a running kernel

Questions

Instrumentation

Documents

Transcript of Instrumentation