Rom Waltonrwalton@ssl.berkeley.edu

September 10, 2008

BOINC Application Lifecycle

•Designing Application•Writing Code•Instrumenting Code•Fixing Bugs

•Running Test Workunits•Checking for Memory Leaks•Checking for Handle Leaks

•Checking Crash Reports•Formulating a Plan of Action

•Uploading Application•Uploading Support Files•Running update_versons

Developing AppsWorker applications should call:

Graphics applications should call:

Benefits of the BOINC Diagnostic FrameworkStack traces on crashes (Win, Linux, Mac)Prevents the JIT Debugger dialog (Win)Memory leak detection (Win – VS Debug Builds)Heap corruption detection (Win – VS Debug

Builds)

boinc_init_diagnostics( BOINC_DIAG_DEFAULTS );

boinc_init_graphics_diagnostics( BOINC_DIAG_DEFAULTS );

Developing Apps (cont’d)Platform neutral tips

Use a separate project to test your applicationsWindows tips

Use the symbol store technology, if possibleLinux tips

Link application with the BOINC-Build-Compat Virtual Machine. Application binds to older glibc but still contains the optimizations of newer compilers

Mac tipsBuild against the Mac OS X 10.3.9 SDK

Developing Apps (cont’d)Windows Symbol Stores

What are they?How do they work?How does the BOINC Framework use them?How do I configure my Windows machine?

Set environment variable _NT_SYMBOL_PATH to:srv*c:\windows\symbols*http://msdl.microsoft .com/download/symbols

Execute:symchk c:\windows\system32\*.dll /ov

Enable Symbols Demo

Developing HighlightsUse the BOINC Diagnostic Framework

Traps structured exceptions (Win)Traps signals (Linux, Mac)Detects memory leaks (Win)Detects heap corruption (Win)

BOINC Application Lifecycle

•Designing Application•Writing Code•Instrumenting Code•Fixing Bugs

•Running Test Workunits•Checking for Memory Leaks•Checking for Handle Leaks

•Checking Crash Reports•Formulating a Plan of Action

•Uploading Application•Uploading Support Files•Running update_versons

Validating AppsRun through your test workunitsCheck for buffer overruns

Win: BOINC_DIAG_HEAPCHECKENABLEDLinux: Valgrind

Check for memory leaksWin:

BOINC_DIAG_MEMORYLEAKCHECKENABLEDLinux: Valgrind

Check for handle leaksWin: Process Explorer, ntsd (!handles), oh.exeLinux: lsof

Validating Apps (cont’d)Global Flags (gflags.exe)

Not useful for debugging inthe field

Tweaks the heap manager(works with any type of application)

Only detects buffer overruns for dynamically allocated memory.

gflags.exe options:

Validating Apps (cont’d)Example memory leak output:

Example handle leak output (oh.exe):

Dumping objects ->..\lib\diagnostics_win.C(642) : {70} normal block at 0x017C3A30, 1068 bytes long. Data: < T > 1C 1A 00 00 54 00 00 00 01 00 00 00 00 00 00 00 Object dump complete.

boinc.exe File 0008 \ProgramData\BOINC\stdoutdae.txtboinc.exe Key 0034 \REGISTRY\MACHINEboinc.exe File 0094 \ProgramData\BOINC\stderrdae.txt

Validating HighlightsTest the following once per application

release cycle:Memory leaksHandle leaksHeap corruption

Test the following once per application release:Exception/Signal handling

(Abort a task from BOINC Manager)

BOINC Application Lifecycle

•Designing Application•Writing Code•Instrumenting Code•Fixing Bugs

•Running Test Workunits•Checking for Memory Leaks•Checking for Handle Leaks

•Checking Crash Reports•Formulating a Plan of Action

•Uploading Application•Uploading Support Files•Running update_versons

Deploying AppsAdd application to

‘$project_root/project.xml’.

$app_name = Whatever you want to call your application

Execute xadd to add your application to the project.

<boinc> <app> <name>$app_name</name> <user_friendly_name></user_friendly_name> </app></boinc>

cd $project_root./bin/xadd

Deploying Apps (cont’d)Upload your application and its dependencies to:

$project_root/apps/$app_name/$app_ver_name$app_ver_name is the name of the worker

executableWorker executable name is defined as:

$name_$version_$platform

Execute update_versions t0 add app versions to project

Windows: uppercase_6.1_windows_intelx86.exeLinux: uppercase_6.1_i686-pc-linux-gnuMac: uppercase_6.1_i686-apple-darwin

cd $project_root./bin/update_versions

Deployment Demo

Deployment HighlightsApplication files are assumed to be immutableupdate_versions performs the following

functions:Copies application files to download directoryReads signature file/generates signature fileAdds application to databaseTouches feeder restart file

BOINC Application Lifecycle

•Designing Application•Writing Code•Instrumenting Code•Fixing Bugs

•Running Test Workunits•Checking for Memory Leaks•Checking for Handle Leaks

•Checking Crash Reports•Formulating a Plan of Action

•Uploading Application•Uploading Support Files•Running update_versons

Debugging Apps4% - 5% failure rates are not uncommon

Machines overheatBad memoryBad processors

Crash reports are stored inside the BOINC DatabaseGo to the project_ops web site to look at crash

reportsLook for reports that have:

PDB symbols for the application and librariesComplete stack traces

Debugging Apps (cont’d)Determine environment is setup correctly

Check that your symbol store is listed

BOINC Windows Runtime Debugger Version 5.5.0

Dump Timestamp : 09/09/08 23:41:39Debugger Engine : 4.0.5.0Symbol Search Path: srv*C:\DOCUME~1\romw\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/alpha/symstore;srv*c:\windows\symbols*http://msdl.microsoft.com/download/symbols;srv*C:\DOCUME~1\romw\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/symstore

Debugging Apps (cont’d)Determine that the loaded modules have

symbols

Symbols may not download Machine is off-line during crash Took longer that 30 seconds to download symbol

file(s)

ModLoad: 00400000 00060000 C:\...\uppercase_6.3_windows_intelx86.exe (PDB Symbols Loaded)ModLoad: 7c800000 000c0000 C:\...\ntdll.dll (5.2.3790.1830) (PDB Symbols Loaded)

Debugging Apps (cont’d)Crashing thread contains an exception record:

Access Violation (0xC0000005) NULL references exceptions Using indexes as pointers Processor flipping bits because of heat

Breakpoint Encountered (0x80000003) User aborted an active task Assertion failure in debug builds

- Unhandled Exception Record -Reason: Breakpoint Encountered (0x80000003) at address 0x7C822583

Debugging Apps (cont’d)Check call stack of crashing thread

Look for the interface point between your application and an OS library. Typically NTDLL.DLL or KERNEL32.DLL

- Callstack -ChildEBP RetAddr Args to Child00aafd60 00402221 00000000 00000000 00000000 00000001 ntdll!_DbgBreakPoint@0+0x000aaffb4 0042684e 77e66063 00000000 00000000 00000000 uppercase_5.10_windows_intelx86!worker+0x0 (c:\boincsrc\main\boinc\samples\uppercase\upper_case.c:181)

Debugging Apps (cont’d)What information is contained in a stack frame

ChildEBP – isn’t really useful RetAddr – Return Address, should be within the

same address space as the previous calling frameArgs to Child – First four parameters to functionModule!FunctionSource File:Line Number

ChildEBP RetAddr Args to Child00aaffb4 0042684e 77e66063 00000000 00000000 00000000 uppercase_5.10_windows_intelx86!worker+0x0 (c:\boincsrc\main\boinc\samples\uppercase\upper_case.c:181)

Debugging Apps (cont’d)When ‘X’ doesn’t mark the spot

Don’t get discouragedSample several crash reportsDebugging applications remotely can be hard

One nasty bug in the Rosetta@home applicationOnly happened during application destructionI looked at 120 crash reports over the course of 5

days before seeing the common themeResource contention between two threads

Debugging Apps (cont’d)Need extra help?

Add trace statements by logging to stderrRelease a new version of your applicationOnly you know how your applications really

work

Debugging HighlightsBOINC’s diagnostic framework

Processes crashes for youDumps relevant information to stderrNo need to manage dump or mini-dump filesNo need to understand the specifics of each

platforms infrastructureEasy to enable

ConclusionBOINC Application Framework:

Mature diagnostic frameworkCompletely automatic feedback loopBOINC Client uses the same framework

We eat our own dog food

Should be compatible with all run-time environmentsC/C++FortranAnything that supports calling a C Function

Questions and AnswersBOINC Website:

http://boinc.berkeley.edu/BOINC Development Mailing List:

boinc_dev@ssl.berkeley.eduBOINC Projects Mailing List:

boinc_projects@ssl.berkeley.eduBOINC Application Debugging:

http://boinc.berkeley.edu/trac/wiki/AppDebug

Any Questions?

ToolsDebugging Tools for Windows (Windows)

gflags.exe, symchk.exe, symstore.exe, windbg.exeWeb site

Windows Resource Kit Tools (Windows)oh.exe, dh.exe, consume.exe, clear.exeWeb site

Process Explorer (Windows)DLL Dependencies, Handle Leak Detection, Real-

Time Thread Monitoring (Backtraces)Web site

Tools (cont’d)Valgrind (Linux)

Memory Leak Detection, Thread Error Detection

Web siteDDD (Linux)

Graphical DebuggerWeb site