DDT: A visual, parallel debugger on Rageco.mines.edu/workshop/march2010/ddt.pdf · DDT 2.5.1 BASH...

IntroductionHistory

The SetupUsage

Summary

DDT: A visual, parallel debugger on Ra

David M. [email protected]

High Performance & Research ComputingCampus Computing, Communications, and Information Technologies

Colorado School of Mines

March, 2010

David M. Larue DDT on Ra 1/44

IntroductionHistory

The SetupUsage

Summary

Outline

1 HistoryAncientMedievalModern

2 The SetupPrerequisitesOne-timeEach time

3 UsageCommon classes of Parallel Programming BugsFeaturesSample Run


IntroductionHistory

The SetupUsage

Summary

Introduction

Programs hang, give incorrect output or exhibit unexpectedbehavior.Parallel computing is not deterministic.

Things can happen in different orders or with differenttimings.

available resources can differdifferent optimizations are appliedjob sizes and inputs can exacerbate these issues.

These differences can alter outcomes.

Hence programs can work some of the time, and fail someof the time.Debugging is needed


IntroductionHistory

The SetupUsage

Summary

AncientMedievalModern

Outline





IntroductionHistory

The SetupUsage

Summary


Old Style

Insertion of statements that write to a file or standardoutput

help ascertain that the program is doing what it wasintended to do.sometimes surprisingly effectiveoften easy to implement

However it has serious limitations.slows and interrupts your thoughtsrequires many recompiles and restartsprevents interactive use


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


Middle Style

Command-line text-oriented serial toolsExample: GDB (the GNU Project debugger)GDB was designed for serial processingIt can be started multiple times to inspect and controlmultiple processes.

Many algorithms in use in a parallel environment have theirbeginnings as serial code

correctness there is neededserial debuggers are then immediately useful


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


High Style

Debuggers withGraphical User InterfacesSupport for Parallel Execution

Advantages:IntuitiveInteractiveManipulate multiple processes as if one entity.

Don’t forget Profilers


IntroductionHistory

The SetupUsage

Summary


DDT

Product du jour: DDTcommercial graphical interfacebuilt on top of GDBdesigned with debugging parallel programs in mind.made by Allinea.

Documentation for DDT:http://allinea.com/downloads/userguide.pdfDDT User Guide, Version 2.5.1

Its home on Ra is/lustre/home/apps/ddt/bin/ddt


http://allinea.com/downloads/userguide.pdf

/lustre/home/apps/ddt/bin/ddt

IntroductionHistory

The SetupUsage

Summary


Other notes

Ra’s DDT License16 processes

More information on debugging on Rahttp://geco.mines.edu/workshop/class2/03wed/debuggers.pdf

Debugging on RaTimothy H. Kaiser, Ph.D., Revised January 2010http://geco.mines.edu/guide/errortracking/index.shtml

Debugging Programs whilst not bugging yourselfTimothy H. Kaiser, Ph.D., January 2010http://geco.mines.edu/guide/body.shtml#h5450

Ra User’s Guide, section on Debugging, includingreferences to workshop resources.


http://geco.mines.edu/workshop/class2/03wed/debuggers.pdf

http://geco.mines.edu/guide/errortracking/index.shtml

http://geco.mines.edu/guide/body.shtml#h5450

IntroductionHistory

The SetupUsage

Summary

PrerequisitesOne-timeEach time

Outline





IntroductionHistory

The SetupUsage

Summary


Things to have and do

ddttools that support debugging and ddt

compilermpi librarygdb

access Ra with X Windowssetup ddtcompile with debugging support


IntroductionHistory

The SetupUsage

Summary


Versions

Combinations of Versionsvarious combinations workmany do notone will be detailed here

OpenMPI 1.3.4Intel Compilers 11.1DDT 2.5.1BASHssh -Xgdb 6.8

Non-standard combination of versions requires extra setup


IntroductionHistory

The SetupUsage

Summary


Modes

ModesDDT submits a job to the queue and attaches to itsprocessesDDT attaches to the processes of an already running job

We will let DDT submit the jobneed a PBS template scriptconfigure DDT to msub


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


One-Time System Setup

Append the following to the end of ~/.bashrc(then source it or logout/login).

### openmpi 1.3.4, intel 11.1 compiler wrapper settings ###### (LD_LIBRARY_PATH, PATH) ###source /lustre/home/apps/mpi/db/openmpi1.3.4/intel11.1/setup.sh

### ddt 2.5.1 settings ###### (LD_LIBRARY_PATH, PATH, DMALLOCPATH) ###source /lustre/home/apps/ddt/setup.sh

This sets the paths and environment variables that point to theexecutables and libraries for the specific versions of OpenMPI,the compiler, GDB and DDT.


IntroductionHistory

The SetupUsage

Summary


One-Time DDT Setup, Part 1/2

Login to Ra enabling X-WindowsFrom most Unixes, type ssh -X ra.

Start DDT for the first timetype ddt &

this creates a ~/.ddt/ directory where settings are stored.

go through the Configuration Wizard.Alternately,

start DDTselect Session / Options... orSession / Configuration Wizard...


IntroductionHistory

The SetupUsage

Summary


One-Time DDT Setup, Part 2/2

Options for DDT:

System SettingsMPI Implementation: OpenMPI (Compatibility)Debugger: GNU (gdb)

Job Submission SettingsSubmit job through queue: checkedSubmission template file: /lustre/home/[username]/workshop/ra.qtfSubmit command: msubRegexpr for job id: (\d+)Cancel command: qdel JOB_ID_TAGDisplay command: qstatTemplate Uses:NUM_NODES_TAG, PROCE_PER_NODE_TAG: selectedPROCS_PER_NODE_TAG: 8 (or less...)

(Version 11 of the Intel Compilers requires selecting

the OpenMPI (Compatibility) MPI Implementation.)


IntroductionHistory

The SetupUsage

Summary


One-Time Submission template file setup, Part 1/4

A simple PBS script prog.pbs to submit a job on Ra reads

#!/bin/bash#PBS -l nodes=2:ppn=8#PBS -l walltime=00:010:00#PBS -N testddt#PBS -o $PBS_JOBID.out.pbs#PBS -e $PBS_JOBID.err.pbs#PBS -r n#PBS -V#-----------------------------------------------------cd $PBS_O_WORKDIRmpirun -np 16 ./prog

One submits this by executing

msub prog.pbs


IntroductionHistory

The SetupUsage

Summary


One-Time Submission template file setup, Part 2/4To have DDT submit the job, we need a template ra.qtf forthe PBS file with named tags that DDT will fill in.

#!/bin/bash#PBS -l nodes=NUM_NODES_TAG:ppn=PROCS_PER_NODE_TAG#PBS -l walltime=00:10:00#PBS -N testddt#PBS -o $PBS_JOBID.out.pbs#PBS -e $PBS_JOBID.err.pbs#PBS -r n#PBS -V#-----------------------------------------------------cd $PBS_O_WORKDIRMPIRUN_TAG \

AUTO_MPI_ARGUMENTS_TAG EXTRA_MPI_ARGUMENTS_TAG \DDTPATH_TAG/bin/ddt-client \DDT_DEBUGGER_ARGUMENTS_TAG \

PROGRAM_TAG \PROGRAM_ARGUMENTS_TAG


IntroductionHistory

The SetupUsage

Summary



Here are what these tags might evaluate to:

PROGRAM_TAG /lustre/home/[username]/workshop/progPROGRAM_ARGUMENTS_TAGNUM_NODES_TAG 2PROCS_PER_NODE_TAG 8DDT_DEBUGGER_ARGUMENTS_TAG

--ddthost ra.mines.edu --ddtport 4242 --ddtsession 1--ddtsessionfile

/lustre/home/[username]/.ddt/session/ra.mines.edu-1MPIRUN_TAG mpirunAUTO_MPI_ARGUMENTS_TAG -np 16EXTRA_MPI_ARGUMENTS_tagDDTPATH_TAG /lustre/home/apps/ddt2.5.1

Other tags are pre-defined, and user-defined tags can bemade.


IntroductionHistory

The SetupUsage

Summary



Takeaway:create a submission template filename and place it as specified in the DDT optionsexample:/lustre/home/[username]/workshop/ra.qtf


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


Compile with the debug flag

Using the appropriate MPI Compiler Wrapper for theIntel Compilers

ifort icc

compile your program with the debug flag, -g:

mpif90 -g -o prog prog.f90mpicc -g -o prog prog.c

Probably you want to compile without optimizations, as these make the connectionbetween your source code and the running code more tenuous.

(A warning about feupdateenv may be harmless, but can be removed by adding the

flag -shared-intel.)


IntroductionHistory

The SetupUsage

Summary


Start DDT, Submit Job

Login to Ra with X-Windows enabled, ssh -X ra.Change to the program directory, cd ~/workshop

Start DDT, ddt &Prepare to run a program

From the Welcome screen menu, selectRun and Debug a ProgramFrom the main menu, selectSession / New Session... / Run...

At the DDT - Run (queue submission mode) window:Set Application to your executable, say/lustre/home/[username]/workshop/progSet Arguments as neededVerify Number of Nodes (1 or 2, say)

Press SubmitDavid M. Larue DDT on Ra 26/44

IntroductionHistory

The SetupUsage

Summary


Wait for Queue, DDT

DDT will submit your job to the queueDDT displays the result of qstat, highlighting your Job IDYou wait until your job runsDDT connects to the processes your job has startedYour processes have been paused after MPI_InitSample screen with successful connection follows


IntroductionHistory

The SetupUsage

Summary


Successful Submission, Run and Connection


IntroductionHistory

The SetupUsage

Summary

Common classes of Parallel Programming BugsFeaturesSample Run

Outline





IntroductionHistory

The SetupUsage

Summary


Serial Bugs, 1/2

Parallel programs suffer from all the same flaws as serialprograms:

out of bound memory accessdivision by 0 and other improper arguments to functionsoverflow or underflowsyntactical and semantical errorsother algorithm flaws


IntroductionHistory

The SetupUsage

Summary


Serial Bugs, 2/2

All the usual debugging techniques for serial programs areavailable here in DDT:

one can set break pointsexamine or change the values of variablesstep over or into statements and function calls

One can do these thingsin individual processes, orkeep all the processes in sync.


IntroductionHistory

The SetupUsage

Summary


Parallel Bugs

Three often cited classes of new types of bugs in parallel code:Data Race ConditionsDeadlockLiveLock


IntroductionHistory

The SetupUsage

Summary


Data Race Conditions

Shared Memory (OpenMP)one process can write to memoryat the same time another process can read itinadequate synchronization methods are presentthe result depends on the sequence of actions

We are focusing here on message passing and MPI, this is notwhere we dwell, but mention that DDT has debugging supportfor

OpenMPthreadsmemory

Debuggers change timing, so these can bevery difficicult to locate.


IntroductionHistory

The SetupUsage

Summary


Deadlock

Shared Memory (OpenMP) andMessage Passing (OpenMPI)

multiple blocked processes are waiting on each otherfor release of locksfor messagesor terminating prematurely

circular locking:A waits for B waits for C waits for A, and the like.protocol mismatch:A expecting from B,B not expecting to send, or sends with wrong tag


IntroductionHistory

The SetupUsage

Summary


Livelock

Shared Memory (OpenMP) andMessage Passing (OpenMPI)

multiple non-blocked processes are repeatedly waiting oneach othertiming coincidence keeps from resolution

This can manifest rarely and occasionaly, without a blockedprocess, and so be hard to find.


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


Features: User Interface

User InterfaceGUITabbed documentsDockable windowsSave/Restore preferencesSearch/Go to/Highlight current/View Multiple source code

Control methodsIcons/ButtonsMenusFunction keysMouse clicks, double-clicks, right-clicks


IntroductionHistory

The SetupUsage

Summary


Features: Groups

GroupsDefault

AllRootWorkersIndividual Processes

Can add/edit/delete groups“Current Group” Affects

Which Processes you start/stop/stepWhich Variables displayedWhich Code and Line, and which Stack, is visible


IntroductionHistory

The SetupUsage

Summary


Features: Controlling Your Processes

Controlling Your ProcessesAdvancing by a line (Stepping over)Stepping into a functionStepping out of a functionFollowing if branchesPausing at breakpointsManually pausing and resuming

Control individually, or by group


IntroductionHistory

The SetupUsage

Summary


Features: Viewing and Editing Data

Viewing and editing dataCurrent lineLocal dataKeeping an expression in view: EvaluateWatch: break on changeEditing data

Structured dataExamine pointersMulti-dimensional arraysVisualizations, StatisticsView Message Queues

Memory debuggingIntercept [de]-allocationsStatistics


IntroductionHistory

The SetupUsage

Summary


Outline





IntroductionHistory

The SetupUsage

Summary


Sample Run

Time and Technology permitting, let’s try a sample run


IntroductionHistory

The SetupUsage

Summary

Summary

We have looked atWhy debug?Different debugging technologiesA modern product available on Ra: DDT

GUIParallel

Multiple modes

SetupOne time: configure tools, template PBS script

Each time: compile with debugging, use X

BugsSerial

Parallel: Race Conditions, Deadlock

DDT FeaturesUser InterfaceProcess Control

Data and Memory Control


IntroductionHistory

The SetupUsage

Summary

The End

Questions?


DDT: A visual, parallel debugger on Rageco.mines.edu/workshop/march2010/ddt.pdf · DDT 2.5.1 BASH...

Documents

Transcript of DDT: A visual, parallel debugger on Rageco.mines.edu/workshop/march2010/ddt.pdf · DDT 2.5.1 BASH...