Debugging with GDB

Sakeeb Sabakka

Agenda

Introduction to GDB GDB commands Malloc debugging Introduction to disassembled code Other debugger tools Remote debugging

What is GDB

The Gnu DeBugger As gdb(1) man page says: gdb “allow

you to see what is going on inside another program while it executes—or what another program was doing at the moment it crashed.

Compile the source code for debugging with -g option.

override CFLAGS +=-g

Getting Started…

Getting Started… `gdb' to start GDB. quit or Ctrl-d to exit.

The most usual way to start GDB is with one argument, specifying an executable program:

# gdb a.out

You can also start with both an executable program and a core file specified:

# gdb a.out core

You can, specify a process ID if you want to debug a running process:

# gdb –p 4321 

Getting Started… Command Completion (gdb) info bre < TAB > (gdb) info breakpoints

gdb) b make_ TAB make_a_section_from_file make_environ make_abs_section

make_function_type make_blockvector make_pointer_type make_cleanup make_reference_type make_command make_symbol_completion_list

(gdb) b make_

Getting Started… (gdb) b 'bubble( M-? bubble(double,double) bubble(int,int)

(gdb) b 'bubble(

Type in quotes and press TAB or type M-?Useful in function overloading scenarios

(gdb) shell [shell command]execute the specified command in shell

Getting Started… (gdb) help (gdb) help command (gdb) aprorpos < regex >

• Search for commands matching the given regex.

(gdb) info (gdb) help info (gdb) info set (gdb) help show

Getting Started… (gdb) run [ args ]

• (gdb) file /home/sakeeb/bin/gdb_ex1• (gdb) run 87 178

(gbd) set args [ args ]• (gdb) file /home/sakeeb/bin/calculate_bms• (gdb) set args 87 178• (gdb) run

(gdb) kill stop the execution of the binary being debugged.

Breakpoints… A breakpoint makes your program stop whenever a

certain point in the program is reached. For each breakpoint, you can add conditions to control in finer detail whether your program stops.

Breakpoints are set using break command (gdb) break locationThe breakpoint location can be any of the following linenum filename:linenum function filename:function

Breakpoints… Conditional Breakpoints (gdb) break if (expression) (gdb) condition bN (expression)

(gdb) until [location] – goto location or skip the loop

Setting break points for a single time (gdb) tbreak args

Using regular expression to set breakpoints (gdb) rbreak regex e.g (gdb) rbreak link* (gdb) rbreak Specify the commands to be executed at a breakpoint

(gdb) commands [ bN ]

Breakpoints…

(gbd) break traverse_tree(gdb) commands> if(node == 0x0)

> print head> else> print *head> end

> continue> end

(gdb) condition 1 (node == 0x0)

(gdb) break random_tree if (branch_count <= 2)

Breakpoints…

continue [COUNT] Continue execution, ignore the current break point for COUNT-1 times.

ignore <BP# [COUNT]> Set ignore-count of breakpoint

save-breakpointsSave current breakpoint definitions as a script

info breakpoints [N]Display information about current breakpoint[s].

Stepping…

Single stepping through the function is done through step or stepi command. Step steps into the functions calls

Stepping over the function call is done using next or nexti command.• stepi and nexti do stepping at assembly instruction

where step and next do stepping at source code level.

The continue statement continues the execution of the program till it reaches next break point or program termination

Function Manipulation… Returning from a function without executing the the

remaining code(gdb) return

Get the control after executing the function (gdb) finish The value of the variable can be changed in the frame

where it is defined.Set the value of i to 20 in the current frame/function

(gdb) print i=20 or

(gdb) set i=20 Invoking a function from gdb.

(gdb) call <function name>

Watchpoints…… Watchpoint stop the program execution whenever the value of

expression changes (gdb) watchpoint temp->next (gdb) watchpoint *(int*)0x56ad52 (gdb) watchpoint ‘a+b / c+d’ Read watch point (gdb ) rwatch expr [thread threadnum]

Read / Write watchpoint (gdb) awatch expr [thread threadnum]

Information on watchpoints (gdb)info watchpoints

Catchpoints….

Catchpoints causes gdb to stop for certain kinds of events.

(gdb) catch event“event” can be

exec call to execfork | vfork call to fork or vforkload libname dynamic loading of any shared libraryunload libname dynamic unloading thread_start | thread_exit | thread join

before create, before exit or after join of threads.

start | exit after create or before exit of procesesthrow throwing of c++ exception

catch catching of c++ exception

Core files… Start gdb with executable and core file

$ gdb a.out core ( or gdb a.out –c core)or (gdb) core core

Stopping execution just before crash happens.$ gdb –crashdebug a.out$ gdb –crashdebug –pid <pid>

Generate the dump of the process(gbd) dumpcore use –lcore or set live-core=1 for using the gdb generated corefile$ gdb –lcore a.out core

Or(gdb) set live-core 1(gdb) core core

Core files… To set the core file size

# ulimit –c# ulimit –c <size>

To dump share memory of the process into the core dump. $ adb -w -k /stand/vmunix /dev/mem(adb) core_addshmem_read/W 1

To append pid of the process to core file while application dumps core$ adb -w -k /stand/vmunix /dev/mem(adb) core_addpid/W 1

Stack Frames... When program stops GDB helps you in finding all the information

stored in the stack frame ( frame pointer register ) Frame contains the arguments given to the function ,

functions local variables and the address at which the function is executing

GDB by default is in the currently executing frame (frame 0) and displays information pertaining to that frame.

Back trace – summary of how your program got to place where it is.

Function Arguments & Environment Variables

StackCollection of stack frames. Grows Downward.

Unused Memory

HeapDynamic memory for programs and libraries.

Grows upward.

Un-initialized Data Segment (BSS)Executable stores only size required for this area, memory will

allocated and initialized to zero on process instantiation.

Initialized Data SegmentInitialized global variables from executable

Text SegmentMachine Code. Shared between all the processes executing the

same binary. Read-Only.

Stack Frames... (gdb) backtrace / bt / where - Prints the entire stack (gdb) bt n - prints the innermost n frames (gdb) bt –n –Prints the outermost n frames (gdb) bt full –Prints the information of local variables

Selecting a frame(gdb) frame <num>

(gdb) frame addr (gdb) up n – towards outermost frame (gdb) down n – towards innermost frame

Stack Frames... Information about frames (gdb) info frame / f

Print -address of the current frame

address of previous and next framesaddress of local variables and parameters

program counter registers saved in the frame(gdb) info args – Prints the argument of the selected frame(gdb) info locals – Prints the local variables(gdb) info scope <function> - info about locals in the function(gdb) info catch - Prints all exception handlers active

Threads… List of threads (gdb) info thread

(gdb) thread <thread-id>(gdb) thread apply <thread-id | all > command

Apply the command on the specified thread.

To view the backtrace of all the threads use the following (gdb) thread apply all backtrace

(gbd) backtrace-other-thread

Examining the Data... Display the value of an expression

(gdb) print[/FMT] (expression)

(gdb) print i(gdb) print/s string

(gdb) output (expression) same as print, but don't put in value history and don't print newline.

Examine the value of memory(gdb) x /FMT <address>

(gdb) x/s string Display the value of expression each time program stops

(gdb) display/FMT (expression)(gdb) display i

(gbd) help set radix

Examining the Data...Print format modifiers x Print in hexadecimal d Print as a signed decimal u Print as an unsigned decimal o Print as octal t Print in binary (t is for two) a Print as an address c Print as character

(gbd) x/FMT (expression)(gdb) x/d &i(gdb) x/s s(gdb) x/c s(gdb) x/4c s(gdb) x/t s(gdb) x/3x s

Examining the Source… Display source code

(gdb) list LINENUM | FILE:LINENUM | FUNCTION | FILE:FUNCTION | *ADDRESS ]

(gdb) list 5,(gbd) list ,28(gbd) list 21, 38(gdb) list parse_args(gbd) list main.c:doit

(gdb) set listsize 24

Searching the source code(gbd) search regex(gdb) reverse-serach regex

Examining the Source… (gdb) whatis insert::node

Display the type of the expression. (gdb) ptype insert::node

Display the description of the type of the expression.Note: - insert::node refers to node variable in function insert

`@' is a binary operator for treating parts of memory as arrayseg:- int *array = (int *) malloc (len * sizeof (int))

(gdb) print *array@len {type}addr is used for display the value at addr as type (type)addr is used for display the value as type

(gdb) p/x (short[2])0x12345678

Signals… GDB has ability to detect any occurrence of signal in execution (gdb) info signals (gdb) info handle (gdb) info signal [Signal num] Print table of how gdb handles signals Change the GDB behavior for signal handling (gdb) handle signal [keywords] signal can be signal name or Num. Keywords can be nostop – Not to stop Program execution

stop- Stop Program execution print / noprint – should / should not print the signal info pass / noignore – allow program to see the signal

nopass / ignore – should not allow program to see the signal

Logging... Logging Output

set logging on • Enable logging.

set logging off • Disable logging.

set logging file file • Change the name of the current logfile. The default logfile is

`gdb.txt'. set logging overwrite [on|off]

• By default, GDB will append to the logfile. Set overwrite if you want set logging on to overwrite the logfile instead.

set logging redirect [on|off] • By default, GDB output will go to both the terminal and the

logfile. Set redirect if you want output to go only to the log file.

show logging • Show the current values of the logging settings.

The .gdbinit File… GDB will look for this file in two places

First, In your home directory Second, In the current directory

comments are started with "#” and blank lines are ignored. Put redundant commands into .gdbinit

Eg:- cat .gdbinitset listsize 24set break mainset sourcedir /home/sakeeb/imported_sourcesset objectdir /home/sakeeb/impored_objectsset logging file /tmp/sakeeb.gdb.logset loggin on

Automate most of debugging tasks Use .gdbinit to automate debugging gdb commands stored in alternate files can be executed using –x option. Make use of –batch option to run gdb in batch mode

eg: - gdb –batch –x /home/sakeeb/audfilter/crash_debug_commands

The child process…

(gdb) set follow-fork-mode [ parent | child | ask | serial ]The unfollowed process will continue to run.By default, the debugger will follow the parent

process.

# inetd –k# chatr +dbg enable /usr/sbin/inetd # gdb /usr/sbin/inetd(gdb) set follow-fork-mode child(gdb) break execve(gdb) continueInvoke the service to be traced (eg;- telnet to the m/c to invoke and debug telnetd)

(gdb) step!!! we are insided the spawned process..

Convenient Breakpoints…

(gdb) rbpSet breakpoints and actions for all functions.

(gdb) rdpDelete breakpoints set by rbp.

(gdb) xbpSet breakpoint for all function return point.

(gdb) xdpDelete breakpoints set by xbp.

(gdb) xbreak <funcion>Set breakpoint at the function exit.

Debugging memory…

(gdb) set heap-check [option] <on | off>option is one of :

leaks – leak detection, defaut with heap-check.bounds – bounds check, defaut with heap-check.free - validation of free, defaut with heap-check.string – validation of arguments to str* functions.scramble – scramble the deallocated blocks with

value 0xfeedface

(gdb) xbp(gdb) set heap-check on(gdb) run

<-- at break-point at function return -->(gdb) info leaks

Debugging memory…

(gdb) set heap-check watch <address>watch allocation/deallocation of address

(gdb) set heap-check null-check-size <num>malloc returns null after allocation of num bytes

(gdb) set heap-check null-check <n | random>instruct malloc to return null after nth invocation

(gdb) set heap-check block-size <num-bytes>Break when allaction size is > num-bytes

(gdb) catch nomemcatch null pointer returned by malloc.

Debugging memory…

(gdb) info corruptionChecks for corruption in the currently allocated heap.

(gdb) info heap <process | arenas>report high level memory usage of a process/arena.

(gdb) help info heap(gdb) info heap

(gdb) help set heap-check

Debugging memory… Use +check=[option] compiler options (IA compiler) to

enable runtime checks Options are:• all | none • bounds – check out of boundary array reference.• malloc – memory leak and heap corruption• stack – writing outside the stack frame• uninit – reading un-initialized variables.

A failed check results in the program abort at runtime with stack trace on stderr.

RTC_NO_ABORT env variable must be set to 1 to continue the program execution after a failed runtime check.

Low level debugging…

(gdb) info shareddetails about shared library mapping into memory.

(gdb) info registersdetails about registry contents.

(gdb) disassemble [address | function]

disassemble the program to assembly instructions.

Low level debugging…

Procedure calling in PA32 binary• The first 4 arguments are passed in registers

(r26==arg0, r25==arg1, r24==arg2, r23==arg3)• Arg4 and above are passed on the stack.• Minimum stack frame size if 0x40. This includes space

for up to 4 optional arguments.• rp (return pointer) register is the only thing most

functions save onto the stack. • Leaf functions ( function which does not call other

functions) doesn’t allocate stack frame.• Structures are passed as pointer to its base address.

Low level debugging…

(gdb) disass fooDump of assembler code for function foo:;;; File: args.c;;;  17 {

<-- saves return pointer0x29ec <foo>:  stw %rp,-0x14(%sp)

<-- creates stack frame (0x40 + space for locals)0x29f0 <foo+0x4>:      ldo 0x140(%sp),%sp

<-- stores the argument to stack ( at 0x24 + frame size)0x29f4 <foo+0x8>:      stw %r26,-0x164(%sp)0x29f8 <foo+0xc>:      stw %r25,-0x68(%sp)0x29fc <foo+0x10>:     stw %r24,-0x6c(%sp)0x2a00 <foo+0x14>:     stw %r23,-0x70(%sp)

PA32 binary function call stack

sp Current stack pointer

sp-0x4 Previous stack pointer

sp-0x14 RP (return pointer)

sp-0x18 External Stub RP (RP’)

sp-0x1C External LPT’ (SR4/Code)

sp-0x1C External Data LPT

sp-0x24 to sp-0x30

arg0 to arg3

sp-(4*(N+9))

Nth arg (for N>=4) optional, allocated only if function takes more than 4 arguments.

Low level debugging…

Low level debugging…

Procedure calling in IPF 32 binary• The first 8 arguments are passed through register stack.• Arg8 and above are passed on the stack.• Unlike PA, arguments are not stored back to stack.• Leaf functions doesn’t allocate stack frame.• The stack on Itanium systems grows the opposite

direction as compared to PA-RISC.

Low level debugging…

(gdb) disass foo<-- saves previous function state to r50.<-- allocates 20 local registers (including its arguments)<-- allocates 5 output registers.

0x4000a00:0 <foo>:       alloc            r50=ar.pfs,0,20,5,0<-- saves return pointer to to r51.

0x4000a00:1 <foo+0x1>:  mov              r51=b0<-- increment stack frame by size 64.

0x4000a00:2 <foo+0x2>:  adds             r12=-64,r12;;<-- stores global pointer (r1) to r36.

0x4000a00:2 <foo+0x3>: addl r36=0,r1 <-- set return value (r8) to 0.

0x4000a00:42 <foo+0x42>: mov r8=0

Low level debugging…

How functions in a shared library are invoked.1. Program calls import stub.2. Search PLT for the function. If not found, invoke dynamic

loader and update PLT.3. The export stub is called.4. The export stub calls the real function5. The real function returns to the export stub6. The export stub returns back to the original caller.

Other Tools...

Change programs internal attributeschatr +dbg enable|disable <executable_name>controls the mapping of shared libraries mapping privately.

and thus enable break point setting and debugging.

chatr +s enable|disable <executable_name> controls whether the SHLIB path can be used to locate

shared libraries.

chatr –z <executable name>Enable run time dereferencing of null pointer to produce SIGSEGV

chatr <executable_name> : shows the attributes.

Other Tools…sh# adb <exe> <core>adb > c

prints the backtraceadb > m

initial and default maps for a valid core file with an indication of which is currently active

Using crashinfo to display stack tracesh# crashinfo -t -l pid=<PID>

Other Tools…

P4/Q4/KWDB

p4> p s curr_file p4> p int audit_state_flagp4> trace pid 1P4> p proc_t 0xe0000001806ba000 | grep

p_auditperproc p4> p s $(( 0xe0000001_806bb1a0 + 32 )) p4> p int cmpt_enabled

Other Tools…

nm – lists the symbols in an object file.• $ nm –A /usr/lib/hpux32/lib*.so | grep milli• Symbols categorized: extern, global or undefined. • Symbols origin: data or text (code).• If the object file is stripped, nm can’t extract symbols.

Use odump or elfdump in those cases

$ odump -slexport /usr/local/lib/libintl.a | grep “__umodsi3”

$ elfdump -n .dynsym -s /usr/local/lib/hpux32/libintl.a | grep “__umodsi3”

Other Tools…

lsof – lists open files and sockets• $ lsof –l /usr/sbin/audomon

fuser - find processes which opened a given file• fuser /var/spool/audfilterd/audfilterd.lock

Other Tools…

tusc – system call tracer• $ tusc –s bind 2134• $ tusc –E /usr/bin/ioscan –funC disk• $ tusc –a –p –f –n –k <deamon-pid> -o

/tmp/tusc.log

ltrace – libc tracer.• Currently only for 32-bit IPF binaries.• libc call tracing similar to tusc.• Available with the linker patch :

11.23: PHSS_38134 11.31: PHSS_38135

Remote Debugging... Normally used when gdb cannot be run on target process or

machine . For example• Debugging Kernel boot process (KWDB)• Embedded systems

Target process is run/attached using gdbserver program• Not available on HP-UX.• For running kernel for remote debugging, specify at the ISL

prompt.ISL> hpux <vmunix> kwdb_flags=lan[.<driver>[.nowait]]ISL> boot <vmunix> kwdb_flags=lan kwdb_hw_path=1/1/1/1HPUX> boot <vmunix> kwdb_flags=sub[.nowait]HPUX> boot <vmunix> kwdb_flags=udp kwdb_hw_path=1/1/1/1

kwdb_udp_ip_addr=15.76.98.42

Remote Debugging… Use target command to connect to remote gdb process Tracepoints helps you to trace the program without

stopping it. • trace command sets the tracepoint.• actions + collect command sequence collects the data.

(gdb) trace tree.c:74(gdb) actions> collect *tree> collect $locals

> while-stepping 3> collect bst>end

>end

• while-stepping command specifies the single stepping behavior.

Remote Debugging... Collect command collects data at tracepoints.

It can collect details like• Registers – via $regs• Function Args – via $args• Local Variables – via $vars• Individual variables – via variable name(gdb) collect $locals(gdb) collect max_index

Passcount command can be used to set the count of data collection

(gdb) passcount 8 2Stop execution at 8th involcation of tracepoint 2

(gdb) passcount 2Stop execution at 2nd involcation of most recent

tracepoint.

Remote Debugging… A trace event is generated whenever a tracepoint is

executed.

Every trace event generates a data record called a trace frame in a buffer called the trace buffer.

If a tracepoint's actions include single-stepping, each single-step also generates a trace event and a trace frame.

Each trace frame has a sequential identifying number starting from zero, plus the address of the event.

Remote Debugging…

Following convenience variables can be used to collect data at the tracepoints.

$trace_frame - the ID of the currently selected frame or -1 if no trace frame is selected.

$tracepoint - the ID of the tracepoint corresponding to $trace_frame

$trace_line - the source line number corresponding to $tracepoint

$trace_file - the source filename corresponding to $tracepoint

$trace_func - the name of the function containing $tracepoint

Remote Debugging…

Tracepoint variables example:

(gdb) tfind start(gdb) while ($trace_frame != -1)

> printf "Tracepoint %d, ", $tracepoint> printf "line %d, ", $trace_line> if ($tracepoint == 1)

> printf ”node value = %d\n", data> end> if ($tracepoint == 2)

> printf ”DLT== %x, ", $r19 > printf ”bst node = 0x%x\n", *bst

> end> tfind

> end

Remote Debugging…(gdb) tstart

Start trace data collection.(gdb) tstop

Stop trace data collection.(gdb) tstatus

Status of trace data collection.(gdb) info tracepoints [number]

List of trace points.(gdb) save-tracepoints <filename>

Save the current tracepoint definitions to a file.

Remote Debugging…(gdb) tdump

Print all of the data collected at the current trace frame.(gdb) tfind [< event id > | start | none | -]

Select frame by id number.(gdb) tfind tracepoint [< tracepoint id >]

Select the next trace for the given tracepoint.(gdb) tfind pc [< address >]

Select the next trace frame for the given code address.(gdb) tfind line [[filename:] linenum]

Select the next trace frame the given source line.

Remote Debugging…

Tracepoint Breakpoint

(gdb) tfind start (gdb) continueTracepoint #12 at tree.c line 34 Breakpoint #12 at tree.c line 34(gdb) print head (gbd) print head$1 = 0x12345678 $1 = 0x12345678(gdb) tfind next (gdb) steptree.c line 45 tree.c line 45(gdb) tfind line tree.c:53 (gdb) until tree.c:53Tracepoint #3 at tree.c:53 tree.c line 53(gdb) print head->data (gdb) head->data$2 = 28 $2 = 28(gdb) print head->left (gdb) head->leftData not collected. $3 = 0x0