[R-sig-ME] Using gdb to debug memory errors

[Douglas Bates]

Some of these instructions are given in section 4.4 of the manual
"Writing R Extensions", which you should also read.

It is best to recompile and reinstall the lme4 package removing any
optimization settings before trying to invoke the debugger on the
compiled code.  It is possible to run the debugger on optimized code
but it can be very confusing to do so.

I define two versions of the CFLAGS macro, one with a -O setting and
one without, in the file ~/.R/Makevars and comment out the one that I
don't want to use.  The R CMD INSTALL process automatically looks for
this file.  My version is

bates at gchq:~$ cat /home/bates/.R/Makevars
CFLAGS = -g -O3 -std=gnu99 -Wall -pedantic
#CFLAGS = -g -std=gnu99 -Wall -pedantic

so I am currently set for optimization.  Switching the position of the
'#' character to the other line sets you up for debugging.

I generally run both R and gdb inside emacs so my sequence is to start
R (using ESS) and attach the lme4 package then start gdb (using M-x
gdb<RET>).  This prompts you for the name of the executable file which
is $R_HOME/bin/exec/R for whatever your value of $R_HOME is.  You need
to attach the debugger to the already running R process for which you
need the process number.  One way to get that is to examine the output
from "ps ux".  You can sometimes use "pgrep R" to list the processes
and grep for 'R' in one step.  Typically that will produce two
numbers, the first of which is the shell script called R and the
second of which is the actual process.

Go to the gdb buffer which will be called *gud-<something>* (that's
not a typo - 'gud' is the grand unified debugger interface) and enter

attach <process number>

If things are going well at this point you will get a lot of output
about the symbols that gdb has read.  When that finishes you can set
breakpoints in the code if you wish.  Typically you set a breakpoint
at the beginning of a function such as

break mer2_getPars

Whether or not you set breakpoints you need to restart the R process
to be able to return control to it.  I do this by sending the 0 signal
from within gdb

signal 0

At that point you can return to the ess buffer and use R again.  There
may be peculiarites in refreshing this window inside ESS (perhaps
Martin will be able to provide more detail on what happens in ESS at
this point).  If the process seems to have hung then use C-c C-c to
see if it is just waiting for an echo.  Once it goes into a function
where you have a breakpoint you can return to the *gud-<whatever>*
window and step through the execution using 'n' (next line) or 's'
(single step).  You can also print out the values of variables with 'p
<name>'.  For vectors there is a convenient notion of an artificial
array produced with the '@' sign so you could ask for

p dims[0]@6

to get 6 elements from an array called dims.

An alternative is to run your R code that causes the memory fault and
let the debugger catch the signal.  That is, attach the debugger to
the process but don't set any breakpoints.  Just let it run until the
problem crops up.  Frequently the problem is detected long after it
has occurred so you need to look at the output of

where

(which could be long indeed) and move up several frames to get to a
place where you recognize the code.