[Bioc-devel] [BioC] Rsubread crashes in 32bit linux

[Robert Castelo]

dear Wei,

thanks for getting quick into this problem, i have a suggestion about 
something you say below but since its very technical i've decided to 
move this particular thread to bioc-devel just in case other more 
experience developers can help, please keep reading.

On 06/06/2012 12:10 PM, Wei Shi wrote:
> Dear Dan,
>
> It didn't seem to be problem of requesting a continuous 1GB block in our
> investigation. We tracked the memory usage of buildindex() function when
> running it on yeast genome using a 32-bit VM, and found that the segfault
> happened right after a request of a few KB of memory was sent to the
> system when the memory parameter was set to 2500. However, the problem was
> gone when the memory parameter was changed to 1000.

i understand from what you write here that the fact that the problem may 
be gone by changing the memory parameter to 1000, does not explain the 
underlying issue that crashes the software.

in my experience, this kind of obscure correlations of behaviour occur 
due to memory leaks elsewhere in the code which in general are very 
difficult identify without using a memory profiling tool.

in case you have not done that yet, i'd recommend you to give it a try 
using valgrind. i've taking the liberty of doing it myself in 64bit 
linux, i.e., where the package does not crash.

i think it points out to some problem in the code. if you want to 
reproduce this please put into a file called test.R the following code 
by Dan which reproduces the problem using the latest R and Rsubread 
versions:

=========================test.R==================
library(Rsubread)
ref <- system.file("extdata","reference.fa",package="Rsubread")
path <- system.file("extdata",package="Rsubread")
buildindex(basename=file.path(path,"reference_index"),reference=ref)
=================================================

and then call valgrind as follows from the shell of a linux box (it will 
take several minutes):

$ R -d "valgrind --tool=memcheck --leak-check=yes --show-reachable=yes 
--track-origins=yes" --vanilla < test.R &> test.out

(notice that this is a single line on the shell, but the email software 
my break that line)

the output dumped in test.out is quite long but i found the following part:

==13386== Syscall param write(buf) points to uninitialised byte(s)
==13386==    at 0x36B8AD3A00: __write_nocancel (in /lib64/libc-2.11.2.so)
==13386==    by 0x36B8A70FC2: _IO_file_write@@GLIBC_2.2.5 (in 
/lib64/libc-2.11.2.so)
==13386==    by 0x36B8A70E89: _IO_file_xsputn@@GLIBC_2.2.5 (in 
/lib64/libc-2.11.2.so)
==13386==    by 0x36B8A6707C: fwrite (in /lib64/libc-2.11.2.so)
==13386==    by 0x390464E3: gvindex_dump (gene-value-index.c:103)
==13386==    by 0x3904760E: build_gene_index (index-builder.c:135)
==13386==    by 0x390481C1: main_buildindex (index-builder.c:703)
==13386==    by 0x3902CE94: R_buildindex_wrapper (R_wrapper.c:31)
==13386==    by 0x4CB4C10: do_dotCode (dotcode.c:1687)
==13386==    by 0x4CE487A: Rf_eval (eval.c:492)
==13386==    by 0x4CEB45F: do_set (eval.c:1711)
==13386==    by 0x4CE469A: Rf_eval (eval.c:466)
==13386==  Address 0x1e80eec54 is 5,188 bytes inside a block of size 
461,615,034 alloc'd
==13386==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==13386==    by 0x390463B6: gvindex_init (gene-value-index.c:14)
==13386==    by 0x39047C08: build_gene_index (index-builder.c:87)
==13386==    by 0x390481C1: main_buildindex (index-builder.c:703)
==13386==    by 0x3902CE94: R_buildindex_wrapper (R_wrapper.c:31)
==13386==    by 0x4CB4C10: do_dotCode (dotcode.c:1687)
==13386==    by 0x4CE487A: Rf_eval (eval.c:492)
==13386==    by 0x4CEB45F: do_set (eval.c:1711)
==13386==    by 0x4CE469A: Rf_eval (eval.c:466)
==13386==    by 0x4CE6ABF: do_begin (eval.c:1409)
==13386==    by 0x4CE469A: Rf_eval (eval.c:466)
==13386==    by 0x4CE7A02: Rf_applyClosure (eval.c:855)
==13386==  Uninitialised value was created by a heap allocation
==13386==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==13386==    by 0x390463B6: gvindex_init (gene-value-index.c:14)
==13386==    by 0x39047C08: build_gene_index (index-builder.c:87)
==13386==    by 0x390481C1: main_buildindex (index-builder.c:703)
==13386==    by 0x3902CE94: R_buildindex_wrapper (R_wrapper.c:31)
==13386==    by 0x4CB4C10: do_dotCode (dotcode.c:1687)
==13386==    by 0x4CE487A: Rf_eval (eval.c:492)
==13386==    by 0x4CEB45F: do_set (eval.c:1711)
==13386==    by 0x4CE469A: Rf_eval (eval.c:466)
==13386==    by 0x4CE6ABF: do_begin (eval.c:1409)
==13386==    by 0x4CE469A: Rf_eval (eval.c:466)
==13386==    by 0x4CE7A02: Rf_applyClosure (eval.c:855)
==13386==
==13386== Warning: set address range perms: large range [0x1e80ed800, 
0x2039287da) (noaccess)

which seems to point to some use of uninitialised values in line 103 of 
the build index-builder.c through a pointer to memory allocated in line 
87 of index-builder.c

i must confess i'm not looking to the source code myself just trying to 
interpret directly the valgrind output, but this looks definitely leaky 
to me. let me know if you and your collaborators have difficulty to spot 
the problem in your code and i'll try to find a moment to look into it.

another issue, in which others in this list may have a more concrete 
advice, is that i'd say that you should replace all the calls to 
malloc() by R_alloc() and if you have any call to calloc()/free() by 
Calloc()/Free(). these are internal R wrappers for the allocation and 
release of memory which i think make the package more friendly to R and 
to the users when, for instance, main memory is exhausted.

cheers,
robert.