[BioC] Rsubread crashes in 32bit linux
Dan Du
tooyoung at gmail.com
Wed Jun 6 13:08:54 CEST 2012
Dear Wei,
Here is a standard bioclite update, I think it is at the last step when
compiling Rsubread.so, the memory usage exceeds 5.5g, then system freeze
and I have to call it off. Same result when runing 'R CMD INSTALL
Rsubread_1.6.3.tar.gz' from shell, or manually compile all .c file and
run the last gcc statement. I guess there might just be a minimum ram
requirement somewhere higher than 6g... I will do some more poking when
I have time.
'gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o
aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o
exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o
gene-value-index.o hashtable.o index-builder.o input-files.o
processExons.o propmapped.o qualityScores.o readSummary.o
removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread
-DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR'
Also down there are the sessionInfo and full gcc version, please let me
know if you need more information.
Regards,
Dan
--------------------------------------------------------------------
> source('http://www.bioconductor.org/biocLite.R')
> biocLite('')
BioC_mirror: http://bioconductor.org
Using R version 2.15, BiocInstaller version 1.4.6.
Installing package(s) ''
Old packages: 'Rsubread'
Update all/some/none? [a/s/n]: a
trying URL
'http://www.bioconductor.org/packages/2.10/bioc/src/contrib/Rsubread_1.6.3.tar.gz'
Content type 'application/x-gzip' length 21891723 bytes (20.9 Mb)
opened URL
==================================================
downloaded 20.9 Mb
WARNING: ignoring environment value of R_HOME
* installing *source* package ‘Rsubread’ ...
** libs
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c R_wrapper.c -o R_wrapper.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c SNP_calling.c -o SNP_calling.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c aligner.c -o aligner.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c atgcContent.c -o atgcContent.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c detectionCall.c -o detectionCall.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c detectionCallAnnotation.c -o detectionCallAnnotation.o
detectionCallAnnotation.c: In function ‘calculateExonGCContent’:
detectionCallAnnotation.c:175: warning: ignoring return value of
‘fgets’, declared with attribute warn_unused_result
detectionCallAnnotation.c: In function ‘calculateIRGCContent’:
detectionCallAnnotation.c:262: warning: ignoring return value of
‘fgets’, declared with attribute warn_unused_result
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c exon-algorithms.c -o exon-algorithms.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c exon-align.c -o exon-align.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c fullscan.c -o fullscan.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c gene-algorithms.c -o gene-algorithms.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c gene-value-index.c -o gene-value-index.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c hashtable.c -o hashtable.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c index-builder.c -o index-builder.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c input-files.c -o input-files.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c processExons.c -o processExons.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c propmapped.c -o propmapped.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c qualityScores.c -o qualityScores.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c readSummary.c -o readSummary.o
readSummary.c: In function ‘readSummary’:
readSummary.c:122: warning: format ‘%d’ expects type ‘int’, but argument
5 has type ‘long int’
readSummary.c:122: warning: format ‘%d’ expects type ‘int’, but argument
6 has type ‘long int’
readSummary.c:39: warning: ignoring return value of ‘getline’, declared
with attribute warn_unused_result
readSummary.c:52: warning: ignoring return value of ‘getline’, declared
with attribute warn_unused_result
readSummary.c:55: warning: ignoring return value of ‘getline’, declared
with attribute warn_unused_result
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c removeDuplicatedReads.c -o removeDuplicatedReads.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c sam2bed.c -o sam2bed.o
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -DMAKE_FOR_EXON -fpic
-O3 -pipe -g -c sorted-hashtable.c -o sorted-hashtable.o
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gene-algorithms.h:23: warning: inline function ‘add_gene_vote_weighted’
declared but never defined
gene-algorithms.h:22: warning: inline function ‘add_gene_vote’ declared
but never defined
gcc -std=gnu99 -shared -o Rsubread.so R_wrapper.o SNP_calling.o
aligner.o atgcContent.o detectionCall.o detectionCallAnnotation.o
exon-algorithms.o exon-align.o fullscan.o gene-algorithms.o
gene-value-index.o hashtable.o index-builder.o input-files.o
processExons.o propmapped.o qualityScores.o readSummary.o
removeDuplicatedReads.o sam2bed.o sorted-hashtable.o -lpthread
-DMAKE_FOR_EXON -L/usr/lib64/R/lib -lR
^Cmake: *** Deleting file `Rsubread.so'
make: *** [Rsubread.so] Interrupt
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
‘Rsubread.Rnw’
** testing if installed package can be loaded
Error in library.dynam(lib, package, package.lib) :
shared object ‘Rsubread.so’ not found
Error: loading failed
Execution halted
--------------------------------------------------------------------
> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: x86_64-pc-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C
[3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8
[5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
--------------------------------------------------------------------
$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
4.4.3-4ubuntu5.1'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --enable-multiarch --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
--program-suffix=-4.4 --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-plugin --enable-objc-gc
--disable-werror --with-arch-32=i486 --with-tune=generic
--enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1)
--------------------------------------------------------------------
On Wed, 2012-06-06 at 20:10 +1000, Wei Shi wrote:
> Dear Dan,
>
> It didn't seem to be problem of requesting a continuous 1GB block in our
> investigation. We tracked the memory usage of buildindex() function when
> running it on yeast genome using a 32-bit VM, and found that the segfault
> happened right after a request of a few KB of memory was sent to the
> system when the memory parameter was set to 2500. However, the problem was
> gone when the memory parameter was changed to 1000.
>
> Removing highly repetitive 16 mers required a continuous 1GB block of
> memory, but this step was always executed successfully. This step also
> included in the old version of Rsubread (1.1.1), and it did not have
> problem there either.
>
> Could you please provide us your complete code for running your test and
> also session info? This will help us to diagnose what the problem could be
> because we couldn't reproduce what you saw from our end.
>
> For the compilation issue on your 64bit laptop, could you provide us more
> details as well, including the message output from gcc?
>
> Thanks,
> Wei
>
> > Dear Wei,
> >
> > Unfortunately reducing the memory parameter to 1000, still causes the
> > segfault. I guess with 3g ram limit on a 32bit system, there is still a
> > fat chance that you can not request a continuous 1g block.
> >
> > For that 64bit laptop, it is still strange about the 6g memory draining.
> > It is happing during the installation when compiling the shared library
> > Rsubread.so, not running the buildindex function. Btw, the gcc version
> > is 4.4.3.
> >
> > Server and opensuse box runs gcc version 4.3.1 and 4.5.0 respectively.
> >
> > Regards,
> > Dan
> >
> > On Wed, 2012-06-06 at 14:56 +1000, Wei Shi wrote:
> >> Dear Dan,
> >>
> >> It is probably because including genome sequences into the index slowed
> >> down your laptop. But I believe it should be alleviated if you give
> >> smaller values to the 'memory ' parameter of the buildindex() function.
> >> Also, the index building is an one-off operation, you do not need to
> >> redo it even when new releases come.
> >>
> >> For your 32-bit opensuse box, I guess the problem will be solved if you
> >> change the amount of memory requested to be 1000MB.
> >>
> >> Cheers,
> >> Wei
> >>
> >> On Jun 5, 2012, at 11:43 PM, Dan Du wrote:
> >>
> >> > Hi Robert,
> >> >
> >> > I have been experiencing something else, possibly related to yours,
> >> > on a 64bit ubuntu laptop with 6g of ram.
> >> >
> >> > As I recall, when bumping to Bioc 2.10, the Rsubread installation kind
> >> > of ate all the memory, basically froze the system so I had to call it
> >> > off, yet building it on the server side turned out fine. So I think I
> >> > just accepted that the new version may be 'computationally heavy' thus
> >> > not suitable for a normal pc, though I did not find any mentioning of
> >> > this increased memory requirement in the NEWS file.
> >> >
> >> > So currently Rsubread stays at 1.4.4 on that pc, all subsequent
> >> versions
> >> > of Rsubread drain the memory in the same way when compiling
> >> Rsubread.so.
> >> >
> >> > Now I think I can confirm this on a 32-bit opensuse box, it did
> >> > successfully built, but when running the example code in the manual,
> >> > same segfault happens.
> >> >
> >> >
> >> >> library(Rsubread)
> >> >> ref <- system.file("extdata","reference.fa",package="Rsubread")
> >> >> path <- system.file("extdata",package="Rsubread")
> >> >> buildindex(basename=file.path(path,"reference_index"),reference=ref)
> >> >
> >> > Building a base-space index.
> >> > Size of memory used=3700 MB
> >> > Base name of the built index
> >> > = /home/opensuse/R-patched/library/Rsubread/extdata/reference_index
> >> >
> >> > *** caught segfault ***
> >> > address 0xdf03ee80, cause 'memory not mapped'
> >> >
> >> > Traceback:
> >> > 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv =
> >> > as.character(cmd), PACKAGE = "Rsubread")
> >> > 2: buildindex(basename = file.path(path, "reference_index"), reference
> >> > = ref)
> >> >
> >> >> sessionInfo()
> >> > R version 2.15.0 Patched (2012-06-04 r59517)
> >> > Platform: i686-pc-linux-gnu (32-bit)
> >> >
> >> > locale:
> >> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> >> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
> >> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
> >> > [7] LC_PAPER=C LC_NAME=C
> >> > [9] LC_ADDRESS=C LC_TELEPHONE=C
> >> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >> >
> >> > attached base packages:
> >> > [1] stats graphics grDevices utils datasets methods
> >> > base
> >> >
> >> > other attached packages:
> >> > [1] Rsubread_1.6.3
> >> >
> >> >
> >> > Regards,
> >> > Dan
> >> >
> >> > On Tue, 2012-06-05 at 09:45 +0200, Robert Castelo wrote:
> >> >> hi,
> >> >>
> >> >> the computer room at my university where we do practicals on R &
> >> Bioconductor runs a 32bit linux distribution and when i tried to run
> >> the latest version of the Rsubread package (1.6.3) it crashes when
> >> calling the buildindex() function on a multifasta file with the yeast
> >> genome. this does *not* happen under a 64bit linux distribution.
> >> >>
> >> >> i have verified that installing the version before (1.4.4) on the
> >> current R 2.15 it also crashes (on the 32bit), but two versions
> >> before, the 1.1.1, it does *not* and it works smoothly on this 32bit
> >> linux distribution.
> >> >>
> >> >> i'm pasting below the output of using the 1.6.3 and 1.1.1 on R 2.15
> >> where allChr.fa is the multifasta file with the yeast genome.
> >> >>
> >> >> so i can manage by now the problem by using the 1.1.1 version on R
> >> 2.15 for my teaching but i wonder whether there would be some easy
> >> solution for this, or even if it could be a symptom of something else
> >> that the Rsubread developers should worry about. i know that using a
> >> 32bit system nowadays is quite obsolete but this is what i got for
> >> teaching :( and i would be happy to let my students play with the
> >> latest version of Rsubread in the future.
> >> >>
> >> >>
> >> >> thanks!!!
> >> >> robert.
> >> >>
> >> >> ======================Rsubread 1.6.3 on R 2.15=======================
> >> >>
> >> >>> library(Rsubread)
> >> >>> sessionInfo()
> >> >> R version 2.15.0 (2012-03-30)
> >> >> Platform: i686-pc-linux-gnu (32-bit)
> >> >>
> >> >> locale:
> >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C
> >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8
> >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8
> >> >> [7] LC_PAPER=C LC_NAME=C
> >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C
> >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C
> >> >>
> >> >> attached base packages:
> >> >> [1] stats graphics grDevices utils datasets methods base
> >> >>
> >> >> other attached packages:
> >> >> [1] Rsubread_1.6.3
> >> >>
> >> >>> buildindex(basename="subreadindex", reference="allChr.fa",
> >> memory=2500)
> >> >>
> >> >> Building a base-space index.
> >> >> Size of memory used=2500 MB
> >> >> Base name of the built index = subreadindex
> >> >>
> >> >> *** caught segfault ***
> >> >> address 0xdf670cc0, cause 'memory not mapped'
> >> >>
> >> >> Traceback:
> >> >> 1: .C("R_buildindex_wrapper", argc = as.integer(n), argv =
> >> as.character(cmd), PACKAGE = "Rsubread")
> >> >> 2: buildindex(basename = "subreadindex", reference = "allChr.fa",
> >> memory = 2500)
> >> >>
> >> >> Possible actions:
> >> >> 1: abort (with core dump, if enabled)
> >> >> 2: normal R exit
> >> >> 3: exit R without saving workspace
> >> >> 4: exit R saving workspace
> >> >> Selection:
> >> >>
> >> >>
> >> >> ======================Rsubread 1.1.1 on R 2.15=======================
> >> >>
> >> >>> library(Rsubread)
> >> >>> buildindex(basename="subreadindex", reference="allChr.fa",
> >> memory=2500)
> >> >>
> >> >> Building the index in the base space.
> >> >> Size of memory requested=2500 MB
> >> >> Index base name = subreadindex
> >> >> INDEX ITEMS PER PARTITION = 275940352
> >> >>
> >> >> completed=40.88%; time used=1.7s; rate=2955.1k bps/s; total=12m bps
> >> completed=81.76%; time used=2.4s; rate=4111.8k
> >> bps/s; total=12m bps
> >> >> All the chromosome files are processed.
> >> >> | Dumping index
> >> [===========================================================>]
> >> >> Index subreadindex is successfully built.
> >> >>> sessionInfo()
> >> >> R version 2.15.0 (2012-03-30)
> >> >> Platform: i686-pc-linux-gnu (32-bit)
> >> >>
> >> >> locale:
> >> >> [1] LC_CTYPE=ca_ES.UTF-8 LC_NUMERIC=C
> >> >> [3] LC_TIME=ca_ES.UTF-8 LC_COLLATE=ca_ES.UTF-8
> >> >> [5] LC_MONETARY=ca_ES.UTF-8 LC_MESSAGES=ca_ES.UTF-8
> >> >> [7] LC_PAPER=C LC_NAME=C
> >> >> [9] LC_ADDRESS=C LC_TELEPHONE=C
> >> >> [11] LC_MEASUREMENT=ca_ES.UTF-8 LC_IDENTIFICATION=C
> >> >>
> >> >> attached base packages:
> >> >> [1] stats graphics grDevices utils datasets methods base
> >> >>
> >> >> other attached packages:
> >> >> [1] Rsubread_1.1.1
> >> >>
> >> >> _______________________________________________
> >> >> Bioconductor mailing list
> >> >> Bioconductor at r-project.org
> >> >> https://stat.ethz.ch/mailman/listinfo/bioconductor
> >> >> Search the archives:
> >> http://news.gmane.org/gmane.science.biology.informatics.conductor
> >> >
> >> > _______________________________________________
> >> > Bioconductor mailing list
> >> > Bioconductor at r-project.org
> >> > https://stat.ethz.ch/mailman/listinfo/bioconductor
> >> > Search the archives:
> >> http://news.gmane.org/gmane.science.biology.informatics.conductor
> >>
> >>
> >> ______________________________________________________________________
> >> The information in this email is confidential and intended solely for
> >> the addressee.
> >> You must not disclose, forward, print or use it without the permission
> >> of the sender.
> >> ______________________________________________________________________
> >
> >
> >
> >
>
>
>
> ______________________________________________________________________
> The information in this email is confidential and inte...{{dropped:5}}
More information about the Bioconductor
mailing list