[Bioc-devel] OpenMP error when using ShortRead::FastqStreamer with parallel::parLapply
Michael Stadler
michael.stadler at fmi.ch
Mon May 27 08:57:34 CEST 2013
Dear bioc core developers,
We have recently (a few months back?) started to get error messages of
the type:
libgomp: Thread creation failed: Resource temporarily unavailable
when using ShortRead::FastqStreamer in combination with
parallel::parLapply (for example in QuasR::preprocessReads or
QuasR::qQCReport). These errors are not fully reproducible, are less
frequent when reducing the number of nodes in the cluster object and
disappear completely when FastqStreamer is used directly rather than
within a parLapply call of the sort:
cl <- makeCluster(n)
results <- parLapply(cl, filenames, function(filename) {
fs <- FastqStreamer(filename, n=1e6)
while (length(chunks <- yield(fs)) != 0L) {
# do work
...
}
close(fs)
})
I guess that this could be a conflict of nested parallelization and/or a
resource limit of our setup. If I am not mistaken, FastqStreamer makes
use of OpenMP for parsing fastq files.
My questions is how the "libgomp" error message can be avoided in the
above situations. Is it possible to switch off the use of OpenMP in
FastqStreamer and friends at run-time, other than switching off OpenMP
at compile time for R or the whole ShortRead package?
With kind regards,
Michael
My sessionInfo():
R version 3.0.1 (2013-05-16)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] ShortRead_1.18.0 latticeExtra_0.6-24 Rsamtools_1.12.3
[4] lattice_0.20-15 Biostrings_2.28.0 GenomicRanges_1.12.3
[7] IRanges_1.18.1 BiocGenerics_0.7.0 RColorBrewer_1.0-5
loaded via a namespace (and not attached):
[1] Biobase_2.20.0 bitops_1.0-5 grid_3.0.1 hwriter_1.3
stats4_3.0.1
[6] tcltk_3.0.1 tools_3.0.1 zlibbioc_1.6.0
More information about the Bioc-devel
mailing list