[Bioc-devel] OpenMP error when using ShortRead::FastqStreamer with parallel::parLapply

[Michael Stadler]

Dear bioc core developers,

We have recently (a few months back?) started to get error messages of
the type:

libgomp: Thread creation failed: Resource temporarily unavailable

when using ShortRead::FastqStreamer in combination with
parallel::parLapply (for example in QuasR::preprocessReads or
QuasR::qQCReport). These errors are not fully reproducible, are less
frequent when reducing the number of nodes in the cluster object and
disappear completely when FastqStreamer is used directly rather than
within a parLapply call of the sort:

cl <- makeCluster(n)
results <- parLapply(cl, filenames, function(filename) {
  fs <- FastqStreamer(filename, n=1e6)
  while (length(chunks <- yield(fs)) != 0L) {
    # do work
    ...
  }
  close(fs)
})

I guess that this could be a conflict of nested parallelization and/or a
resource limit of our setup. If I am not mistaken, FastqStreamer makes
use of OpenMP for parsing fastq files.

My questions is how the "libgomp" error message can be avoided in the
above situations. Is it possible to switch off the use of OpenMP in
FastqStreamer and friends at run-time, other than switching off OpenMP
at compile time for R or the whole ShortRead package?

With kind regards,
Michael


My sessionInfo():
R version 3.0.1 (2013-05-16)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
[1] C

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

other attached packages:
[1] ShortRead_1.18.0     latticeExtra_0.6-24  Rsamtools_1.12.3
[4] lattice_0.20-15      Biostrings_2.28.0    GenomicRanges_1.12.3
[7] IRanges_1.18.1       BiocGenerics_0.7.0   RColorBrewer_1.0-5

loaded via a namespace (and not attached):
[1] Biobase_2.20.0 bitops_1.0-5   grid_3.0.1     hwriter_1.3
stats4_3.0.1
[6] tcltk_3.0.1    tools_3.0.1    zlibbioc_1.6.0