[Bioc-devel] OpenMP error when using ShortRead::FastqStreamer with parallel::parLapply

[Michael Stadler]

Hi Martin,

Thank you for the quick response. Now it works like a charm.

Best wishes,
Michael

On 27.05.2013 17:29, Martin Morgan wrote:
> On 05/26/2013 11:57 PM, Michael Stadler wrote:
>> Dear bioc core developers,
>>
>> We have recently (a few months back?) started to get error messages of
>> the type:
>>
>> libgomp: Thread creation failed: Resource temporarily unavailable
>>
>> when using ShortRead::FastqStreamer in combination with
>> parallel::parLapply (for example in QuasR::preprocessReads or
>> QuasR::qQCReport). These errors are not fully reproducible, are less
>> frequent when reducing the number of nodes in the cluster object and
>> disappear completely when FastqStreamer is used directly rather than
>> within a parLapply call of the sort:
>>
>> cl <- makeCluster(n)
>> results <- parLapply(cl, filenames, function(filename) {
>>    fs <- FastqStreamer(filename, n=1e6)
>>    while (length(chunks <- yield(fs)) != 0L) {
>>      # do work
>>      ...
>>    }
>>    close(fs)
>> })
>>
>> I guess that this could be a conflict of nested parallelization and/or a
>> resource limit of our setup. If I am not mistaken, FastqStreamer makes
>> use of OpenMP for parsing fastq files.
>>
>> My questions is how the "libgomp" error message can be avoided in the
>> above situations. Is it possible to switch off the use of OpenMP in
>> FastqStreamer and friends at run-time, other than switching off OpenMP
>> at compile time for R or the whole ShortRead package?
> 
> Hi Michael --
> 
> Your diagnosis sounds on target. It is possible to set the number of
> openMP threads, which I think will address the problem. You'd like to
> not interfere with others' use of openMP, so reset the thread count when
> your function exits. Along the lines of
> 
>     nthreads <- .Call(ShortRead:::.set_omp_threads, 1L)
>     on.exit(.Call(ShortRead:::.set_omp_threads, nthreads))
> 
> ShortRead's srapply function does this internally when it detects that
> the `parallel` package is in use (reasoning that mclapply and forked
> processes would be the common use case for single-machine parallel
> evaluation). I'll give some thought to more aggressively discovering
> when it is necessary to invoke FastqStreamer with this option set.
> 
> Sorry for the trouble,
> 
> Martin
> 
>>
>> With kind regards,
>> Michael
>>
>>
>> My sessionInfo():
>> R version 3.0.1 (2013-05-16)
>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>
>> locale:
>> [1] C
>>
>> attached base packages:
>> [1] parallel  stats     graphics  grDevices utils     datasets  methods
>> [8] base
>>
>> other attached packages:
>> [1] ShortRead_1.18.0     latticeExtra_0.6-24  Rsamtools_1.12.3
>> [4] lattice_0.20-15      Biostrings_2.28.0    GenomicRanges_1.12.3
>> [7] IRanges_1.18.1       BiocGenerics_0.7.0   RColorBrewer_1.0-5
>>
>> loaded via a namespace (and not attached):
>> [1] Biobase_2.20.0 bitops_1.0-5   grid_3.0.1     hwriter_1.3
>> stats4_3.0.1
>> [6] tcltk_3.0.1    tools_3.0.1    zlibbioc_1.6.0
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
> 
> 

-- 
--------------------------------------------
Michael Stadler, PhD
Head of Computational Biology
Friedrich Miescher Institute
Basel (Switzerland)
Phone : +41 61 697 6492
Fax   : +41 61 697 3976
Mail  : michael.stadler at fmi.ch