[Bioc-devel] BatchJobs and BiocParallel

Laurent Gatto lg390 at cam.ac.uk
Wed Feb 19 13:07:34 CET 2014

Thank you all for your replies. I will experiment with the list of
BiocParallelParam objects and BatchJobs package. 

Best wishes,


> We discussed with Michael Lang at the Cambridge meeting and he agreed
> to think about potential abstractions for nested parallelization
> requests in BatchJobs. Not sure whether that ever took off. Ideally
> one would be able to infer the number of requested nodes AND cpus
> directly from the BiocParallel object. In BatchJobs this would then be
> exposed as a variable that could be injected in the appropriate
> template scripts. E.g., on our SGE I would ask for something like
> this: #$ -pe smp <%=resources$cores%>
> Currently I am already doing this by manually providing the number of
> requested cores in the resources list, but from an API point of view it
> would be very beneficial to have that information in a standardized form
> somewhere inside the BiocParallel object which can be queried in the
> executing code.
> Florian
> On 2/14/14 7:23 PM, "Martin Morgan" <mtmorgan at fhcrc.org> wrote:
>>On 02/14/2014 09:20 AM, Laurent Gatto wrote:
>>> Dear all,
>>> Looking back in my (sparse) notes from the last European Bioc meeting, I
>>> saw two points that were mentioned during the BatchJobs and BiocParallel
>>> talk that are of particular interest to me.
>>> - BatchJobs: is there support, or are there any plans to support the
>>>    HTCondor infrastructure [1], and will this back-end be supported in
>>>    BiocParallel?
>>> - BiocParallel: is there support, or are there any plans to support
>>>    nested parallelisation. The use case I have in mind is to have a
>>>    level of parallelisation to distribute n sets jobs on n different
>>>    machines using snow for example, and then each set of jobs would be
>>>    distributed on their respective machine via multicore?
>>> Any comments on the above solutions or their feasibility would be very
>>> welcome.
>>I opened a related issue on github from some off-hub commentary from
>>Lang (QuasaR).
>>     https://github.com/Bioconductor/BiocParallel/issues/36
>>and I think this is a reasonable request. At least sometimes it seems
>>like there 
>>will be a natural transition from `bplapply` at the cluster level to
>>`bpvec` at 
>>the machine level.
>>> Thank you very much in advance.
>>> Best wishes,
>>> Laurent
>>> [1]  http://research.cs.wisc.edu/htcondor/
>>Computational Biology / Fred Hutchinson Cancer Research Center
>>1100 Fairview Ave. N.
>>PO Box 19024 Seattle, WA 98109
>>Location: Arnold Building M1 B861
>>Phone: (206) 667-2793
>>Bioc-devel at r-project.org mailing list

More information about the Bioc-devel mailing list