[Bioc-devel] BatchJobs and BiocParallel
lg390 at cam.ac.uk
Wed Feb 19 13:07:34 CET 2014
Thank you all for your replies. I will experiment with the list of
BiocParallelParam objects and BatchJobs package.
> We discussed with Michael Lang at the Cambridge meeting and he agreed
> to think about potential abstractions for nested parallelization
> requests in BatchJobs. Not sure whether that ever took off. Ideally
> one would be able to infer the number of requested nodes AND cpus
> directly from the BiocParallel object. In BatchJobs this would then be
> exposed as a variable that could be injected in the appropriate
> template scripts. E.g., on our SGE I would ask for something like
> this: #$ -pe smp <%=resources$cores%>
> Currently I am already doing this by manually providing the number of
> requested cores in the resources list, but from an API point of view it
> would be very beneficial to have that information in a standardized form
> somewhere inside the BiocParallel object which can be queried in the
> executing code.
> On 2/14/14 7:23 PM, "Martin Morgan" <mtmorgan at fhcrc.org> wrote:
>>On 02/14/2014 09:20 AM, Laurent Gatto wrote:
>>> Dear all,
>>> Looking back in my (sparse) notes from the last European Bioc meeting, I
>>> saw two points that were mentioned during the BatchJobs and BiocParallel
>>> talk that are of particular interest to me.
>>> - BatchJobs: is there support, or are there any plans to support the
>>> HTCondor infrastructure , and will this back-end be supported in
>>> - BiocParallel: is there support, or are there any plans to support
>>> nested parallelisation. The use case I have in mind is to have a
>>> level of parallelisation to distribute n sets jobs on n different
>>> machines using snow for example, and then each set of jobs would be
>>> distributed on their respective machine via multicore?
>>> Any comments on the above solutions or their feasibility would be very
>>I opened a related issue on github from some off-hub commentary from
>>and I think this is a reasonable request. At least sometimes it seems
>>will be a natural transition from `bplapply` at the cluster level to
>>the machine level.
>>> Thank you very much in advance.
>>> Best wishes,
>>>  http://research.cs.wisc.edu/htcondor/
>>Computational Biology / Fred Hutchinson Cancer Research Center
>>1100 Fairview Ave. N.
>>PO Box 19024 Seattle, WA 98109
>>Location: Arnold Building M1 B861
>>Phone: (206) 667-2793
>>Bioc-devel at r-project.org mailing list
More information about the Bioc-devel