[Bioc-devel] BatchJobs and BiocParallel

[Hahne, Florian]

We discussed with Michael Lang at the Cambridge meeting and he agreed to
think about potential abstractions for nested parallelization requests in
BatchJobs. Not sure whether that ever took off. Ideally one would be able
to infer the number of requested nodes AND cpus directly from the
BiocParallel object. In BatchJobs this would then be exposed as a variable
that could be injected in the appropriate template scripts. E.g., on our
SGE I would ask for something like this:
#$ -pe smp <%=resources$cores%>

Currently I am already doing this by manually providing the number of
requested cores in the resources list, but from an API point of view it
would be very beneficial to have that information in a standardized form
somewhere inside the BiocParallel object which can be queried in the
executing code.
Florian




On 2/14/14 7:23 PM, "Martin Morgan" <mtmorgan at fhcrc.org> wrote:

>On 02/14/2014 09:20 AM, Laurent Gatto wrote:
>>
>> Dear all,
>>
>> Looking back in my (sparse) notes from the last European Bioc meeting, I
>> saw two points that were mentioned during the BatchJobs and BiocParallel
>> talk that are of particular interest to me.
>>
>> - BatchJobs: is there support, or are there any plans to support the
>>    HTCondor infrastructure [1], and will this back-end be supported in
>>    BiocParallel?
>>
>> - BiocParallel: is there support, or are there any plans to support
>>    nested parallelisation. The use case I have in mind is to have a
>>first
>>    level of parallelisation to distribute n sets jobs on n different
>>    machines using snow for example, and then each set of jobs would be
>>    distributed on their respective machine via multicore?
>>
>> Any comments on the above solutions or their feasibility would be very
>> welcome.
>
>I opened a related issue on github from some off-hub commentary from
>Michael 
>Lang (QuasaR).
>
>     https://github.com/Bioconductor/BiocParallel/issues/36
>
>and I think this is a reasonable request. At least sometimes it seems
>like there 
>will be a natural transition from `bplapply` at the cluster level to
>`bpvec` at 
>the machine level.
>
>Martin
>
>>
>> Thank you very much in advance.
>>
>> Best wishes,
>>
>> Laurent
>>
>> [1]  http://research.cs.wisc.edu/htcondor/
>>
>>
>
>
>-- 
>Computational Biology / Fred Hutchinson Cancer Research Center
>1100 Fairview Ave. N.
>PO Box 19024 Seattle, WA 98109
>
>Location: Arnold Building M1 B861
>Phone: (206) 667-2793
>
>_______________________________________________
>Bioc-devel at r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/bioc-devel