[Bioc-devel] BatchJobs and BiocParallel

[Laurent Gatto]

Thank you all for your replies. I will experiment with the list of
BiocParallelParam objects and BatchJobs package. 

Best wishes,

Laurent

> We discussed with Michael Lang at the Cambridge meeting and he agreed
> to think about potential abstractions for nested parallelization
> requests in BatchJobs. Not sure whether that ever took off. Ideally
> one would be able to infer the number of requested nodes AND cpus
> directly from the BiocParallel object. In BatchJobs this would then be
> exposed as a variable that could be injected in the appropriate
> template scripts. E.g., on our SGE I would ask for something like
> this: #$ -pe smp <%=resources$cores%>
>
> Currently I am already doing this by manually providing the number of
> requested cores in the resources list, but from an API point of view it
> would be very beneficial to have that information in a standardized form
> somewhere inside the BiocParallel object which can be queried in the
> executing code.
> Florian
>
>
>
>
> On 2/14/14 7:23 PM, "Martin Morgan" <mtmorgan at fhcrc.org> wrote:
>
>>On 02/14/2014 09:20 AM, Laurent Gatto wrote:
>>>
>>> Dear all,
>>>
>>> Looking back in my (sparse) notes from the last European Bioc meeting, I
>>> saw two points that were mentioned during the BatchJobs and BiocParallel
>>> talk that are of particular interest to me.
>>>
>>> - BatchJobs: is there support, or are there any plans to support the
>>>    HTCondor infrastructure [1], and will this back-end be supported in
>>>    BiocParallel?
>>>
>>> - BiocParallel: is there support, or are there any plans to support
>>>    nested parallelisation. The use case I have in mind is to have a
>>>first
>>>    level of parallelisation to distribute n sets jobs on n different
>>>    machines using snow for example, and then each set of jobs would be
>>>    distributed on their respective machine via multicore?
>>>
>>> Any comments on the above solutions or their feasibility would be very
>>> welcome.
>>
>>I opened a related issue on github from some off-hub commentary from
>>Michael 
>>Lang (QuasaR).
>>
>>     https://github.com/Bioconductor/BiocParallel/issues/36
>>
>>and I think this is a reasonable request. At least sometimes it seems
>>like there 
>>will be a natural transition from `bplapply` at the cluster level to
>>`bpvec` at 
>>the machine level.
>>
>>Martin
>>
>>>
>>> Thank you very much in advance.
>>>
>>> Best wishes,
>>>
>>> Laurent
>>>
>>> [1]  http://research.cs.wisc.edu/htcondor/
>>>
>>>
>>
>>
>>-- 
>>Computational Biology / Fred Hutchinson Cancer Research Center
>>1100 Fairview Ave. N.
>>PO Box 19024 Seattle, WA 98109
>>
>>Location: Arnold Building M1 B861
>>Phone: (206) 667-2793
>>
>>_______________________________________________
>>Bioc-devel at r-project.org mailing list
>>https://stat.ethz.ch/mailman/listinfo/bioc-devel