[Bioc-devel] BiocParallel: BatchJobs backend (Was: Re: BiocParallel)

Thu Jun 6 22:56:20 CEST 2013

Hi, I'd like to pick up the discussion on a BatchJobs backend for
BiocParallel where it was left back in Dec 2012 (Bioc-devel thread
'BiocParallel' [https://stat.ethz.ch/pipermail/bioc-devel/2012-December/003918.html]).

Florian, would you mind sharing your BatchJobs backend code?  Is it
independent of BiocParallel and/or have you tried it with the most
recent BiocParallel implementation
[https://github.com/Bioconductor/BiocParallel/]?

/Henrik

On Tue, Dec 4, 2012 at 12:38 PM, Henrik Bengtsson <hb at biostat.ucsf.edu> wrote:
> Thanks.
>
> On Tue, Dec 4, 2012 at 3:47 AM, Vincent Carey
> <stvjc at channing.harvard.edu> wrote:
>> I have been booked up so no chance to deploy but I do have access to SGE and
>> LSF so will try and will report ASAP.
>
> ...and I'll try it out on PBS (... but I most likely won't have time
> to do this until the end of the year).
>
> Henrik
>
>>
>>
>> On Tue, Dec 4, 2012 at 4:08 AM, Hahne, Florian <florian.hahne at novartis.com>
>> wrote:
>>>
>>> Hi Henrik,
>>> I have now come up now with a relatively generic version of this
>>> SGEcluster approach. It does indeed use BatchJobs under the hood and
>>> should thus support all available cluster queues, assuming that the
>>> necessary batchJobs routines are available. I could only test this on our
>>> SGE cluster, but Vince wanted to try other queuing systems. Not sure how
>>> far he got. For now the code is wrapped in a little package called
>>> Qcluster with some documentation. If you want to I can send you a version
>>> in a separate mail. Would be good to test this on other systems, and I am
>>> sure there remain some bugs that need to be ironed out. In particular the
>>> fault tolerance you mentioned needs to be addressed properly. Currently
>>> the code may leave unwanted garbage if things fail in the wrong places
>>> because all the communication is file-based.
>>> Martin, I'll send you my updated version in case you want to include this
>>> in biocParallel for others to contribute.
>>> Florian
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 12/4/12 5:46 AM, "Henrik Bengtsson" <hb at biostat.ucsf.edu> wrote:
>>>
>>> >Picking up this thread in lack of other places (= were should
>>> >BiocParallel be discussed?)
>>> >
>>> >I saw Martin's updates on the BiocParallel - great.  Florian's SGE
>>> >scheduler was also mentioned; is that one built on top of BatchJobs?
>>> >If so I'd be interested in looking into that/generalizing that to work
>>> >with any BatchJobs scheduler.
>>> >
>>> >I believe there is going to be a new release of BatchJobs rather soon,
>>> >so it's probably worth waiting until that is available.
>>> >
>>> >The main use case I'm interested in is to launch batch jobs on a
>>> >PBS/Torque cluster, and then use multicore processing on each compute
>>> >node.  It would be nice to be able to do this using the BiocParallel
>>> >model, but maybe it is too optimistic to get everything to work under
>>> >same model.  Also, as Vince hinted, fault tolerance etc needs to be
>>> >addressed and needs to be addressed differently in the different
>>> >setups.
>>> >
>>> >/Henrik
>>> >
>>> >On Tue, Nov 20, 2012 at 6:59 AM, Ramon Diaz-Uriarte <rdiaz02 at gmail.com>
>>> >wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Sat, 17 Nov 2012 13:05:29 -0800,"Ryan C. Thompson"
>>> >><rct at thompsonclan.org> wrote:
>>> >>
>>> >>> On 11/17/2012 02:39 AM, Ramon Diaz-Uriarte wrote:
>>> >>> > In addition to Steve's comment, is it really a good thing that "all
>>> >>>code
>>> >>> > stays the same."?  I mean, multiple machines vs. multiple cores are,
>>> >>> > often, _very_ different things: for instance, shared vs. distributed
>>> >>> > memory, communication overhead differences, whether or not you can
>>> >>>assume
>>> >>> > packages and objects to be automagically present in the slaves/child
>>> >>> > process, etc. So, given they are different situations, I think it
>>> >>> > sometimes makes sense to want to write different code for each
>>> >>>situation
>>> >>> > (I often do); not to mention Steve's hybrid cases ;-).
>>> >>> >
>>> >>> >
>>> >>> > Since BiocParallel seems to be a major undertaking, maybe it would
>>> >>> > be
>>> >>> > appropriate to provide a flexible approach, instead of hard wiring
>>> >>>the
>>> >>> > foreach approach.
>>> >>> Of course there are cases where the same code simply can't work for
>>> >>>both
>>> >>> multicore and multi-machine situations, but those generally don't fall
>>> >>> into the category of things that can be done using lapply. Lapply and
>>> >>> all of its parallelized buddies like mclapply, parLapply, and foreach
>>> >>> are designed for data-parallel operations with no interdependence
>>> >>> between results, and these kinds of operations generally parallelize
>>> >>> as
>>> >>> well across machines as across cores, unless your network is not fast
>>> >>> enough (in which case you would choose not to use multi-machine
>>> >>> parallelism). If you want a parallel algorithm for something like the
>>> >>> disjoin method of GRanges, you might need to write some special
>>> >>> purpose
>>> >>> code, and that code might be very different for multicore vs
>>> >>>multi-machine.
>>> >>
>>> >>> So yes, sometimes there is a fundamental reason that you have to
>>> >>> change
>>> >>> the code to make it run on multiple machines, and neither foreach nor
>>> >>> any other parallelization framework will save you from having to
>>> >>>rewrite
>>> >>> your code. But often there is no fundamental reason that the code has
>>> >>>to
>>> >>> change, but you end up changing it anyway because of limitations in
>>> >>>your
>>> >>> parallelization framework. This is the case that foreach saves you
>>> >>>from.
>>> >>
>>> >>
>>> >>
>>> >> Hummm... I guess you are right, and we are talking about "often" or
>>> >>"most
>>> >> of the time", which is where all this would fit. Point taken.
>>> >>
>>> >>
>>> >> Best,
>>> >>
>>> >> R.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Ramon Diaz-Uriarte
>>> >> Department of Biochemistry, Lab B-25
>>> >> Facultad de Medicina
>>> >> Universidad Autónoma de Madrid
>>> >> Arzobispo Morcillo, 4
>>> >> 28029 Madrid
>>> >> Spain
>>> >>
>>> >> Phone: +34-91-497-2412
>>> >>
>>> >> Email: rdiaz02 at gmail.com
>>> >>        ramon.diaz at iib.uam.es
>>> >>
>>> >> http://ligarto.org/rdiaz
>>> >>
>>> >> _______________________________________________
>>> >> Bioc-devel at r-project.org mailing list
>>> >> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>
>>