[Bioc-devel] BiocParallel

[Ryan C. Thompson]

On 11/17/2012 02:39 AM, Ramon Diaz-Uriarte wrote:
> In addition to Steve's comment, is it really a good thing that "all code
> stays the same."?  I mean, multiple machines vs. multiple cores are,
> often, _very_ different things: for instance, shared vs. distributed
> memory, communication overhead differences, whether or not you can assume
> packages and objects to be automagically present in the slaves/child
> process, etc. So, given they are different situations, I think it
> sometimes makes sense to want to write different code for each situation
> (I often do); not to mention Steve's hybrid cases ;-).
>
>
> Since BiocParallel seems to be a major undertaking, maybe it would be
> appropriate to provide a flexible approach, instead of hard wiring the
> foreach approach.
Of course there are cases where the same code simply can't work for both 
multicore and multi-machine situations, but those generally don't fall 
into the category of things that can be done using lapply. Lapply and 
all of its parallelized buddies like mclapply, parLapply, and foreach 
are designed for data-parallel operations with no interdependence 
between results, and these kinds of operations generally parallelize as 
well across machines as across cores, unless your network is not fast 
enough (in which case you would choose not to use multi-machine 
parallelism). If you want a parallel algorithm for something like the 
disjoin method of GRanges, you might need to write some special purpose 
code, and that code might be very different for multicore vs multi-machine.

So yes, sometimes there is a fundamental reason that you have to change 
the code to make it run on multiple machines, and neither foreach nor 
any other parallelization framework will save you from having to rewrite 
your code. But often there is no fundamental reason that the code has to 
change, but you end up changing it anyway because of limitations in your 
parallelization framework. This is the case that foreach saves you from.