[Bioc-devel] BiocParallel

[Ramon Diaz-Uriarte]

On Sat, 17 Nov 2012 00:45:46 -0500,Steve Lianoglou <mailinglist.honeypot at gmail.com> wrote:
> Ding!

> On Fri, Nov 16, 2012 at 11:42 PM, Ryan C. Thompson <rct at thompsonclan.org> wrote:
> > The difference is that in the parallel package, you use mclapply for
> > multicore and parLapply for multi-machine parallelism. If you want to switch
> > from one to the other, you have to change all your code that uses either
> > function to the other one. If you use llply(..., .parallel=TRUE), then all
> > you have to do is register a different backend, which is one line of code to
> > load the new backend and a second one to register it, and the rest of your
> > code stays the same.


In addition to Steve's comment, is it really a good thing that "all code
stays the same."?  I mean, multiple machines vs. multiple cores are,
often, _very_ different things: for instance, shared vs. distributed
memory, communication overhead differences, whether or not you can assume
packages and objects to be automagically present in the slaves/child
process, etc. So, given they are different situations, I think it
sometimes makes sense to want to write different code for each situation
(I often do); not to mention Steve's hybrid cases ;-).


Since BiocParallel seems to be a major undertaking, maybe it would be
appropriate to provide a flexible approach, instead of hard wiring the
foreach approach.


Best,


R.



> This is the same reason I've also floated use of foreach in the past.

> But then you have the situation of multi-machines w/ multiple cores --
> is this (2) or (3) here?

> How do you explicitly write code for that w/ foreach mojo? I guess the
> answer to that is that you let your "grid engine" (or whatever your
> SGE equivalent is that you might be running) handle that for you.

> -- 
> Steve Lianoglou
> Graduate Student: Computational Systems Biology
>  | Memorial Sloan-Kettering Cancer Center
>  | Weill Medical College of Cornell University
> Contact Info: http://cbio.mskcc.org/~lianos/contact

> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina 
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02 at gmail.com
       ramon.diaz at iib.uam.es

http://ligarto.org/rdiaz