[BioC] pooling for parallel hierarchical operations

[Cook, Malcolm]

Michael,

Have you seen  http://cran.r-project.org/web/packages/doRedis/index.html ??

If you take a look and come across a description of internals/architecture, please share....

Cheers,

~Malcolm


> -----Original Message-----
> From: bioconductor-bounces at r-project.org [mailto:bioconductor-bounces at r-project.org] On Behalf Of Michael Lawrence
> Sent: Wednesday, November 14, 2012 8:41 AM
> To: Bioconductor List
> Subject: [BioC] pooling for parallel hierarchical operations
> 
> We often execute nested operations in parallel. For example, first by
> sample, then by chromosome. Fixed allocation of resources to each level
> will often result in waste. For example, if one sample finishes quickly,
> its CPUs are not available to help the other samples along. Perhaps the
> most expedient solution is to expand.grid() the hierarchy and create one
> job for every combination, i.e., flatten the hierarchy. A more ideal
> solution might be a pool of resources (cores) that are allocated more
> fluidly. Is there any sort of pooling system for R? I know that the
> parallel package supports the declaration of resources in cluster objects,
> but there is no central manager. This is a general R question, but it's
> worth discussing in the context of how we can make better use of
> parallelism in the low-level infrastructure, which would cause these
> hierarchies to arise. It's also relevant to the discussion of specifying
> parallelization modes or strategies. Pools themselves could be hierarchical
> and heterogeneous (hosts, cores). Declaring available resources is fairly
> straight-forward. Deciding how to use them is context dependent and
> requires user control.
> 
> Michael
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at r-project.org
> https://stat.ethz.ch/mailman/listinfo/bioconductor
> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor