[Bioc-devel] reporting progress while running in parallel

[Robert Castelo]

hi Martin,

On Fri, 2010-08-27 at 20:04 -0700, Martin Morgan wrote:
> On 08/27/2010 07:27 AM, Robert Castelo wrote:
> > dear developers,
> > 
> > i'm adding parallel support to my package by copying & pasting the way
> > in which this is done by other packages like ShortRead. i'm currently
> > using 'snow' and since i'm providing this support for
> > computing-intensive calculations i'd like that while these calculations
> > are running in parallel, some sort of progress is reported.
> > 
> > i'd like to ask whether anybody here has added parallel support to
> > his/her package including some sort of progress reporting functionality.
> > 
> > my first guess is that probably the loop of slave R processes created by
> > 'snow' should be replaced by something else, but i hope somebody has a
> > more clear idea about what could be done.
> 
> Hi Robert --
> 
> I don't have a solution, but if I were starting on this with snow I'd
> look at clusterApplyLB and dynamicClusterApply -- break the overall set
> of tasks into a number of jobs > the number of nodes, and then in a loop
> like that in dynamicClusterApply report to the user as each job comes
> back from recvOneResult.

the progress reporting i'm looking for is not for knowing when each job
starts and ends since care is taken to split the entire computation into
the available processors with a nearly perfect balance so that there
will be only as many jobs as available processors and every processor
will take about the same time to finish up the job. therefore, i'd like
to report the % of the entire computation being done, which implies that
each slave tells the master now and then how far has gone with its
computation.

> In Rmpi a fun project might be to figure out how to use a separate comm
> to send errors.

i have some experience with C & MPI and i could imagine how to go on
with MPI but i think it would be nice to solve it within snow's parallel
support in order to be "parallel-support-type" independent.

robert.