Martin, thanks for your points. Being in bioinformatics like you, by biggest
task is usually to apply the same function(s) to rows of a large matrix
(gene expressions in my case) so the individual tasks would take roughly the
same amount of time.

I have to admit that I have not yet had time to read the vignette as
suggested, but Martin's point has prompted me to ask a question I was
thinking about. I believe from a previous thread on here (which I can't
find) I remember someone pointing out the figuring out the appropriate chunk
size is a bit of trial and error.

It seems to me that there might be a programmatic approach to this that
could be built into parallel functions. Before actually executing code "for
real", the function could send out tasks in various chunk sizes and report
back an estimate of what fraction of time was taken up by actual processing
of the task and how much by system over-head. I would think the curve of the
plot of these percentages would be asymptotic, with diminishing returns for
larger chunks. As Martin points out, in some situations there may be a
reason to not make chucks too big. This graph could be produced for the user
to determine chunk size or there could be some sort of default built in
based on the slope of the curve.

I'm not a computer scientist so forgive me if I haven't used the right lingo
for my ideas, but I would appreciate feedback as to whether they have
already been implemented somewhere, if they might be useful in certain
situations, etc.

Mark

Mark W. Kimpel MD  ** Neuroinformatics ** Dept. of Psychiatry
Indiana University School of Medicine

15032 Hunter Court, Westfield, IN  46074

(317) 490-5129 Work, & Mobile & VoiceMail
(317) 399-1219 Skype No Voicemail please


On Tue, Jan 26, 2010 at 12:10 PM, Martin Morgan <mtmorgan@fhcrc.org> wrote:

> On 01/26/2010 07:48 AM, Stephen Weston wrote:
> > In parallel computing, "chunking" is used to bundle multiple messages
> > together, since big messages can very often be sent much more efficiently
> > than short messages.  That means that in systems like "snow" and
> "foreach",
> > if your task executes very quickly, most of the time may be spent moving
> > data around, rather than doing useful work.  By bundling many tasks
> > together, you might be able to do the communication efficiently enough so
> > that you get a benefit from doing the tasks in parallel.  However, if you
> > have short tasks and large inputs and/or outputs, chunking won't really
> > help you, since it isn't going to make the communication more efficient.
> > You need to figure out some way to decrease the amount of data that is
> > being moved around.
>
> Another perhaps more common use is for 'cheap' load-balancing, where the
> individual tasks may take variable amounts of time (e.g., during
> simulations). Say each of the first 100 tasks take 10 times longer than
> the next 900. Dividing the 1000 tasks equally between 10 nodes means
> that the entire computation is limited by the first node, which takes
> (100 x 10) / 1 time units, versus (900 x 1) / 9 for the remaining 9
> nodes. Choose a smaller chunk size, e.g., 10. All nodes get 10 big
> tasks, and when complete come back for more. Ignoring communication time
> the elapsed time is something like (100 x 10) / 10 (for the 100 big jobs
> each taking 10 time units, divided between 10 processors) + (900 x 1) /
> 10  (for the remaining 900 small jobs, each taking 1 time unit, divided
> across 10 nodes).
>
> Since not explicitly mentioned elsewhere, the Rmpi mpi.*apply functions
> have a job.num argument for this purpose.
>
> Martin
>
> >
> > The nws package supports "chunking" via the eachElem "chunkSize" element
> > of the "eo" argument.  The multicore package supports chunking as an all
> > or nothing thing via the "mc.preschedule" argument to mclapply.  The doMC
> > package uses the backend-specific "preschedule" option, which it passes
> on
> > to mclapply via the "mc.preschedule" argument.  The doMPI package uses
> > the backend-specific "chunkSize" option to specify any chunk size, much
> > like nws.
> >
> > The iterators and itertools packages contain various functions that
> create
> > iterators that allow you to split up data in chunks, so they support
> "chunking"
> > in their own way.  That allows you to do manual chunking, as I call it,
> with
> > any of the foreach backends.
> >
> > The snow package has some internal functions that split vectors and
> matrices
> > into chunks.  They are used in functions such as parMM, parCapply, and
> > parRapply.
> >
> > - Steve
> >
> >
> > On Tue, Jan 26, 2010 at 9:44 AM, Brian G. Peterson <brian@braverock.com>
> wrote:
> >> Mark Kimpel wrote:
> >>>
> >>> I have seen references on this list to "chunking" parallel tasks. If I
> am
> >>> interpreting this correctly that is to decrease the overhead of
> multiple
> >>> system calls. For instance, if I  have a loop of 10000 simple tasks and
> 10
> >>> processors, then 10 chunks of 1000 would be processed.
> >>>
> >>> Which of the parallel packages has the ability to take "chunk" (or its
> >>> equivalent) as an argument? I've googled chunk with R and come up with
> >>> everything but want I'm interested in.
> >>>
> >>
> >> Google "nesting foreach loops"
> >>
> >> The foreach package will do what you want.  Steve Weston has posted some
> >> examples to this list on this topic as well.
> >>
> >> Regards,
> >>
> >>  - Brian
> >>
> >> _______________________________________________
> >> R-sig-hpc mailing list
> >> R-sig-hpc@r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>
> >
> > _______________________________________________
> > R-sig-hpc mailing list
> > R-sig-hpc@r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
>
> --
> Martin Morgan
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M1 B861
> Phone: (206) 667-2793
>

	[[alternative HTML version deleted]]

