Thank you all for your advice, this is exactly the sort of information I was
looking for. Mark

Mark W. Kimpel MD  ** Neuroinformatics ** Dept. of Psychiatry
Indiana University School of Medicine

15032 Hunter Court, Westfield, IN  46074

(317) 490-5129 Work, & Mobile & VoiceMail
(317) 399-1219 Skype No Voicemail please


On Tue, Jan 26, 2010 at 10:48 AM, Stephen Weston <stephen.b.weston@gmail.com
> wrote:

> In parallel computing, "chunking" is used to bundle multiple messages
> together, since big messages can very often be sent much more efficiently
> than short messages.  That means that in systems like "snow" and "foreach",
> if your task executes very quickly, most of the time may be spent moving
> data around, rather than doing useful work.  By bundling many tasks
> together, you might be able to do the communication efficiently enough so
> that you get a benefit from doing the tasks in parallel.  However, if you
> have short tasks and large inputs and/or outputs, chunking won't really
> help you, since it isn't going to make the communication more efficient.
> You need to figure out some way to decrease the amount of data that is
> being moved around.
>
> The nws package supports "chunking" via the eachElem "chunkSize" element
> of the "eo" argument.  The multicore package supports chunking as an all
> or nothing thing via the "mc.preschedule" argument to mclapply.  The doMC
> package uses the backend-specific "preschedule" option, which it passes on
> to mclapply via the "mc.preschedule" argument.  The doMPI package uses
> the backend-specific "chunkSize" option to specify any chunk size, much
> like nws.
>
> The iterators and itertools packages contain various functions that create
> iterators that allow you to split up data in chunks, so they support
> "chunking"
> in their own way.  That allows you to do manual chunking, as I call it,
> with
> any of the foreach backends.
>
> The snow package has some internal functions that split vectors and
> matrices
> into chunks.  They are used in functions such as parMM, parCapply, and
> parRapply.
>
> - Steve
>
>
> On Tue, Jan 26, 2010 at 9:44 AM, Brian G. Peterson <brian@braverock.com>
> wrote:
> > Mark Kimpel wrote:
> >>
> >> I have seen references on this list to "chunking" parallel tasks. If I
> am
> >> interpreting this correctly that is to decrease the overhead of multiple
> >> system calls. For instance, if I  have a loop of 10000 simple tasks and
> 10
> >> processors, then 10 chunks of 1000 would be processed.
> >>
> >> Which of the parallel packages has the ability to take "chunk" (or its
> >> equivalent) as an argument? I've googled chunk with R and come up with
> >> everything but want I'm interested in.
> >>
> >
> > Google "nesting foreach loops"
> >
> > The foreach package will do what you want.  Steve Weston has posted some
> > examples to this list on this topic as well.
> >
> > Regards,
> >
> >  - Brian
> >
> > _______________________________________________
> > R-sig-hpc mailing list
> > R-sig-hpc@r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >
>

	[[alternative HTML version deleted]]

