Hello Ryan,

Thank you for looking at the example and even forking the gist. I have
updated the example with your approach and also calculated that it uses 6.794G
RAM when starting from scratch using 20 cores and thus beating the other 3
approaches under that scenario.

Also note that just generating the 'data' object uses 558.938M RAM. So at
least none of the approaches are at the ~ 550 Mb * 20 level.


So, without resorting to I/O the approach you suggested seems like the best
option. It still gets beaten by saving the data, re-starting R, then
mclapplying but this has the problem of having to re-start R.

Thank you,
Leo






On Thu, Nov 14, 2013 at 4:09 AM, Ryan <rct@thompsonclan.org> wrote:

> The minimize the additional memory used by mclapply, remember that
> mclapply works by forking processes, and the advantage of this is that
> as long as an object is not modified in either the parent or child,
> they will share the memory for that object, which effectively means
> that a child process *only* uses a significant amount of memory when it
> modifies existing objects (triggering creation of a copy) or creates a
> new object.
>
> In your case, there's no point in splitting the data (which results in
> creating copies). You only have to split the indices using
> parallel::splitIndices. I've tried to incorporate this into your gist:
> https://gist.github.com/DarwinAwardWinner/7463652
>
> The key line is:
>
>         res4 <- mclapply(splitIndices(nrow(data), opt$mcores), function(i)
> rowMeans(data[i,]), mc.cores=opt$mcores)
>
> Also, for concatenating the results, you can use "do.call(c,
> unname(res4))".
>
> On Thu Nov 14 00:13:41 2013, Leonardo Collado Torres wrote:
> > Dear BioC developers,
> >
> > I am trying to understand how to use mclapply() without blowing up the
> > memory usage and need some help.
> >
> > My use case is splitting a large IRanges::DataFrame() into chunks, and
> > feeding these chunks to mclapply(). Let say that I am using n cores and
> > that the operation I am doing uses K memory units.
> >
> > I understand that the individual jobs in mclapply() cannot detect how the
> > others are doing and if they need to run gc(). While this coupled n * K
> >   could explain a higher memory usage, I am running into higher than
> > expected memory loads.
> >
> > I have tried
> > 1) pre-splitting the data into a list (one element per chunk),
> > 2) assigning the elements of the list as elements of an environment and
> the
> > using mclapply() over a set of indexes,
> > 3) saving each chunk on its own Rdata file, then using mclapply with a
> > function that loads the appropriate chunk and then performs the operation
> > of interest.
> >
> > Strategy 3 performs best in terms of max memory usage, but I am afraid
> that
> > it is more error prone due to having to write to disk.
> >
> > Do you have any other ideas/tips on how to reduce the memory load? In
> other
> > words, is there a strategy to reduce the number of copies as much as
> > possible when using mclapply()?
> >
> >
> > I have a full example (with data.frame instead of DataFrame) and code
> > comparing the three options described above at http://bit.ly/1ar71yA
> >
> >
> > Thank you,
> > Leonardo
> >
> > Leonardo Collado Torres, PhD student
> > Department of Biostatistics
> > Johns Hopkins University
> > Bloomberg School of Public Health
> > Website: http://www.biostat.jhsph.edu/~lcollado/
> > Blog: http://bit.ly/FellBit
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

	[[alternative HTML version deleted]]

