[Rd] mclapply memory leak?

[Gabriel Becker]

Well it's only a leak if you don't get the memory back after it returns,
right?

Anyway, one (untested by me) possibility is the copying of memory pages
when the garbage collector touches objects, as pointed out by Radford Neal
here:
http://r.789695.n4.nabble.com/Re-R-devel-Digest-Vol-149-Issue-22-td4710367.html

If so, I don't think this would be easily avoidable, but there may be
mitigation strategies.

~G

On Wed, Sep 2, 2015 at 10:12 AM, Toby Hocking <tdhock5 at gmail.com> wrote:

> Dear R-devel,
>
> I am running mclapply with many iterations over a function that modifies
> nothing and makes no copies of anything. It is taking up a lot of memory,
> so it seems to me like this is a bug. Should I post this to
> bugs.r-project.org?
>
> A minimal reproducible example can be obtained by first starting a memory
> monitoring program such as htop, and then executing the following code
> while looking at how much memory is being used by the system
>
> library(parallel)
> seconds <- 5
> N <- 100000
> result.list <- mclapply(1:N, function(i)Sys.sleep(1/N*seconds))
>
> On my system, memory usage goes up about 60MB on this example. But it does
> not go up at all if I change mclapply to lapply. Is this a bug?
>
> For a more detailed discussion with a figure that shows that the memory
> overhead is linear in N, please see
> https://github.com/tdhock/mclapply-memory
>
> > sessionInfo()
> R version 3.2.2 (2015-08-14)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu precise (12.04.5 LTS)
>
> locale:
>  [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C
>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_CA.UTF-8
>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_CA.UTF-8
>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
>  [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] parallel  graphics  utils     datasets  stats     grDevices methods
> [8] base
>
> other attached packages:
> [1] ggplot2_1.0.1      RColorBrewer_1.0-5 lattice_0.20-33
>
> loaded via a namespace (and not attached):
>  [1] Rcpp_0.11.6             digest_0.6.4            MASS_7.3-43
>  [4] grid_3.2.2              plyr_1.8.1              gtable_0.1.2
>  [7] scales_0.2.3            reshape2_1.2.2          proto_1.0.0
> [10] labeling_0.2            tools_3.2.2             stringr_0.6.2
> [13] dichromat_2.0-0         munsell_0.4.2
>  PeakSegJoint_2015.08.06
> [16] compiler_3.2.2          colorspace_1.2-4
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>



-- 
Gabriel Becker, PhD
Computational Biologist
Bioinformatics and Computational Biology
Genentech, Inc.

	[[alternative HTML version deleted]]