[R-sig-hpc] mclapply: rm intermediate objects and returning memory

Ramon Diaz-Uriarte rdiaz02 at gmail.com
Mon Oct 15 16:47:03 CEST 2012



Dear All,


It seems that, in some cases, mclapply can result in out of memory
conditions that can be avoided by having the "X" argument split into smaller
pieces, and running mclapply several times.  This is an example (on a
machine with 12 cores and 64 Gb of RAM [similar examples can be created
for different numbers of cores and RAM]):


#########################
library(parallel)
data.500 <- matrix(1.1, nrow = 3*10^6, ncol = 500)
print(object.size(data.500), units = "Mb")

f1 <- function(index, data) {
  x <- data[, index]
  u <- 2 * x
  return(u)
}

## This will not run
tmp1 <- mclapply(1:500, f1, data.500,
                 mc.cores = detectCores())


## These will run


tmp1a <- mclapply(1:100, f1, data.500,
                 mc.cores = detectCores())
tmp1b <- mclapply(101:200, f1, data.500,
                 mc.cores = detectCores())
tmp1c <- mclapply(201:300, f1, data.500,
                 mc.cores = detectCores())
tmp1d <- mclapply(301:400, f1, data.500,
                 mc.cores = detectCores())
tmp1e <- mclapply(401:500, f1, data.500,
                 mc.cores = detectCores())

tmp <- c(tmp1a, tmp1b, tmp1c, tmp1d, tmp1e)

########################

Notice that the problem is not simply memory usage by the master, since
the last concatenation of the five lists works.


However, if we delete intermediate objects as in:

##############

f1 <- function(index, data) {
  x <- data[, index]
  u <- 2 * x
  rm(x); gc() ### This is the only change
  return(u)
}
#############

then it does run.




It also runs if we use the original function (i.e., we do not delete
intermediate results) but change the scheduling:


#################

f1 <- function(index, data) {
  x <- data[, index]
  u <- 2 * x
  return(u)
}

tmp1 <- mclapply(1:500, f1, data.500,
                 mc.cores = detectCores(),
                 mc.preschedule = FALSE)

###################




So it seems that, with preschedule = TRUE, each slave does not return the
memory to the OS until all the jobs are collected by the master. Before
that, in terms of memory, it is as if a list as long as X is kept, but not
just with the return objects, but also with all the intermediate (and non
deleted) objects (i.e., it is as if each function invocation has not
really fully returned?).


In the help file, mc.preschedule = TRUE is the recommended setting for
large number of values in X (and in my experiments setting it to FALSE
makes things slower). I guess, then, that the recommended way of dealing
with these issues is carefully deleting (and gc'ing) any non-needed
intermediate result? Is this correct?



Best,

R.


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina 
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02 at gmail.com
       ramon.diaz at iib.uam.es

http://ligarto.org/rdiaz



More information about the R-sig-hpc mailing list