[R-sig-hpc] Seeing some memory leak with foreach...

Jonathan Greenberg jgrn at illinois.edu
Tue Feb 26 15:49:14 CET 2013


r-sig-geo'ers:

I always hate doing this, but the test function/dataset is going to be
hard to pass along to the list.  Basically: I have a foreach call that
has no superassignments or strange environmental manipulations, but
resulted in the nodes showing a slow but steady memory creep over
time.  I was using a parallel backend for foreach via doParallel.  Has
anyone else seen this behavior (unexplained memory creep)?  Is there a
good way to "flush" a node?  I'm trying to embed gc() at the top of my
foreach function, but this process took about 24 hours to get to a
memory overuse stage (multiple iterations would have passed, e.g. the
function would have been called more than one time on a single node)
so I'm not sure if this will work so I figured I'd ask the group about
it.  I've seen other people post about this on various boards with no
clear response/solution to it (gc() apparently didn't work).

Some other notes: there should be no resultant output of data, because
the output is being written from within the foreach function (e.g. the
output of the function that foreach executes is NULL).

I'll see if I can work up a faster executing example later, but wanted
to see if there are some general pointers for dealing with memory
leaks using a parallel system.

--j

-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007



More information about the R-sig-hpc mailing list