[Rd] Re: [R] Memory Fragmentation in R

[Luke Tierney]

On Sat, 19 Feb 2005, Nawaaz Ahmed wrote:

> Thanks Brian. I looked at the code (memory.c) after I sent out the first 
> email and noticed the malloc() call that you mention in your reply.
> Looking into this code suggested a possible scenario where R would fail in 
> malloc() even if it had enough free heap address space.
>
> I noticed that if there is enough heap address space (memory.c:1796, 
> VHEAP_FREE() > alloc_size) then the garbage collector is not run. So malloc 
> could fail (since there is no more address space to use), even though R 
> itself has enough free space it can reclaim. A simple fix is for R to try 
> doing garbage collection if malloc() fails.
>
> I hacked memory.c() to look in R_GenHeap[LARGE_NODE_CLASS].New if malloc() 
> fails (in a very similar fashion to ReleaseLargeFreeVectors())
> I did a "best-fit" stealing from this list and returned it to allocVector(). 
> This seemed to fix my particular problem - the large vectors that I had 
> allocated in the previous round were still sitting in  this list. Of course, 
> the right thing to do is to check if there are any free vectors of the right 
> size before calling malloc() - but it was simpler to do it my way (because I 
> did not have to worry about how efficient my best-fit was; memory allocation 
> was anyway going to fail).
>
> I can look deeper into this and provide more details if needed.

Thanks.  It looks like it would be a good idea to modify the malloc at
that point to try a GC if the malloc fails, then retry the malloc and
only bail if the second malloc fails.  I want to think this through a
bit more before going ahead, but I think it will be the right thing to
do.

Best,

luke


>
> Nawaaz
>
>
>
>
>
> Prof Brian Ripley wrote:
>> BTW, I think this is really an R-devel question, and if you want to pursue 
>> this please use that list.  (See the posting guide as to why I think so.)
>> 
>> This looks like fragmentation of the address space: many of us are using 
>> 64-bit OSes with 2-4Gb of RAM precisely to avoid such fragmentation.
>> 
>> Notice (memory.c line 1829 in the current sources) that large vectors are 
>> malloc-ed separately, so this is a malloc failure, and there is not a lot R 
>> can do about how malloc fragments the (presumably in your case as you did 
>> not say) 32-bit process address space.
>> 
>> The message
>>   1101.7 Mbytes of heap free (51%)
>> is a legacy of an earlier gc() and is not really `free': I believe it means 
>> something like `may be allocated before garbage collection is triggered': 
>> see memory.c.
>> 
>> 
>> On Sat, 19 Feb 2005, Nawaaz Ahmed wrote:
>> 
>>> I have a data set of roughly 700MB which during processing grows up to 2G 
>>> ( I'm using a 4G linux box). After the work is done I clean up (rm()) and 
>>> the state is returned to 700MB. Yet I find I cannot run the same routine 
>>> again as it claims to not be able to allocate memory even though gcinfo() 
>>> claims there is 1.1G left.
>>> 
>>>     At the start of the second time
>>>     ===============================
>>>               used  (Mb) gc trigger   (Mb)
>>>     Ncells  2261001  60.4    3493455   93.3
>>>     Vcells 98828592 754.1  279952797 2135.9
>>> 
>>>     Before Failing
>>>     ==============
>>>     Garbage collection 459 = 312+51+96 (level 0) ...
>>>     1222596 cons cells free (34%)
>>>     1101.7 Mbytes of heap free (51%)
>>>     Error: cannot allocate vector of size 559481 Kb
>>> 
>>> This looks like a fragmentation problem. Anyone have a handle on this 
>>> situation? (ie. any work around?) Anyone working on improving R's 
>>> fragmentation problems?
>>> 
>>> On the other hand, is it possible there is a memory leak? In order to make 
>>> my functions work on this dataset I tried to eliminate copies by coding 
>>> with references (basic new.env() tricks). I presume that my cleaning up 
>>> returned the temporary data (as evidenced by the gc output at the start of 
>>> the second round of processing). Is it possible that it was not really 
>>> cleaned up and is sitting around somewhere even though gc() thinks it has 
>>> been returned?
>>> 
>>> Thanks - any clues to follow up will be very helpful.
>>> Nawaaz
>> 
>> 
>
> ______________________________________________
> R-devel at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Luke Tierney
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu