[Rd] R Memory Management Under Windows (PR#3980)

[Prof Brian Ripley]

Why is this a bug report?  It does not fit the definition of bugs in the 
FAQ.  If you want to help, don't misuse R-bugs for questions/comments.

On Mon, 25 Aug 2003 dteller at affinnova.com wrote:

> Full_Name: David Teller
> Version: 1.7.1
> OS: Windows XP
> Submission from: (NULL) (12.110.141.194)
> 
> 
> I've noticed several issues with the R memory management under Windows.
> It frequently runs out of memory, even when there is plenty of available memory
> on the system and even plenty of free process virtual address space.
> This tends to happen far more often when allocation a number of large objects
> early in the processing.

But that is not a bug -- it is even documented as likely.

> Investigating the malloc.c routines indicates that when expanding the heap to
> allocate a large object that will not fit, the windows sbrk emulation routine
> calls an internal routine findregion with the object size (rounded up for
> overhead). This routine essentially searches the virtual address space for an
> unused place large enough to put the object, looking towards higher addresses,
> until it finds a suitable location and always starting at the end of the last
> chunk allocated.
> 
> The issue is that any available space that is not large enough for the object in
> question is bypassed and can NEVER be used by the memory allocator. This can
> lead to a huge amount of address space wastage if several large objects are
> allocated early in the process.
> 
> It seems that the search cannot just be restarted at the beginning of memory
> each time because the malloc implementation assumes that each chunk of memory
> allocated from the system will be at a higher address than the last.
> 
> I have several possible solutions to this and am thinking about implementing one
> of them:
> 1) Completely scrap the routines in malloc.c and replace them with a thin
> wrapper around the core Windows memory allocation routines.

We did this precisely because that approach (used prior to 1.2.0) was far 
too slow.  We saw speed hits of at least 4x in that version until this 
malloc.c was used.

> 2) Attempt to implement a windows version of mmap() and enable that feature for
> large allocations.
> 3) Attempt to clean up the malloc routines so they are less picky about the
> address ordering of chunks allocated from the system.
> 
> Any feedback would be appreciated.
> I don't see any other bugs on this issue and don't want to duplicate efforts if
> someone else is already working on this.

I don't see any bugs at all.  I suspect most people using large amounts of 
memory use a better OS anyway: if Windows provided an adequate malloc in 
the first place this would all be moot.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595