[Rd] R Memory Management Under Windows (PR#3980)

[dteller at affinnova.com]

Full_Name: David Teller
Version: 1.7.1
OS: Windows XP
Submission from: (NULL) (12.110.141.194)


I've noticed several issues with the R memory management under Windows.
It frequently runs out of memory, even when there is plenty of available memory
on the system and even plenty of free process virtual address space.
This tends to happen far more often when allocation a number of large objects
early in the processing.

Investigating the malloc.c routines indicates that when expanding the heap to
allocate a large object that will not fit, the windows sbrk emulation routine
calls an internal routine findregion with the object size (rounded up for
overhead). This routine essentially searches the virtual address space for an
unused place large enough to put the object, looking towards higher addresses,
until it finds a suitable location and always starting at the end of the last
chunk allocated.

The issue is that any available space that is not large enough for the object in
question is bypassed and can NEVER be used by the memory allocator. This can
lead to a huge amount of address space wastage if several large objects are
allocated early in the process.

It seems that the search cannot just be restarted at the beginning of memory
each time because the malloc implementation assumes that each chunk of memory
allocated from the system will be at a higher address than the last.

I have several possible solutions to this and am thinking about implementing one
of them:
1) Completely scrap the routines in malloc.c and replace them with a thin
wrapper around the core Windows memory allocation routines.
2) Attempt to implement a windows version of mmap() and enable that feature for
large allocations.
3) Attempt to clean up the malloc routines so they are less picky about the
address ordering of chunks allocated from the system.

Any feedback would be appreciated.
I don't see any other bugs on this issue and don't want to duplicate efforts if
someone else is already working on this.

- Dave Teller