[R] allocating vector memory > 1 GByte on Windows XP / Vista / 7

Jochen Albrecht jochen at hunter.cuny.edu
Mon Nov 30 18:34:26 CET 2009


Let me begin stating that I read all help files and faq's on the subject 
matter (there aren't more than about a dozen) but either did not find 
solutions or found them not to work.
Here is the issue. I am trying to run a spatial regression on a 
medium-sized dataset. Part of the functions in the spdep package I use 
require me to allocate a vector of 1.1 Gb (mine is not a spatial SIG 
question, I am using this just as an example). I have been trying my 
luck on two different machines, each with 4 GB of real RAM, plus a 
hundred GB of swap space. One machine runs XP, the other Windows 7.
I have tried to work with startup parameters --min-vsize, --min-nsize, 
--max-vsize, max-nsize. The limit here seems to be 2048 M - I am getting 
error messages if I try to allocate more. I can increase the general 
memory allocation using memory.limit(3072), and gc() then reports that I 
have 3 Gigs available, although I don't know how this is split into cons 
and heap cells.
In any case, I still do not get to allocate my 1.1 GB, and I am at a 
loss explaining why.
The help pages for memory() also do not explain what actually happens 
with the command line options min-vsize and max-vsize, or at least did 
not prepare me for the following odd observation. If I specify 
min-vsize=2048M, then I do not get to read in more than 384 MB of data; 
it is as if the allocation of a minimum of 2 Gig of RAM actually reduces 
the amount available. If I specify max-vize=2048M, then I can at least 
read my data, but still run out of memory trying to run my spatial 
regression. In other words, the specification of a minimum renders my 
nice machine unusable, whereas the specification of a maximum seems to 
have not the desired effect of increasing the memory allocation for R.
So here are my questions:
1. What is the purpose of min-vsize and min-nsize? (please do not point 
me to help pages, I read them)
2. Why do I get error messages for min-nsize > 2G even on machines whose 
OS support more? (I tried the BCDEdit switch suggested by Kingsford 
Jones but it did not have any visible effect)
3. How can I increase vsize within R rather than at startup 
(memory.limit does not allow to specify types of memory)?
4. Why do I still get a "cannot allocate vector of size 1.1 Gb" even 
when gc() tells me that I have used only 240 out of my 2048 Mb allocated?
5. At this point, I am actually using only a small test dataset to 
develop my models. Once I have a functioning, well-tested model, I want 
to apply it to a dataset 100 times larger and I know that I will have to 
move onto a Linux platform. Given that I won't have 400 Gb of RAM 
available there, will I run into similar problems there or will the OS 
take care of memory allocation issues using swap space?
Cheers,
      Jochen




More information about the R-help mailing list