[R] allocating vector memory > 1 GByte on Windows XP / Vista / 7

[Duncan Murdoch]

In answer to your 4th question:  you are trying to allocate an object 
whose size is 1/4 of the addressable memory on your machine.  You might 
be lucky and have that much available in one piece, but I'm guessing you 
don't, which is why the allocation fails.  So don't worry about 
questions 1,2, or 3, switch to a 64 bit build of R on a 64 bit OS, where 
your allocation is a tiny bit of the addressable memory.

I don't know the answer to 5.

Duncan Murdoch

On 30/11/2009 12:34 PM, Jochen Albrecht wrote:
> Let me begin stating that I read all help files and faq's on the subject 
> matter (there aren't more than about a dozen) but either did not find 
> solutions or found them not to work.
> Here is the issue. I am trying to run a spatial regression on a 
> medium-sized dataset. Part of the functions in the spdep package I use 
> require me to allocate a vector of 1.1 Gb (mine is not a spatial SIG 
> question, I am using this just as an example). I have been trying my 
> luck on two different machines, each with 4 GB of real RAM, plus a 
> hundred GB of swap space. One machine runs XP, the other Windows 7.
> I have tried to work with startup parameters --min-vsize, --min-nsize, 
> --max-vsize, max-nsize. The limit here seems to be 2048 M - I am getting 
> error messages if I try to allocate more. I can increase the general 
> memory allocation using memory.limit(3072), and gc() then reports that I 
> have 3 Gigs available, although I don't know how this is split into cons 
> and heap cells.
> In any case, I still do not get to allocate my 1.1 GB, and I am at a 
> loss explaining why.
> The help pages for memory() also do not explain what actually happens 
> with the command line options min-vsize and max-vsize, or at least did 
> not prepare me for the following odd observation. If I specify 
> min-vsize=2048M, then I do not get to read in more than 384 MB of data; 
> it is as if the allocation of a minimum of 2 Gig of RAM actually reduces 
> the amount available. If I specify max-vize=2048M, then I can at least 
> read my data, but still run out of memory trying to run my spatial 
> regression. In other words, the specification of a minimum renders my 
> nice machine unusable, whereas the specification of a maximum seems to 
> have not the desired effect of increasing the memory allocation for R.
> So here are my questions:
> 1. What is the purpose of min-vsize and min-nsize? (please do not point 
> me to help pages, I read them)
> 2. Why do I get error messages for min-nsize > 2G even on machines whose 
> OS support more? (I tried the BCDEdit switch suggested by Kingsford 
> Jones but it did not have any visible effect)
> 3. How can I increase vsize within R rather than at startup 
> (memory.limit does not allow to specify types of memory)?
> 4. Why do I still get a "cannot allocate vector of size 1.1 Gb" even 
> when gc() tells me that I have used only 240 out of my 2048 Mb allocated?
> 5. At this point, I am actually using only a small test dataset to 
> develop my models. Once I have a functioning, well-tested model, I want 
> to apply it to a dataset 100 times larger and I know that I will have to 
> move onto a Linux platform. Given that I won't have 400 Gb of RAM 
> available there, will I run into similar problems there or will the OS 
> take care of memory allocation issues using swap space?
> Cheers,
>       Jochen
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>