[R] large data set, error: cannot allocate vector

[Marc Schwartz (via MN)]

On Fri, 2006-05-05 at 17:56 +0200, Uwe Ligges wrote:
> Robert Citek wrote:
> 
> > Why am I getting the error "Error: cannot allocate vector of size  
> > 512000 Kb" on a machine with 6 GB of RAM?
> 
> 1. The message means that you cannot allocate *further* 512Mb of RAM 
> right now for the next step, but not what is required nor what R is 
> currently consuming.
> 
> 2. This seems to be a 32-bit OS. It limits the maximal allocation for 
> the *single* R process to < 4Gb (if all goes very well).
> 
> 
> > I'm playing with some large data sets within R and doing some simple  
> > statistics.  The data sets have 10^6 and 10^7 rows of numbers.  R  
> 
> 3. 10^7 rows is not large, if you have one column...
> 
> 4. 10^7 needs 10 times what is needed for 10^6. Hence comparing 10^6 and 
> 10^7 is quite a difference.
> 
> Uwe Ligges
> 
> > reads in and performs summary() on the 10^6 set just fine.  However,  
> > on the 10^7 set, R halts with the error.  My hunch is that somewhere  
> > there's an setting to limit some memory size to 500 MB.  What setting  
> > is that, can it be increased, and if so how?  Googling for the error  
> > has produced lots of hits but none with answers, yet.  Still browsing.
> > 
> > Below is a transcript of the session.
> > 
> > Thanks in advance for any pointers in the right direction.
> > 
> > Regards,
> > - Robert
> > 
> > $ uname -sorv ; rpm -q R ; R --version
> > Linux 2.6.11-1.1369_FC4smp #1 SMP Thu Jun 2 23:08:39 EDT 2005 GNU/Linux

          ^^^^^^^^^^^^^^^^^^^^
<snip>

I might throw out one more pointer in addition to Uwe's comments above,
which _should_ not affect this issue, but as an FYI. Note that I said
"should not" versus "will not".

You are about 17 kernel versions behind. 2.6.11-1.1369_FC4smp was the
original FC4 SMP kernel.

The current FC4 kernel version is 2.6.16-1.2107_FC4smp.

This might suggest that your system in general may require some
substantial updating, which may more generally affect system behavior.

FC4 was rather unstable when first released and has improved notably
since then. It is one of the reasons that some folks are still running
FC3, even though it is EOL.

The current FC4 kernel release (noted above) has some issues with it at
present. A new kernel release version by Dave Jones, 2111, should be out
"any time now", but in the mean time, I would suggest updating your
kernel to version 2.6.16-1.2096smp and doing a full system update
generally.

You may very well find that some behaviors (related and/or unrelated to
this issue) do change for the better.

HTH,

Marc Schwartz