[R] large data set, error: cannot allocate vector

[Prof Brian Ripley]

On Fri, 5 May 2006, Robert Citek wrote:

>
> On May 5, 2006, at 11:30 AM, Thomas Lumley wrote:
>> In addition to Uwe's message it is worth pointing out that gc()
>> reports
>> the maximum memory that your program has used (the rightmost two
>> columns).
>> You will probably see that this is large.
>
> Reloading the 10 MM dataset:

Ah, but the comment was about the 100 MM dataset, the one which gave you a 
problem.

> R > foo <- read.delim("dataset.010MM.txt")
>
> R > object.size(foo)
> [1] 440000376
>
> R > gc()
>            used  (Mb) gc trigger  (Mb) max used  (Mb)
> Ncells 10183941 272.0   15023450 401.2 10194267 272.3
> Vcells 20073146 153.2   53554505 408.6 50086180 382.2
>
> Combined, Ncells or Vcells appear to take up about 700 MB of RAM,
> which is about 25% of the 3 GB available under Linux on 32-bit
> architecture.

Please re-read help("Memory-limits").  You have a 3Gb address space, and 
are looking for at least one 500Mb chunk.  Fragmentation will come into 
play here and it is quite likely that malloc will be unable to find a 
500Mb chunk once you have allocated 1Gb.  In attempting to read the 
100 MM dataset you probably did go over 1Gb.

> Also, removing foo seemed to free up "used" memory, but didn't change 
> the "max used":

Well, it doesn't change history does it?  You were not expecting removing 
objects to increase the memory used, I hope.  From ?gc:

      The final two columns show the maximum space used since the last
      call to 'gc(reset=TRUE)' (or since R started).

> R > rm(foo)
>
> R > gc()
>          used (Mb) gc trigger  (Mb) max used  (Mb)
> Ncells 186694  5.0   12018759 321.0 10194457 272.3
> Vcells  74095  0.6   44173915 337.1 50085563 382.2

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595