[R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)

[Prof Brian Ripley]

On Tue, 23 Nov 2010, derek eder wrote:

> Hello,
>
> I am facing the dreaded "Error: cannot allocate vector of size x Gb" and 
> don't understand
> enough about R (or operating system) memory management to diagnose and solve 
> the problem
> -- despite studying previous posts and relevant R help -- e.g.:
>
> "Error messages beginning cannot allocate vector of size indicate a failure 
> to obtain memory,
> either because the size exceeded the address-space limit for a process or, 
> more likely,
> because the system was unable to provide the memory.
> [...] On all builds of R, the maximum length (number of elements)
> of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers.
> In addition, the storage space cannot exceed the address limit."
> - from Memory-limits {Base}
>
>
> Simple question:  Given 64-bit R (AMD64 Linux) with a ulimit of "unlimited", 
> can the size of an R object exceed the amount of availlable RAM
> memory?
>
> Empirically my system with 4Gb RAM and ample Swap, is failing:
>
>>  x <- integer(10^9)
>
>> object.size(x)
> 4000000040 bytes
>
>> gc()
>            used   (Mb) gc trigger   (Mb)  max used   (Mb)
> Ncells    121195    6.5     350000   18.7    350000   18.7
> Vcells 500124024 3815.7  606849099 4629.9 550124408 4197.2
>
>> matrix(x, ncol=16)
> Error: cannot allocate vector of size 3.7 Gb
>
> I don't understand how this operation violates the limits detailed in the 
> Memory-limit help (above).

It doesn't, and that example works on my (32GB RAM) 64-bit system

> gc()
              used   (Mb) gc trigger    (Mb)   max used    (Mb)
Ncells     134054    7.2     350000    18.7     350000    18.7
Vcells 1000131158 7630.4 1575382823 12019.3 1500131550 11445.1

So maybe you don't have 'ample swap' and/or have a VM limit set.

Note that

dim(x) <- c(length(x)/16, 16)

is much more efficient use of memory.


>
> Thank you!
>
>
> Derek Eder
>
>
>
> -------------------------------------------------------------------------------------------------
>
>> version
>               _
> platform       x86_64-pc-linux-gnu
> arch           x86_64
> os             linux-gnu
> system         x86_64, linux-gnu
> status
> major          2
> minor          11.1
> year           2010
> month          05
> day            31
> svn rev        52157
> language       R
> version.string R version 2.11.1 (2010-05-31)
>
>
>
> derek at papanca:~$ top
>
> top - 09:10:18 up 51 min,  4 users,  load average: 0.51, 0.51, 0.45
> Tasks: 160 total,   2 running, 158 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.0%us, 25.0%sy,  0.0%ni, 75.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
> 0.0%st
> Mem:   3796484k total,  3764852k used,    31632k free,    14204k buffers
> Swap:  2929660k total,   834240k used,  2095420k free,    94800k cached
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
> 2854 derek     20   0  239m 9260 5448 S    6  0.2   0:05.53 gnome-terminal
> 1164 root      20   0  218m  31m  10m S    4  0.8   1:29.71 Xorg
> 3331 derek     20   0 19276 1324  944 R    1  0.0   0:00.6  top
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595