[R] Error: cannot allocate vector of size x Gb (64-bit ... yet again)

[Uwe Ligges]

On 23.11.2010 09:26, derek eder wrote:
> Hello,
>
> I am facing the dreaded "Error: cannot allocate vector of size x Gb" and
> don't understand
> enough about R (or operating system) memory management to diagnose and
> solve the problem
> -- despite studying previous posts and relevant R help -- e.g.:
>
> "Error messages beginning cannot allocate vector of size indicate a
> failure to obtain memory,
> either because the size exceeded the address-space limit for a process
> or, more likely,
> because the system was unable to provide the memory.
> [...] On all builds of R, the maximum length (number of elements)
> of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers.
> In addition, the storage space cannot exceed the address limit."
> - from Memory-limits {Base}
>
>
> Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of
> "unlimited", can the size of an R object exceed the amount of availlable
> RAM
> memory?
>
> Empirically my system with 4Gb RAM and ample Swap, is failing:
>
>  > x <- integer(10^9)
>
>  > object.size(x)
> 4000000040 bytes
>
>  > gc()
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 121195 6.5 350000 18.7 350000 18.7
> Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2
>
>  > matrix(x, ncol=16)
> Error: cannot allocate vector of size 3.7 Gb
>
> I don't understand how this operation violates the limits detailed in
> the Memory-limit help (above).


Why? If you have a limit of 4Gb, 4Gb are already used and you try to 
work with a 4Gb objeczt in the matrix function which makes a copy and 
creates another copy (with dim attributed), so 8Gb additionally. You 
will need at least 12 of your 4Gb.  Probably R does not get a sufficient 
amount of non-fragmented memory from the OS or your swap space is not 
"ample".

To make the above more efficient, just use:

dim(x) <- c(10^8/16, 16)

and you won't get any copies.

Uwe


> Thank you!
>
>
> Derek Eder
>
>
>
> -------------------------------------------------------------------------------------------------
>
>
>  > version
> _
> platform x86_64-pc-linux-gnu
> arch x86_64
> os linux-gnu
> system x86_64, linux-gnu
> status
> major 2
> minor 11.1
> year 2010
> month 05
> day 31
> svn rev 52157
> language R
> version.string R version 2.11.1 (2010-05-31)
>
>
>
> derek at papanca:~$ top
>
> top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45
> Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 3796484k total, 3764852k used, 31632k free, 14204k buffers
> Swap: 2929660k total, 834240k used, 2095420k free, 94800k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2854 derek 20 0 239m 9260 5448 S 6 0.2 0:05.53 gnome-terminal
> 1164 root 20 0 218m 31m 10m S 4 0.8 1:29.71 Xorg
> 3331 derek 20 0 19276 1324 944 R 1 0.0 0:00.6 top
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.