[R] Memory Problems in R

[Roger D. Peng]

There is a limit on how long a single vector can be, and I think it's 
2GB (even on 64-bit platforms).  Not sure on how the gc trigger is set....

-roger

Scott Gilpin wrote:
> Hello everyone -
> 
> I have a couple of questions about memory management of large objects.
> Thanks in advance for your response.
> 
> I'm running R version 1.9.1 on solaris 8, compiled as a 32 bit app. 
> My system has 12.0 GB of memory, with usually ~ 11GB free.  I checked
> system limits using ulimit, and there is nothing set that would limit
> the maximum amount of memory for a process (with the exception of an
> 8MB stack size).  I've also checked the amount of memory available to
> R using mem.limits(), and there is no limit set.
> 
> I'm running into two problems.  The first is the error  "cannot
> allocate vector of size XXXXX" - I know this has been discussed
> several times on this mailing list, but it usually seems the user does
> not have enough memory on their system, or does not have the memory
> limits set correctly.  I don't believe this is the case in this
> situation.  I verified that I don't have any objects in memory when R
> starts up, and that memory limits are set to NA.  Here is some output:
> 
> 
>>ls()
> 
> character(0)
> 
>>mem.limits()
> 
> nsize vsize 
>    NA    NA 
> 
>>gc()
> 
>          used (Mb) gc trigger (Mb)
> Ncells 432197 11.6     531268 14.2
> Vcells 116586  0.9     786432  6.0
> 
>>v<-rep(0,268435431)
> 
> Error: cannot allocate vector of size 2097151 Kb
> 
>>v<-rep(0,268435430)
>>object.size(v)
> 
> [1] 2147483468
> 
>>gc()
> 
>             used   (Mb) gc trigger   (Mb)
> Ncells    432214   11.6     741108   19.8
> Vcells 268552029 2048.9  268939773 2051.9
> 
> 
> Does R have a limit set on the size of an object that it will
> allocate?  I know that the entire application will only be able to use
> 4GB of memory (because it's only 32bit), but I haven't found anything
> in the R documentation or the help lists that indicates there is
> maximum on the size of an object.  I understand there will be problems
> if an object is greater than 2GB and needs to be copied - but will R
> limit the creation of such an object?  It's also my understanding that
> the garbage collector won't move objects and this may cause memory to
> become fragmented - but I'm seeing these issues on startup when there
> are no objects in memory.
> 
> 
> My second problem is with matrices and the garbage collector, and the
> limits it sets for gc trigger after a matrix is created.  When I
> create a vector of approximately 500MB, R sets the gc trigger to be
> slightly above this amount.  The gc trigger also seems to correspond
> to the process size (as output by top).  When I create a matrix of
> approximately 500MB, R sets the gc trigger to be roughly 3 times the
> size of the matrix (and the process size is ~ 1.5GB).  Therefor, when
> I try to create larger matrices, where 3x the size of the matrix is
> greater than 4GB, R gives me an error.  Is there anything I can do to
> create large matrices?  Or do I have to manipulate large objects as a
> vector?
> 
> Output from the 3 different scenarios is below:
> 
> 1) - can't create a matrix, but can create a vector
> 
> [Previously saved workspace restored]
> 
> 
>>m<-matrix(rep(0,25000*10000),nrow=10000)
> 
> Error: cannot allocate vector of size 1953125 Kb
> 
>>v<-rep(0,25000*10000)
>>object.size(v)/1024
> 
> [1] 1953125
> 
> 
> 2) gc trigger is set slightly higher than the size of the vector
> 
> 
>>ls()
> 
> character(0)
> 
>>mem.limits()
> 
> nsize vsize 
>    NA    NA 
> 
>>gc()
> 
>          used (Mb) gc trigger (Mb)
> Ncells 432197 11.6     531268 14.2
> Vcells 116586  0.9     786432  6.0
> 
>>v<-rep(0,(2510)*(25000))
>>object.size(v)
> 
> [1] 5.02e+08
> 
>>gc()
> 
>            used  (Mb) gc trigger  (Mb)
> Ncells   432210  11.6     667722  17.9
> Vcells 62866589 479.7   63247172 482.6
> 
> 
> 3) gc trigger is set ~ 3x the size of the matrix
> 
> 
>>ls()
> 
> character(0)
> 
>>mem.limits()
> 
> nsize vsize 
>    NA    NA 
> 
>>gc()
> 
>          used (Mb) gc trigger (Mb)
> Ncells 432197 11.6     531268 14.2
> Vcells 116586  0.9     786432  6.0
> 
>>A<-matrix(rep(0,(2510)*(25000)),nrow=(2510),ncol=(25000))
>>object.size(A)
> 
> [1] 502000120
> 
>>gc()
> 
>            used  (Mb) gc trigger   (Mb)
> Ncells   432213  11.6     741108   19.8
> Vcells 62866590 479.7  188640940 1439.3
> 
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>