[R] memory error with 64-bit R in linux

[zhihua li]

Thanks for replying!
i don't think i'm paging. i tried to use a smaller version of my matrix and 
do all the checkings as suggested by jim. The smaller matrix caused another 
problem, for which I've opened another thread. But i've found something 
about memory that I don't understand.
> gc()
          used (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  269577 14.4    5570995 297.6  8919855 476.4
Vcells 3353395 25.6    9493567  72.5 15666095 119.6

Does this mean the maximum memory I can use for variables is only 120 M?
However, when I tried to check the memory limits:
> mem.limits()
nsize vsize
   NA    NA

Here it seems the maximum memory is not limited?

When there is no R function is being executed, I checked the system process 
by:
ps u

PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
7821  0.0  0.1  10048  2336 pts/0    Ss   Jul18   0:00 -bash
8076  2.9 24.5 523088 504004 pts/0   S+   Jul18   2:46 /usr/lib64/R/bi
8918  1.5  0.1   9912  2328 pts/1    Ss   00:44   0:00 -bash
8962  0.0  0.0   3808   868 pts/1    R+   00:45   0:00 ps u

Does this mean R is using 25% of my memory? But my RAM is 2 GB and the 
objects in R only occupy 40 MB from gc().

Did I interpret it wrong?

Thanks a lot!



>From: "jim holtman" <jholtman at gmail.com>
>To: "zhihua li" <lzhtom at hotmail.com>
>CC: r-help at stat.math.ethz.ch
>Subject: Re: [R] memory error with 64-bit R in linux
>Date: Wed, 18 Jul 2007 17:50:31 -0500
>
>Are you paging?  That might explain the long run times. How much 
>space
>are your other objects taking up?  The matrix by itself should only
>require about 13MB if it is numeric.  I would guess it is some of 
>the
>other objects that you have in your working space.  Put some gc() in
>your loop to see how much space is being used.  Run it with a subset
>of the data and see how long it takes.  This might give you an
>estimate of the time, and space, that might be needed for the entire
>dataset.
>
>Do a 'ps' to see how much memory your process is using.  Do one 
>every
>couple of minutes to see if it is growing.  You can alway use 
>Rprof()
>to get an idea of where time is being spent (use it on a small
>subset).
>
>On 7/18/07, zhihua li <lzhtom at hotmail.com> wrote:
>>Hi netters,
>>
>>I'm using the 64-bit R-2.5.0 on a x86-64 cpu, with an RAM of 2 GB.  
>>The
>>operating system is SUSE 10.
>>The system information is:
>>-uname -a
>>Linux someone 2.6.13-15.15-smp #1 SMP Mon Feb 26 14:11:33 UTC 2007 
>>x86_64
>>x86_64 x86_64 GNU/Linux
>>
>>I used heatmap to process a matrix of the dim [16000,100].  After 3 
>>hours
>>of desperating waiting, R told me:
>>cannot allocate vector of size 896 MB.
>>
>>I know the matrix is very big, but since I have 2 GB of RAM and in 
>>a 64-bit
>>system, there should be no problem to deal with a vector smaller 
>>than 1 GB?
>>(I was not running any other applications in my system)
>>
>>Does anyone know what's going on?  Is there a hardware limit where 
>>I have
>>to add more RAM, or is there some way to resolve it softwarely? 
>>Also is it
>>possible to speed up the computing (I don't wanna wait another 3 
>>hours to
>>know I get another error message)
>>
>>Thank you in advance!
>>
>>_________________________________________________________________
>>ÏíÓÃÊÀ½çÉÏ×î´óµÄµç×ÓÓʼþϵͳ¡ª MSN Hotmail¡£  
>>http://www.hotmail.com
>>
>>
>>______________________________________________
>>R-help at stat.math.ethz.ch mailing list
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide 
>>http://www.R-project.org/posting-guide.html
>>and provide commented, minimal, self-contained, reproducible code.
>>
>>
>
>
>--
>Jim Holtman
>Cincinnati, OH
>+1 513 646 9390
>
>What is the problem you are trying to solve?