[R] 64-bit R on 64-bit Windows box... Still not enough memory?!

[David Winsemius]

On Aug 5, 2010, at 7:40 PM, noclue_ wrote:

>
> I have a 64-bit windows box -
>    Intel Xeon CPU E7340 @ 2.4GHz 31.9GB of RAM
> I have R 2.11.1 (64bit) running on it.

Dear noclue_;

  What does this return?:

.Machine$sizeof.pointer

>
> My csv data is 3.6 GB (with about 15 million obs, 120 variables.)

On my 64 bit setup with 24GB of RAM I can comfortably work with a  
dataset that is around the same number of columns but (only)  4.5  
million rows. Working with this size data.frame in 18GB of RAM was  
somewhat uncomfortable because it would often "roll over" into virtual  
memory and then modeling calls took forever.... well,  twenty minutes  
anyway.

I think you may be under-estimating the space requirements when  
working with larger objects. Numerics take 8 bytes. Objects often need  
to be copied and so space consumption can quickly double or triple.

 > 8*15000000*120
[1] 1.44e+10

So that's 14 GB just to hold the object, not to do anything useful  
with it.

-- 
David.
> ------------------------------------------------
> I have successfully imported the data above into R. No problem.
>
> Now I am trying to run 'rpart' on my data.  But I got the following  
> error :
>
> Error: cannot allocate vector of size 53.5 Mb
> In addition: Warning messages:
> 1: In lapply(x, "is.na") :
>  Reached total allocation of 32764Mb: see help(memory.size)
> 2: In lapply(x, "is.na") :
>  Reached total allocation of 32764Mb: see help(memory.size)
> 3: In lapply(x, "is.na") :
>  Reached total allocation of 32764Mb: see help(memory.size)
> 4: In lapply(x, "is.na") :
>  Reached total allocation of 32764Mb: see help(memory.size)
>>
> ===========================================================
> Can anybody give me a hint on how to solve this?

Post better details?:

?sessionInfo


> Thanks!
> =========================================================
>

David Winsemius, MD
West Hartford, CT