[R] Matrix Size

[Duncan Murdoch]

On 14/07/2010 5:23 PM, paul s wrote:
> hi -
> 
> i just started using R as i am trying to figure out how perform a linear 
> regression on a huge matrix.
> 
> i am sure this topic has passed through the email list before but could 
> not find anything in the archives.
> 
> i have a matrix that is 2,000,000 x 170,000 the values right now are 
> arbitray.
> 
> i try to allocate this on a x86_64 machine with 16G of ram and i get the 
> following:
> 
>  > x <- matrix(0,2000000,170000);
> Error in matrix(0, 2e+06, 170000) : too many elements specified
>  >
> 
> is R capable of handling data of this size? am i doing it wrong?

It is capable of handling large data, but not that large in a single 
matrix.  The limit on the number of entries in any vector (and matrices 
are stored as vectors) is about 2^31 ~ 2 billion.  Your matrix needs 
about 340 billion entries, so it's too big.  (It won't all fit in 
memory, either:  you've only got space for 2 billion numeric values in 
your 16G of RAM, and you also need space for the OS, etc.  But the OS 
can use disk space as virtual memory, so you might be able to get that 
much memory, it would just be very, very slow.)

You need to break up the work into smaller pieces.

Duncan Murdoch

> 
> cheers
> paul
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.