[R] Matrix Size

[Douglas Bates]

On Wed, Jul 14, 2010 at 4:23 PM, paul s <r-project.org at queuemail.com> wrote:
> hi -

> i just started using R as i am trying to figure out how perform a linear
> regression on a huge matrix.

> i am sure this topic has passed through the email list before but could not
> find anything in the archives.

> i have a matrix that is 2,000,000 x 170,000 the values right now are
> arbitray.

> i try to allocate this on a x86_64 machine with 16G of ram and i get the
> following:

>> x <- matrix(0,2000000,170000);
> Error in matrix(0, 2e+06, 170000) : too many elements specified

R stores matrices and other data objects in memory.  A matrix of that
size would require
> 2e+06*170000*8/2^30
[1] 2533.197

gigabytes of memory.  Start looking for a machine with at least 5
terabytes of memory (you will need more than one copy of the matrix to
be stored) or, probably easier, rethink your problem.

Results from a linear regression producing 170,000 coefficient
estimates are unlikely to be useful.  The model matrix is essentially
guaranteed to be rank deficient.

>>
>
> is R capable of handling data of this size? am i doing it wrong?
>
> cheers
> paul
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>