[R] processing a large matrix

[Charles C. Berry]

On Mon, 12 Feb 2007, andy1983 wrote:

>
> I would like to compare every column in my matrix with every other column and
> get the r-squared.
>
> I tried using the following formula and looping through every column:
>> summary(lm(matrix[,x]~matrix[,y]))$r.squared
> If I have 10,000 columns, the loops (10,000 * 10,000) take forever even if
> there is no formula inside.
>
> Then, I attempted to vectorize my code:
>> cor(matrix)^2
> With 10,000 columns, this works great. With 30,000, R tells me it cannot
> allocate vector of that length even if the memory limit is set to 4 GBs.


30000^2 doubles * 8 Bytes/double > 6.5 GBs.

And that's just to store the result; you will need some space to work 
in, too.


>
> Is there anything else I can do to resolve this issue?
>
> Thanks.
> -- 
> View this message in context: http://www.nabble.com/processing-a-large-matrix-tf3216447.html#a8932591
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

Charles C. Berry                        (858) 534-2098
                                          Dept of Family/Preventive Medicine
E mailto:cberry at tajo.ucsd.edu	         UC San Diego
http://biostat.ucsd.edu/~cberry/         La Jolla, San Diego 92093-0901