[R] Symmetric Matrix classes

[Gad Abraham]

On Wed, Jan 20, 2010 at 8:18 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Tue, Jan 19, 2010 at 9:26 AM, Martin Maechler
> <maechler at stat.math.ethz.ch> wrote:
>> Scanning for 'Matrix' in old R-help e-mail, I found
>>
>>>>>>> "GA" == Gad Abraham <gabraham at csse.unimelb.edu.au>
>>>>>>>     on Fri, 27 Nov 2009 13:45:00 +1100 writes:
>>
>>    GA> Hi,
>>    GA> I'd like to store large covariance matrices using Matrix classes.
>>
>>    GA> dsyMatrix seems like the right one, but I want to specify just the
>>    GA> upper/lower triangle and diagonal and not have to instantiate a huge
>>    GA> n^2 vector just for the sake of having half of it ignored:
>>
>>    GA> Dumb example:
>>    GA> M <- new("dsyMatrix", uplo="U", x=rnorm(1e4), Dim=as.integer(c(100, 100)))
>>    GA> diag(M) <- 1
>>
>>    GA> This doesn't work:
>>    GA> M <- new("dsyMatrix", uplo="U", x=0, Dim=as.integer(c(100, 100)))
>>    GA> Error in validObject(.Object) :
>>    GA> invalid class "dsyMatrix" object: length of x slot != prod(Dim)
>>
>>    GA> Is there an easier way of doing this?
>>
>> I think you want a  "dspMatrix"  ("sp" == "symmetric packed")
>> instead.
>>
>> Before going into details: Is this topic still interesting to
>> those involved almost two months ago?
>>
>> Regards,
>> Martin
>
> Also, I fail to understand the advantage of allocating storage for
> roughly half the elements in the matrix instead of the whole matrix
> when the matrix is so large.  If dense storage of a symmetric matrix
> of size 20000 takes up too much memory (approx 3 GB for each copy) it
> is unlikely that the packed symmetric storage form, using about 1.5 GB
> per copy, is going to save the day.
>
>> 8 * (20000 * 20000)/2^30  # size in gigabytes for full array
> [1] 2.980232
>> 4 * (20000 * 20001)/2^30  # size in gigabytes for sp array
> [1] 1.490191

The difference between 16GB and 32 GB will determine whether I can
store the matrix in RAM or not.

>
> If the matrix is sparse, the dsCMatrix class may help.

It's a covariance matrix, it's dense.

>
> And there is also the issue of what exactly do you want to do with the
> matrix once you have stored it?
>

I'm mainly interested in doing things like partial correlation using
(taking the pseudoinverse of the matrix using SVD or Lanczos).


-- 
Gad Abraham
PhD Student, Dept. CSSE and NICTA
The University of Melbourne
Parkville 3010, Victoria, Australia
email: gabraham at csse.unimelb.edu.au
web: http://www.csse.unimelb.edu.au/~gabraham