[R] Construction of a large sparse matrix

[Huntsinger, Reid]

Your statements for sample.size=10,000 try to construct a matrix with 3
dense 10,000 x 10,000 blocks. That's approximately 10*10*8 MB each and very
likely explains your error message. 

Since you have a simple formula for the matrix, why not define a function to
implement multiplication by this matrix (and whatever else you want to do
with it), rather than use general sparse matrix representations? 

Reid Huntsinger

-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Doran, Harold
Sent: Monday, April 18, 2005 4:55 PM
To: r-help at stat.math.ethz.ch
Subject: [R] Construction of a large sparse matrix


Dear List:

I'm working to construct a very large sparse matrix and have found
relief using the SparseM package. I have encountered an issue that is
confusing to me and wonder if anyone may be able to suggest a smarter
solution. The matrix I'm creating is a covariance matrix for a larger
research problem that is subsequently used in a simulation. Below is the
latex form of the matrix if anyone wants to see the pattern I am trying
to create. 

The core of my problem seems to localize to the last line of the
following portion of code. 

n<-sample.size*4
k<-n/4
vl.mat <- as.matrix.csr(0, n, n) 
block <- 1:k #each submatrix size
for(i in 1:3) vl.mat[i *k + block, i*k + block] <- LE

When the variable LE is 0, the matrix is easily created. For example,
when sample.size = 10,000 this matrix was created on my machine in about
1 second. Here is the object size.

> object.size(vl.mat)
[1] 160692

However, when LE is any number other than 0, the code generates an
error. For example, when I try LE <- 2 I get

Error: cannot allocate vector of size 781250 Kb
In addition: Warning message: 
Reached total allocation of 1024Mb: see help(memory.size) 
Error in as.matrix.coo(as.matrix.csr(value, nrow = length(rw), ncol =
length(cl))) : 
        Unable to find the argument "x" in selecting a method for
function "as.matrix.coo" 

I'm guessing that single digit integers should occupy the same amount of
memory. So, I'm thinking that the matrix is "less sparse" and the
problem is related to the introduction of a non-zero element (seems
obvious). However, the matrix still retains a very large proportion of
zeros. In fact, there are still more zeros than non-zero elements. 

Can anyone suggest a reason why I am not able to create this matrix? I'm
at the limit of my experience and could use a pointer if anyone is able
to provide one.

Many thanks,
Harold


P.S. The matrix above is added to another matrix to create the
covariance matrix below. The code above is designed to create the
portion of the matrix \sigma^2_{vle}\bm{J} . 


\begin{equation}
\label{vert:cov}
\bm{\Phi} = var
\left [ 
\begin{array}{c}
Y^*_{1}\\
Y^*_{2}\\
Y^*_{3}\\
Y^*_{4}\\
\end{array}
\right ]
=
\left [ 
\begin{array}{cccc}
\sigma^2_{\epsilon}\bm{I}& \sigma^2_{\epsilon}\rho\bm{I} & \bm{0} &
\bm{0}\\
\sigma^2_{\epsilon}\rho\bm{I} &
\sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} &
\sigma^2_{\epsilon}\rho^2\bm{I} & \bm{0}\\
\bm{0} & \sigma^2_{\epsilon}\rho^2\bm{I} &
\sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J}&
\sigma^2_{\epsilon}\rho^3\bm{I}\\
\bm{0} & \bm{0} & \sigma^2_{\epsilon}\rho^3\bm{I}&
\sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} \\
\end{array}
\right]
\end{equation}

where $\bm{I}$ is the identity matrix, $\bm{J}$ is the unity matrix, and
$\rho$ is the autocorrelation.



	[[alternative HTML version deleted]]

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html