[R] R help
Petr Savicky
savicky at cs.cas.cz
Sun Feb 19 11:01:01 CET 2012
On Sat, Feb 18, 2012 at 06:00:53PM -0500, li li wrote:
> Dear all,
> I need to generate numbers from multivariate normal with large dimensions
> (5,000,000).
> Below is my code and the error I got from R. Sigma in the code is the
> covariance
> matrix. Can anyone give some idea on how to take care of this error. Thank
> you.
> Hannah
>
> > m <- 5000000
> > m1 <- 0.5*m
> > rho <- 0.5
> > Sigma <- rho* matrix(1, m, m)+diag(1-rho, m)
> Error in matrix(1, m, m) : too many elements specified
Hi.
The matrix of dimension m times m does not fit into memory,
since it requires 8*m^2 = 2e+14 bytes = 2e+05 GB.
Generating a multivariate normal with a covariance matrix
with 1 on the diagonal and rho outside of the diagonal may
be done also as follows.
m <- 10 # can be 5000000
rho <- 0.5
# single vector
x <- rnorm(1, sd=sqrt(rho)) + rnorm(m, sd=sqrt(1 - rho))
# several vectors
a <- t(replicate(10000, rnorm(1, sd=sqrt(rho)) + rnorm(m, sd=sqrt(1 - rho))))
# check the sample covariance matrix if m is not too large
sigma <- cov(a)
range(diag(sigma)) # elements on the diagonal
[1] 0.9963445 1.0196015
diag(sigma) <- NA
range(sigma, na.rm=TRUE) # elements outside of the diagonal
[1] 0.4935129 0.5162836
Generating several vectors using replicate() may not be efficient.
The following can be used instead.
n <- 10000
a <- matrix(rnorm(n*m, sd=sqrt(1 - rho)), nrow=n, ncol=m) + rnorm(n, sd=sqrt(rho))
Note that the size of "a" is n times m and it should fit into the memory.
Hope this helps.
Petr Savicky.
More information about the R-help
mailing list