[R] Data Simulation in R

[Doran, Harold]

Thanks. But, I think I am doing that. I use rm() and gc() as the code
moves along. The datasets are stored as a list. Is there a way that I
can save the list object and call each dataset within a list one at a
time, or must the entire list be in memory at once?

Harold

-----Original Message-----
From: Uwe Ligges [mailto:ligges at statistik.uni-dortmund.de] 
Sent: Wednesday, January 19, 2005 5:51 AM
To: Doran, Harold
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] Data Simulation in R

Doran, Harold wrote:

> Dear List:
> 
> A few weeks ago I posted some questions regarding data simulation and 
> received some very helpful comments, thank you. I have modified my 
> code accordingly and have made some progress.
> 
> However, I now am facing a new challenge along similar lines. I am 
> attempting to simulate 250 datasets and then run the data through a 
> linear model. I use rm() and gc() as I move along to clean up the 
> workspace and preserve memory. However, my aim is to use sample sizes 
> of 5,000 and 10,000. By any measure this is a huge task.
> 
> My machine has 2GB RAM and a Pentium 4 2.8 GHz machine with Windows
XP.
> I have the following in the "target" section of the Windows shortcut 
> --max-mem-size=1812M
> 
> With such large samples, R is unable to perform the analysis, at least

> with the code I have developed. It halts when it runs out of memory. A

> collegue subsequently constructed the simulation using another 
> software program with a similar computer and, while it took over night

> (and then some), the program produced the results desired.
> 
> I am curious if it is the case that such large simulations are out of 
> the grasp of R or if my code is not adequately organized to perform 
> the simulation.
> 
> I would appreciate any thoughts or advice.


Don't hold all datasets (and results, if they are big) in the memory at
the same time!!!

Either generate them when you use them and delete them afterwards, or
save them to disc an only load one by one for further analyses.
Also, you might want to call gc() after you removed large objects...

Uwe Ligges



> Harold
> 
> 
> 
> library(MASS)
> library(nlme)
> mu<-c(100,150,200,250)
> Sigma<-matrix(c(400,80,80,80,80,400,80,80,80,80,400,80,80,80,80,400),4
> ,4
> )
> mu2<-c(0,0,0)
> Sigma2<-diag(64,3)
> sample.size<-5000
> N<-250 #Number of datasets
> #Take a single draw from VL distribution vl.error<-mvrnorm(n=N, mu2, 
> Sigma2)
> 
> #Step 1 Create Data
> Data <- lapply(seq(N), function(x)
> as.data.frame(cbind(1:10,mvrnorm(n=sample.size, mu, Sigma))))
> 
> #Step 2 Add Vertical Linking Error
> for(i in seq(along=Data)){
> Data[[i]]$V6 <- Data[[i]]$V2
> Data[[i]]$V7 <- Data[[i]]$V3 + vl.error[i,1]
> Data[[i]]$V8 <- Data[[i]]$V4 + vl.error[i,2]
> Data[[i]]$V9 <- Data[[i]]$V5 + vl.error[i,3] }
> 
> #Step 3 Restructure for Longitudinal Analysis long <- lapply(Data, 
> function(x) reshape(x, idvar="Data[[i]]$V1", 
> varying=list(c(names(Data[[i]])[2:5]),c(names(Data[[i]])[6:9])),
> v.names=c("score.1","score.2"), direction="long"))
> 
> #####################
> #Clean up Workspace
> rm(Data,vl.error)
> gc()
> #####################
> 
> # Step 4 Run GLS
> 
> glsrun1 <- lapply(long, function(x) gls(score.1~I(time-1), data=x, 
> correlation=corAR1(form=~1|V1), method='ML'))
> 
> # Extract intercepts and slopes
> int1 <- sapply(glsrun1, function(x) x$coefficient[1])
> slo1 <- sapply(glsrun1, function(x) x$coefficient[2])
> 
> ################
> #Clean up workspace
> rm(glsrun1)
> gc()
> 
> glsrun2 <- lapply(long, function(x) gls(score.2~I(time-1), data=x, 
> correlation=corAR1(form=~1|V1), method='ML'))
> 
> # Extract intercepts and slopes
> int2 <- sapply(glsrun2, function(x) x$coefficient[1])
> slo2 <- sapply(glsrun2, function(x) x$coefficient[2])
> 
>  
> #Clean up workspace
> rm(glsrun2)
> gc()
> 
> 
> 
> # Print Results
> 
> cat("Original Standard Errors","\n", "Intercept","\t", 
> sd(int1),"\n","Slope","\t","\t", sd(slo1),"\n")
> 
> cat("Modified Standard Errors","\n", "Intercept","\t", 
> sd(int2),"\n","Slope","\t","\t", sd(slo2),"\n")
> 
> rm(list=ls())
> gc()
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html