Thanks for the quick reply! See my answers below...

On Sep 16, 2011, at 2:24 PM, Douglas Bates wrote:

> On Fri, Sep 16, 2011 at 10:46 AM,  <jpl2136@columbia.edu> wrote:
>> Hi,
>> 
>> I am running some pretty big mlm models using glmer right now and they are taking forever. So I really would like to speed things up.
>> Here is the data setup: I have around 130 datasets on the dyad level with 3,600 to 4,000,000 observations, the outcome is binary, I am using two cross-classified random intercepts (for i and j in the dyad) with equal group sizes, and about 35 covariates. For each dataset, I am currently fitting 4 models with slightly different specifications and there will be more. Right now, I am running all this in the cloud on 9 computers. I started the job over 4 days ago and I still only have results for 26 of the datasets.
>> 
>> [...]
>> 
>> 3) Speeding up the fitting process itself. I am not sure about this option and would be happy about any suggestions.
> 
> Which version of lme4?  Could you send the output of sessionInfo() in
> R after loading the lme4 package?
> 
> The 35 covariates are likely the cause of the slow fitting.  In a
> linear mixed-effects model you can remove the fixed-effects parameters
> from the general optimization of the deviance because the conditional
> estimates of the fixed-effects, given the other parameters, can be
> determined in a direct computation.  In a generalized linear
> mixed-effects model you can do the same thing but you aren't
> guaranteed to get exactly the conditional estimate.  The experimental
> versions of glmer allow you to use this short-cut method to fit the
> model and usually produce answers that are sufficiently close to the
> optimum that the difference in the deviance is nearly negligible
> whereas the difference in time for fitting the model can be enormous.

That sounds like a very good solution! Can you point me to the corresponding argument or give some more details about setting this up in the experimental version of glmer (below is also the code for my model)? Is the experimental version available for download? (when I end up using this, I am also happy to report back about eventual problems and my experiences with using the stable and the experimental version).
Is that solution scaleable - i.e. does it also work when I include 200 covariates (fixed effects instead of one of the random effects). 

fit=lmer(y ~ X + (1 | i) + (1 | j),family=binomial(link="logit"))	
whereby X is a n x k matrix of covariates so that the expanded version looks like this
fit=lmer(y ~ female.ego + female.alter + dyad.female.same + ... + (1 | i) + (1 | j),family=binomial(link="logit"))	



lme4 version
Here is the output from sessionInfo() from the main computational node. Some of the other nodes use older versions of R but I think the same lme4 package. I have to check that though, which is not necessarily straight forward because I can't control to which node my job is send.  

R version 2.13.0 (2011-04-13)
Platform: x86_64-redhat-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=C              LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] lme4_0.999375-39   Matrix_0.999375-50 lattice_0.19-23   

loaded via a namespace (and not attached):
[1] grid_2.13.0   nlme_3.1-100  stats4_2.13.0



> 
>> 4) The size of the 'mer' objects get pretty big and I would like to reduce the size to make later post-processing easier. I am using the function below to do that, which reduces the object size in R to ~5% of the original size. So everything seems to be fine but when I save the 'mer' object using save(fit1,fit2,file=[PATH]), the RData file is still up to 400 MB big, which is roughly the size of the R objects before using the mer.clean function. Any suggestions?
> 
> save(fit1, fit2, file=[PATH}, compress="xz")
> 
> tends to provide the best file compression in the saved object,
> although it will often take longer to save the object because of the
> calculations involved in the compression.
>> mer.clean = function(x) {
>>        x@frame=data.frame()
>>        x@A=new("dgCMatrix")
>>        x@Zt=new("dgCMatrix")
>>        x@X=matrix() x@mu=as.numeric(NA)
>>        x@muEta=as.numeric(NA)
>>        x@resid=as.numeric(NA)
>>        x@y=as.numeric(NA)
>>        x@eta=as.numeric(NA)
>>        x@pWt=as.numeric(NA)
>>        x@Cx=as.numeric(NA)
>>        x@var=as.numeric(NA)
>>        x@sqrtXWt=matrix()
>>        x@sqrtrWt=as.numeric(NA)
>>        x@flist=data.frame()
>>        x
>> }
> 
> Rather than rewriting all of those slots I would just create a new
> object that had the slots that I wanted to save.  Modifying the slots
> in an object will probably result in something that does not satisfy
> validObject() so it will not be of much value to load the saved object
> as it will fail in most methods for the mer class.

Thanks! I wil do that instead...


>> Thanks!
>> Joscha
>> 
>> ps: From the description of this list I understand that it is for development related discussions. When I read threads about lme4 on the main R list, people always get referred to this list though. So feel free to refer me to another list.
>> 
>> _______________________________________________
>> R-sig-mixed-models@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> 


	[[alternative HTML version deleted]]


