[R-sig-ME] When is sparseX implementation expected?

[Doran, Harold]

Curiously, though, how much of a need is there for sparseX? The model matrix for the random effects can often involve thousands of columns (sometimes even per level), and thousands of rows. Clearly, if the pattern of non-zeroes is large, there is a significant benefit.

But, the dimensions of the model matrix for the fixed effects are typically very small relative to the model matrix for the random effects, often involving only a handful of columns. 

One of the major benefits of the sparse storage for the random effect matrices are the methods that benefit, such as Cholesky(). In an iterative process, it only updates the numeric factorization at each iteration as it has already identified the symbolic representation in the first iteration. 

One thing I can imagine is if the number of rows of X is, say millions, and still involve only a handful of columns and suppose many of those rows are zeros. Then perhaps it could be useful? But even still when doing X'X on that matrix, its dimensions are (in theory) usually small and so doing things on that small p x p matrix is not too expensive.

On the other hand, if one has an X matrix with thousands of columns, I would question what the underlying statistical issue is and if the model is even good.

 

-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org [mailto:r-sig-mixed-models-bounces at r-project.org] On Behalf Of Ben Bolker
Sent: Tuesday, October 15, 2013 9:06 AM
To: r-sig-mixed-models at r-project.org
Subject: Re: [R-sig-ME] When is sparseX implementation expected?

Pete Meyer <petemeyer at ...> writes:

> 
> For some of our large models (>= 5k factor levels) I found that 
> sparseX in lme4a (under R 2.14.2) made a big difference in compute 
> time.  I see that it isn't yet implemented for this update to lme4.  
> Any idea how long it will be until it is available again?
> 

  It has not been near the top of the priority list.  We have looked at it and thought (more or less) "oh, that shouldn't be *too hard*, but it looks somewhat annoying" -- see https://github.com/lme4/lme4/issues/6
If you want to chime in on the issues list that might help push the priority up slightly.

  cheers
    Ben Bolker

_______________________________________________
R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models