Options like sparseX quickly get us into a flexibility versus efficiency
trade-off.  Representing a model as R structures and manipulating these
structures in R code such as is available in the Matrix package provides
flexibility but there is a cost.  Even moderately large data sets can
completely wedge a computer when you try to fit models to them, because of
the time that is spent allocating storage for intermediate results and then
freeing it.

Much of the model representation in lme4-1.0 is frozen at the level of the
R code and only manipulated in compiled code where the model structures are
represented as C++ types.  This allows for greater efficiency but at the
cost of using a strongly typed language, instead of a dynamic language.
 Unfortunately, allowing for an option like sparseX becomes much more
difficult in a strongly typed language.  I'm sure there are ways of using
C++ templates to allow for sparse or dense matrices in structures but that
type of C++ template metaprogramming is above my pay grade.

In short, I wouldn't hold my breath waiting for it.  Sorry I can't be more
encouraging as I remember that this facility is important to you.


On Mon, Oct 14, 2013 at 5:51 PM, Pete Meyer <petemeyer@google.com> wrote:

> For some of our large models (>= 5k factor levels) I found that sparseX in
> lme4a (under R 2.14.2) made a big difference in compute time.  I see that
> it isn't yet implemented for this update to lme4.  Any idea how long it
> will be until it is available again?
>
> --
> Pete
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]

