[R-sig-ME] Upcoming changes in lme4

[Douglas Bates]

We (the lme4 authors) have mentioned on this list that we are
preparing a new version of lme4 that will provide enhanced
capabilities.  That's the good news.  The bad news is that the
internal representation of the model has been changed yet again.  If
your use of the lme4 package is through the exported functions only
you should be okay.  However, if you find problems then please report
them to us either in email <lme4-authors at r-forge.wu-wien.ac.at> or on
the bug-tracker at R-forge,

https://r-forge.r-project.org/tracker/?func=add&group_id=60&atid=298

One major, and unfortunately inevitable, problem that will affect many
users is the inability of the new code to load saved objects created
with the old code.  Unfortunately that is what happens when you change
the class representation.  If possible it is best to save the code and
data that generated the fitted model and re-fit after the change.

If your usage involved access to components or slots in the object
then there will be changes.  To ease the transition, Martin created a
function called getME that is available in the current lme4 and in the
new lme4, available as lme4Eigen on R-forge.  This function takes a
fitted model and a character variable naming a component and returns
the desired component.  For example, if you want the fixed-effects
model matrix from fitted model fm1 then use

getME(fm1, "X")

Similarly for the random-effects model matrix, "Z", or its transpose,
"Zt", the sparse Cholesky factor, "L" and many others.  Use

library(lme4)
example(getME)

to get some examples.

The new lme4 will provide

 - profiling of the deviance function with respect to the parameters
   in a linear mixed model (and soon generalized linear mixed models).
   The profiles allow for creation of realistic confidence intervals
   on the parameters and for various plots that show the sensitivity
   of the deviance to the values of the parameters.

 - a more reliable and flexible implementation of generalized linear
   mixed models, including the use of adaptive Gauss-Hermite
   quadrature for evaluating an approximation to the deviance.

 - short-cut functions such as refitML and refit to re-evaluate a
   model under the maximum likelihood criterion or with a new response
   vector.

 - a cleaner internal representations that provides, in most cases,
   faster and more reliable model fits.

 - choice of optimizer when estimating the parameters.  Current
   choices are "NelderMead" and "bobyqa".  Both are generally faster
   and more reliable than the optimizer used in the current lme4,
   which is based on the code in R's nlminb() function, the one that
   gives those annoying "false convergence" messages.

 - smaller memory footprint.  The need for copying large objects is
   reduced through the use of reference classes in R.  In the past
   there were circumstances where it was possible to fit a model to a
   large data set but not to print or show the results because a new
   copy of the entire object was created during the process of
   creating a summary.  This no longer occurs.

 - nlmer has been improved and will, by the time of release, allow
   adaptive Gauss-Hermite quadrature.

An alpha-test release will be made available on the R-forge archive
but *not* uploaded to CRAN.  If you use lme4 extensively please test
this release by installing from the R-forge archive and telling us if
there are problems with your code.  This way you can still back out to
the current release by removing the lme4 package and reinstalling from
CRAN.  Once the new lme4 has been released to CRAN it will be much
more difficult to back out that installation.

Authors of packages that depend on lme4 will get a separate message
off-list about testing and suggested modifications.

Thanks for your cooperation.  We do honestly believe that this change
will provide an improved capability for fitting and analyzing
mixed-effects models.