[R-sig-ME] Upcoming changes in lme4

[Douglas Bates]

On Tue, Feb 7, 2012 at 11:11 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
> We (the lme4 authors) have mentioned on this list that we are
> preparing a new version of lme4 that will provide enhanced
> capabilities.  That's the good news.  The bad news is that the
> internal representation of the model has been changed yet again.  If
> your use of the lme4 package is through the exported functions only
> you should be okay.  However, if you find problems then please report
> them to us either in email <lme4-authors at r-forge.wu-wien.ac.at> or on
> the bug-tracker at R-forge,
>
> https://r-forge.r-project.org/tracker/?func=add&group_id=60&atid=298
>
> One major, and unfortunately inevitable, problem that will affect many
> users is the inability of the new code to load saved objects created
> with the old code.  Unfortunately that is what happens when you change
> the class representation.  If possible it is best to save the code and
> data that generated the fitted model and re-fit after the change.
>
> If your usage involved access to components or slots in the object
> then there will be changes.  To ease the transition, Martin created a
> function called getME that is available in the current lme4 and in the
> new lme4, available as lme4Eigen on R-forge.  This function takes a
> fitted model and a character variable naming a component and returns
> the desired component.  For example, if you want the fixed-effects
> model matrix from fitted model fm1 then use
>
> getME(fm1, "X")
>
> Similarly for the random-effects model matrix, "Z", or its transpose,
> "Zt", the sparse Cholesky factor, "L" and many others.  Use
>
> library(lme4)
> example(getME)
>
> to get some examples.
>
> The new lme4 will provide
>
>  - profiling of the deviance function with respect to the parameters
>   in a linear mixed model (and soon generalized linear mixed models).
>   The profiles allow for creation of realistic confidence intervals
>   on the parameters and for various plots that show the sensitivity
>   of the deviance to the values of the parameters.
>
>  - a more reliable and flexible implementation of generalized linear
>   mixed models, including the use of adaptive Gauss-Hermite
>   quadrature for evaluating an approximation to the deviance.
>
>  - short-cut functions such as refitML and refit to re-evaluate a
>   model under the maximum likelihood criterion or with a new response
>   vector.
>
>  - a cleaner internal representations that provides, in most cases,
>   faster and more reliable model fits.
>
>  - choice of optimizer when estimating the parameters.  Current
>   choices are "NelderMead" and "bobyqa".  Both are generally faster
>   and more reliable than the optimizer used in the current lme4,
>   which is based on the code in R's nlminb() function, the one that
>   gives those annoying "false convergence" messages.
>
>  - smaller memory footprint.  The need for copying large objects is
>   reduced through the use of reference classes in R.  In the past
>   there were circumstances where it was possible to fit a model to a
>   large data set but not to print or show the results because a new
>   copy of the entire object was created during the process of
>   creating a summary.  This no longer occurs.
>
>  - nlmer has been improved and will, by the time of release, allow
>   adaptive Gauss-Hermite quadrature.
>
> An alpha-test release will be made available on the R-forge archive
> but *not* uploaded to CRAN.  If you use lme4 extensively please test
> this release by installing from the R-forge archive and telling us if
> there are problems with your code.  This way you can still back out to
> the current release by removing the lme4 package and reinstalling from
> CRAN.  Once the new lme4 has been released to CRAN it will be much
> more difficult to back out that installation.
>
> Authors of packages that depend on lme4 will get a separate message
> off-list about testing and suggested modifications.
>
> Thanks for your cooperation.  We do honestly believe that this change
> will provide an improved capability for fitting and analyzing
> mixed-effects models.

I forgot to mention an important point for Mac OS X users.  The new
lme4, which is based on the Eigen linear algebra library, requires the
RcppEigen package for which there is no binary Mac OS X package on
CRAN.  It's a ridiculous situation that comes about because Apple
ships an antique version of gcc in their XCode development
environment, and won't upgrade because of licensing disagreements.
Even that very old version of g++ compiles the package successfully
for Intel Macs but it croaks when trying to cross-compile for the ppc
architecture.  You can see at
http://www.r-project.org/nosvn/R.check/r-release-macosx-ix86/RcppEigen-00install.html
that the advice it to inform Apple of the compiler error but they
ignore such reports.  Things may improve when XCode 4 is in use
because it provides clang++ in addition to the same very old version
of g++.  In the meantime we will provide a binary i386 Mac OS X
package for RcppEigen.