[R-sig-ME] Alpha test version of the latest (and last - trust me on this) reformulation of lmer and friends

[Douglas Bates]

At one point in the movie "The Big Chill" Nick (William Hurt) is
talking with Chloe (Meg Tilly) and says, "Well, yes, I do have a small
but deeply disturbed following."

So I would like to announce to the small but deeply disturbed
following of the lme4 package that the version of the package based on
the most recent redesign of the computational methods (and this is the
last time I'm going to tear it apart and rebuild it - I promise -
really) is ready for a few brave testers.  Currently lmer works but
glmer and nlmer need a bit more tuning up.  Also mcmcsamp doesn't work
at present in this version.

I would very much appreciate those who have written papers or books
that quote results from earlier versions of lmer checking those
results in the new lmer.  The answers from the new lmer should be at
least as good (in the sense of returning parameter estimates with a
log-likelihood or log-restricted-likelihood at least as large) as
those from earlier versions.  If the results are not as good as the
previous results, please let me know so we can work out what the
problem is.

If you can install source packages, the best way to get alpha test
releases of lme4 is from the SVN repository

 https://svn.r-project.org/R-packages/branches/gappy-lmer

using whatever svn client is handy for you.

A source tarball and a Windows binary package are also available as

 http://www.stat.wisc.edu/~bates/lme4_0.999375-0.tar.gz

and

 http://www.stat.wisc.edu/~bates/lme4_0.999375-0.zip

respectively.  The Windows binary package is prepared on Uwe Ligges'
wonderful win-builder facility at win-builder.R-project.org

I have again observed peculiar timing results when using accelerated
BLAS with lmer.  I haven't done intensive profiling of the C code to
determine exactly what is happening but the bottom line is that on my
machine (Athlon-64 dual-core, Ubuntu 7.10 "gutsy" devel) using the
reference BLAS compiled with gfortran is faster than using the AMD
Core Mathematics Library (ACML) "accelerated" BLAS, even the single
threaded version.  My typical test is shown in the enclosed
transcript.  The timing results there, about 55 - 60 seconds user
time, are with R's reference BLAS.  If I switch to single-threaded
ACML the time jumps to about 2 minutes.  With multithreaded BLAS it
takes about 4 minutes.

The computational methods are derived and described in the vignette
"Computational methods for mixed models" included with the package.
The "Implementation" vignette is in transition between a description
of earlier and current versions so don't believe everything in there.

I am also quite interested in timings.  Although I consider
robustness, reliability and accuracy to be more important than speed,
I would like to know how the speed of this version compares to earlier
versions on models fit to large data sets.