[R-sig-ME] Sorry for the false alarm - results from glmer in the released lme4 are consistent with other software

[Douglas Bates]

On Fri, Mar 5, 2010 at 8:28 AM, Gabor Grothendieck
<ggrothendieck at gmail.com> wrote:
> In general S4 seems slow to load relative to S3.  You can generally
> tell whether a package is S3 or S4 just by the time it takes to load.
> However, if there is some special situation that makes S4 take even
> longer than usual it would be useful to know what it is.so others can
> avoid the problem in situations related or unrelated to lme4a.

There is a lot more involved in loading a package with S4 classes and
methods than in loading a package that uses S3 methods only.  S3
methods are lightweight - they only involve determining names of
functions, which is going to be done anyway.  S3 classes are even more
lightweight since they don't exist, in a formal sense.  Of course, the
downside of having everything in the S3 world so lightweight is that
S3 is limited (an S3 version of the Matrix package would be very
limited because there is no concept of multiple dispatch) and provides
no consistency checks.

>From the point of view of loading a package, S3 exists only in the
namespace.  S4 requires that tables of classes and methods be updated
and checked for consistency, which obviously takes longer.  I consider
it to be analogous to the difference between calling lm and calling
lm.fit directly.  Of course lm.fit is much faster but it requires the
user to construct the model frame and the model matrix themselves and
allows them the opportunity to make any number of subtle mistakes when
doing so.

The thing we found out was that a large portion of the time in loading
the lme4 package was being spent in the insertMethod function and
being driven by the fact that lme4 imported the entire Matrix
namespace.  The long-term fix will be to modify or perhaps even
eliminate insertMethod.  It is, for the most part, vestigial.  The
short term fix is to import only the classes and methods from Matrix
that are needed in lme4.  I'm not sure of the details but I think that
the problem originated in checking all the method tables imported from
Matrix and not modified in lme4.


> On Fri, Mar 5, 2010 at 9:05 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> In an earlier thread with the subject "lme4a, glmer and all that" I
>> stated that the results of fitting generalized linear mixed models in
>> the lme4 and lme4a packages were different and I was beginning to
>> doubt the results from the released version 0.999375-32 of lme4.  I
>> did discover a programming mistake in the development version, lme4a,
>> and am working on correcting it.  The results from the released
>> version, lme4, are consistent with those reported by other software.
>>
>> In an unrelated development we have discovered one of the reasons that
>> the lme4 package takes a very long time to load, relative to other
>> packages.  It has always seemed peculiar that the Matrix package with
>> dozens of S4 classes and hundreds of methods takes a couple of seconds
>> to load whereas the much simpler lme4 package (in terms of classes and
>> methods) takes 5 to 10 times as long.  We have isolated why this
>> occurs and John Chambers is testing a modification in the methods
>> package that can avoid this.
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>