[R-sig-ME] Crossed random effects

[Kevin Wright]

Thanks for the clarification.  It is no secret that large
plant-breeding programs (both corporate and governmental--see
http://www.dpi.nsw.gov.au/__data/assets/pdf_file/113474/annual_report_part_3.pdf)
have adopted ASREML, probably due to the "war stories" with crossed
random effects that you mention.  I have heard several people say that
ASREML is often orders of magnitude (100-1000) times better than SAS
for handling large datasets with crossed random effects.  My limited
experience suggests ASREML/Genstat/SAMM and lme4 are in the same
order-of-magnitude performance-wise.

P.S.  I offer sincere appreciation for the "Mixed-effects modeling
with crossed random effects for subjects and items" paper,
particularly the MCMC approach and the corresponding interpretations
and discussions.  Very nice.

K Wright


On 3/13/07, Douglas Bates <bates at stat.wisc.edu> wrote:
> On 3/13/07, Kevin Wright <kw.statr at gmail.com> wrote:
> > I am confused by some apparent contradictions about fitting crossed
> > random effects in software.  Consider this quote from
> > http://www.mpi.nl/world/persons/private/baayen/publications/baayenDavidsonBates.pdf
> > "To our knowledge, the only software currently available for fitting
> > mixed-effects models with crossed random effects is the lme4 package"
>
> That statement should have been more carefully worded.  It is in
> reference to the types of experimental situations described in that
> paper where random effects are associated with subject and item,
> subjects are crossed with item and the numbers of both the subjects
> and the items can be very large.
>
> > Yet, nlme and GLIMMIX appear to claim that crossed-random effects can
> > be fit by those respective tools:
> >
> > In Mixed Effects Models in S and S-Plus:
> > "The crossed random-effects structure is represented in lme by a
> > combination of pdBlocke3d and pdIdent objects" (page 163)
>
> It is possible to fit a model with crossed random effects with lme
> provided that the number of levels of both of the crossed factors is
> small.  Otherwise you end up with huge, sparse model matrices that are
> being treated as dense matrices and you quickly run out of memory or
> time or both.
>
> Really, doesn't a random effects specification like
> pdBlocked(list(pdIdent(~ rows - 1), pdIdent(~ columns - 1))) smell
> like a kludge to you?
>
> > http://support.sas.com/rnd/app/papers/glimmix.pdf
> > "The GLIMMIX procedure, on the other hand, determines by default the
> > marginal log likelihood as that of an approximate linear mixed model.
> > This allows multiple random effects, nested and crossed random
> > effects, multiple cluster types, and R-side random components."  [and]
> >  "Example 2. Mating Experiment with Crossed Random Effects"
>
> I think that several readers of this list could tell you war stories
> of trying to fit models with crossed random effects using SAS PROC
> MIXED or SAS PROC NLMIXED versus fitting the same model in lmer or
> lmer2.  You are correct that one can specify a model with crossed
> random effects in SAS PROC MIXED and that we overstated the uniqueness
> of the capabilities of lmer to fit such models.  However, if you want
> to try to fit such a model in SAS PROC MIXED when you have large
> numbers of subjects and large numbers of items you had better be
> prepared to wait for a long time.
>
>
> > Are these three quotes using different definitions of "crossed random
> > effects"?  Have I taken the quotes out of context?  Any clarifications
> > would be appreciated.
> >
> > Thanks,
> >
> > K Wright
> >
> > _______________________________________________
> > R-sig-mixed-models at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>