[R-sig-ME] nlme4 vs. nlme question

[Douglas Bates]

On 6/19/07, Sofie Van Gijsel <Sofie.VanGijsel at arts.kuleuven.ac.be> wrote:

> >Dear List members,

> >I have a question regarding the difference between nlme & lme4 to
> >which I don't seem to find an answer in the previous posts about this topic.

> >I fitted a mixed-effects model with the different observations in my
> >dataset as random effect, in a by-subject analysis. Basically, I have
> >the following model:

I would say that the problem is using a model with random effects for
each observation.  The "per-observation" noise term already provides a
random effect for each observation.  For a linear mixed model or a
nonlinear mixed model this "residual variance" has a Gaussian
distribution, the same as the random effect.  Thus the random effect
variance is completely confounded with the residual variance.

In the nlme package there was no check on this situation and the
algorithm would apparently converge to an estimate of the variance
components.  However, this convergence would be spurious because any
combination of the residual variance and the random effect variance
that had the same sum would produce the same predictions and the same
value of the log-likelihood.

For the lme4 package I added a check on the number of levels of the
grouping factor for a random effects term and throw an error if that
equals the number of observations.  This is not declared to be an
error in a generalized linear mixed model with a binomial family.

Someone with more experience in mathematical statistics may be able to
tell us if this should or should not be an error for the Poisson
family.  It seems that this may not be necessarily be a case of
confounding of two variance components because the mean of the Poisson
determines the variance.

> >set_dimb.lmer_pois <- lmer(type_wf  ~dim + region + edu + sex +
> >(1|subcorp), family = "poisson", data = set_dimb)
> >
> >In this model, dim, region, edu & sex are all categorical variables.
> >subcorp = the subcorpora or observations in the dataset. Type_wf is
> >the "number of types per subcorpus".

I have created version 0.99875-2 of the lme4 package with this error
removed for the Poisson family.  I have tested the new version under
Linux.  Once it has been tested on Windows I will upload it to CRAN.
A preview version of the source package is at

http://www.stat.wisc.edu/~bates/lme4_0.99875-2.tar.gz

and it will be joined by the Windows binary package at

http://www.stat.wisc.edu/~bates/lme4_0.99875-2.zip

when the package has been compiled for Windows.  If you are running
under Windows you will need to download the .zip file and save it to
disk then select Packages->Install from local zip file  (or some name
like that, I don't use Windows so I am writing this from memory) to
install it.

> >With nlme, this worked fine, and plotting the ranefs gives insight in
> >which subcorpora behave in an anomolous way.
> >However, if I attempt to do the same with lme4, the error message
> >tells me that the model cannot fit:
> >
> >Error in lmerFactorList(formula, mf, fltype) :
> >          number of levels in grouping factor(s) 'subcorp' is too large
> >
> >I think the problem might be that for different combinations of the
> >factor levels, I have more than one sample, so for example for one
> >level of dim, region, education & sex, the dataset contains several
> >subcorpora. In fact, if I include the subcorpus types as random
> >effect (so not on the individual level of the subcorpora but on the
> >'higher' level of the different types of combinations of the
> >independent variables), the analysis does work and gives
> >interpretable results.
> >
> >So my question is: why does nlme allow this, but lme4 not? And if
> >lme4 does not allow this analysis, is there a theoretical reason,
> >viz. is it "wrong" to fit this type of by-subject analysis? Could this
> >indicate a problem with the sampling method (viz. with the dataset)?
> >
> >I hope this is clear (I am not exactly a statistician :-)),
> >Many thanks,
> >
> >Kind regards,
> >Sofie VG
>
>
> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>