[R] lme4: apparently different results between 0.8-2 and 0.95-6

[Douglas Bates]

Steve Cumming wrote:
> I've been using lme4 to fit Poisson GLMMs with crossed random effects. The
> data are counts(y) sampled at 55 sites over 4 (n=12) or 5 (n=43) years. Most
> models use three fixed effects: x1 is a two level factor; x2 and x3 are
> continuous. We are including random intercepts for YEAR and SITE. On
> subject-matter considerations, we are also including a random coefficient
> for x3 within YEAR.
> 
> Neglecting the log link, the model is
> 
> 	y_{i,j} = x'_i \beta + \eta_i + z'_i \phi_j + \epsilon_{i,j}
> 
> where
> 	i indexes SITE and j indexes YEAR,
>       \beta is the vector of fixed effects
> 	\eta_i in the random intercept for SITE
> and
> 	\phi_j are the random intercept and coefficient for YEAR.
> 
> I have written x'_i because the covariates are assumed (reasonably) to be
> constant over the 5 years. Thus, obviously, the z'_i = (1, x3_i) are
> constant over j as well.
> 
> 
> Using lme4 0.8-2 and R 1.9.0 (under Windows), the call
> 
> 	GLMM(y~x1 + x2 + x3,random = list(YEAR=~1+x3, SITE=~1), data=foo,
> family=poisson, offset=log(reps))
> 
> seemed to work correctly, so far as I can tell. The fixed effects were
> more-or-less consistent with those estimated by an ordinary GLM, and the
> random YEAR effects had signs, magnitudes and correlation appeared to be
> sensible and consistent with my expectations.
> 
> Earlier today, we updated to lme4 0.95-6 and R 2.1.0. When we try to use
> lmer to fit the same model, it complains bitterly:
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR) + (1 | SITE), data=foo,
> family=poisson, offset=log(reps))
> 	Error: Unable to invert singular factor of downdated X'X
> 
> Simpler models still work (or at least return):
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR), ...)
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 | YEAR) + (1 | SITE), ...)
> 
> As I mentioned, the design is unbalanced. But, we get same "invert singular
> factor" Error using the balanced subset.
> 
> Can anybody advise? Are we using lmer incorrectly? Or is the new error
> perhaps telling us that GLMM in 0.8-2 wasn't actually working in some sense?

I think you are using lmer correctly and that deep in the code there is 
a glitch related to exactly the circumstances you outlined - a model 
matrix with more than one column for one grouping factor (i.e. 
(x3|YEAR)) and a model matrix with a different number of columns for 
another grouping factor (1|SITE)).  Because, as you mentioned, YEAR and 
SITE are crossed, the internal representation gets more complicated than 
  it would be if you had nested grouping factors.

The fact that you can fit the simpler models (thanks for checking that) 
is a strong indication where the problem is.

Is it possible for you to share the data with Deepayan Sarkar and me? 
If so it would help us track down the bug.  Please contact us off list 
if you can do so.  In the meantime I suggest using GLMM for this 
particular problem if you still have access to the old copy of the lme4 
package.  If not, contact us off list and we will provide you with a 
replacement package that can be used in conjunction with the current lme4.

This is an exemplary bug report.  You did a great job of isolating the 
problem.