[R-sig-ME] lmer fails when too many observations

[Asaf Weinstein]

Thank you very much Ben for the detailed explanation, this is so helpful.
I did set warn=2, as you mention. Anyhow, this is very useful information.

Asaf

On 11 March 2015 at 17:26, Ben Bolker <bbolker at gmail.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>   Just a quick update on this.
>   This 'failure' is coming because (1) the current version of lme4 has
> too strict a test for convergence and gives what we think are
> false-positive warnings, especially for large data sets; (2) you have
> set options(warn=2) so that warnings get converted into errors.  Until
> we get the issue fixed (which could take a while, as we haven't yet
> given up on finding a more principled way than just increasing the
> default tolerance a lot) you can either (a) convert your warnings back
> to regular warnings (i.e. options(warn=1) or options(warn=0) or (b)
> increase the tolerance level, e.g.
>
>  control= lmerControl(check.conv.grad = .makeCC("warning", tol = 1e-2))
>
> or
>
>  control = lmerControl(check.conv.grad = "ignore")
>
>
>
> On 15-03-09 09:52 PM, Asaf Weinstein wrote:
> > Dear lmer community,
> >
> > I am trying to run a simulation for a two-way random-effects model
> > with unbalanced design (ie, unequal number of observations per
> > cell) and no interaction. It's especially important for me to be
> > able to run the lmer/blmer functions when the number of (column and
> > row) random effects is large, say 100, and with possible replicates
> > in each cell. The problem is that lmer() works with the full vector
> > of observations, as opposed to working with the cell averages
> > (which is a sufficient statistic), and the methods fails pretty
> > quickly when there are replicates (because the response vector is
> > too big, I suppose). I get the following error:
> >
> > *Error in get("checkConv", lme4Env)(attr(opt, "derivs"), opt$par,
> > ctrl = control$checkConv,  : * *  (converted from warning) Model
> > failed to converge with max|grad| = 0.00244385 (tol = 0.002)*
> >
> > Just to give an example: suppose there are R=100 row effects, C=100
> > column effects, and 5 replicates in each cell. The vector of
> > individual observations is of length 100^5 (lmer fails), while the
> > vector of cell averages is of length 100^2 (a size which causes no
> > problem for lmer). My question is whether there is a way to tell
> > lmer() to work with the sufficient statistic (of course, the
> > conditional covariance is no longer c*Identity, a fact which is
> > used in the implementation of lmer (according to documentation) ).
> >
> > Thank you very much and I hope I was clear!
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
>
> iQEcBAEBAgAGBQJVALMaAAoJEOCV5YRblxUHnRwH/j8/MFNfYKgvgzbBBCdmfXOi
> nfRGRpGWrFUM7MznFWFf4hdA0SFPUOTS9yZYxoGuyJHos1q5WeAyR9kdlqNQjdDx
> lPV+zYy9gsj+uMl5XBVSurDrfJY+JXfnqtOzZECMbrZjNxHgTMnSxzMdRNVZAVm6
> DaCAaNGJkeTI96k4jg7ucj1p/xGgk3deVW8NMY3bKMasd+dtFzTXqPDvEYovBRQT
> sbQ7SavzjJY93QYHabaaDkpsD8N86qe5fJgOE9uVLNkvuOGVPELvwG8lCaC/kj7y
> wdd4+HOum1AgeBUosCRfs+Ld41V4wBUySxFW9coLRUVwzQGfU8rOTjIpKs4iCpY=
> =jitR
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]