[R-sig-ME] Model specification help

[Douglas Bates]

On 3/12/07, Andrew Perrin <clists at perrin.socsci.unc.edu> wrote:
> Update: I tried the same model on a 10-percent random sample and it
> completed fine:
>
> > grades.10pct.lmer<-lmer(grade.pt ~ (1|stud.id) + (1|instr.id) +
> (1|cour.dep),n
> ewgrades.10pct,control=list(gradient = FALSE, niterEM = 0, msVerbose = 1))
>    0      368526.: 0.114490 0.0870205 0.00154286
>    1      353650.: 0.125362 0.292639 0.980115
>    2      351398.: 0.222252 0.783158 0.978312
>    3      351303.: 0.186460 0.779040 0.977594
>    4      351290.: 0.151424 0.770734 0.976153
>    5      351274.: 0.168863 0.739213 0.975204
>    6      351273.: 0.161381 0.726354 0.973485
>    7      351273.: 0.166878 0.716955 0.963202
>    8      351269.: 0.167807 0.742046 0.904732
>    9      351269.: 0.162145 0.739476 0.903375
>   10      351268.: 0.165429 0.737807 0.898187
>   11      351238.: 0.165235 0.751137 0.505946
>   12      351233.: 0.158653 0.728281 0.461245
>   13      351223.: 0.162827 0.729269 0.360067
>   14      351221.: 0.165015 0.728947 0.324127
>   15      351219.: 0.158737 0.728858 0.252385
>   16      351218.: 0.169401 0.730054 0.229288
>   17      351217.: 0.164054 0.726687 0.217478
>   18      351217.: 0.163587 0.723005 0.242674
>   19      351217.: 0.164458 0.725164 0.235951
>   20      351217.: 0.164303 0.725227 0.234643
>   21      351217.: 0.164319 0.725163 0.234738
> > summary(grades.10pct.lmer)
> Linear mixed-effects model fit by REML
> Formula: grade.pt ~ (1 | stud.id) + (1 | instr.id) + (1 | cour.dep)
>     Data: newgrades.10pct
>      AIC    BIC  logLik MLdeviance REMLdeviance
>   351225 351265 -175608     351212       351217
> Random effects:
>   Groups   Name        Variance Std.Dev.
>   instr.id (Intercept) 0.067380 0.25958
>   stud.id  (Intercept) 0.297357 0.54530
>   cour.dep (Intercept) 0.096256 0.31025
>   Residual             0.410055 0.64036
> number of obs: 167654, groups: instr.id, 7198; stud.id, 5471; cour.dep, 97
>
> Fixed effects:
>              Estimate Std. Error t value
> (Intercept)  3.19796    0.03476   92.01

The fact that this fit converged in 21 iterations and the earlier fits
to the full data set ran out of memory after 20 iterations makes me
suspect that the memory problem is not in the convergence but in the
construction of the object to be returned.  Right now the only way to
cut down on the size of that object is to use the optional argument
frame = FALSE in the call to lmer or lmer2.

If you have a chance to do a traceback on a fit that runs out of
memory, it would be very valuable to know if the insufficient memory
condition occurred within the call to nlminb (lmer or old version of
lmer2) or .Call(mer2_optimize, mer, cv$msVerbose) (new version of
lmer2) versus within a call to new.  I suspect it is the latter.