Hi all,

I am currently trying to fit lmer models in the following setting:

> mem.limits()
nsize vsize
  NA    NA

> sessionInfo()
R version 2.9.0 (2009-04-17)
x86_64-unknown-linux-gnu

locale:
LC_CTYPE=en_US;LC_NUMERIC=C;
LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=en_US;LC_PAPER=en_US;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] lme4_0.999375-32   Matrix_0.999375-26 lattice_0.17-22

loaded via a namespace (and not attached):
[1] grid_2.9.0

I actually have two machines I'm working on, a small one with 4G
physical memory (10G swap) and a larger one with 8G physical memory
(6G swap).

Here's some "top" output from the small machine
Mem:   3985760k total,  3540380k used,   445380k free,    18508k buffers
Swap: 10634988k total,   345168k used, 10289820k free,   259944k cached

And here's some from the large machine:
Mem:   8223208k total,  2491756k used,  5731452k free,   344328k buffers
Swap:  6024332k total,     9252k used,  6015080k free,  1626640k cached

The dataset "frm" is under 5M bytes and has less than 22k rows.  I am
fitting a simple model of the form:

mod <- lmer(y ~ factorA + (1 | factorB), data=frm)

What makes this a memory hog is that I factorA has 4000 levels (and
yes - I do care about the specific levels, and no - they aren't
normally distributed).  FactorB has only about 400 levels, so that is
not really a problem.  After fitting the model I am interested in
calculating things like t(vec) %*% fixef(mod), where vec is a 1 column
matrix and its variance (using t(vec) %*% vcov(mod) %*% vec).  For
example, if there are N levels for factorA, vec could be rep(1/N,
times=N), so that I get back the simple average of the levels with its
SE.

If I check object size in R (assuming that it actually fits the
model), it says it is only about 860M bytes, which would be fine.  The
model completes (if successful) in about 3 hours - but I wouldn't be
writing this email if things worked smoothly.

Models crash more often than they are successfully fit, always with
the dreaded "Error: cannot allocate vector of size ..." message.  I've
tried model=FALSE and x=FALSE to save memory space, but that saves
only about 100M bytes.  I get rid of all unnecessary objects in memory
before trying to fit a model (and any unnecessary libraries) and then
immediately save a model if it is fit successfully.  I have been able
to fit a model, calculate the vcov and save both - a couple of times.
However, when I try to read the model or vcov matrix, I often get an
"Error: vector memory exhausted (limit reached?)" message from load.
I'm wondering if the retrieval approach load uses is somehow reading
things first into a vector for processing and that vector exceeds a
2*10^9 limit in vector length (
http://www.iiap.res.in/astrostat/School07/R/html/base/html/Memory-limits.html
makes this assertion).  Or is load is making multiple copies in the loading
process resulting in a crash at some point?

Here is the last gasp from an attempt to calculate the vcov after
successfully fitting a model:
>   fixefvar <- try(vcov(lme.1))
Garbage collection 281 = 115+54+112 (level 2) ...
39.1 Mbytes of cons cells used (57%)
2206.1 Mbytes of vectors used (73%)
-Garbage collection 282 = 116+54+112 (level 0) ...
39.2 Mbytes of cons cells used (57%)
2648.1 Mbytes of vectors used (88%)
Garbage collection 283 = 116+55+112 (level 1) ...
39.2 Mbytes of cons cells used (57%)
2648.1 Mbytes of vectors used (88%)
Garbage collection 284 = 116+55+113 (level 2) ...
39.2 Mbytes of cons cells used (57%)
2316.6 Mbytes of vectors used (73%)
Garbage collection 285 = 117+55+113 (level 0) ...
39.2 Mbytes of cons cells used (57%)
2648.2 Mbytes of vectors used (84%)
Garbage collection 286 = 117+56+113 (level 1) ...
39.2 Mbytes of cons cells used (57%)
2537.7 Mbytes of vectors used (80%)
Garbage collection 287 = 117+56+114 (level 2) ...
39.2 Mbytes of cons cells used (57%)
2648.2 Mbytes of vectors used (80%)
Garbage collection 288 = 118+56+114 (level 0) ...
39.2 Mbytes of cons cells used (57%)
3200.8 Mbytes of vectors used (96%)
Garbage collection 289 = 118+57+114 (level 1) ...
39.2 Mbytes of cons cells used (57%)
3090.3 Mbytes of vectors used (93%)
Garbage collection 290 = 118+57+115 (level 2) ...
39.2 Mbytes of cons cells used (57%)
2869.3 Mbytes of vectors used (82%)
Garbage collection 291 = 118+57+116 (level 2) ...
39.2 Mbytes of cons cells used (57%)
3090.3 Mbytes of vectors used (84%)
Garbage collection 292 = 118+57+117 (level 2) ...
39.2 Mbytes of cons cells used (57%)
3090.3 Mbytes of vectors used (80%)
Garbage collection 293 = 118+57+118 (level 2) ...
39.2 Mbytes of cons cells used (57%)
3090.3 Mbytes of vectors used (77%)
Garbage collection 294 = 118+57+119 (level 2) ...
39.2 Mbytes of cons cells used (57%)
3090.3 Mbytes of vectors used (73%)
Garbage collection 295 = 118+57+120 (level 2) ...
39.2 Mbytes of cons cells used (57%)
3311.4 Mbytes of vectors used (74%)
Error : cannot allocate vector of size 110.5 Mb

So in this case it looks like it failed about the time it would have
exhausted the physical memory - which begs the question why it wasn't
able to access some of the swap memory under Ubuntu.  I get
the same kind of behavior on either machine.

So I'm open for suggestions :-)

*Suggestions for how to fit more compact lmer models (ie is it
possible to save fewer attributes/slots but still be able to calculate
fixef and vcov?)

*Are there any load tricks (like setting "environment" to a particular
value that prevents R from making some of its copies when loading)
that would allow the loading of a model with less overhead?

There is no guarranteed structure among the fixed and random
effects that would allow the problem to be separated into smaller
subproblems.

If no one has any suggestions, I can try to construct a toy dataset
that duplicates the effect.

Many thanks in advance - Pete.

	[[alternative HTML version deleted]]


