[R-sig-ME] Summarizing the fitted model takes more RAM than fitting it

[Douglas Bates]

The summary object contains a copy of the original fitted model.  At
the time of return it must be an actual copy.  My guess is that it is
the need to create two copies of the fitted model object that is
causing the memory overflow.  You may want to set a trace on the
summary method for signature(object = "mer") and step through the
evaluation to see where the problem arises.  My guess is that it will
be at the point where the new "summary.mer" object is created.

The need to create copies of very large objects is a problem that I
have wrestled with in the design of the lme4 package.  To maintain the
functional language semantics of R  you cannot modify an argument to a
function, even a replacement function.  If your model-fitting function
is based on iterative optimization and the objects involved are very
large then the need to create copies often is a problem.  The current
version of lme4 gets around this problem by cheating and modifying the
arguments to certain .Call functions.  The development version (in the
allcoef branch of the SVN tree for lme4 at R-forge) is cleaner in that
it uses an environment to hold the objects representing the model
during the optimization.  Environments are special in that they are
never copied so they provide a legitimate way of implementing call by
reference semantics.

On Sat, Dec 13, 2008 at 8:13 AM, Gorjanc Gregor
<Gregor.Gorjanc at bfro.uni-lj.si> wrote:
> Dear lme4 developers,
>
> I am a big fan/user of your lme4 package. Thank you very much for this package and
> other R related work. I would like to report my experience that summarizing the fitted
> model via summary() takes more RAM than fitting it with lmer(). I noticed this several
> times. The model is quite large, but not that large. Am I doing something wrong or ...?
>
> Bellow is an example of the model I am fitting along with the "necessary" code
> to summarize the fitted model. I am currently using R 2.8.0 on MS Windows Vista on
> Dell Inspiron I1520 laptop having Intel(R) Core(TM) Duo CPU T7500 @ 2.20GHz 2.20GHz
> and 2046 MB of RAM. Version of lme4 is 0.999375-27 as of date 2008-09-27.
>
>> ## Fit the model
>> fit9b <- lmer(tezaroj ~ pasma + roj2 + zj2 + spol + reja + jagLM +
>>                                      (1 | rejec) + (1 | hy) + (1 | hys), data=podatki)
>>
>> summary(fit9b)
> Error: cannot allocate vector of size 130.4 Mb
> In addition: There were 22 warnings (use warnings() to see them)
>> warnings()
> Warning messages:
> 1: In slot(from, what) <- slot(value, what) ... :
>  Reached total allocation of 1535Mb: see help(memory.size)
>>
>> ## Make sure R can allocate enough memory
>> memory.limit(size=1900)
>>
>> ## Sink the output - since correlation matrix of fixed effects hides
>> sink(file="tmp.txt")
>> summary(fit9b)
>> sink()
>
> ## Linear mixed model fit by REML
> ## Formula: tezaroj ~ pasma + roj2 + zj2 + spol + reja + jagLM + (1 | rejec) +      (1 | hy) + (1 | hys)
> ##    Data: podatki
> ##     AIC    BIC logLik deviance REMLdev
> ##  150646 152563 -75118   149507  150236
> ## Random effects:
> ##  Groups   Name        Variance Std.Dev.
> ##  hys      (Intercept) 0.057519 0.23983
> ##  hy       (Intercept) 0.135671 0.36834
> ##  rejec    (Intercept) 0.268200 0.51788
> ##  Residual             0.297427 0.54537
> ## Number of obs: 85035, groups: hys, 9653; hy, 2036; rejec, 346
> ...
>
> I also tried using the following things before calling summary():
> - gc() and summary();
>
> - removing all unnecessary objects in the workspace, gc() and summary();
>
> - saving the object, starting new instance of R, loading the object and summary();
>
> but could still not summarize the fitted object. I am really suprised that summarising the
> fitted model consumes more memory than fittng it. Am I missing something?
>
> Lep pozdrav / With regards,
>    Gregor Gorjanc
> ----------------------------------------------------------------------
> University of Ljubljana     PhD student
> Biotechnical Faculty        www: http://gregor.gorjanc.googlepages.com
> Animal Science Department   blog: http://ggorjan.blogspot.com
> Groblje 3                   mail: gregor.gorjanc <at> bfro.uni-lj.si
> SI-1230 Domzale             fax: +386 (0)1 72 17 888
> Slovenia, Europe            tel: +386 (0)1 72 17 861
> ----------------------------------------------------------------------
>