[R-sig-ME] problems with allocate memory

[Douglas Bates]

On Mon, Dec 19, 2011 at 5:54 PM,  <cumuluss at web.de> wrote:
> Hi Douglas,
>
> thanky you for your reply. This is "mydata"
>
> 'data.frame':   3909896 obs. of  19 variables:
>  $ gr.b            : int  0 0 0 0 0 0 0 0 0 0 ...
>  $ o.ag      : num  -0.651 -0.651 -0.651 -0.651 -0.651 ...
>  $ o.rar      : num  -0.935 -0.935 -0.935 -0.935 -0.935 ...
>  $ si       : num  0.299 0.299 0.299 0.299 0.299 ...
>  $ f.ag       : num  -1.25 -1.36 -1.33 -1.26 -1.21 ...
>  $ f.se       : Factor w/ 2 levels "F","M": 1 2 1 2 2 2 1 1 1 1 ...
>  $ o.se       : Factor w/ 2 levels "F","M": 1 1 1 1 1 1 1 1 1 1 ...
>  $ diff        : num  -0.536 -0.514 -0.521 -0.534 -0.545 ...
>  $ exp.r           : num  -0.168 -0.168 -0.163 -0.168 -0.168 ...
>  $ f.rar      : num  -0.911 0.215 1.224 -1.086 1.107 ...
>  $ f.si: num  1.0008 1.1583 0.0561 -0.4163 0.371 ...
>  $ kl              : Factor w/ 3 levels "mat","nonkin",..: 1 2 2 2 3 2 1 2 2 2 ...
>  $ sn          : Factor w/ 2 levels "BS","MS": 1 1 1 1 1 1 1 1 1 1 ...
>  $ MP_y_n          : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
>  $ ratio       : num  -0.0506 -0.0506 -0.0506 -0.0506 -0.0506 ...
>  $ f           : Factor w/ 55 levels "0A0","0A1","0A2",..: 1 6 7 8 9 10 11 13 15 16 ...
>  $ o           : Factor w/ 552 levels "","00T","00Z",..: 2 2 2 2 2 2 2 2 2 2 ...
>  $ d            : int  9099 9099 9099 9099 9099 9099 9099 9099 9099 9099 ...
>  $ MP              : num  6 6 6 6 5 4 6 6 6 6 ...
>
>
> formula for the model:
> mfit=lmer(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se * (o.se + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) + (1|f)+(1|o)+(1|d) + offset(log(MP)), data=c.data, family=binomial)

> I hope this is what you want to see. Thank you for your help.

My guess is that the problem is with creating the fixed-effects model
matrix, of which there could be several copies created during the
evaluation and optimization of the deviance.

Just as a test, could you fit the model for the fixed-effects only
using glm and check on what the size of the model matrix is?
Something like

glm1 <- glm(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
(o.se + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
offset(log(MP)), data=c.data, family=binomial)
object.size(model.matrix(glm1)

Also, could you convert 'd' to a factor and run str again so we can
learn how many levels there are?  Either that or send the result of

length(unique(mydata$d))


> -------- Original-Nachricht --------
>> Datum: Mon, 19 Dec 2011 14:50:06 -0600
>> Von: Douglas Bates <bates at stat.wisc.edu>
>> An: cumuluss at web.de
>> CC: r-sig-mixed-models at r-project.org
>> Betreff: Re: [R-sig-ME] problems with allocate memory
>
>> On Sun, Dec 18, 2011 at 3:17 PM,  <cumuluss at web.de> wrote:
>> > Hi to everyone,
>>
>> > I have been trying to fit a glmm with a binomial error structure. My
>> model is a little bit complex. I have 8 continuous predictor variables one of
>> them as nonlinear term, 5 categorical predictor variables with some
>> three-way interactions between them. Additional I have 3 random effects and one
>> offset variable in the model. Number of obs is greater than 3million.
>> > I’m working with the latest version of R 2.14.0 on a 64 bit windows
>> system with 8Gb ram.
>> > Everything I tried (reducing model complexity, different 64bit PC with
>> even more memory) nothing leads to a fitted model, always the Error occurs:
>> cannot allocate vector of size 2GB.
>> > Is there anything I can do? I would be very grateful for any commentary.
>>
>> You probably have multiple copies of some large objects hanging
>> around.  Can you send us the output of
>>
>> str(myData)
>>
>> where 'myData' is the name of the model frame containing the data you
>> are using and the formula for the model you are trying to fit?
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models