[R-sig-ME] problems with allocate memory

[cumuluss at web.de]

Hi Douglas,

The variable‘d’ has about 710 levels.

For your other request I tried to fit the suggested model but it was not possible. I tried it with different approaches, first without any interactions and non nonlinear term. It fitted. The object size was about 731571664 bytes. Then I successive made the model more complex. With one two way interaction, with one three way interaction or with the nonlinear term it was slightly the same as before. With five two way interaction always with the nonlinear term the object size went up to 1075643424 bytes. With one additional two way interaction the model won’t fit anymore with the known error.

Perhaps another hint: Yesterday I attempted to fit a much simpler model with lmer, just to see if this works. (mfit=lmer(gr.b ~ f.ag + f.se + o.se + diff + exp.r + kl + (1|f)+(1|o), data=c.data, family=binomial)). It fitted but I could not open mfit. Trying to see only the coefficients also did not work. I saved the image and this one is unfamiliar huge about 1.7 GB.

By the way: After reloading the image some interesting things happened. An error occurred: slot coefs are not an S4 object. It seems to me that it is not possible to save the model results in an R image. Is that right?

I hope this will make your guess clearer. And again thank you very much for help.
Regards
Paul


-------- Original-Nachricht --------
> Datum: Tue, 20 Dec 2011 09:23:37 -0600
> Von: Douglas Bates <bates at stat.wisc.edu>
> An: cumuluss at web.de
> CC: r-sig-mixed-models at r-project.org
> Betreff: Re: [R-sig-ME] problems with allocate memory

> On Mon, Dec 19, 2011 at 5:54 PM,  <cumuluss at web.de> wrote:
> > Hi Douglas,
> >
> > thanky you for your reply. This is "mydata"
> >
> > 'data.frame':   3909896 obs. of  19 variables:
> >  $ gr.b            : int  0 0 0 0 0 0 0 0 0 0 ...
> >  $ o.ag      : num  -0.651 -0.651 -0.651 -0.651 -0.651 ...
> >  $ o.rar      : num  -0.935 -0.935 -0.935 -0.935 -0.935 ...
> >  $ si       : num  0.299 0.299 0.299 0.299 0.299 ...
> >  $ f.ag       : num  -1.25 -1.36 -1.33 -1.26 -1.21 ...
> >  $ f.se       : Factor w/ 2 levels "F","M": 1 2 1 2 2 2 1 1 1 1
> ...
> >  $ o.se       : Factor w/ 2 levels "F","M": 1 1 1 1 1 1 1 1 1 1
> ...
> >  $ diff        : num  -0.536 -0.514 -0.521 -0.534 -0.545 ...
> >  $ exp.r           : num  -0.168 -0.168 -0.163 -0.168 -0.168
> ...
> >  $ f.rar      : num  -0.911 0.215 1.224 -1.086 1.107 ...
> >  $ f.si: num  1.0008 1.1583 0.0561 -0.4163 0.371 ...
> >  $ kl              : Factor w/ 3 levels "mat","nonkin",..:
> 1 2 2 2 3 2 1 2 2 2 ...
> >  $ sn          : Factor w/ 2 levels "BS","MS": 1 1 1 1 1 1 1 1
> 1 1 ...
> >  $ MP_y_n          : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2
> 2 2 2 ...
> >  $ ratio       : num  -0.0506 -0.0506 -0.0506 -0.0506 -0.0506
> ...
> >  $ f           : Factor w/ 55 levels "0A0","0A1","0A2",..: 1
> 6 7 8 9 10 11 13 15 16 ...
> >  $ o           : Factor w/ 552 levels "","00T","00Z",..: 2 2
> 2 2 2 2 2 2 2 2 ...
> >  $ d            : int  9099 9099 9099 9099 9099 9099 9099
> 9099 9099 9099 ...
> >  $ MP              : num  6 6 6 6 5 4 6 6 6 6 ...
> >
> >
> > formula for the model:
> > mfit=lmer(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se * (o.se
> + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
> (1|f)+(1|o)+(1|d) + offset(log(MP)), data=c.data, family=binomial)
>
> > I hope this is what you want to see. Thank you for your help.
>
> My guess is that the problem is with creating the fixed-effects model
> matrix, of which there could be several copies created during the
> evaluation and optimization of the deviance.
>
> Just as a test, could you fit the model for the fixed-effects only
> using glm and check on what the size of the model matrix is?
> Something like
>
> glm1 <- glm(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
> (o.se + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
> offset(log(MP)), data=c.data, family=binomial)
> object.size(model.matrix(glm1)
>
> Also, could you convert 'd' to a factor and run str again so we can
> learn how many levels there are?  Either that or send the result of
>
> length(unique(mydata$d))
>
>
> > -------- Original-Nachricht --------
> >> Datum: Mon, 19 Dec 2011 14:50:06 -0600
> >> Von: Douglas Bates <bates at stat.wisc.edu>
> >> An: cumuluss at web.de
> >> CC: r-sig-mixed-models at r-project.org
> >> Betreff: Re: [R-sig-ME] problems with allocate memory
> >
> >> On Sun, Dec 18, 2011 at 3:17 PM,  <cumuluss at web.de> wrote:
> >> > Hi to everyone,
> >>
> >> > I have been trying to fit a glmm with a binomial error structure. My
> >> model is a little bit complex. I have 8 continuous predictor variables
> one of
> >> them as nonlinear term, 5 categorical predictor variables with some
> >> three-way interactions between them. Additional I have 3 random effects
> and one
> >> offset variable in the model. Number of obs is greater than 3million.
> >> > I’m working with the latest version of R 2.14.0 on a 64 bit windows
> >> system with 8Gb ram.
> >> > Everything I tried (reducing model complexity, different 64bit PC
> with
> >> even more memory) nothing leads to a fitted model, always the Error
> occurs:
> >> cannot allocate vector of size 2GB.
> >> > Is there anything I can do? I would be very grateful for any
> commentary.
> >>
> >> You probably have multiple copies of some large objects hanging
> >> around.  Can you send us the output of
> >>
> >> str(myData)
> >>
> >> where 'myData' is the name of the model frame containing the data you
> >> are using and the formula for the model you are trying to fit?
> >
> > _______________________________________________
> > R-sig-mixed-models at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models