[R-sig-ME] problems with allocate memory

Douglas Bates bates at stat.wisc.edu
Thu Dec 22 16:42:31 CET 2011 

On Wed, Dec 21, 2011 at 6:22 PM,  <cumuluss at web.de> wrote:
> Hi Douglas,
> Ok, those are not the expected beautiful news :'(. If I understand it right Andrews implementation needs your check first before it is usably with lme4.

I believe Andrew checks his CXXR builds against several packages so
perhaps he has already tried to install lme4.  I am just suggesting
the possibility that CXXR may be more effective in memory usage - I
don't know and really don't want to stop other development to check it
out.

Basically you are going to need to consider your model more carefully.
 The fact that you can specify a model with dozens of fixed-effects
parameters doesn't mean it will be meaningful.

> And please one more questions what would be a rough time estimation for a new R-core with 64-bit indices for atomic R vectors?

That hasn't been established yet.  The current plan is to make some
changes and see what breaks then decide if it is even feasible to try
to patch everything back together.

> Thank you very much for your help and suggestions.
> All the Best
> Paul
>
>
> -------- Original-Nachricht --------
>> Datum: Wed, 21 Dec 2011 17:10:49 -0600
>> Von: Douglas Bates <bates at stat.wisc.edu>
>> An: cumuluss at web.de
>> CC: r-sig-mixed-models at r-project.org, a.r.runnalls at kent.ac.uk
>> Betreff: Re: [R-sig-ME] problems with allocate memory
>
>> On Wed, Dec 21, 2011 at 5:03 PM,  <cumuluss at web.de> wrote:
>> > Hi Douglas,
>> >
>> > thank you for your reply. But it sounds not that good for me. Could you
>> please suggest me something what I could do more or maybe different. You
>> said: There are no simple solutions at present. Is there a complicated
>> available which I could try?
>> > In the third part of your answer where you mentioned Andrew Runnalls and
>> the “reimplementing of R” Could this also be helpful for my non
>> fitting models problem or is this for opening the model results issue only?
>>
>> By "no easy solutions" I mean that I can't think of any approach that
>> doesn't involve reimplementing the code from scratch, which would
>> definitely take a long time.  Consider how long we have been working
>> on getting a 1.0 version of lme4 :-)
>>
>> I am not sure if Andrew's CXXR implementation of R would be more
>> effective or not.  I think it has a better garbage collection scheme
>> but I haven't tried it and I don't know if lme4 would build in that
>> system.  I have an item on the "ToDo" list to try it but I have a lot
>> of items on the "ToDo" list.
>>
>> Basically you will need to use a simpler model or fit to a sample of
>> your data or wait for R-core to determine if it is possible to use
>> 64-bit indices for atomic R vectors.
>>
>> > -------- Original-Nachricht --------
>> >> Datum: Wed, 21 Dec 2011 15:47:36 -0600
>> >> Von: Douglas Bates <bates at stat.wisc.edu>
>> >> An: cumuluss at web.de
>> >> CC: r-sig-mixed-models at r-project.org, "a.r.runnalls"
>> <a.r.runnalls at kent.ac.uk>
>> >> Betreff: Re: [R-sig-ME] problems with allocate memory
>> >
>> >> On Tue, Dec 20, 2011 at 5:25 PM,  <cumuluss at web.de> wrote:
>> >> > Hi Douglas,
>> >> >
>> >> > The variable‘d’ has about 710 levels.
>> >> >
>> >> > For your other request I tried to fit the suggested model but it was
>> not
>> >> possible. I tried it with different approaches, first without any
>> >> interactions and non nonlinear term. It fitted. The object size was
>> about 731571664
>> >> bytes. Then I successive made the model more complex. With one two way
>> >> interaction, with one three way interaction or with the nonlinear term
>> it was
>> >> slightly the same as before. With five two way interaction always with
>> the
>> >> nonlinear term the object size went up to 1075643424 bytes. With one
>> >> additional two way interaction the model won’t fit anymore with the
>> known
>> >> error.
>> >>
>> >> Which is an indication that the fixed-effects model matrix is getting
>> >> to be too large.  There are no simple solutions at present.  You may
>> >> find that some packages allow you to fit such large models by working
>> >> with horizonal chunks of the data and accumulating the result but
>> >> extending those to GLMMs would be decidedly non-trivial.
>> >>
>> >> > Perhaps another hint: Yesterday I attempted to fit a much simpler
>> model
>> >> with lmer, just to see if this works. (mfit=lmer(gr.b ~ f.ag + f.se +
>> o.se
>> >> + diff + exp.r + kl + (1|f)+(1|o), data=c.data, family=binomial)). It
>> >> fitted but I could not open mfit. Trying to see only the coefficients
>> also did
>> >> not work. I saved the image and this one is unfamiliar huge about 1.7
>> GB.
>> >>
>> >> The problem there is that the implicit print(mfit) (which is what I
>> >> imagine you mean when you say "could not open") ends up taking copies
>> >> of the whole object, which will eat up all your memory.  Development
>> >> versions of lme4 may eventually help with that.
>> >>
>> >> > By the way: After reloading the image some interesting things
>> happened.
>> >> An error occurred: slot coefs are not an S4 object. It seems to me that
>> it
>> >> is not possible to save the model results in an R image. Is that right?
>> >>
>> >> It should be possible to save and load such an object but there is
>> >> always the problem that when you have an object that is a sizable
>> >> fraction of the total available memory then you can get bitten if
>> >> something behind the scenes happens to take a copy at some point in
>> >> the calculation.  The original design in R for keeping track of when a
>> >> copy must be made is not the greatest and, as a result, R is somewhat
>> >> conservative when deciding whether or not to copy an object.  Getting
>> >> around that limitation would mean reimplementing R, more-or-less from
>> >> scratch and Andrew Runnalls is the only person I know who is willing
>> >> to embark on that.
>> >>
>> >> > -------- Original-Nachricht --------
>> >> >> Datum: Tue, 20 Dec 2011 09:23:37 -0600
>> >> >> Von: Douglas Bates <bates at stat.wisc.edu>
>> >> >> An: cumuluss at web.de
>> >> >> CC: r-sig-mixed-models at r-project.org
>> >> >> Betreff: Re: [R-sig-ME] problems with allocate memory
>> >> >
>> >> >> On Mon, Dec 19, 2011 at 5:54 PM,  <cumuluss at web.de> wrote:
>> >> >> > Hi Douglas,
>> >> >> >
>> >> >> > thanky you for your reply. This is "mydata"
>> >> >> >
>> >> >> > 'data.frame':   3909896 obs. of  19 variables:
>> >> >> >  $ gr.b            : int  0 0 0 0 0 0 0 0 0 0 ...
>> >> >> >  $ o.ag      : num  -0.651 -0.651 -0.651 -0.651 -0.651 ...
>> >> >> >  $ o.rar      : num  -0.935 -0.935 -0.935 -0.935 -0.935 ...
>> >> >> >  $ si       : num  0.299 0.299 0.299 0.299 0.299 ...
>> >> >> >  $ f.ag       : num  -1.25 -1.36 -1.33 -1.26 -1.21 ...
>> >> >> >  $ f.se       : Factor w/ 2 levels "F","M": 1 2 1 2 2 2 1 1
>> 1
>> >> 1
>> >> >> ...
>> >> >> >  $ o.se       : Factor w/ 2 levels "F","M": 1 1 1 1 1 1 1 1
>> 1
>> >> 1
>> >> >> ...
>> >> >> >  $ diff        : num  -0.536 -0.514 -0.521 -0.534 -0.545
>> ...
>> >> >> >  $ exp.r           : num  -0.168 -0.168 -0.163 -0.168
>> >> -0.168
>> >> >> ...
>> >> >> >  $ f.rar      : num  -0.911 0.215 1.224 -1.086 1.107 ...
>> >> >> >  $ f.si: num  1.0008 1.1583 0.0561 -0.4163 0.371 ...
>> >> >> >  $ kl              : Factor w/ 3 levels
>> >> "mat","nonkin",..:
>> >> >> 1 2 2 2 3 2 1 2 2 2 ...
>> >> >> >  $ sn          : Factor w/ 2 levels "BS","MS": 1 1 1 1 1
>> 1
>> >> 1 1
>> >> >> 1 1 ...
>> >> >> >  $ MP_y_n          : Factor w/ 2 levels "0","1": 2 2 2 2
>> 2
>> >> 2 2
>> >> >> 2 2 2 ...
>> >> >> >  $ ratio       : num  -0.0506 -0.0506 -0.0506 -0.0506
>> -0.0506
>> >> >> ...
>> >> >> >  $ f           : Factor w/ 55 levels
>> "0A0","0A1","0A2",..:
>> >> 1
>> >> >> 6 7 8 9 10 11 13 15 16 ...
>> >> >> >  $ o           : Factor w/ 552 levels
>> "","00T","00Z",..: 2
>> >> 2
>> >> >> 2 2 2 2 2 2 2 2 ...
>> >> >> >  $ d            : int  9099 9099 9099 9099 9099 9099
>> >> 9099
>> >> >> 9099 9099 9099 ...
>> >> >> >  $ MP              : num  6 6 6 6 5 4 6 6 6 6 ...
>> >> >> >
>> >> >> >
>> >> >> > formula for the model:
>> >> >> > mfit=lmer(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
>> >> (o.se
>> >> >> + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
>> >> >> (1|f)+(1|o)+(1|d) + offset(log(MP)), data=c.data, family=binomial)
>> >> >>
>> >> >> > I hope this is what you want to see. Thank you for your help.
>> >> >>
>> >> >> My guess is that the problem is with creating the fixed-effects
>> model
>> >> >> matrix, of which there could be several copies created during the
>> >> >> evaluation and optimization of the deviance.
>> >> >>
>> >> >> Just as a test, could you fit the model for the fixed-effects only
>> >> >> using glm and check on what the size of the model matrix is?
>> >> >> Something like
>> >> >>
>> >> >> glm1 <- glm(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
>> >> >> (o.se + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
>> >> >> offset(log(MP)), data=c.data, family=binomial)
>> >> >> object.size(model.matrix(glm1)
>> >> >>
>> >> >> Also, could you convert 'd' to a factor and run str again so we can
>> >> >> learn how many levels there are?  Either that or send the result of
>> >> >>
>> >> >> length(unique(mydata$d))
>> >> >>
>> >> >>
>> >> >> > -------- Original-Nachricht --------
>> >> >> >> Datum: Mon, 19 Dec 2011 14:50:06 -0600
>> >> >> >> Von: Douglas Bates <bates at stat.wisc.edu>
>> >> >> >> An: cumuluss at web.de
>> >> >> >> CC: r-sig-mixed-models at r-project.org
>> >> >> >> Betreff: Re: [R-sig-ME] problems with allocate memory
>> >> >> >
>> >> >> >> On Sun, Dec 18, 2011 at 3:17 PM,  <cumuluss at web.de> wrote:
>> >> >> >> > Hi to everyone,
>> >> >> >>
>> >> >> >> > I have been trying to fit a glmm with a binomial error
>> structure.
>> >> My
>> >> >> >> model is a little bit complex. I have 8 continuous predictor
>> >> variables
>> >> >> one of
>> >> >> >> them as nonlinear term, 5 categorical predictor variables with
>> some
>> >> >> >> three-way interactions between them. Additional I have 3 random
>> >> effects
>> >> >> and one
>> >> >> >> offset variable in the model. Number of obs is greater than
>> >> 3million.
>> >> >> >> > I’m working with the latest version of R 2.14.0 on a 64 bit
>> >> windows
>> >> >> >> system with 8Gb ram.
>> >> >> >> > Everything I tried (reducing model complexity, different 64bit
>> PC
>> >> >> with
>> >> >> >> even more memory) nothing leads to a fitted model, always the
>> Error
>> >> >> occurs:
>> >> >> >> cannot allocate vector of size 2GB.
>> >> >> >> > Is there anything I can do? I would be very grateful for any
>> >> >> commentary.
>> >> >> >>
>> >> >> >> You probably have multiple copies of some large objects hanging
>> >> >> >> around.  Can you send us the output of
>> >> >> >>
>> >> >> >> str(myData)
>> >> >> >>
>> >> >> >> where 'myData' is the name of the model frame containing the data
>> >> you
>> >> >> >> are using and the formula for the model you are trying to fit?
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > R-sig-mixed-models at r-project.org mailing list
>> >> >> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> >> > ___________________________________________________________
>> >> > SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
>> >> > kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192
>> > ___________________________________________________________
>> > SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
>> > kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192
>
> ___________________________________________________________
> SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
> kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192

More information about the R-sig-mixed-models mailing list