
Have you tried using gc() between iterations?  It has been my experience that R lulls you to sleep about garbage collection.  I haven't actually noticed that it will invoke automatic garbage collection when your code is in a loop or otherwise hard at work, like say, Java does [any correction on this would be welcome].    In general when you know you will be taxing your memory over time, it seems to be a prudent in practice to accumulate a counter of loops (or equivalent) without a gc(), and then stop and do one from time to time.  I have written example code that will bog down horribly when you don't do this, but will fly through easily when you do.   Can't hurt to try.

 

GL.
 
> Date: Tue, 21 Jul 2009 10:02:40 -0400
> From: juliet.hannah@gmail.com
> To: h.wickham@gmail.com
> CC: r-sig-mixed-models@r-project.org
> Subject: Re: [R-sig-ME] strategy to iterate over repeated measures/longitudinal data
> 
> Hi Hadley,
> 
> I wanted to check if there is anything obviously incorrect/inefficient with the
> strategy I outlined. I can, of course, try out various wide to long
> conversions.
> But apart from that, I was
> worried that I should try to convert the data all at once beforehand,
> and iterate
> through the converted data somehow.
> 
> If you see nothing horribly wrong with my current strategy, then
> that is helpful in itself. :)
> 
> My main concern is efficiency and memory use. Speed is a secondary
> concern. Really, I just want it to work. In looping through
> this data (when I wasn't using repeated measures), I have had
> several problems with memory usage going up (about 20G). I never
> figured it out (using GEEs for that one), but I suspected that
> maybe I wasn't iterating in a proper manner. Now when I am going
> to use all of the measurements, I fear I am going to run into
> more problems.
> 
> Thanks for responding.
> 
> Regards,
> 
> Juliet
> 
> On Tue, Jul 21, 2009 at 9:42 AM, hadley wickham<h.wickham@gmail.com> wrote:
> >> For each predictor:
> >> 1. create a long data set using the predictor and all measurements
> >> (using make.univ function from  multilevel package)
> >> 2. run lmer, extract the coefficient of interest
> >> 3. go to next predictor
> >>
> >> The end result is a vector of 400,000 coefficients.
> >>
> >> Do you have any suggestions on how I can improve this strategy?
> >
> > What sort of improvement are you looking for?  Speed?  Elegance?  Ease of use?
> >
> > Hadley
> >
> > --
> > http://had.co.nz/
> >
> 
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

_________________________________________________________________



	[[alternative HTML version deleted]]


