[R] load huge image
hb at maths.lth.se
Mon Mar 27 12:43:48 CEST 2006
On 3/27/06, Martin Maechler <maechler at stat.math.ethz.ch> wrote:
> >>>>> "Gottfried" == Gottfried Gruber <ggruber at terminal.at>
> >>>>> on Sun, 26 Mar 2006 10:27:35 +0200 writes:
> Gottfried> hello, i have run around 65000 regressions and
> Gottfried> stored them in a list. then i stored the session
> Gottfried> with save.image on my hard disk. the file is
> Gottfried> almost 1GB. when i now want to load the image it
> Gottfried> took tons of time. even after 12h of loading it
> Gottfried> was not done, although the saving was done fairly
> Gottfried> fast.
> I'm sure it takes so lang because you (i.e. R) run out of RAM
> and the machine starts to swap.
> Try to get access to a Linux (or other Unix-alike) machine with
> a 64-bit version of R and about 8 GB of RAM (maybe 4 GB is
> already sufficient). I guess then you should be able to read it
> much more quickly.
> For 65000 regressions, do you need more than the estimated
> coefficients or -- a bit more informatively -- the
> coef(summary( <lm-fit> )) result ?
> If you had only saved these coefficient matrices, I'm sure you'd
> have need **much** less memory.
> Gottfried> i fear i have to run the regressions again and
> Gottfried> store them in a database ...
> or really store what you need instead of everything ...
> Gottfried> can i load this file? any suggestions?
Do you need to have all of them in memory at once? Instead of using
save.image() can't you use save()/load() on each of the regression
fits? You can name the files using sprintf("regression%05d.Rdata",
idx) or similar.
Also, as Martin says, "fit" objects contains a lot of information that
you might not need; remove these before saving by setting the elements
you don't want to NULL.
> Gottfried> thanks & bets regards, gg --
> Gottfried> ---------------------------------------------------
> Gottfried> Gottfried Gruber
> Gottfried> mailto:gottfried.gruber at terminal.at www:
> Gottfried> http://gogo.sehrsupa.net
> Gottfried> ______________________________________________
> Gottfried> R-help at stat.math.ethz.ch mailing list
> Gottfried> https://stat.ethz.ch/mailman/listinfo/r-help
> Gottfried> PLEASE do read the posting guide!
> Gottfried> http://www.R-project.org/posting-guide.html
> R-help at stat.math.ethz.ch mailing list
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Mobile: +46 708 909208 (+2h UTC)
More information about the R-help