[R] R memory and CPU requirements

[John Fox]

Dear Alexander,


At 01:29 AM 10/17/2003 -0700, Alexander Sirotkin \[at Yahoo\] wrote:
>I agree completely.
>
>In fact, I have about 5000 observations, which should
>be enough.
>I was using 200 samples because of RAM limitations and
>  I'm afraid to think about what amount of RAM I'll
>need to fit an aov() for such data.
>


OK -- I didn't realize that you have 5000 observations. Perhaps I didn't 
read some of the earlier messages carefully enough.

At the risk of getting you to repeat information that you've already 
provided, how many degrees of freedom are there in the model that you're 
trying to fit? I can create a 5000 by 5000 model matrix on my relatively 
anemic Windows machine, and surely (unless there's some specification 
error) your model should have many fewer df than that if it includes just 
the main effects and two-way interactions (or by all interactions, do you 
mean higher-order interactions as well?).

Perhaps providing the following information would help: What is the model 
formula? Which variables are factors? How many levels does each factor have?

Regards,
  John

>--- John Fox <jfox at mcmaster.ca> wrote:
> > Dear Alexander,
> >
> > If I understand you correctly, you have a sample of
> > 200 observations. Even
> > if you had only two factors with 40 levels each, the
> > main effects and
> > interactions of these factors would require about
> > 1600 degrees of freedom
> > -- that is, more than the number of observations.
> > This doesn't make a whole
> > lot of sense.
> >
> > I hope that this helps,
> >   John
> >
> > At 05:03 PM 10/16/2003 -0700, Alexander Sirotkin
> > \[at Yahoo\] wrote:
> >
> > >--- Deepayan Sarkar <deepayan at stat.wisc.edu> wrote:
> > > > On Thursday 16 October 2003 17:59, Alexander
> > > > Sirotkin \[at Yahoo\] wrote:
> > > > > Thanks for all the help on my previous
> > questions.
> > > > >
> > > > > One more (hopefully last one) : I've been very
> > > > > surprised when I tried to fit a model (using
> > > > aov())
> > > > > for a sample of size 200 and 10 variables and
> > > > their
> > > > > interactions.
> > > >
> > > > That doesn't really say much. How many of these
> > > > variables are factors ? How
> > > > many levels do they have ? And what is the order
> > of
> > > > the interaction ? (Note
> > > > that for 10 numeric variables, if you allow all
> > > > interactions, then there will
> > > > be a 100 terms in your model. This increases for
> > > > factors.)
> > > >
> > > > In other words, how big is your model matrix ?
> > (See
> > > > ?model.matrix)
> > > >
> > > > Deepayan
> > > >
> > >
> > >
> > >I see...
> > >
> > >Unfortunately, model.matrix() ran out of memory :)
> > >I have 10 variables, 6 of which are factor, 2 of
> > which
> > >
> > >have quite a lot of levels (about 40). And I would
> > >like
> > >to allow all interactions.
> > >
> > >I understand your point about categorical
> > variables,
> > >but
> > >still - this does not seem like too much data to
> > me.
> > >
> > >
> > >I remmeber fitting all kinds of models (mostly
> > >decision
> > >trees) for much, much larger data sets.
> > >
> > >______________________________________________
> > >R-help at stat.math.ethz.ch mailing list
> >
> >https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> >
> >
>-----------------------------------------------------
> > John Fox
> > Department of Sociology
> > McMaster University
> > Hamilton, Ontario, Canada L8S 4M4
> > email: jfox at mcmaster.ca
> > phone: 905-525-9140x23604
> > web: www.socsci.mcmaster.ca/jfox
> >
>-----------------------------------------------------
> >
>
>
>__________________________________
>Do you Yahoo!?

>http://shopping.yahoo.com

-----------------------------------------------------
John Fox
Department of Sociology
McMaster University
Hamilton, Ontario, Canada L8S 4M4
email: jfox at mcmaster.ca
phone: 905-525-9140x23604
web: www.socsci.mcmaster.ca/jfox