[R] R memory and CPU requirements

[Alexander Sirotkin [at Yahoo]]

Thanks for all the responses. 

After re-examining my data I came to realize that
second order interactions would be enough in my
particular case. With second order instructions I
managed to fit a model with less then 512MB RAM.

Thanks to everybody.


--- John Fox <jfox at mcmaster.ca> wrote:
> Dear Alexander,
> 
> 
> At 01:29 AM 10/17/2003 -0700, Alexander Sirotkin
> \[at Yahoo\] wrote:
> >I agree completely.
> >
> >In fact, I have about 5000 observations, which
> should
> >be enough.
> >I was using 200 samples because of RAM limitations
> and
> >  I'm afraid to think about what amount of RAM I'll
> >need to fit an aov() for such data.
> >
> 
> 
> OK -- I didn't realize that you have 5000
> observations. Perhaps I didn't 
> read some of the earlier messages carefully enough.
> 
> At the risk of getting you to repeat information
> that you've already 
> provided, how many degrees of freedom are there in
> the model that you're 
> trying to fit? I can create a 5000 by 5000 model
> matrix on my relatively 
> anemic Windows machine, and surely (unless there's
> some specification 
> error) your model should have many fewer df than
> that if it includes just 
> the main effects and two-way interactions (or by all
> interactions, do you 
> mean higher-order interactions as well?).
> 
> Perhaps providing the following information would
> help: What is the model 
> formula? Which variables are factors? How many
> levels does each factor have?
> 
> Regards,
>   John
> 
> >--- John Fox <jfox at mcmaster.ca> wrote:
> > > Dear Alexander,
> > >
> > > If I understand you correctly, you have a sample
> of
> > > 200 observations. Even
> > > if you had only two factors with 40 levels each,
> the
> > > main effects and
> > > interactions of these factors would require
> about
> > > 1600 degrees of freedom
> > > -- that is, more than the number of
> observations.
> > > This doesn't make a whole
> > > lot of sense.
> > >
> > > I hope that this helps,
> > >   John
> > >
> > > At 05:03 PM 10/16/2003 -0700, Alexander Sirotkin
> > > \[at Yahoo\] wrote:
> > >
> > > >--- Deepayan Sarkar <deepayan at stat.wisc.edu>
> wrote:
> > > > > On Thursday 16 October 2003 17:59, Alexander
> > > > > Sirotkin \[at Yahoo\] wrote:
> > > > > > Thanks for all the help on my previous
> > > questions.
> > > > > >
> > > > > > One more (hopefully last one) : I've been
> very
> > > > > > surprised when I tried to fit a model
> (using
> > > > > aov())
> > > > > > for a sample of size 200 and 10 variables
> and
> > > > > their
> > > > > > interactions.
> > > > >
> > > > > That doesn't really say much. How many of
> these
> > > > > variables are factors ? How
> > > > > many levels do they have ? And what is the
> order
> > > of
> > > > > the interaction ? (Note
> > > > > that for 10 numeric variables, if you allow
> all
> > > > > interactions, then there will
> > > > > be a 100 terms in your model. This increases
> for
> > > > > factors.)
> > > > >
> > > > > In other words, how big is your model matrix
> ?
> > > (See
> > > > > ?model.matrix)
> > > > >
> > > > > Deepayan
> > > > >
> > > >
> > > >
> > > >I see...
> > > >
> > > >Unfortunately, model.matrix() ran out of memory
> :)
> > > >I have 10 variables, 6 of which are factor, 2
> of
> > > which
> > > >
> > > >have quite a lot of levels (about 40). And I
> would
> > > >like
> > > >to allow all interactions.
> > > >
> > > >I understand your point about categorical
> > > variables,
> > > >but
> > > >still - this does not seem like too much data
> to
> > > me.
> > > >
> > > >
> > > >I remmeber fitting all kinds of models (mostly
> > > >decision
> > > >trees) for much, much larger data sets.
> > > >
> > > >______________________________________________
> > > >R-help at stat.math.ethz.ch mailing list
> > >
> >
>
>https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> > >
> > >
>
>-----------------------------------------------------
> > > John Fox
> > > Department of Sociology
> > > McMaster University
> > > Hamilton, Ontario, Canada L8S 4M4
> > > email: jfox at mcmaster.ca
> > > phone: 905-525-9140x23604
> > > web: www.socsci.mcmaster.ca/jfox
> > >
>
>-----------------------------------------------------
> > >
> >
> >
> >__________________________________
> >Do you Yahoo!?

> search
> >http://shopping.yahoo.com
> 
>
-----------------------------------------------------
> John Fox
> Department of Sociology
> McMaster University
> Hamilton, Ontario, Canada L8S 4M4
> email: jfox at mcmaster.ca
> phone: 905-525-9140x23604
> web: www.socsci.mcmaster.ca/jfox
>
-----------------------------------------------------
>