[R] R memory and CPU requirements

[kjetil@entelnet.bo]

On 17 Oct 2003 at 1:33, Alexander Sirotkin [at Yahoo] wrote:

You mentioned in an earlier post that at least one of your factors 
have 40 levels. If you use the default contrast, contrast.traetment, 
the design matrix for this factor will be dominated by zeros. Maybe 
you shoukd look at tha CRAN package SparseM, which have 
function slm for linear models with sparse matrices?

(I did'nt try this , but it could be worthwile)

Still, I don't think it makes much sense to start with a model with 
all the interactions in!

Kjetil Halvorsen

> 
> --- Deepayan Sarkar <deepayan at stat.wisc.edu> wrote:
> > On Thursday 16 October 2003 19:03, Alexander
> > Sirotkin \[at Yahoo\] wrote:
> > 
> > > > > Thanks for all the help on my previous
> > questions.
> > > > >
> > > > > One more (hopefully last one) : I've been very
> > > > > surprised when I tried to fit a model (using
> > > > > aov())
> > > > > for a sample of size 200 and 10 variables and
> > > > > their interactions.
> > > >
> > > > That doesn't really say much. How many of these
> > > > variables are factors ? How
> > > > many levels do they have ? And what is the order
> > of
> > > > the interaction ? (Note
> > > > that for 10 numeric variables, if you allow all
> > > > interactions, then there will
> > > > be a 100 terms in your model. This increases for
> > > > factors.)
> > > >
> > > > In other words, how big is your model matrix ?
> > (See
> > > > ?model.matrix)
> > > >
> > > > Deepayan
> > >
> > > I see...
> > >
> > > Unfortunately, model.matrix() ran out of memory :)
> > > I have 10 variables, 6 of which are factor, 2 of
> > which
> > >
> > > have quite a lot of levels (about 40). And I would
> > > like to allow all interactions.
> > >
> > > I understand your point about categorical
> > variables,
> > > but still - this does not seem like too much data
> > to me.
> > 
> > That's one way to look at it. You don't have enough
> > data for the model you are 
> > trying to fit. The usual approach under these
> > circumstances is to try 
> > 'simpler' models.
> > 
> > Please try to understand what you are trying to do
> > (in this case by reading an 
> > introductory linear model text) before blindly
> > applying a methodology.
> > 
> > Deepayan
> > 
> > 
> 
> 
> I did study ANOVA and I do have enough observations.
> 200 was only a random sample of more then 5000 which I
> think should be enough. However, I'm afraid to even
> think about amount of RAM I will need with R to fit a
> model for this data.
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help