[R] help: cannot allocate vector of length 828310236

[Prof Brian Ripley]

Does it make any statistical sense to do polr or probit regression (not 
the same thing) on `really huge data'?  There are few regression-like 
problems in which model inadequacy does not swamp estimation uncertainty 
for as few as a 1000 cases.

If you want to do that sort of thing, by all means use SAS to do it.
But if you are not prepared to spend a few $$ on adequate RAM, don't 
expect free technical consultancy, especially not from those whose work 
you are using and not crediting.

- The uncredited author of polr().


On Mon, 14 Aug 2006, T Mu wrote:

> Hi all,
> 
> I was trying a probit regression using polr() and got this message,

polr is a strange choice of tool for 'probit regression' as the term is 
usually used.  It does 'ordered probit regression'.

> Error in model.matrix.default(Terms, m, contrasts) :
>         cannot allocate vector of length 828310236
> 
> The data is about 20M (a few days ago I asked a question about large file,
> thank you for responses, then I use MS Access to select those columns I
> would use).
> 
> R is 2.3.1, Windows XP, 512M Ram.
> 
> I am going to read some help on memory use in R, but hope anybody can give
> me some quick hints.

Quick hint: read and follow the posting guide BEFORE posting.

> Is it because iphysical memory runs out, or some other things could be wrong
> with data or polr()?
> Does R use virtual memory? If so, what options can I set?
> If not, can R deal with really huge data (except adding RAM according to
> data size)? If this is the case, it is too bad that I have to tell my boss
> to go back to SAS. Now it is not a speed issue yet.
> 
> Thank you.
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595