[R] memory limits in R loading a dataset and using the package tree

[Prof Brian Ripley]

Please read the rw-FAQ Q2.9.  There are ways to raise the limit, and you 
have not told us that you used them (nor the version of R you used, which 
matters as the limits are version-specific).

Beyond that, there are ways to use read.table more efficiently: see its 
help page and the 'R Data Import/Export' manual.  In particular, did you 
set nrows and colClasses?

But for the size of problem you have I would use a 64-bit build of R.


On Thu, 4 Jan 2007, domenico pestalozzi wrote:

> I think the question is discussed in other thread, but I don't exactly find
> what I want .
> I'm working in Windows XP with 2GB of memory and a Pentium 4 - 3.00Ghx.
> I have the necessity of working with large dataset, generally from 300,000
> records to 800,000 (according to the project), and about 300 variables
> (...but a dataset with 800,000 records could not be "large" in your
> opinion...). Because of we are deciding if R will be the official software
> in our company, I'd like to say if the possibility of using R with these
> datasets depends only by the characteristics of the "engine" (memory and
> processor).
> In this case we can improve the machine (for example, what memory you
> reccomend?).
>
> For example, I have a dataset of 200,000 records and 211 variables but I
> can't load the dataset because R doesn't work : I control the loading
> procedure (read.table in R) by using the windows task-manager and R is
> blocked when the file paging is 1.10 GB.
> After this I try with a sample of 100,000 records and I can correctly load
> tha dataset, but I'd like to use the package tree, but after some seconds (
> I use this tree(variable1~., myDataset) )   I obtain the message "Reached
> total allocation of 1014Mb".
>
> I'd like your opinion and suggestion, considering that I could improve (in
> memory) my computer.
>
> pestalozzi
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595