Mulholland, Tom <Tom.Mulholland@health.wa.gov.au> wrote:
> 
> I would suggest that you make a more thorough search of the
> R-Archives.
> (http://finzi.psych.upenn.edu/search.html) If you do you will find
> this
> discussion has been had several times and that the type of machine
> you
> are running will have an impact upon what you can do. My feeling is
> that
> you are going have to knuckle down with the documentation and
> understand
> how R works and then when you have specific issues that show you have
> read all the appropriate documentation, you might try another message
> to
> the list.
> 
> Ciao, Tom

Another approach is to not try to bring all your data into R at once - it is unlikely 
that you actually need every column of every row in your dataset to undertake any 
particular analysis. The trick is to bring into R only those rows and columns which 
you need at a particular moment, and then discard them.

The best way to do this is to manage your data in an SQl database such as 
MySQL or PostgreSQL, and then use one of the R database interfaces to issue 
queries against this database and to surface the query results as a data frame.  
Remeber to compose your queries in such as way as to only retreive the rows 
and columns you really need at any particular moment, and don't forget to delete 
these data frames as soon as you have finished with them (or at least, as soon 
as you need more space in your R session).

There is also an (experimental I think) package which allows lazy or virtual 
loading of database queries into data frames, so that the query results are paged 
into memory as they are needed. But I doubt you will need that.

Tim C

> 
> _________________________________________________
>  
> Tom Mulholland
> Senior Policy Officer
> WA Country Health Service
> Tel: (08) 9222 4062
>  
> The contents of this e-mail transmission are confidential and may be
> protected by professional privilege. The contents are intended only
> for
> the named recipients of this e-mail. If you are not the intended
> recipient, you are hereby notified that any use, reproduction,
> disclosure or distribution of the information contained in this
> e-mail
> is prohibited. Please notify the sender immediately.
> 
> 
> -----Original Message-----
> From: Edward McNeil [mailto:edward@ratree.psu.ac.th] 
> Sent: Tuesday, 2 December 2003 8:45 AM
> To: r-help@stat.math.ethz.ch
> Subject: [R] R and Memory
> 
> 
> Dear all,
> This is my first post.
> We have started to use R here and have also started teaching it to
> our
> PhD students. Our unit will be the HQ for developing R throughout
> Thailand.
> 
> I would like some help with a problem we are having. We have one
> sample
> of data that is quite large in fact - over 2 million records (ok ok
> it's
> more like a population!). The data is stored in SPSS. The file is
> over
> 350Mb but SPSS happily stores this much data. Now when I try to read
> it
> into R it grunts and groans for a few seconds and then reports that
> there is not enough memory (the computer has 250MB RAM). I have tried
> setting the memory in the command line (--max-vsize and
> --max-mem-size)
> but all to no avail.
> 
> Any help would be muchly appreciated!
> 
> Edward McNeil (son of Don)
> Epidemiology Unit
> Faculty of Medicine
> Prince of Songkhla University
> Hat Yai  90110
> THAILAND
> 
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> 
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help


