[R] RAM usage

[Vaidotas Zemlys]

Hi,

I'm having problems while working with large data sets with R 1.5.1 in
windows 2000. Given a integer matrix size of 30 columns and  15000 rows
my function should return a boolean matrix size of about 5000 rows and
15000 columns.

First of all I tried to run this function on computer with 256 MB of 
RAM. I increased memory limit of R with memory.limit() up to 512 MB. I 
was inspecting memory and processor usage through Windows  task manager. 
At first R was using 100% of processor and memory usage was constantly 
increasing. When there were no physical memory left, R began using 
virtual memory. Then the processor usage dropped, but there was 
instensive work with hard drive. Of course that slowed down 
calculations. Yet the memory used by R always changed, and that was I
think the  sign, that R was calculating. But after a while the task
manager showed that R uses constant size of memory. The Rgui was not
responding, so I assumed that R crashed.

So I tried to run the calculations on another win2k box with 1024 MB of 
RAM with the same R version 1.5.1.  This time virtual memory was not 
used, yet still R froze. The memory usage grew to about 450 MB and then 
R stopped. Memory usage was not changing, Rgui did not respond, yet 
processor was used 100%. Task manager showed that peak memory usage was 
about 760 MB.

On smaller data sets there were no problems, memory usage was constantly 
  increasing and processor was used 100%. My function does not use fancy 
functions. Basically it just sums, finds minimum and maximum, uses 
subseting of a matrix, and calculates correlation matrix between 10 
columns of a given matrix.

So I would like to ask can R at all perform such calculations where a 
lot of memory must be used? And if R can do such calculations, what are 
specific problems, topics or tips which should be known before letting R 
to do these calculations?

Thanks in advance, for any help and special thanks for reading such a 
long letter.

Vaidotas Zemlys

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._