[R] RAM usage

[ripley@stats.ox.ac.uk]

On Fri, 18 Oct 2002, Vaidotas Zemlys wrote:

> I'm having problems while working with large data sets with R 1.5.1 in
> windows 2000. Given a integer matrix size of 30 columns and  15000 rows
> my function should return a boolean matrix size of about 5000 rows and
> 15000 columns.

That's  75million items of 4bytes each, hence almost 300Mb for that one
object.

> First of all I tried to run this function on computer with 256 MB of
> RAM. I increased memory limit of R with memory.limit() up to 512 MB. I
> was inspecting memory and processor usage through Windows  task manager.
> At first R was using 100% of processor and memory usage was constantly
> increasing. When there were no physical memory left, R began using
> virtual memory. Then the processor usage dropped, but there was
> instensive work with hard drive. Of course that slowed down
> calculations. Yet the memory used by R always changed, and that was I
> think the  sign, that R was calculating. But after a while the task
> manager showed that R uses constant size of memory. The Rgui was not
> responding, so I assumed that R crashed.

Don't think so.  More likely that Windows is having problems managing the
memory requirements.  You are trying to access an object too big to fit
into RAM, and that going to cause severe strain.

> So I tried to run the calculations on another win2k box with 1024 MB of
> RAM with the same R version 1.5.1.  This time virtual memory was not
> used, yet still R froze. The memory usage grew to about 450 MB and then
> R stopped. Memory usage was not changing, Rgui did not respond, yet
> processor was used 100%. Task manager showed that peak memory usage was
> about 760 MB.

Again, there is likely a problem with Windows allocating a contiguous
chunk of 300Mb of memory.  Try this sort of thing only after a fresh
reboot.

> On smaller data sets there were no problems, memory usage was constantly
>   increasing and processor was used 100%. My function does not use fancy
> functions. Basically it just sums, finds minimum and maximum, uses
> subseting of a matrix, and calculates correlation matrix between 10
> columns of a given matrix.
>
> So I would like to ask can R at all perform such calculations where a
> lot of memory must be used? And if R can do such calculations, what are
> specific problems, topics or tips which should be known before letting R
> to do these calculations?

R can. The question is `can Windows'?  If possible use a Unix-based OS.

You have not told us your problem, so has not demonstrated that
`a lot of memory must be used'.   Hard to help when we don't know what you
are attempting, but few problems cannot be done in pieces.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._