[R] a quick Q about memory limit in R

[Paul Gilbert]

Yan Yu wrote:
...
> 
> (2) what decides the memory limit in R, How can increase that?

In recent versions of R this is controlled by the operating system, unless you
start R with an option that sets a lower limit than the OS allows. In Linux and
Unix this is controlled by

1/ ulimit (or limit in some shells). This can typically be relaxed by a normal
user to the system limits. On Linux the default max memory is usually not
limited (by ulimit) but it is sometimes necessary to relax the default stack
size.

2/ A combination of the amount of memory and amount of swap space. Roughly, on
Linux, these are added together to give the limit. This has changed over the
years in Linux and may vary on different version of Unix, but typically swap
space increases the size of problem you can handle. Physical memory is faster,
but swap works. Given prices these days you might consider having these add to
around 4G if you want to work on large problems with R.

3/ The architecture of the processor (e.g. 32-bit vs 64-bit). A program cannot
exceed the address space of the architecture (2^32 = 4G bytes on a typical PC
32-bit processor, 2^64=a lot more on a 64-bit workstation). The OS itself needs
some of this, so I believe the practical limit on 32-bit Linux is around 3G. (In
any case, there is not much point in have more than 4G of swap+memory on a
32-bit machine.) On a 64-bit architecture with a 64-bit Unix (most workstations)
the application (R) must be compiled as a 64-bit application. Thus the fixing of
Solaris bugs in gcc 3.2.3 has meant that much larger problems can now be handled
with R on Solaris (I believe this was possible before with Sun compilers.) It
should also be possible to compile 64-bit R under (64-bit) Linux on Intel
Itanium and AMD Opteron processors. I have no experience with this (but would be
interested in hearing from anyone that does).

On Windows the situation is different and I am much less familiar with it (and
look forward to being corrected). I believe applications must fit into physical
memory on Windows, that is, they can be swapped out but not partly swapped out.
This means that it is necessary to buy more memory to run bigger R problems. (Of
course, with physical memory problems will run much faster, so you should
consider buying more memory even in Unix.) Windows itself demands some of the
memory, so I believe the practical limit for applications in Windows is 2G
bytes. I understand there is a 64-bit version of Windows under development, but
I don't think it has been released yet.

Paul Gilbert