[R] Memory allocation, IBM-AIX and R-1.6.2 - addendum

[Luke Tierney]

On 10 Feb 2003, Peter Dalgaard BSA wrote:

> Laurent Gautier <laurent at cbs.dtu.dk> writes:
> 
> > ...sorry for the spam, but answers I get to my previous
> > question suggest that I should specify that the
> > machine has a *lot* of memory and should be able
> > to the instanciation...
> > 
> > Anybody with an IBM mainframe, R-1.6.2 and large
> > matrices ?
> 
> For whatever it's worth, that size does not appear to be a problem in
> Linux. Takes a while, and a lot of memory, to get a summary of the
> matrix though (I'm slightly puzzled by that: The memory footprint
> appears to shoot up to 1.7G for something that is just working
> on the columns of a 319M matrix?).

If the AIX you are using is 32 bit (I forget if they support 64) then
you have to remember that, no matter how much memory you have, you are
starting to run up against address space limits.  It is only fairly
recent Linux malloc's that are able to give a process more than 1GB of
address space, and the way things are done when the 1G threshold is
crossed is not necessarily very efficient.  (1G is usually the point
in the address space where the traditional heap has to stop because
the shared library loader is mapped there).  AIX will be different in
detail but the basic issue is the same: 2^32 = 4G only leave you so
much room to do the things the OS needs to do.

luke

-- 
Luke Tierney
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
   Actuarial Science
241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu