[Rd] (PR#8861) Memory allocation fails in R 2.2.1 and R 2.3.0 on SGI Irix, while plenty of memory available

[Prof Brian Ripley]

(Your reply to R-bugs started a new report, so I am aiming to re-file it.)

On Mon, 15 May 2006, Thomas Lumley wrote:

> On Mon, 15 May 2006, m.vroonhoven at erasmusmc.nl wrote:
>
>> Dear R developers,
>>
>> We have a big SGI Origin computation server with 32 cpu's and 64 Gb of
>> RAM. In R 2.0.0 we could run large jobs, allocating 8 Gb of RAM was not
>> a problem, for example by running:
>> > v1 <- seq(1,2^29)
>> > v2 <- seq(1,2^29)
>> > v3 <- seq(1,2^29)
>> > v4 <- seq(1,2^29)
>> This yields an R process, consuming about 8 Gb of RAM:
>>       PID       PGRP USERNAME PRI  SIZE   RES STATE    TIME WCPU% CPU% COMMAND
>>    177484     177484 mirjam    20 8225M 8217M sleep    1:18  29.3 0.00 R
>>
>> After upgrading from R 2.0.0 to R 2.2.1, we cannot allocate more than
>> about 1300 M of memory, as shown below:
>> > v1 <- seq(1,2^29)
>>  Error: cannot allocate vector of size 2097152 Kb
>> > v1 <- seq(1,2^28)
>> > v2 <- seq(1,2^27)
>>  Error: cannot allocate vector of size 524288 Kb
>> > v2 <- seq(1,2^25)
>> > v3 <- seq(1,2^24)
>> > v4 <- seq(1,2^23)
>> > v5 <- seq(1,2^22)
>>  Error: cannot allocate vector of size 16384 Kb
>> > v5 <- seq(1,2^21)
>> > v6 <- seq(1,2^20)
>> > v7 <- seq(1,2^19)
>> > v8 <- seq(1,2^18)
>> > q()
>>  Save workspace image? [y/n/c]: n
>> Upgrading to R 2.3.0 yields the same results.
>> This yields an R executable taking 1284M of RAM, refusing to allocate
>> more RAM, with about 30Gb free on the machine.
>
>
> You can tell if you have a 64bit build of R by looking at
> .Machine$sizeof.pointer in R, which should be 8.

And if this is a 32-bit build, it is working as expected given the limited 
address space.

> If not, then you need to set whatever C and Fortran compilation flags give
> a 64bit system. It doesn't look to me as if R's configure script has any
> special handling for C compiler flags on SGI.

Well, the R-admin manual says (under IRIX)

   @R{} 2.1.0 has been successfully built on IRIX64 6.5 using both
   @command{gcc} and the native (MipsPro 7.4) compiler. However, neither
   version has passed @command{make check} due to a problem with time
   zones (see below).  A 64-bit executable has not been successfully
   built.

so we could not use special handling for a system we have not been told 
how to build in 64-bit mode.

Here we don't know the OS, the compiler, the flags used .... and that 
definitely is a bug.

> If you have a 64bit build then something strange is happening. The message
> you quote happens only when malloc() returns NULL, so it is hard to see
> how R could be causing it, though.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595