[Rd] allocMatrix error

[Prof Brian Ripley]

On Tue, 17 Feb 2009, Hamid Ashafi wrote:

> On Sat, Feb 14, 2009 at 00:17,  <ashrafi at ucdavis.edu> wrote:
>
> Hi,
>
> I was trying to read ~400 chips in an affybatch and I got the same message.
> Could you find a remedy for that. My server has 128 GB of RAM. However, R
> halted ever before it uses the memory.

We don't have anything like sufficient details (please do read the 
posting guide).

If the issue is the size of matrices, you possibly (depending on the 
compiler) could arrange to compile R (and any relevant system 
libraries) to use 64-bit ints.  For C code in R there is typedef to 
change, and you would need integer*8 in the Fortran.  We would be 
interested to know the results if you do so, but the developers are 
unlikely to do so for you.

In any case, since you mention 'affybatch' it looks like this might 
be a design issue in that BioC package and the BioC lists might be the 
appropriate place to discuss it.  It is not obvious to me why ~400 
datasets need a single large R object rather than, say, a list of 400 
smaller ones, if that is indeed the problem.  So, to return to my
first point:

> We don't have anything like sufficient details.

Please give us the full details of your system, the memory in use (see 
?gc) and what you were trying to do.


> I have been able to load upto 250 CEL files but this time I wanted to test
> what would happen if I want to normalize 400 chips.

R can handle up to 16GB objects, which even for a 64-bit OS and 128GB 
of RAM are pretty large objects and do not arise naturally from many 
small files.

> Thanks for your prompt response.
>
>
>
> Hamid
>
>>
>
>>
>
>>
>
>> Martin Maechler wrote:
>
>>>
>
>>>>>>>> "VK" == Vadim Kutsyy <vadim at kutsyy.com>
>
>>>>>>>>     on Fri, 01 Aug 2008 07:35:01 -0700 writes:
>
>>>
>
>>>     VK> Martin Maechler wrote:
>
>>>    >>
>
>>>     VK> The problem is in array.c, where allocMatrix check for
>
>>>     VK> "if ((double)nrow * (double)ncol > INT_MAX)".  But why
>
>>>     VK> itn is used and not long int for indexing? (max int is
>
>>>     VK> 2147483647, max long int is 9223372036854775807)
>
>>>    >>
>
>>>    >> Well, Brian gave you all info:
>
>>>    >>
>
>>>     VK> exactly, and given that most modern system used for
>
>>>     VK> computations (i.e.  64bit system) have long int which is
>
>>>     VK> much larger than int, I am wondering why long int is not
>
>>>     VK> used for indexing (I don't think that 4 bit vs 8 bit
>
>>>     VK> storage is an issue).
>
>>>    >> Did you really carefully read ?Memory-limits ??
>
>>>    >>
>
>>>     VK> Yes, it is specify that 4 bit int is used for indexing
>
>>>     VK> in all version of R, but why? I think 2147483647
>
>>>     VK> elements for a single vector is OK, but not as total
>
>>>     VK> number of elements for the matrix.  I am running out of
>
>>>     VK> indexing at mere 10% memory consumption.
>
>>>
>
>>> Hmm, do you have 160 GBytes of RAM?
>
>>> But anyway, let's move this topic from R-help to R-devel.
>
>>>
>
>>>    [...........]
>
>>>
>
>>>     VK> PS: I have no problem to go and modify C code, but I am
>
>>>     VK> just wondering what are the reasons for having such
>
>>>     VK> limitation.
>
>>>
>
>>> This limitation and its possible remedies are an interesting topic,
>
>>> but really not for R-help:
>
>>>
>
>>> It will be a lot about C programming the internal represenation of R
>
>>> objects, etc.
>
>>> Very fascinating .... but for R-devel.
>
>>>
>
>>> "See you there!"
>
>>> Martin
>
>>>
>
>>> ______________________________________________
>
>>> R-help at r-project.org mailing list
>
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>
>>> PLEASE do read the posting guide
>
>>> http://www.R-project.org/posting-guide.html
>
>>> and provide commented, minimal, self-contained, reproducible code.
>
>>>
>
>>>
>
>> Quoted from:
>
>> http://www.nabble.com/allocMatrix-limits-tp18763791p18776531.html
>
>>
>
>>
>
>
>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595