[BioC] Memory limit (vector size) o linux 64bit

[James W. MacDonald]

Hi Ivan,

Ivan Porro wrote:
> Hi all,
> 
> I'm running a script that try to normalise 448 HGU133A Affymetrix arrays, and I
> have "The Error" during the ReadAffy():
> 
>   Error: cannot allocate vector of size 1770343 Kb
> 
> I know about R and OSs adressing limitations, so (according to several posts on
> the mailing list) I'm doing that on a  64bit server:
> 
> x86_64 GNU/Linux (2x AMD Opteron 275)
> R 2.3.1 compiled from source
> MemTotal:         7819808 kB
> VmallocTotal: 34359738367 kB
> 
> that is, 8GB RAM (the difference is probably kept by a on board video) and up to
> 34TB swap.
> 
> I know from R FAQ that  "There are also limits on individual objects. On all
> versions of R, the maximum length (number of elements) of a vector is 2^31 - 1
> ~ 2*10^9"
> 
> But: 2.147.483.648  = 2^31  is bigger than 1.770.343.000 bytes (my vector size)

I think you are comparing apples and oranges here. The number of 
elements in a vector is different than the amount of RAM that the vector 
occupies.

> 
> I'm near (above) R physical limitations?
> 
> I use batch<-ReadAffy() then I should normalize it with gcrma(), invariantset

With that number of arrays I don't think you have enough RAM. The error 
you see doesn't mean that there is only one thing of 1.7 Gb being 
allocated. What it means is that R is trying to allocate 1.7 Gb for a 
vector, and you have less than that available (the rest has already been 
allocated).

If you want to do gcrma(), you will probably be able to use justGCRMA(), 
which skips the AffyBatch altogether. If you want to do gcrma() with an 
invariantset normalization you might be able to hack just.gcrma() enough 
to use a modified normalize.AffyBatch.invariantset. Personally, I don't 
think it would be worth the effort, but if you really want an 
invariantset normalization, that would probably be your best bet.

HTH,

Jim



> 
> thank you in advance,
> 
>     Ivan
> 
> 
> 
> 


-- 
James W. MacDonald, M.S.
Biostatistician
Affymetrix and cDNA Microarray Core
University of Michigan Cancer Center
1500 E. Medical Center Drive
7410 CCGC
Ann Arbor MI 48109
734-647-5623


**********************************************************
Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues.