[BioC] affy & problems with memory

[Kevin Coombes]

The maximum number of hgu133plus2 arrays that can be processed using 
expresso-like tools in R on a 32-bit Windows machine is about 30.

The maximum number of hgu95a arrays that can be processed using 
expresso-like tools in R on a 32-bit Windows machine is about 90.

The rough computation is simple: each hgu133plus 2 array contains 1164*1164 
= 1,354,896 probes. Allowing at least 10 bytes to store the probe intenisty 
as a double precision float, the probe-level data for one array requires 
about 13.5 MB. Thus, 30 arrays requires at least 400 MB just to store the 
probe level data. Because R implements pass-by-value rather than 
pass-by-reference (and because BioConductor has followed the usual good 
programming practice of writing and reusing modular functions), you can 
easily get as many as 3 copies of this data structure floating around in 
memory at the same time, putting us at the 1.2 GB level. This doesn't 
include any additional overhead for other parts of the AffyBatch structure 
or the amount of space needed to store the code objects, and anything else 
in your current session.

Now, a 32-bit machine is limited to adressing 4GB of RAM, but as far as I 
can tell, Windows won't let a single program see more than 2GB, so you can 
begin to see why expresso can't handle more than about 30 of these arrays 
at once.

The specialized functions like justrma get around this problem by passing 
control over to routines written in C, which presimably use 
pass-by-reference to avoid making extra copies of the data.

  -- Kevin

--On Wednesday, January 11, 2006 10:30 AM +0100 Benjamin Otto 
<b.otto at uke.uni-hamburg.de> wrote:

> Hi Nick,
>
> I'm currently working on that chip type myself on a system with 512MB RAM
> and had to increase the memory assigned to R too. However
> "--max-mem-size=1024Mb" seems to be ok in my case. Besides the problem of
> how much memory is necessary for the hgu133plus2 maybe somebody out there
> could answer me this two related questions:
> * I'm running R under WinXp AND under Linux on the same machine. However
> there has never been any memory problem in linux. So is the memory
> allocation or the  assignment of the max. memory size that can be used by
> R different in the two systems?
> * With my 512MB RAM system the old maximum memory value under WinXP was
> 512MB. Is this just a coincidence or does this maximum value rise to
> 1024MB if I upgrade my system to 1024MB RAM?
>
> regards,
> Benjamin
>
>> -----Original Message-----
>> From: bioconductor-bounces at stat.math.ethz.ch
>> [mailto:bioconductor-bounces at stat.math.ethz.ch]On Behalf Of James W.
>> MacDonald
>> Sent: 10 January 2006 20:02
>> To: Ettinger, Nicholas
>> Cc: bioconductor at stat.math.ethz.ch
>> Subject: Re: [BioC] affy & problems with memory
>>
>>
>> Ettinger, Nicholas wrote:
>> > Hello all!
>> >
>> > I have 16 Affy Hg-U133plus2 cel files that I am trying to analyze.  I
>> > am having difficulties generating expression data because of memory
>> > problems.  I am working on a Dell desktop running WinXP.  I can
>> > successfully generate expression values if I use "RMA" but if I try
>> > MAS5 or GCRMA or LiWong instead then I consistently get an error
>> > message with "Cannot allocate vector of size 169362 Kb."  To try to
>> > increase available memory I have already done two things:  (a) I
>> > right-clicked on the R icon and changed the box labeled 'Target' to
>> > read "C:\Program Files\R\R-2.2.0\bin\Rgui.exe" --sdi
>> > --max-mem-size=2000M" and (b) I have increased the virtual memory
>> > Total Page File size to 2000MB.  But it still runs out of memory.
>> >
>> > Help! Any suggestions -- I don't really have access to any bigger
>> > computers?
>>
>> You don't need a bigger computer, you need more RAM, which is pretty
>> cheap these days. I can't imagine it would take more than 512 Mb to do
>> 16 U133 plus 2 chips, so you might look into adding a 512 Mb stick.
>>
>> Best,
>>
>> Jim
>>
>>
>> >
>> > Thanks.
>> > ---Nick
>> >
>> > _______________________________________________
>> > Bioconductor mailing list
>> > Bioconductor at stat.math.ethz.ch
>> > https://stat.ethz.ch/mailman/listinfo/bioconductor
>>
>>
>> --
>> James W. MacDonald
>> Affymetrix and cDNA Microarray Core
>> University of Michigan Cancer Center
>> 1500 E. Medical Center Drive
>> 7410 CCGC
>> Ann Arbor MI 48109
>> 734-647-5623
>>
>> _______________________________________________
>> Bioconductor mailing list
>> Bioconductor at stat.math.ethz.ch
>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>>
>
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/bioconductor