[BioC] affy & problems with not enough avaliable memory

[Kort, Eric]

A few comments, in no particular order:

1. I follow the recommendations of Choe et al in their article "Preferred analysis methods for Affymetrix GeneChips revealed by a wholly defined control dataset", EXCEPT that I use mas5 instead of RMA specifically because this eliminates the need to normalize all arrays at once.

2. I do my normalization on a 64 bit Linux server so I can allocate > 4GB of memory to the task (4GB being the absolute ceiling of addressable memory using 32bit addresses).  As mentioned in another thread, this is a little pricey, although I would say the 64GB RAM recommendation of that thread is excessive (I can do 200 hgu133plus2 chips with 16GB of RAM without much difficulty).  Then again, more is more where RAM is concerned.

3. However, if I didn't have a 64 bit machine at my institution, there are various 64 bit machines at institutions near me that I could have used for this sort of periodic normalization.  It seems to me that there must be such machines available to many/most/nearly all of us just waiting to be identified.

4. Finally, in concert with Dr. Gentleman's data centralization efforts which he described in an earlier post (see http://www.sgdi.org), it occurs to me that it would be incredibly useful if there was some sort of shared computing resources available to the BioC community.  I know our 64bit machine is underutilized 95% of the time, and I imagine this is the case for many of these systems.  I envision a 64bit version of "Folding At Home"..."Normalizing at Home"?  Or are these just fantastical musings of an idle dreamer?

-Eric




-----Original Message-----
From: bioconductor-bounces at stat.math.ethz.ch on behalf of Seth Falcon
Sent: Mon 1/16/2006 7:16 PM
To: bioconductor at stat.math.ethz.ch
Subject: Re: [BioC] affy & problems with not enough avaliable memory
 
Hi Nick,

I'm having some deja vu:
https://stat.ethz.ch/pipermail/bioconductor/2006-January/011471.html


On 15 Jan 2006, nicholas-ettinger at uiowa.edu wrote:
> I have 16 Affy Hg-U133-plus-2 CEL files that I am trying to analyze.
> I am having difficulties generating expression data because of
> memory problems.  I am working on a Dell desktop running WinXP.

How much RAM is installed on your system?  As Jim previously
commented, purchasing more RAM may be the easiest way to get going.

> What can I do?  Do I have to normalize all 16 arrays at once?  Is it
> still valid to do them piecemeal?  My gut says no. 

The currently available procedures are intended to normalize a set of
arrays all at once.  Trust your gut.

We have in the works a normalization routine that will work in limited
memory environments, but it isn't available yet.

+ seth

_______________________________________________
Bioconductor mailing list
Bioconductor at stat.math.ethz.ch
https://stat.ethz.ch/mailman/listinfo/bioconductor

This email message, including any attachments, is for the so...{{dropped}}