For those of you who prefer to keep everything in R, here is a costly option: get a 64-bit machine and plenty of memory (> 64GB). In a large clinical trail setting or pharmaceuticals, this kind of machine is easily available. 

With that being said, I have run many "stupidly inefficient" codes. It is efficient for your brain, though. 

A recent run was 250 arrays of human U133A. The peak memory usage was above 15 GB. No crash, the SAME code as I run 8 arrays.

-Simon

Message: 18
Date: Sat, 14 Jan 2006 09:22:02 -0000
From: "Xiaofan Li" <xiaofan.mlist@gmail.com>
Subject: Re: [BioC] affy & problems with memory
To: <kfbargad@ehu.es>
Cc: bioconductor@stat.math.ethz.ch
Message-ID: <43c8c2bb.30dbc6c5.0fb3.ffff945e@mx.gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"

Hi,

I have met with similar problems (memory allocation limit) dealing with
affy. I have seen that the maximum memory allocation for R in Windows XP is
the same size as your physical memory, say, in my case it is 768M. Reading
some 40 U133A CEL files will cause the error. I managed to *avoid* this
error by *separately loading the files*, for example, I load 20 files at a
time then save the image, close R, re-open it and reload the image. I have
successfully loaded 1.4G CEL files (121 chips) on my 768M laptop, creating a
450M RData image. 

However there are further problems with various data pre-processing methods.
With a certain pile of chips (about 20), it is very easy to cause a memory
allocation fault when doing rma(), expresso() and normalize(), both in
Windows XP and Linux. And the problem is no longer a typical "768M limit"
one, but related to the size of the target object. Since R seems to have
used no object compressing approaches, this might have been the real memory
limit (in Windows: physical mem + virtual mem size; in Linux: physical mem +
swap disk size).

Since R doesn't seem to have a data compressing approach as well as cleaning
and garbage collection mechanism, I would suggest that adding more RAMs into
your computer is the only way to solve the problem (alternatively you can
increase the swap/virtual memory size up to (4GB - physical mem), bearing
that extremely poor system performance).

Xiaofan Li
DAMTP, University of Cambridge, CB3 0WA, UK Tel +44 7886 614030, Email
xl252@cam.ac.uk
	[[alternative HTML version deleted]]

