[BioC] memory bottleneck on Linux

[Rafael A. Irizarry]

i dont know about taking advantage of the cluster, but
you can easily create a function that reads cel files in one by one (not
keeping them in memory) an computes mas5 values for each one... then
normalizes at the end. for RMA try justRMA. looking at the code for
justRMA can help you write the function you want for mas5.

i believe that with justRMA you can handle 100 hgu133 cels with less than
1 gig.


On Fri, 4 Jul 2003, Tapan Mehta wrote:

> I tried using rma instead of mas5 to check whether the
> memory restrictions still apply. Unfortunately they do
> apply and right now I am finding it impossible to
> process all the 100 files together. However mas5 works
> for smaller batch of 20 or 25 chips. However it would
> be great to figure out a solution that can process 100
> files since for rma I need to process all the 100
> files in a single batch. Are these
> algorithms(mas5,rma) parallelizable or can the
> programs be modified for making the optimum use of the
> cluster?
> 
> --- Ben Bolstad <bolstad at stat.berkeley.edu> wrote:
> > I am surprised that you can not get 100 CEL files to
> > run successfully. 4
> > GB (on a single machine) should be more than
> > sufficient to do this.
> > Because the MAS5 method is not a multi-chip method,
> > you could compute
> > the expression measures in smaller groups (say
> > groups of 25 for
> > example).  Join the expression measures together for
> > each group and then
> > scale them appropriately.
> > 
> > If you are using the mas5() function from the affy
> > package remember to
> > set normalize=FALSE when doing each of the groups
> > (so as not to scale
> > the groups individually).
> > 
> > Ben
> > 
> > 
> > On Wed, 2003-07-02 at 09:52, Tapan Mehta wrote:
> > > Hello,
> > > 
> > > I am trying to run a task of 100 CEL files each of
> > 10
> > > MB on Linux based Bewoulf cluster with each node
> > of  4
> > > GB. I am trying to use the mas5 method in the affy
> > > library. However I am unable to finish it gives
> > memory
> > > problems. I am unable to increase the memory as
> > the
> > > function of memory in R is only for Windows( as
> > posted
> > > on the R mailing list). Please could anybody guide
> > me
> > > in this problem? 
> > > 
> > > Regards,
> > > 
> > > Tapan Mehta
> > > 
> > > _______________________________________________
> > > Bioconductor mailing list
> > > Bioconductor at stat.math.ethz.ch
> > >
> >
> https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor
> > -- 
> > Ben Bolstad <bolstad at stat.berkeley.edu>
> > http://www.stat.berkeley.edu/~bolstad
> >
> 
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at stat.math.ethz.ch
> https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor
>