[BioC] memory bottleneck on Linux

[Tapan Mehta]

Thanks a lot for the help however I am getting the
same problem even after using the justRMA method and
the ulimit for the shell. I am working on hgu95 CEL
files each of which is of nearly 10 MB. I will look
into how I can modify the mas5 for the cluster and
give you an update.

Thanks again,

Tapan Mehta
--- "Rafael A. Irizarry" <ririzarr at jhsph.edu> wrote:
> i dont know about taking advantage of the cluster,
> but
> you can easily create a function that reads cel
> files in one by one (not
> keeping them in memory) an computes mas5 values for
> each one... then
> normalizes at the end. for RMA try justRMA. looking
> at the code for
> justRMA can help you write the function you want for
> mas5.
> 
> i believe that with justRMA you can handle 100
> hgu133 cels with less than
> 1 gig.
> 
> 
> On Fri, 4 Jul 2003, Tapan Mehta wrote:
> 
> > I tried using rma instead of mas5 to check whether
> the
> > memory restrictions still apply. Unfortunately
> they do
> > apply and right now I am finding it impossible to
> > process all the 100 files together. However mas5
> works
> > for smaller batch of 20 or 25 chips. However it
> would
> > be great to figure out a solution that can process
> 100
> > files since for rma I need to process all the 100
> > files in a single batch. Are these
> > algorithms(mas5,rma) parallelizable or can the
> > programs be modified for making the optimum use of
> the
> > cluster?
> > 
> > --- Ben Bolstad <bolstad at stat.berkeley.edu> wrote:
> > > I am surprised that you can not get 100 CEL
> files to
> > > run successfully. 4
> > > GB (on a single machine) should be more than
> > > sufficient to do this.
> > > Because the MAS5 method is not a multi-chip
> method,
> > > you could compute
> > > the expression measures in smaller groups (say
> > > groups of 25 for
> > > example).  Join the expression measures together
> for
> > > each group and then
> > > scale them appropriately.
> > > 
> > > If you are using the mas5() function from the
> affy
> > > package remember to
> > > set normalize=FALSE when doing each of the
> groups
> > > (so as not to scale
> > > the groups individually).
> > > 
> > > Ben
> > > 
> > > 
> > > On Wed, 2003-07-02 at 09:52, Tapan Mehta wrote:
> > > > Hello,
> > > > 
> > > > I am trying to run a task of 100 CEL files
> each of
> > > 10
> > > > MB on Linux based Bewoulf cluster with each
> node
> > > of  4
> > > > GB. I am trying to use the mas5 method in the
> affy
> > > > library. However I am unable to finish it
> gives
> > > memory
> > > > problems. I am unable to increase the memory
> as
> > > the
> > > > function of memory in R is only for Windows(
> as
> > > posted
> > > > on the R mailing list). Please could anybody
> guide
> > > me
> > > > in this problem? 
> > > > 
> > > > Regards,
> > > > 
> > > > Tapan Mehta
> > > > 
> > > >
> _______________________________________________
> > > > Bioconductor mailing list
> > > > Bioconductor at stat.math.ethz.ch
> > > >
> > >
> >
>
https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor
> > > -- 
> > > Ben Bolstad <bolstad at stat.berkeley.edu>
> > > http://www.stat.berkeley.edu/~bolstad
> > >
> > 
> > _______________________________________________
> > Bioconductor mailing list
> > Bioconductor at stat.math.ethz.ch
> >
>
https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor
> >
> 
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at stat.math.ethz.ch
>
https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor