[BioC] Clustering of 30,000+ genes

[Thorsten Forster]

Hi January

As previous posters have pointed out, it is normal practice to remove 
noise genes prior to measuring co-expression.

However, if you are doing research, standard practice often does not 
apply. We have been working on this computational issue for a while now 
and if you have a good reason to want to compute all pairwise 
similarities, maybe you can make use of our SPRINT package. It is 
basically applying brute computing power to run any number of 
correlations you wish.

Within SPRINT, we have implemented a parallelised version of the basic 
cor() function in addition to a few machine learning algorithms, boot(), 
apply(), and some statistical tests.

Once installed, all you need to do is load the SPRINT library and in 
your script replace cor() with pcor().

Computation time is much reduced through the parallelisation scheme, and 
we get around RAM limits by incorporating the "ff" package.


Caveats:
a) You'll probably need to use the cor() function in a separate step, I 
don't know how it fits with your CoXpress or GSCA packages.

b) Someone (with system administrator skills) needs to configure the 
SPRINT package for your high performance computing platform of choice, 
be that a multi-processor desktop or something like UK supercomputer HECToR.

c) No Windows at the moment, you are limited to a Unix-based OS (we are 
just about done with a Mac version)


You can find SPRINT on CRAN, and further information here:

http://www.r-sprint.org/

  Thorsten



On 10/09/2011 11:00, bioconductor-request at r-project.org wrote:
> Subject:
> Re: [BioC] Clustering of 30,000+ genes
> From:
> Sean Davis <sdavis2 at mail.nih.gov>
> Date:
> 09/09/2011 11:08
>
> To:
> January Weiner <january.weiner at gmail.com>
> CC:
> bioconductor at r-project.org
>
>
> Hi, January.
>
> One common way of reducing the number of features is to choose the top
> X% by variance or coefficient of variation.  A large percentage of
> genes are not even expressed in a given tissue type and another large
> percentage do not vary across a sample set.  You can use the
> genefilter package to perform such filtering.
>
> Sean
>
> On Wed, Sep 7, 2011 at 5:29 PM, January Weiner<january.weiner at gmail.com>  wrote:
>> >  Hello,
>> >
>> >  I'm struggling with co-expression analysis, and for that I would like
>> >  to try to cluster all the genes I have in my microarray set, including
>> >  those which are not differentially expressed between the study groups.
>> >  I am using CoXpress at the moment and will try my luck with GSCA as
>> >  well, but both packages seem to have been layed out for 3000 rather
>> >  than 30000 genes.
>> >
>> >  How do you do that in R? I get errors about R not being able to
>> >  allocate enough memory. Clearly, the amount of memory required to
>> >  calculate all correlations the simple way might be a bit on the large
>> >  side, but I can think of one or two tricks to get this done; I wonder
>> >  whether it has been implemented already.
>> >
>> >  Other than that -- how should I reasonably limit the number of genes
>> >  to study? i don't want to bias the outcome of the analysis by
>> >  selecting only genes that are DE, actually -- I would be very
>> >  interested in genes that  show differential co-expression, but no
>> >  differences in expression.
>> >
>> >  Kind regards,
>> >
>> >  j.
>> >
>> >  --
>> >
>> >  _______________________________________________
>> >  Bioconductor mailing list
>> >  Bioconductor at r-project.org
>> >  https://stat.ethz.ch/mailman/listinfo/bioconductor
>> >  Search the archives:http://news.gmane.org/gmane.science.biology.informatics.conductor
>> >
>
>
> Part 1.2
>
> Subject:
> Re: [BioC] Clustering of 30,000+ genes
> From:
> "Tim Triche, Jr." <tim.triche at gmail.com>
> Date:
> 09/09/2011 12:48
>
> To:
> Sean Davis <sdavis2 at mail.nih.gov>
> CC:
> bioconductor at r-project.org, January Weiner <january.weiner at gmail.com>
>
>
> That said, there are a number of differential coexpression papers out there
> noting that, among the remaining transcripts, calculating (shrunken or
> unshrunken) estimates of the covariance matrices can be... interesting.
>
> 'corpcor', 'glasso', 'huge', and 'WGCNA' may come in handy for the latter
> task, with WGCNA explicitly designed for finding differential coexpression.
>   The authors of one such (throwaway -- no implementation released) paper
> note that they crammed 128GB of physical RAM into the machine used for the
> analyses in the paper, but it's quite possible the authors did not realize
> that filtering could have saved them a lot of time and memory.
>
>
>
> On Fri, Sep 9, 2011 at 3:08 AM, Sean Davis<sdavis2 at mail.nih.gov>  wrote:
>
>> >  Hi, January.
>> >
>> >  One common way of reducing the number of features is to choose the top
>> >  X% by variance or coefficient of variation.  A large percentage of
>> >  genes are not even expressed in a given tissue type and another large
>> >  percentage do not vary across a sample set.  You can use the
>> >  genefilter package to perform such filtering.
>> >
>> >  Sean
>> >
>> >  On Wed, Sep 7, 2011 at 5:29 PM, January Weiner<january.weiner at gmail.com>
>> >  wrote:
>>> >  >  Hello,
>>> >  >
>>> >  >  I'm struggling with co-expression analysis, and for that I would like
>>> >  >  to try to cluster all the genes I have in my microarray set, including
>>> >  >  those which are not differentially expressed between the study groups.
>>> >  >  I am using CoXpress at the moment and will try my luck with GSCA as
>>> >  >  well, but both packages seem to have been layed out for 3000 rather
>>> >  >  than 30000 genes.
>>> >  >
>>> >  >  How do you do that in R? I get errors about R not being able to
>>> >  >  allocate enough memory. Clearly, the amount of memory required to
>>> >  >  calculate all correlations the simple way might be a bit on the large
>>> >  >  side, but I can think of one or two tricks to get this done; I wonder
>>> >  >  whether it has been implemented already.
>>> >  >
>>> >  >  Other than that -- how should I reasonably limit the number of genes
>>> >  >  to study? i don't want to bias the outcome of the analysis by
>>> >  >  selecting only genes that are DE, actually -- I would be very
>>> >  >  interested in genes that  show differential co-expression, but no
>>> >  >  differences in expression.
>>> >  >
>>> >  >  Kind regards,
>>> >  >
>>> >  >  j.



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.