[BioC] multtest MTP: Error: cannot allocate vector

[Shtatland, Timur]

Dear all,

I am looking for differentially expressed genes using multtest
package, MTP procedure. I am computing raw and adjusted t-test P
values (group 1 = 7 samples, group 2 = 3 samples) using bootstrap. R
is running out of memory ('Error: cannot allocate vector') when the
number of genes (5413) combined with the number of bootstrap
iterations (B=10000) produce a large matrix. I am running R on
PowerBook G4, Mac OS 10.4.11. See the code and the output below.

Are there any other possible solutions, except for 1-3 listed below?
For example, can I enable only part of the null matrix, rather than
the entire matrix, to be held in RAM at any given time?

The other possible solutions to this problem are:

1. Buy more memory - my choice #1 unless another solution is easily
available. I assume that MTP is trying to read the entire null matrix
into memory: 10000 iterations * 5413 genes * 10 samples * 8
bytes/(sample*gene*iteration) = 4.3 GB, and it obviously does not fit
into the current RAM (1.25 GB).

2. Use fewer bootstrap iterations - my choice #2, because with fewer
iterations many genes have raw P values equal to exactly 0:
https://stat.ethz.ch/pipermail/bioconductor/2008-March/021396.html
https://stat.ethz.ch/pipermail/bioconductor/2008-March/021436.html

3. Use fewer genes - my choice #3, because it is not clear exactly
what effect a more restrictive filter will have on the false positive
rate (the rate of truly differentially expressed genes that will be
filtered out). Currently I already use genefilter to reduce the
number of genes for MTP input from 22283 to 5413:

ffun <- filterfun(pOverA(p = 0.5, A = 100), cv(a = 0.3))

I was not running any process other than R at the time of the error,
to maximize available memory. The error occurs always at the end of
the bootstrap iterations (which take 18-24 hours).

I searched the MTP help page and its vignettes, as well as
Bioconductor mailing list archives. The solution 'buy more memory'
appears to be the most commonly suggested on this mailing list for
other assorted 'out of memory' problems, but I was wondering if there
is an easy way around it.

Thank you for your help.

Best regards,

Timur
--
Timur Shtatland, PhD
Center for Molecular Imaging Research
Massachusetts General Hospital
149 13th Street, Room 5408
Charlestown, MA 02129
tshtatland at mgh dot harvard dot edu

############################################################

## read *only* the gcrma-processed dataset (nothing else) into a new R session 
## to reduce the number of objects in memory:
> load("esetGcrma.rda")
...
> B=10000
...
> ffun <- filterfun(pOverA(p = 0.5, A = 100), cv(a = 0.3))
> filtered <- genefilter(2^exprs(esetGcrma), ffun)
...
> TTBoot <- MTP(X=esetGcrmaExprsFiltered, Y=TT, test = "t.twosamp.unequalvar",
alternative = "two.sided", typeone="fdr", method="ss.maxT",
fdr.method="conservative", keep.nulldist = FALSE, B=B, seed=seed)
running bootstrap...
iteration = 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500
1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100
3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700
4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300
6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900
8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500
9600 9700 9800 9900 10000

Error: cannot allocate vector of size 413.0 Mb
R(202,0xa000ed88) malloc: *** vm_allocate(size=433041408) failed (error code=3)
R(202,0xa000ed88) malloc: *** error: can't allocate region
R(202,0xa000ed88) malloc: *** set a breakpoint in szone_error to debug
2008-03-08 18:56:10.979 R[202] tossing reply message sequence 2 on thread
0x4d2a110
> 
> traceback()
7: apply(null, 2, max)
6: ss.maxT(nulldistn, obs, alternative, get.cutoff, get.cr, pind, 
       alpha)
5: MTP(X = esetGcrmaExprsFiltered, Y = TT, test = "t.twosamp.unequalvar", 
       alternative = "two.sided", typeone = "fdr", method = "ss.maxT", 
       fdr.method = "conservative", keep.nulldist = FALSE, B = B, 
       seed = seed)
4: multTest(esetGcrma = esetGcrma, B = 10000)
3: eval.with.vis(expr, envir, enclos)
2: eval.with.vis(ei, envir)
1: source("~/bin/computeRestrictedMa2.R")
> sessionInfo()
R version 2.6.0 (2007-10-03) 
powerpc-apple-darwin8.10.1 

locale:
en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] splines   tools     stats     graphics  grDevices utils     datasets
methods   base     

other attached packages:
[1] multtest_1.18.0   genefilter_1.16.0 survival_2.32     Biobase_1.16.1   

loaded via a namespace (and not attached):
[1] AnnotationDbi_1.0.6 DBI_0.2-4           RSQLite_0.6-4       annotate_1.16.1

> version
               _                           
platform       powerpc-apple-darwin8.10.1  
arch           powerpc                     
os             darwin8.10.1                
system         powerpc, darwin8.10.1       
status                                     
major          2                           
minor          6.0                         
year           2007                        
month          10                          
day            03                          
svn rev        43063                       
language       R                           
version.string R version 2.6.0 (2007-10-03)
> .Machine$sizeof.pointer == 8
[1] FALSE
> R.version$arch
[1] "powerpc"
> .Platform$r_arch
[1] "ppc"

The information transmitted in this electronic communica...{{dropped:16}}