[Bioc-devel] Memory limits for individual objects

[Tim Rayner]

Hi,

I'm running R 2.9.0 on Mac OS X, and attempting to import a large
number of SNP calls into a snp.matrix object as supported by the
snpMatrix package. I've run into a problem where I'd like the final
matrix object to contain around 5e9 elements, but of course the
maximum vector (and matrix) size in R is 2^31-1 (approx. 2e9), and I
get the dreaded "allocMatrix: too many elements specified" error
message. An obvious workaround is to split the analysis up into parts
which will fit within this limit, but I feel I should ask whether
there's a better way. I'm using a 64-bit build of R and I was
wondering whether anyone had experience changing the indexing of R
vectors and matrices from signed 32-bit integers to signed 64-bit
integers? Or should I just head over to R-devel directly (in which
case, apologies for the mispost)?

There was a not-terribly-helpful exchange regarding this question on
the main Bioconductor list last year:

http://www.nabble.com/allocMatrix-limits-td18763791.html

Since the snpMatrix package stores each element as type 'raw', the
final memory consumption for the snp.matrix object should only be a
handful of gigabytes, readily available in modern desktop computers.
It would be nice to be able to use that memory.

Many thanks,

Tim Rayner


Bioinformatician - Smith Lab
Cambridge Institute for Medical Research
University of Cambridge