[R] R's memory capabilities

[Ben Bolker]

peter dalgaard <pdalgd <at> gmail.com> writes:

> 
> 
> On Feb 6, 2012, at 11:15 , Alaios wrote:
> 
> > Dear all,
> > I have tried to create a diagonal matrix of size diag(65536)
> > 

  [snip ...]
 
> Well, it means what it says. Indexing is limited by the size of a
> 32-bit integer and (2^16)^2 is too big for that. So you are limited
> to roughly 16 GB for any single object in R.  This issue has been
> foreseeable for some time. It can only be fixed by internal changes
> in the R engine, but switching to long integers has various issues,
> so it is not a straightforward modification.  Meanwhile, you might
> want to consider whether you really do need a diagonal matrix of
> that size stored in full. It's an awful lot of zeroes to carry
> around....

  What Peter is hinting at (I think) is that you depending
on your application you may be able to get away with using
sparse matrices, which only store the positions and values
of non-zero elements rather than the values of all elements.
For example:

library(Matrix)
d <- Diagonal(2^16)
str(d)

  Formal class 'ddiMatrix' [package "Matrix"] with 4 slots
  ..@ diag    : chr "U"
  ..@ Dim     : int [1:2] 65536 65536
  ..@ Dimnames:List of 2
  .. ..$ : NULL
  .. ..$ : NULL
  ..@ x       : num(0) 

object.size(d)
  652 bytes


vignette("Intro2Matrix")