[R] memory

[Sean O'Riordain]

Ferran,

What are you trying to do with such a large matrix?  with 7e9 cells
and a linear algorithm which is quite unlikely, your problem solution
is likely to take a "very long time"(tm)... just quickly... at one
micro-second per operation (very optimistic?) and 7e9 operations,
thats
> 7e9/1e6/60
[1] 116.6667
minutes...

if we're doing something a little more complicated than linear, say
O(n^2.5) on a square matrix of 7e9 cells, then we're talking
> (7e9^.5)^2.5/1e6/60
[1] 33745.92
minutes...

As Brian Ripley said, if you really want to to this then you must use
another operating system which can handle more than 32-bit addressing,
one such would be linux running and built for a 64-bit platform - of
which there are a few.

cheers!
Sean


On 30/08/05, Ferran Carrascosa <ferran.carrascosa at gmail.com> wrote:
> Thanks Prof Brian for your answers,
> I have read about 'ref' package to work with more efficient memory
> work. Anybody know if this package could help me to work with a
> 700.000 x 10.000 matrix?
> 
> I will have problems with ref package on:
> - Limit of 2 Gb in R for Windows.
> -The maximum cells in one object 2*10^9 (aprox.)
> 
> Thanks in advance,
> --
> Ferran Carrascosa
> 
> 
> 2005/8/30, Prof Brian Ripley <ripley at stats.ox.ac.uk>:
> > On Mon, 29 Aug 2005, Ferran Carrascosa wrote:
> >
> > > Hi,
> > >
> > > I have a matrix with 700.000 x 10.000 cells with floating point data.
> > > I would like to work with the entire table but I have a lot of memory
> > > problems. I have read the ?memory
> > > I work with Win 2000 with R2.1.0
> > >
> > > The only solution that I have applied is:
> > >> memory.limit(size=2048)
> > >
> > > But now my problems are:
> > > - I need to work with more than 2 Gb. How I can exceed this limit?
> >
> > Re-read the rw-FAQ, or (preferably) get a more capable OS on a 64-bit CPU.
> >
> > > - When apply some algorithms, the maximum cells in one object 2*10^9
> > > (aprox.) is reached.
> >
> > You will never get that many cells (that is the address space in bytes,
> > and they are several bytes each).  Please do as the posting guide asks
> > and report accurately what happened.
> >
> > > Please could you send me some advises/strategies about the work with
> > > large amount of data in R?
> > >
> > > R have a way to work with less memory needs?
> >
> > Your matrix has 7e09 cells (assuming you are using . as a thousands
> > separator) and needs 5.6e10 bytes to store.  Your OS has a memory address
> > limit of 3.2e09 bytes.  Don't blame R for being limited by your OS.
> >
> > --
> > Brian D. Ripley,                  ripley at stats.ox.ac.uk
> > Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> > University of Oxford,             Tel:  +44 1865 272861 (self)
> > 1 South Parks Road,                     +44 1865 272866 (PA)
> > Oxford OX1 3TG, UK                Fax:  +44 1865 272595
> >
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>