[R] maximum matrix size if it runs with 64-bit R

Thu Oct 7 01:01:09 CEST 2010

On Oct 6, 2010, at 6:08 PM, Carrie Li wrote:

> Thanks a lot!!
>
> Any integer is single element,

Not the way I would have expressed it. An integer is a 4 byte number  
(range 2^31 + sign) and a double has a 8 byte  floating point  
representation (probably specified by some IEEE designation that I  
cannot find in the three manual i just searched.) . Numeric vectors  
can have any number of elements up to the range of the integer mode  
and can be either of mode "integer" or "double". The same maximum  
vector length exists for 32 bit and 64 bit versions of R

> whereas any number with decimal point is double element ? Is that  
> right ?

The second part is true, but many double numbers will still be  
displayed without decimal points so that is not a good test.

>
> below is what I tried
> so, basically, simply force it to be integer would save some space ?

It would be at considerable loss of accuracy if the underlying data is  
not well represented as integers.

-- 
David
>
> P=10000
>
> a=rnorm(P*P)*1000
> a=as.integer(a)
> D1=matrix(a, nrow=P)
> object.size(D1)
> >4000112 bytes
>
>
> b=rnorm(P*P)*1000
> D2=matrix(b, nrow=P)
> object.size(D2)
> >8000112 bytes
>

>
> On Wed, Oct 6, 2010 at 5:44 AM, David Winsemius <dwinsemius at comcast.net 
> > wrote:
>
> On Oct 5, 2010, at 11:17 PM, Carrie Li wrote:
>
> Thank you, Henrik! That makes more sense now.
> You mentioned that every double value needs 8 bytes. So, in R, how  
> many
> decimal point, or any number smaller than, say 10^4 are considered  
> as double
> value ? (Sorry I don't have any C or Java language background, and  
> couldn't
> find it for R. )
>
> You can specify an integer mode for a vector by using an "L" after  
> the digits used in its definition or by using mode = integer in the  
> call to the vector function. You can test for integer statsus with  
> is.integer and coerce to integer status with as.integer. IN all  
> other instances you should assume that the storage mode is double.  
> The storage mode does not vary by element, and an integer vector can  
> quickly be coerced to double if you make an assignment of double mode.
>
> > a <- c(1L, 2L, 3L)
> > a
> [1] 1 2 3
> > is.integer(a)
> [1] TRUE
> > a[3] <- 1.2
> > is.integer(a)
> [1] FALSE
>
>
> After coercion to double the vector will now take up the full 8  
> bytes (plus overhead) per element, but is rather set at the vector  
> level:
>
> > b <- rep(1L, 100)
> > object.size(b)
> 440 bytes
> > b[100] <- 1.2    # change single element to double.
> > object.size(b)
> 840 bytes
>
> -- 
> David.
>
>
>
>
>
> I appreciate your explanation and helps!
>
> On Tue, Oct 5, 2010 at 8:32 PM, Henrik Bengtsson  
> <hb at stat.berkeley.edu>wrote:
>
> On Tue, Oct 5, 2010 at 5:21 PM, Carrie Li <carrieandstat at gmail.com>  
> wrote:
> I am sorry that it has been couple days.
> I've read the website you provided below, but still don't quite know  
> if
> this
> is doable.
> The maximum vector length is 2^31-1, so here is what I tired, and it
> returned errors as below.
>
>
> P=20000
> D=matrix(rep(0, P*P), nrow=P)
>
> Error: cannot allocate vector of size 1.5 Gb
> In addition: Warning messages:
> 1: In as.vector(data) :
> Reached total allocation of 1535Mb: see help(memory.size)
>
> On the manuals, it says "32-bit OSes by default limit file sizes to  
> 2GB
> ",
> so why P=20000 is not working here ?
>
> Every double value (e.g. 0) needs 8 bytes.  So the total memory needed
> for that matrix is 8*P*P = 3.2e+09 bytes = 3.2e+09/1024^3 Gb = 2.98Gb.
> Now, in order to do anything useful you also need space for creating
> an internal copy or two of that object.  That is, you basically need
> 2-3 times more *free* (and *contiguous/non-fragmented*) RAM than that
> to do anything useful.
>
> /Henrik
>
>
> Thanks for any helps. I appreciate.
>
>
> On Sat, Oct 2, 2010 at 12:22 PM, Marc Schwartz <marc_schwartz at me.com>
> wrote:
>
> On Oct 2, 2010, at 11:14 AM, Carrie Li wrote:
>
> Hi everyone,
>
> If I run on a 64-bit R, what is the maximum matrix size that it can
> handle ?
> Is a matrix 20,000 x 20,000 possible on 32 bit ?
> Thanks for answering!
>
>
> A matrix is a vector with 'dim' attributes. The maximum vector  
> length is
> 2^31 - 1 and that does not change between 32 and 64 bit R. The primary
> advantage of 64 bit R is the larger memory address space.
>
> See:
>
>
>
> http://cran.r-project.org/doc/manuals/R-admin.html#Choosing-between-32_002d-and-64_002dbit-builds
>
> HTH,
>
> Marc Schwartz
>
>
>
>      [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius, MD
> West Hartford, CT
>
>

David Winsemius, MD
West Hartford, CT