[R] maximum matrix size if it runs with 64-bit R

[David Winsemius]

On Oct 5, 2010, at 11:17 PM, Carrie Li wrote:

> Thank you, Henrik! That makes more sense now.
> You mentioned that every double value needs 8 bytes. So, in R, how  
> many
> decimal point, or any number smaller than, say 10^4 are considered  
> as double
> value ? (Sorry I don't have any C or Java language background, and  
> couldn't
> find it for R. )

You can specify an integer mode for a vector by using an "L" after the  
digits used in its definition or by using mode = integer in the call  
to the vector function. You can test for integer statsus with  
is.integer and coerce to integer status with as.integer. IN all other  
instances you should assume that the storage mode is double. The  
storage mode does not vary by element, and an integer vector can  
quickly be coerced to double if you make an assignment of double mode.

 > a <- c(1L, 2L, 3L)
 > a
[1] 1 2 3
 > is.integer(a)
[1] TRUE
 > a[3] <- 1.2
 > is.integer(a)
[1] FALSE


After coercion to double the vector will now take up the full 8 bytes  
(plus overhead) per element, but is rather set at the vector level:

 > b <- rep(1L, 100)
 > object.size(b)
440 bytes
 > b[100] <- 1.2    # change single element to double.
 > object.size(b)
840 bytes

-- 
David.



>
> I appreciate your explanation and helps!
>
> On Tue, Oct 5, 2010 at 8:32 PM, Henrik Bengtsson  
> <hb at stat.berkeley.edu>wrote:
>
>> On Tue, Oct 5, 2010 at 5:21 PM, Carrie Li <carrieandstat at gmail.com>  
>> wrote:
>>> I am sorry that it has been couple days.
>>> I've read the website you provided below, but still don't quite  
>>> know if
>> this
>>> is doable.
>>> The maximum vector length is 2^31-1, so here is what I tired, and it
>>> returned errors as below.
>>>
>>>
>>> P=20000
>>> D=matrix(rep(0, P*P), nrow=P)
>>>
>>> Error: cannot allocate vector of size 1.5 Gb
>>> In addition: Warning messages:
>>> 1: In as.vector(data) :
>>> Reached total allocation of 1535Mb: see help(memory.size)
>>>
>>> On the manuals, it says "32-bit OSes by default limit file sizes  
>>> to 2GB
>> ",
>>> so why P=20000 is not working here ?
>>
>> Every double value (e.g. 0) needs 8 bytes.  So the total memory  
>> needed
>> for that matrix is 8*P*P = 3.2e+09 bytes = 3.2e+09/1024^3 Gb =  
>> 2.98Gb.
>> Now, in order to do anything useful you also need space for creating
>> an internal copy or two of that object.  That is, you basically need
>> 2-3 times more *free* (and *contiguous/non-fragmented*) RAM than that
>> to do anything useful.
>>
>> /Henrik
>>
>>>
>>> Thanks for any helps. I appreciate.
>>>
>>>
>>> On Sat, Oct 2, 2010 at 12:22 PM, Marc Schwartz  
>>> <marc_schwartz at me.com>
>> wrote:
>>>
>>>> On Oct 2, 2010, at 11:14 AM, Carrie Li wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> If I run on a 64-bit R, what is the maximum matrix size that it  
>>>>> can
>>>> handle ?
>>>>> Is a matrix 20,000 x 20,000 possible on 32 bit ?
>>>>> Thanks for answering!
>>>>
>>>>
>>>> A matrix is a vector with 'dim' attributes. The maximum vector  
>>>> length is
>>>> 2^31 - 1 and that does not change between 32 and 64 bit R. The  
>>>> primary
>>>> advantage of 64 bit R is the larger memory address space.
>>>>
>>>> See:
>>>>
>>>>
>>>>
>> http://cran.r-project.org/doc/manuals/R-admin.html#Choosing-between-32_002d-and-64_002dbit-builds
>>>>
>>>> HTH,
>>>>
>>>> Marc Schwartz
>>>>
>>>>
>>>
>>>       [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius, MD
West Hartford, CT