[R] Memory in R on windows running 64bit XP

[Duncan Murdoch]

On 9/16/2009 10:54 AM, michael watson (IAH-C) wrote:
> Hi Duncan
> 
> Many thanks for the reply.  Here are the results of the commands:
> 
>> Rprofmem() 
> Error in Rprofmem() : memory profiling is not available on this system
>> memory.size(NA)
> [1] 3500
>> memory.size() 
> [1] 16.19
> 
> 
> Clearly Rprofmem() is a dead end, though if there is some way of enabling it, I will be happy to.

I forgot that you need to recompile R to enable it.  Not impossible, but 
probably not worth the trouble just for this problem.  Using 
gcinfo(TRUE) may give you enough information:  it will report whenever 
there's a garbage collection.

> I am only actually using 16.19Mb of memory, as all I do is open R and read in some data (a 2.7Mb text file).

I would guess that mclust does a lot of other allocations before it fails.

> I am not sure how to guard against the memory fragmentation problem.  The machine has 16Gb of RAM and I am only running R and Internet Explorer, but I don't know if that helps.  Is there a way to deliver contiguous memory to R?

R doesn't give you any real control over that.  I would guess that R is 
getting 3.5 Gb in a contiguous chunk based on your memory.size report, 
but then it is fragmenting it.

> Is 64bit R available for Windows?  Sorry to bang on about Windows, I'm waiting for my IT department to build Linux machines, and this is what I have in the meantime.

Revolution Computing sells one.  There's no (reliable) 64 bit version of 
gcc for Windows, so we don't distribute one.

Duncan Murdoch

> 
> Thanks
> Mick
> 
> ________________________________________
> From: Duncan Murdoch [murdoch at stats.uwo.ca]
> Sent: 16 September 2009 15:43
> To: michael watson (IAH-C)
> Cc: r-help at r-project.org
> Subject: Re: [R] Memory in R on windows running 64bit XP
> 
> On 9/16/2009 10:16 AM, michael watson (IAH-C) wrote:
>> Hi
>> I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states:
>>
>> If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB.
>>
>> So, using the help which states the size parameter can go up to 4095:
>>
>> memory.limit(size=4095)
>>
>> When I run mclust I get:
>>
>> Error: cannot allocate vector of size 1.3 Gb
>>
>> So, now I set the max-mem-size flag:
>>
>> --max-mem-size=3500M
>>
>> Repeat and I still get the same error:
>>
>> Error: cannot allocate vector of size 1.3 Gb
>>
>> So, can anyone help?  32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb.
> 
> You are misreading things in a couple of places.  The most important one
> is the error message: it says allocation of a 1.3 Gb object failed.  You
> may have 4 Gb available in total, but have used so much of it already
> that Windows won't allocate a 1.3 Gb piece to R.  Using Rprofmem() might
> show how the allocations are happening.
> 
> Another possibility is that your particular version of Windows doesn't
> support giving 4 Gb to R.  Run memory.size(NA) to find out the limit,
> memory.size() to find out the current amount in use.
> 
> And even if the limit is close to 4 Gb and you have more than 1.3 Gb
> free, the allocation may fail because there is no sufficiently large
> contiguous block available.  Once memory is allocated, that address is
> in use and is unavailable.  Since the 32 bit address space only covers 4
> Gb, you can fairly easily end up with allocations that are spread out
> across it leaving no large blocks available.  That's the advantage of
> switching to 64 bit R:  even if you still only had 4 Gb of memory
> available, the address space is so much bigger that fragmentation
> probably won't be a problem.
> 
> Duncan Murdoch
> 
>>
>>> sessionInfo()
>> R version 2.9.2 (2009-08-24)
>> i386-pc-mingw32
>> locale:
>> LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>> other attached packages:
>> [1] mclust_3.3.1
>>
>> Thanks
>> Mick
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.