[R-sig-Geo] memory Usage setting

[elw at stderr.org]

> These days in GIS on may have to manipulate big datasets or arrays.
>
> Here I am on WINDOWS I have a 4Gb
> my aim was to have an array of dim 298249 12 10 22 but that's 2.9Gb


It used to be (maybe still is?) the case that a single process could only 
'claim' a chunk of max size 2GB on Windows.


Also remember to compute overhead for R objects... 58 bytes per object, I 
think it is.


> It is also strange that once a dd needed 300.4Mb and then 600.7Mb (?) as 
> also I made some room in removing ZZ?


Approximately double size - many things the interpreter does involve 
making an additional copy of the data and then working with *that*.  This 
might be happening here, though I didn't read your code carefully enough 
to be able to be certain.


> which I don't really know if it took into account as the limit is 
> greater than the physical RAM of 4GB. ...?

:)

> would it be easier using Linux ?

possibly a little bit - on a linux machine you can at least run a PAE 
kernel (giving you a lot more address space to work with) and have the 
ability to turn on a bit more virtual memory.

usually with data of the size you're trying to work with, i try to find a 
way to preprocess the data a bit more before i apply R's tools to it. 
sometimes we stick it into a database (postgres) and select out the bits 
we want our inferences to be sourced from.  ;)

it might be simplest to just hunt up a machine with 8 or 16GB of memory in 
it, and run those bits of the analysis that really need memory on that 
machine...

--e