The thing is I am using a function a recursive algorithm on multi array 
which could probably be rewritten the way you are describing (subsetting 
via a database for storage)
but ... that would have to be rewritten!

thanks again I thanks to the other comments as well
Didier

Roger Bivand wrote:
> On Thu, 13 Sep 2007, Didier Leibovici wrote:
>
>>
>> Thanks Roger
>>
>> I feel we've got a low RAM machine which would need a bit of an 
>> uplift (recent server though)!
>> The linux machine is unfortunately also with 4Gb of RAM
>> But  I persist to say it would be interesting to have within R a way 
>> of automatically performing swapping memory if needed ...
>
> On many OS, virtual memory will cut in at a certain point, but here 
> the object is in any case too large to represent on a 32-bit system, 
> and indeed R sets a limit on single object size of roughly 2Gb - for 
> full details see:
>
> ?"Memory-limits"
>
> However, the real question remains why you cannot save time by 
> subsetting first, which is equivalent to swapping to virtual memory, 
> but under your own control. There were some comments on this in my 
> earlier reply, and other replies make similar suggestions. You still 
> have not said why all the data must be in memory at the same time, and 
> I strongly doubt that there are no viable or even superior alternatives.
>
> Roger
>
>>
>> Didier
>>
>> Roger Bivand wrote:
>>>  On Tue, 11 Sep 2007, elw@stderr.org wrote:
>>>
>>> > > >  These days in GIS on may have to manipulate big datasets or 
>>> arrays.
>>> > > > >  Here I am on WINDOWS I have a 4Gb
>>> > >  my aim was to have an array of dim 298249 12 10 22 but that's 
>>> 2.9Gb
>>> >
>>>  Assuming double precision (no single precision in R), 5.8Gb.
>>>
>>> > >  It used to be (maybe still is?) the case that a single process 
>>> could >  only
>>> >  'claim' a chunk of max size 2GB on Windows.
>>> > > >  Also remember to compute overhead for R objects... 58 bytes 
>>> per object, >  I
>>> >  think it is.
>>> > > > >  It is also strange that once a dd needed 300.4Mb and then 
>>> 600.7Mb (?) > >  as
>>> > >  also I made some room in removing ZZ?
>>> > > >  Approximately double size - many things the interpreter does 
>>> involve
>>> >  making an additional copy of the data and then working with 
>>> *that*. >  This
>>> >  might be happening here, though I didn't read your code carefully 
>>> enough
>>> >  to be able to be certain.
>>> > > > >  which I don't really know if it took into account as the 
>>> limit is
>>> > >  greater than the physical RAM of 4GB. ...?
>>> > > :) > > >  would it be easier using Linux ?
>>> > >  possibly a little bit - on a linux machine you can at least run 
>>> a PAE
>>> >  kernel (giving you a lot more address space to work with) and 
>>> have the
>>> >  ability to turn on a bit more virtual memory.
>>> > >  usually with data of the size you're trying to work with, i try 
>>> to find >  a
>>> >  way to preprocess the data a bit more before i apply R's tools to 
>>> it.
>>> >  sometimes we stick it into a database (postgres) and select out 
>>> the bits
>>> >  we want our inferences to be sourced from.  ;)
>>> > >  it might be simplest to just hunt up a machine with 8 or 16GB 
>>> of memory >  in
>>> >  it, and run those bits of the analysis that really need memory on 
>>> that
>>> >  machine...
>>>
>>>  Yes, if there is no other way, a 64bit machine with lots of RAM 
>>> would not
>>>  be so contrained, but maybe this is a matter of first deciding why 
>>> doing
>>>  statistics on that much data is worth the effort? It may be, but just
>>>  trying to read large amounts of data into memory is perhaps not 
>>> justified
>>>  in itself.
>>>
>>>  Can you tile or subset the data, accumulating intermediate results? 
>>> This
>>>  is the approach the biglm package takes, and the R/GDAL interface also
>>>  supports subsetting from an external file.
>>>
>>>  Depending on the input format of the data, you should be able to do 
>>> all
>>>  you need provided that you do not try to keep all the data in memory.
>>>  Using a database may be a good idea, or if the data are multiple 
>>> remote
>>>  sensing images, subsetting and accumulating results.
>>>
>>>  Roger
>>>
>>> > >  --e
>>> >
>>>> _______________________________________________
>>> >  R-sig-Geo mailing list
>>> >  R-sig-Geo@stat.math.ethz.ch
>>> >  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>> >
>>
>>
>>
>


-- 
Dr Didier Leibovici    
	http://www.nottingham.ac.uk/cgs/leibovici.shtml
Centre for Geospatial Science     
Sir Clive Granger Building
University of Nottingham,
University Park
Nottingham NG7 2RD, UK
	Tel: +44 - (0)115 84 66058   Fax: +44 (0)115 95 15249


	[[alternative HTML version deleted]]


