[R-sig-Geo] memory Usage setting

[Agustin Lobo]

Any particular advice for setting up the kernel
(or other things) for such a machine (i.e., the PAE kernel)?

Agus

Edzer J. Pebesma escribió:
> I think R will never do it's own memory swapping, as that is a typical 
> OS task. There are however several developments (provided in add-on 
> packages) that will not load all data in memory at start-up, but instead 
> call some data base whenever a data element is needed. You might search 
> r-help for rsqlite or biglm, and there are others; also look at the 
> award winners at useR this year.
> 
> Here, we've run pretty successful R sessions needing 10-11 Gb of memory 
> on a 8Gb RAM 64 bits linux machine with lots of swap space. Needs some 
> patience, and still R might crash other parts of the system when memory 
> usage becomes too excessive.
> 
> Best regards,
> --
> Edzer
> 
> Didier Leibovici wrote:
>> Thanks Roger
>>
>> I feel we've got a low RAM machine which would need a bit of an uplift 
>> (recent server though)!
>> The linux machine is unfortunately also with 4Gb of RAM
>> But  I persist to say it would be interesting to have within R a way of 
>> automatically performing swapping memory if needed ...
>>
>> Didier
>>
>> Roger Bivand wrote:
>>   
>>> On Tue, 11 Sep 2007, elw at stderr.org wrote:
>>>
>>>     
>>>>> These days in GIS on may have to manipulate big datasets or arrays.
>>>>>
>>>>> Here I am on WINDOWS I have a 4Gb
>>>>> my aim was to have an array of dim 298249 12 10 22 but that's 2.9Gb
>>>>>         
>>> Assuming double precision (no single precision in R), 5.8Gb.
>>>
>>>     
>>>> It used to be (maybe still is?) the case that a single process could 
>>>> only
>>>> 'claim' a chunk of max size 2GB on Windows.
>>>>
>>>>
>>>> Also remember to compute overhead for R objects... 58 bytes per 
>>>> object, I
>>>> think it is.
>>>>
>>>>
>>>>       
>>>>> It is also strange that once a dd needed 300.4Mb and then 600.7Mb 
>>>>> (?) as
>>>>> also I made some room in removing ZZ?
>>>>>         
>>>> Approximately double size - many things the interpreter does involve
>>>> making an additional copy of the data and then working with *that*.  
>>>> This
>>>> might be happening here, though I didn't read your code carefully enough
>>>> to be able to be certain.
>>>>
>>>>
>>>>       
>>>>> which I don't really know if it took into account as the limit is
>>>>> greater than the physical RAM of 4GB. ...?
>>>>>         
>>>> :)
>>>>
>>>>       
>>>>> would it be easier using Linux ?
>>>>>         
>>>> possibly a little bit - on a linux machine you can at least run a PAE
>>>> kernel (giving you a lot more address space to work with) and have the
>>>> ability to turn on a bit more virtual memory.
>>>>
>>>> usually with data of the size you're trying to work with, i try to 
>>>> find a
>>>> way to preprocess the data a bit more before i apply R's tools to it.
>>>> sometimes we stick it into a database (postgres) and select out the bits
>>>> we want our inferences to be sourced from.  ;)
>>>>
>>>> it might be simplest to just hunt up a machine with 8 or 16GB of 
>>>> memory in
>>>> it, and run those bits of the analysis that really need memory on that
>>>> machine...
>>>>       
>>> Yes, if there is no other way, a 64bit machine with lots of RAM would 
>>> not be so contrained, but maybe this is a matter of first deciding why 
>>> doing statistics on that much data is worth the effort? It may be, but 
>>> just trying to read large amounts of data into memory is perhaps not 
>>> justified in itself.
>>>
>>> Can you tile or subset the data, accumulating intermediate results? 
>>> This is the approach the biglm package takes, and the R/GDAL interface 
>>> also supports subsetting from an external file.
>>>
>>> Depending on the input format of the data, you should be able to do 
>>> all you need provided that you do not try to keep all the data in 
>>> memory. Using a database may be a good idea, or if the data are 
>>> multiple remote sensing images, subsetting and accumulating results.
>>>
>>> Roger
>>>
>>>     
>>>> --e
>>>>
>>>> _______________________________________________
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo at stat.math.ethz.ch
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>
>>>>       
>>
>>
> 
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> 

-- 
Dr. Agustin Lobo
Institut de Ciencies de la Terra "Jaume Almera" (CSIC)
LLuis Sole Sabaris s/n
08028 Barcelona
Spain
Tel. 34 934095410
Fax. 34 934110012
email: Agustin.Lobo at ija.csic.es
http://www.ija.csic.es/gt/obster