[R-sig-Geo] geostatistic memory issue

[Fernando Santo]

Dear Giuseppe,

I am using 2 processor of 2 Gb.  I did not know about this problem.  Is 
there a way to allocate more memory in linux?  I believe that the answer 
is in R help ?Memory.  But I did not understand how to do it.  Starting 
R from the shell with R --min-vsize=100M --max-vsize=3G --min-nsize=100M 
--max-nsize=3G did not make difference...

Fernando
> Yes,
> in almost all "prediction" command you can apply to a tiles grids, as
> is suggested by Robert.
>
> Anyway be sure that you are using all your 4 G. If you have 2
> processor of 2 G, and you run R in a shell you will use only the 2 G
> of one processor. In this case the  multicore/parallel packages can
> help you.
> A nice presentation you can find on http://geostat2010.info/node/421
>
> Ciao
> Giuseppe
>
> On Tue, Nov 30, 2010 at 3:11 AM, Robert J. Hijmans<r.hijmans at gmail.com>  wrote:
>> Fernando,
>>
>> You can use the raster package for dealing with memory issues like this:
>>
>> library(fields)
>> fit<- Krig(ozone$x, ozone$y, theta=20)
>>
>> library(raster)
>> # create a target RasterLayer
>> r<- raster(xmn=min(ozone$x), xmx=max(ozone$x), ymn=min(ozone$y),
>> ymx=max(ozone$y))
>> res(r)<- 1/6
>> x<- interpolate(r, fit)
>> plot(x)
>> x
>>
>>
>> in your case, that would be something like this:
>> ...
>> r<- raster(xmn=723604,xmx=724904,ymn=9661020,ymx=9662348)
>> res(r)<- 4
>> ...
>>
>> Robert
>>
>> On Mon, Nov 29, 2010 at 3:54 PM, Fernando Santo<fdbesanto at gmail.com>  wrote:
>>> Dear All,
>>>
>>> I am running a geostatistical analysis using the package "fields" of
>>> n=980 and grid of approximately 1x1 km with 4 m of spatial resolution.
>>> Because the fine grid resolution of 4 meters, which I really need for my
>>> experiment, I am having problems with memory allocation.  I am using an
>>> OS linux 32 bit (i686-pc-linux-gnu) with 4 Gb of memory and 2 Gb of swap
>>> and R version 2.12.0.  Running the R command gc() in my computer, I have
>>> the following:
>>>
>>>   >gc()
>>>              used        (Mb)   gc      trigger      (Mb)      max used
>>> (Mb)
>>> Ncells   147153      4.0              350000      9.4           350000   9.4
>>> Vcells    86617       0.7              786432       6.0
>>> 434604   3.4
>>>
>>> I tried to work around of my memory issue (?Memory) for Linux, start R
>>> with the command *R --min-vsize=100M --max-vsize=3G --min-nsize=100M
>>> --max-nsize=3G*, but still I did not saved the problem of memory
>>> allocation.  I know that the solution could be: (1) install a OS 64bits
>>> or (2) reduce the spatial resolution of my grid (e.g. for 10x10 m).
>>> However, I can not format my computer right now and I need a grid of 4 x
>>> 4 m.
>>>
>>> I would like to ask you the following:
>>> 1 - How I could increase the memory allocation of my linux?  Is there
>>> something wrong with my work around of memory allocation showed above?
>>> 2 - Is possible to apply the use of the R package  e.g. "bigmemory" in
>>> my geostatistic script (please see bellow)?  If yes, how it could be?
>>> My problem is only in the last command of my script when I call the
>>> function "predict" of fields.
>>>
>>> My script is bellow:
>>>
>>> Any help will be very welcome.
>>> Thanks for your attention.
>>> Fernando
>>>
>>> ###########################################
>>> dat<- read.csv("/home/fernando/Desktop/data.csv")
>>> library(fields)
>>>
>>> # set  x-y  coordinates of image, considering the center of each pixel
>>> col_xcoord<- seq(723606,724902,by=4)         #  correct: 325
>>> row_ycoord<- seq(9661022,9662346,by=4)   #  correct: 332
>>> xmin = min(col_xcoord)
>>> ymin = min(row_ycoord)
>>> locs.x = col_xcoord-xmin
>>> locs.y = row_ycoord-ymin
>>> newlocs = expand.grid(locs.x,locs.y)
>>>
>>> locs = data.frame(dat$long-xmin,dat$lat-ymin)  # adjusted location as
>>> data frame
>>> sqrt.gf = sqrt(dat$gap.frac)                                   # The
>>> response variable
>>>
>>> #################################################
>>> # Produce the Kriging estimates using exponential variogram with scale
>>> parameter 24.15
>>> #################################################
>>>
>>> fit = Krig(locs,sqrt.gf,theta=24.15)
>>> # I used the range parameter 24.15 estimated earlier
>>>
>>> summary(fit) # summary of fit
>>>
>>> # HERE I HAD THE ERROR OF MEMORY ALLOCATION
>>> # "predict" the kriging values
>>> krig.out = predict(fit,newlocs)
>>> # Error: cannot allocate vector of size 806.7 Mb
>>>
>>> # This takes about 2 minutes on Mac of my friend
>>> # but it did not work in my linux
>>>
>>>
>>>
>>>
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>