[R-sig-Geo] where to tweak memory allocation settings

[Roger Bivand]

On Thu, 8 Oct 2009, Jochen Albrecht wrote:

> Fellow spatial statisticians:
> I am trying to align a bunch of world-wide datasets at 0.1 degree resolution. 
> I figured that the easiest way to do that is to create a dataset where the 
> rownames form the index that allows to relate all my variable-sized point 
> datasets to. I therefore created a shapefile base dataset that contains appr. 
> 6.5 million points and had hoped that I could read this into R to then link 
> all my other data to. Unfortunately, I ran into memory allocation problems 
> (see error messages attached to this email).

You have not reported sessionInfo() or your operating system (Windows?). 
See the R for Windows FAQ for memory management there. In fact the problem 
isn't the number of points (here 3.5 million, presumably the raster cells 
with data, not 6.5 million for global coverage), it is 
likely the combination of character columns and their conversion into 
factor form - do they contain long character strings?

> My hunch is that these can be overcome by just changing some settings 
> (hopefully without having to recompile R sources). Do you have suggestions 
> what these are and what the practically hard limit is? 6.5 million points is 
> large but not uncommon these days, so I figured that this should be doable 
> without embarking on major efforts.
> Alternatively, would it save me a lot of memory space if I tried to read this 
> into a SpatialPixel or SpatialGrid structure?

To read into a SpatialPixelDataFrame object, you still need points, or go 
through a SpatialGridDataFrame. For a SpatialGridDataFrame, store the data 
as a multiband GeoTiff for example. If however there is something odd - 
long character strings as attributes, the problem will be the same. One 
advantage of going through a SpatialGridDataFrame is that the 3.5 million 
row names fo not get generated and do not take up space - readOGR() always 
generates feature IDs from the input geometry FIDs. If the support is grid 
support, use a gridded representation, as in Robert's suggestion for 
using the raster package.

Hope this helps,

Roger

> Cheers,
>    Jochen
>
>> basepoints = readOGR(".", "basepoints")
> OGR data source with driver: ESRI Shapefile
> Source: ".", layer: "basepoints"
> with  3465355  rows and  7  columns
> Feature type: wkbPoint with 2 dimensions
> Warning in data.frame(dlist) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in data.frame(dlist) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in data.frame(dlist) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in data.frame(dlist) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in as.data.frame.integer(x[[i]], optional = TRUE) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in as.data.frame.integer(x[[i]], optional = TRUE) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in as.data.frame.integer(x[[i]], optional = TRUE) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Warning in as.data.frame.integer(x[[i]], optional = TRUE) :
> Reached total allocation of 1535Mb: see help(memory.size)
> Error: cannot allocate vector of size 13.2 Mb
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no