[R-sig-Geo] Memory limit problems in R / import of maps

[Roger Bivand]

On Tue, 22 Apr 2008, Tomislav Hengl wrote:

>
> Dear list,
>
> I know that much has already been said about the memory limit problems. 
> If there is any progress about this problem, we would be interested to 
> hear.
>
> In our project, we are importing 24 maps/bands, each consists of 
> 1,450,000 pixels. We further would like to glue all maps into a single 
> data frame (e.g. 'ksc' class in adehabitat package; or 
> 'SpatialGridDataFrame' in sp package), but this seems to be impossible.
>
> We tried to run this under windows (after following 
> http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-be-a-limit-on-the-memory-it-us 
> es_0021 and setting the --max-mem-size) and under Linux Ubuntu, but 
> still get the same error message (seems that there is no difference in 
> use of memory under the two OS):
>
> "Error: Cannot allocate vector of size 11.1 Mb"
>
> The R workspace with 24 loaded grids is also really small (18 MB), but 
> any further gluing and calculation is blocked due the vector size error 
> message.

No, about 11MB per grid times 24 is not 18MB, you are probably looking at 
the wrong row in gc(). A lot depends on how you are reading them, are you 
using the rgdal package, or reading ASCII grids? The question is mostly 
how to organise the workflow to reduce copying to a minimum.

>
> For a comparison, in a GIS such as ArcGIS or SAGA/ILWIS (open source) we 
> have no problems of loading and processing 3-4 times more grids.
>

Recall that these are different kinds of systems, and very likely not 
holding and copying all the data in memory.

> Should we simply give up on running spatial analysis using large grids 
> (>10 million grids) in R?

Do you mean grids with over 10M grid cells, or 10K grids each of 1M grid 
cells?

If the analyses can be done in tiles, rgdal provides access to the full 
ability of GDAL to select subscenes.

Hope this helps,

Roger

>
> Any suggestion or comment is more than welcome,
>
>
>
> Tom Hengl
> http://spatial-analyst.net
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no