[R-sig-Geo] memory issue on R with Linux 64

[Roger Bivand]

On Thu, 29 Jan 2009, Robert Hijmans wrote:

> Herry,
>
> This is how you can do it in package{raster}. (revision 209;  a build
> should be available within 24 hours).
>
> Following Edzer's example:
>
> require(raster)
> library(maptools)
> # read a SpatialPolygonsDataFrame
> nc <- readShapePoly(system.file("shapes/sids.shp",
>      package="maptools")[1], proj4string=CRS("+proj=longlat +datum=NAD27"))
> # create a new RasterLayer object from the polygon bounding box and
> set row / cols.
> rs <- rasterFromBbox(nc, nrows=54, ncols=176)
> # transfer polygons to raster, use values of column 13 in the polygon dataframe
> rs <- polygonsToRaster(nc, rs, field=13)
> # plot, either directly
> plot(rs)
> plot(nc, add=T, border="blue")
> # or via sp for a real map
> x11()
> spplot(asSpGrid(rs), col.regions=bpy.colors())
>
> also see the example in ?polygonsToRaster
>
> The polygonsToRaster function works for very large rasters (row by row
> processing if you provide an output file name). I have only tested if
> for a very limited number of very simple cases, so user beware...
> The algorithm needs optimization for speed, so that might be a problem
> for your very large grids. (particularly of your polygons are also
> complex). It also needs some tweaking (and options) for deciding when
> a polygon is in a grid cell. As is, the intention is that a polygon
> has to overlap with the center of a cell to be considered inside.

It would be very useful to compare the output of this procedure, which 
looks very promising, with Starspan:

http://starspan.casil.ucdavis.edu/doku/

to see which subpixel heuristics may be helpful.

Roger


>
> Robert
>
>
>
>
> On Thu, Jan 29, 2009 at 3:09 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote:
>> On Thu, 29 Jan 2009, Alexander.Herr at csiro.au wrote:
>>
>>>
>>> Hi List,
>>>
>>> I get an error using readGDAL{rgdal}: cannot allocate vector of size 3.1
>>> Gb
>>
>> This is a tile of your 73K by 80K raster, right? One possibility is to use
>> smaller tiles, another to get more memory (as Edzer wrote), a third to use
>> lower level functions in rgdal to avoid duplication (and repeated gc()) - in
>> readGDAL the data read by getRasterData() are copied, so at least doubling
>> memory usage.
>>
>> Do you need to read the raster? If this is the overlay problem, you should
>> be able to use the output of GDALinfo for your raster to build a
>> GridTopology and SpatialGrid, and overlay (tiles of) that on the
>> SpatialPolygons (tiles of that because overlay() will generate cell centre
>> coordinates and do point in polygon, so you're still stuck with too many
>> coordinates). The next issue would be to copy out the polygon IDs, or the
>> extracted values, as a raster - here the forthcoming raster package on
>> R-Forge may be what you need.
>>
>> Roger
>>
>>
>>>
>>> I am using Linux 64bit (opensuse 11) with 4 gig swap and 4 gig Ram and R
>>> 2.8.0.
>>>
>>> The load monitor shows that most of Ram is used up and then when Swap use
>>> starts increasing, R returns the error.
>>>
>>> Is there anything I should do within R to circumvent this?
>>>
>>>
>>> Any help appreciated
>>> Thanks
>>> Herry
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>> --
>> Roger Bivand
>> Economic Geography Section, Department of Economics, Norwegian School of
>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>> e-mail: Roger.Bivand at nhh.no
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at stat.math.ethz.ch
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no