[R-sig-Geo] reading ascii grid memory issues

[Roger Bivand]

On Fri, 5 Dec 2008, Alexander.Herr at csiro.au wrote:

> Thanks Barry
>>
>> Hi List,
>>
>> I am unable to read in a 7.8Gb ascii grid using readAsciiGrid 
>> {maptools}/ read.asciigrid {sp}; R runs out of memory. I have 4Gb ram 
>> and 4Gb swap, so things are getting tight.
>>
>> I am wondering if anyone has alternative options (preferably with 
>> example) that enable to read in large grids, do some calculations and 
>> save a new grid?

This is another case of posting on r-help and R-sig-geo, leading to the 
thread getting confused. I replied on r-help:

https://stat.ethz.ch/pipermail/r-help/2008-December/181686.html

saying that using tiling with rgdal functions - which have a driver for 
this data - is the obvious way to go, and that the raster package under 
development on R-forge looks very promising for letting users automate the 
process of reading rasters in tiles.

See also the relevant thread here:

https://stat.ethz.ch/pipermail/r-sig-geo/2008-November/004486.html

Roger

>
> Depends on what you want to do with it. What's the dimension of the grid in rows x columns?
>
> -->dimensions 80264x73772 for the largest grid and 28848x47030 for current grid
>
> Firstly, can you convert it to a more efficient format? Ascii Grid files 
> are generally long lines of numbers, looking like 2.333535 4.457574 
> 2.332336 5.445775 and so on. Maybe 10 characters per grid square. You 
> might be able to squeeze it to 4 characters per grid square by storing 
> as raw floating point numbers.
>
> -->yes, depends on what formats R is able to read in
>
> Next, can you just process it serially, one line or grid square at a 
> time? Do you need to store it all in memory at the same time?
>
> --> I can process serially, I don't need to store it in memory for the 
> simple math calculations, only for subsequent analysis. But once 
> calculations are done I can resample/aggregate to higher resolution.
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no