[R-sig-Geo] reading ascii grid memory issues

[Barry Rowlingson]

2008/12/4  <Alexander.Herr at csiro.au>:
>
> Hi List,
>
> I am unable to read in a 7.8Gb ascii grid using readAsciiGrid {maptools}/ read.asciigrid {sp}; R runs out of memory. I have 4Gb ram and 4Gb swap, so things are getting tight.
>
> I am wondering if anyone has alternative options (preferably with example) that enable to read in large grids, do some calculations and save a new grid?

 Depends on what you want to do with it. What's the dimension of the
grid in rows x columns?

 Firstly, can you convert it to a more efficient format? Ascii Grid
files are generally long lines of numbers, looking like 2.333535
4.457574 2.332336 5.445775 and so on. Maybe 10 characters per grid
square. You might be able to squeeze it to 4 characters per grid
square by storing as raw floating point numbers.

 Next, can you just process it serially, one line or grid square at a
time? Do you need to store it all in memory at the same time?

 Finally, how much would you lose if you resampled the grid to a
smaller size? Take every tenth or hundredth cell. I'd certainly not
want to develop any analysis with a grid that size. Maybe you've
already worked out what you want to do with a small grid and the
problem is just processing this monster.

 R is probably not the right thing for this job, you might be better
off writing it as 4-byte floating point numbers, and writing some C
code and using the C seek() functions to read numbers from the grid.

Hope that helps...

Barry