[R-sig-Geo] Issue with rgrass7 and readRAST

[Roger Bivand]

On Sun, 12 Jul 2020, Thomas Adams wrote:

> Hi Roger,
>
> I believe I may have been correct about the memory issue... it's possible,
> in GRASS, that I did not restrict my REGION sufficiently, using g.region.
> Additionally, I applied a MASK for my country of interest, Mozambique,
> before starting R, which yielded:
>
> srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
> Creating BIL support files...
> Exporting raster as integer values (bytes=4)
> 100%
> Warning messages:
> 1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
>  Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
> +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
> 2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
>  Discarded datum unknown in CRS definition
>> image(srtm)
> Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2], interpolate
> = interpolate,  :
>  cannot allocate memory block of size 16777216 Tb

I'd got to an errant g.region before r.out.bin too.

graphics::rasterImage() is reporting the error. The metric is wrong, but 
the problem is real. The object is about 650M x 4 bytes, which image() 
needs to manipulate too, creating several copies (in the imported object 
it is a vector, but is cast to a matrix and transposed).

>
> Still, a huge amount of memory needed (is 16777216 Tb correct??)
>
> My thought was to resample the SRTM data in GRASS to 1km resolution, then
> start R, and proceed, which quite honestly, for my purposes, really is
> sufficient. By doing this, everything worked perfectly! I was able to read
> the srtm data quickly and generate an image. I am puzzled by the "cannot
> allocate memory block of size 16777216 Tb" message with my GRASS Region
> settings in place.

Please try r.out.gdal in 90m, then stars::read_stars(..., proxy=TRUE) 
(outside rgrass7). We need to try to get stars to work in rgrass7, but we 
had not gone into that yet. In stars and gdalcubes, the idea is to do work 
on the back end, and only bring data to (say) display in the feasible 
resolution. So I think something like:

library(stars)
big <- read_stars("big.tif", proxy=TRUE)
plot(big)

automates the downsampling. The default arguments to readRAST() assume 
that uncompressed binary is the optimal transfer strategy, and that all 
the data should be in R's workspace.

I can speculate that if GDAL used the GRASS plugin, it might be possibble 
to proxy the actual GRASS CELL or DCELL. However, someone else would need 
to take responsibility for implementing prototypes for that. The same 
applies to support for stars, alternatively for terra in addition (raster 
is used by many, but with limited resources, stars and terra should be 
preferred).

Roger

>
> Thank you for all your help!
>
> Best,
> Tom
>
>
>
> On Sat, Jul 11, 2020 at 11:02 PM Thomas Adams <tea3rd using gmail.com> wrote:
>
>> Hi Roger,
>>
>> r.out.bin from GRASS did succeed... r.out.bin
>> input=mozambique_srtm_patch using teaiii output=mozambique_srtm_patch null=-999
>> Exporting raster as integer values (bytes=2)
>>
>> A very large file!!
>> -rw-r--r--   1 teaiii teaiii 1045368225220 Jul 11 13:26
>> mozambique_srtm_patch
>>
>> Tom
>>
>>
>> On Sat, Jul 11, 2020 at 12:12 PM Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>
>>> Briefly, please try to run r.out.bin manually, to the same output or
>>> another. This is what is failing on theGRASS side.
>>>
>>> Roger
>>>
>>> -------
>>> Roger Bivand
>>> Norwegian School of Economics, Bergen, Norway
>>> roger.bivand using nhh.no
>>>
>>> ------------------------------
>>> *Fra:* Thomas Adams <tea3rd using gmail.com>
>>> *Sendt:* lørdag 11. juli 2020, 17:59
>>> *Til:* Roger Bivand
>>> *Kopi:* r-sig-Geo using r-project.org
>>> *Emne:* Re: [R-sig-Geo] Issue with rgrass7 and readRAST
>>>
>>> Hi Roger!
>>>
>>> Here is what was written in
>>> /media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS:
>>>
>>> drwxr-xr-x 3 teaiii teaiii   4096 Sep 23  2019 ..
>>> -rw-r--r-- 1 teaiii teaiii 131072 Jul  9 10:02 801.0.gpkg
>>> -rw-r--r-- 1 teaiii teaiii 131072 Jul  9 10:04 466.0.gpkg
>>> -rw-r--r-- 1 teaiii teaiii      0 Jul  9 20:32 801.0
>>> -rw-r--r-- 1 teaiii teaiii      0 Jul  9 22:40 188.0
>>> -rw-r--r-- 1 teaiii teaiii      0 Jul  9 22:51 222.0
>>>
>>> And there is plenty of space on the disk drive...
>>>
>>> Tom
>>>
>>> On Sat, Jul 11, 2020 at 9:26 AM Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>>
>>>> Tom
>>>>
>>>> On Fri, 10 Jul 2020, Thomas Adams wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm using R in combination with GRASS GIS 7.8.3, attempting to read a
>>>> GRASS
>>>>> raster file into R with the rgrass7 interface. I have done this kind of
>>>>> thing countless times previously without problems. It's quite possible
>>>> I am
>>>>> running into memory issues, but my Ubuntu 18.04 Linux computer has 32
>>>> GB of
>>>>> RAM.
>>>>>
>>>>
>>>> I have not succeeded in reproducing this, I'm afraid. I created a
>>>> location
>>>> with cell counts to match, and ran r.out.bin first to check whether
>>>> there
>>>> were any issues writing the data to a temporary file. Then I ran the
>>>> equivalent of your command (readRAST()), again without problems. I'm on
>>>> a
>>>> 16 GB PC, so the problem is puzzling. I'm curious about the free space
>>>> on
>>>> your
>>>>
>>>> /media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/
>>>> teaiii-Leopard-WS
>>>>
>>>> directory. Did you check whether any output arrived there, since the
>>>> error
>>>> message is coming from r.out.bin?
>>>>
>>>> The error is on line 53 in raster/r.out.bin/main.c, when the system
>>>> function fwrite() returns an error.
>>>>
>>>> Thanks for your patience.
>>>>
>>>> Roger
>>>>
>>>>
>>>>> My GRASS raster file:
>>>>>
>>>>> Type of Map:  raster               Number of Categories: 0
>>>>> Data Type:    CELL
>>>>> Rows:         22663
>>>>> Columns:      28794
>>>>> Total Cells:  652558422
>>>>> ~90m resolution
>>>>>
>>>>> Using...
>>>>>
>>>>> srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
>>>>>
>>>>> After quite some time, I get...
>>>>>
>>>>> Error in execGRASS("r.out.bin", flags = rOutBinFlags, input =
>>>> vname[i],  :
>>>>>  The command:
>>>>> r.out.bin -b -i input=mozambique_srtm_patch
>>>>>
>>>> output=/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS/mozambique_srtm_patch
>>>>> bytes=4 null=-81
>>>>> produced an error (1) during execution:
>>>>> Creating BIL support files...
>>>>> Exporting raster as integer values (bytes=4)
>>>>> ERROR: Error writing data
>>>>> In addition: Warning messages:
>>>>> 1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
>>>>>  Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0
>>>> +lon_0=0
>>>>> +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
>>>>> 2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
>>>>>  Discarded datum unknown in CRS definition
>>>>>
>>>>> Libraries loaded include: sp, rgrass7, rgdal, raster...
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> Best,
>>>>> Tom
>>>>>
>>>>>
>>>>
>>>> --
>>>> Roger Bivand
>>>> Department of Economics, Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>> https://orcid.org/0000-0003-2392-6140
>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>
>>>
>>>
>>> --
>>> Thomas E Adams, III
>>> 1724 Sage Lane
>>> Blacksburg, VA 24060
>>> tea3rd using gmail.com (personal)
>>> tea using terrapredictions.org (work)
>>>
>>> 1 (513) 739-9512 (cell)
>>>
>>>
>>>
>>
>> --
>> Thomas E Adams, III
>> 1724 Sage Lane
>> Blacksburg, VA 24060
>> tea3rd using gmail.com (personal)
>> tea using terrapredictions.org (work)
>>
>> 1 (513) 739-9512 (cell)
>>
>>
>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en