[R-sig-Geo] Issue with rgrass7 and readRAST
Thomas Adams
te@3rd @end|ng |rom gm@||@com
Mon Jul 13 01:47:00 CEST 2020
Hi Roger,
I did run with a 90m resolution, with the same error:
> srtm <- readRAST("mozambique_srtm_90m",cat=FALSE)
Creating BIL support files...
Exporting raster as integer values (bytes=4)
100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded datum unknown in CRS definition
> image(srtm)
Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2], interpolate
= interpolate, :
cannot allocate memory block of size 16777216 Tb
However, with a 250m resolution the import and visualization worked fine:
> srtm <- readRAST("mozambique_srtm_250m",cat=FALSE)
Creating BIL support files...
Exporting raster as integer values (bytes=4)
100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded datum unknown in CRS definition
> image(srtm,col = terrain.colors(20))
Thank you for your help!
Best regards,
Tom
On Sun, Jul 12, 2020 at 9:19 AM Roger Bivand <Roger.Bivand using nhh.no> wrote:
> On Sun, 12 Jul 2020, Thomas Adams wrote:
>
> > Hi Roger,
> >
> > I believe I may have been correct about the memory issue... it's
> possible,
> > in GRASS, that I did not restrict my REGION sufficiently, using g.region.
> > Additionally, I applied a MASK for my country of interest, Mozambique,
> > before starting R, which yielded:
> >
> > srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
> > Creating BIL support files...
> > Exporting raster as integer values (bytes=4)
> > 100%
> > Warning messages:
> > 1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
> > Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
> > +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
> > 2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
> > Discarded datum unknown in CRS definition
> >> image(srtm)
> > Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2],
> interpolate
> > = interpolate, :
> > cannot allocate memory block of size 16777216 Tb
>
> I'd got to an errant g.region before r.out.bin too.
>
> graphics::rasterImage() is reporting the error. The metric is wrong, but
> the problem is real. The object is about 650M x 4 bytes, which image()
> needs to manipulate too, creating several copies (in the imported object
> it is a vector, but is cast to a matrix and transposed).
>
> >
> > Still, a huge amount of memory needed (is 16777216 Tb correct??)
> >
> > My thought was to resample the SRTM data in GRASS to 1km resolution, then
> > start R, and proceed, which quite honestly, for my purposes, really is
> > sufficient. By doing this, everything worked perfectly! I was able to
> read
> > the srtm data quickly and generate an image. I am puzzled by the "cannot
> > allocate memory block of size 16777216 Tb" message with my GRASS Region
> > settings in place.
>
> Please try r.out.gdal in 90m, then stars::read_stars(..., proxy=TRUE)
> (outside rgrass7). We need to try to get stars to work in rgrass7, but we
> had not gone into that yet. In stars and gdalcubes, the idea is to do work
> on the back end, and only bring data to (say) display in the feasible
> resolution. So I think something like:
>
> library(stars)
> big <- read_stars("big.tif", proxy=TRUE)
> plot(big)
>
> automates the downsampling. The default arguments to readRAST() assume
> that uncompressed binary is the optimal transfer strategy, and that all
> the data should be in R's workspace.
>
> I can speculate that if GDAL used the GRASS plugin, it might be possibble
> to proxy the actual GRASS CELL or DCELL. However, someone else would need
> to take responsibility for implementing prototypes for that. The same
> applies to support for stars, alternatively for terra in addition (raster
> is used by many, but with limited resources, stars and terra should be
> preferred).
>
> Roger
>
> >
> > Thank you for all your help!
> >
> > Best,
> > Tom
> >
> >
> >
> > On Sat, Jul 11, 2020 at 11:02 PM Thomas Adams <tea3rd using gmail.com> wrote:
> >
> >> Hi Roger,
> >>
> >> r.out.bin from GRASS did succeed... r.out.bin
> >> input=mozambique_srtm_patch using teaiii output=mozambique_srtm_patch
> null=-999
> >> Exporting raster as integer values (bytes=2)
> >>
> >> A very large file!!
> >> -rw-r--r-- 1 teaiii teaiii 1045368225220 Jul 11 13:26
> >> mozambique_srtm_patch
> >>
> >> Tom
> >>
> >>
> >> On Sat, Jul 11, 2020 at 12:12 PM Roger Bivand <Roger.Bivand using nhh.no>
> wrote:
> >>
> >>> Briefly, please try to run r.out.bin manually, to the same output or
> >>> another. This is what is failing on theGRASS side.
> >>>
> >>> Roger
> >>>
> >>> -------
> >>> Roger Bivand
> >>> Norwegian School of Economics, Bergen, Norway
> >>> roger.bivand using nhh.no
> >>>
> >>> ------------------------------
> >>> *Fra:* Thomas Adams <tea3rd using gmail.com>
> >>> *Sendt:* lørdag 11. juli 2020, 17:59
> >>> *Til:* Roger Bivand
> >>> *Kopi:* r-sig-Geo using r-project.org
> >>> *Emne:* Re: [R-sig-Geo] Issue with rgrass7 and readRAST
> >>>
> >>> Hi Roger!
> >>>
> >>> Here is what was written in
> >>>
> /media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS:
> >>>
> >>> drwxr-xr-x 3 teaiii teaiii 4096 Sep 23 2019 ..
> >>> -rw-r--r-- 1 teaiii teaiii 131072 Jul 9 10:02 801.0.gpkg
> >>> -rw-r--r-- 1 teaiii teaiii 131072 Jul 9 10:04 466.0.gpkg
> >>> -rw-r--r-- 1 teaiii teaiii 0 Jul 9 20:32 801.0
> >>> -rw-r--r-- 1 teaiii teaiii 0 Jul 9 22:40 188.0
> >>> -rw-r--r-- 1 teaiii teaiii 0 Jul 9 22:51 222.0
> >>>
> >>> And there is plenty of space on the disk drive...
> >>>
> >>> Tom
> >>>
> >>> On Sat, Jul 11, 2020 at 9:26 AM Roger Bivand <Roger.Bivand using nhh.no>
> wrote:
> >>>
> >>>> Tom
> >>>>
> >>>> On Fri, 10 Jul 2020, Thomas Adams wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I'm using R in combination with GRASS GIS 7.8.3, attempting to read a
> >>>> GRASS
> >>>>> raster file into R with the rgrass7 interface. I have done this kind
> of
> >>>>> thing countless times previously without problems. It's quite
> possible
> >>>> I am
> >>>>> running into memory issues, but my Ubuntu 18.04 Linux computer has 32
> >>>> GB of
> >>>>> RAM.
> >>>>>
> >>>>
> >>>> I have not succeeded in reproducing this, I'm afraid. I created a
> >>>> location
> >>>> with cell counts to match, and ran r.out.bin first to check whether
> >>>> there
> >>>> were any issues writing the data to a temporary file. Then I ran the
> >>>> equivalent of your command (readRAST()), again without problems. I'm
> on
> >>>> a
> >>>> 16 GB PC, so the problem is puzzling. I'm curious about the free space
> >>>> on
> >>>> your
> >>>>
> >>>> /media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/
> >>>> teaiii-Leopard-WS
> >>>>
> >>>> directory. Did you check whether any output arrived there, since the
> >>>> error
> >>>> message is coming from r.out.bin?
> >>>>
> >>>> The error is on line 53 in raster/r.out.bin/main.c, when the system
> >>>> function fwrite() returns an error.
> >>>>
> >>>> Thanks for your patience.
> >>>>
> >>>> Roger
> >>>>
> >>>>
> >>>>> My GRASS raster file:
> >>>>>
> >>>>> Type of Map: raster Number of Categories: 0
> >>>>> Data Type: CELL
> >>>>> Rows: 22663
> >>>>> Columns: 28794
> >>>>> Total Cells: 652558422
> >>>>> ~90m resolution
> >>>>>
> >>>>> Using...
> >>>>>
> >>>>> srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
> >>>>>
> >>>>> After quite some time, I get...
> >>>>>
> >>>>> Error in execGRASS("r.out.bin", flags = rOutBinFlags, input =
> >>>> vname[i], :
> >>>>> The command:
> >>>>> r.out.bin -b -i input=mozambique_srtm_patch
> >>>>>
> >>>>
> output=/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS/mozambique_srtm_patch
> >>>>> bytes=4 null=-81
> >>>>> produced an error (1) during execution:
> >>>>> Creating BIL support files...
> >>>>> Exporting raster as integer values (bytes=4)
> >>>>> ERROR: Error writing data
> >>>>> In addition: Warning messages:
> >>>>> 1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
> >>>>> Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0
> >>>> +lon_0=0
> >>>>> +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
> >>>>> 2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
> >>>>> Discarded datum unknown in CRS definition
> >>>>>
> >>>>> Libraries loaded include: sp, rgrass7, rgdal, raster...
> >>>>>
> >>>>> Any thoughts?
> >>>>>
> >>>>> Best,
> >>>>> Tom
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Roger Bivand
> >>>> Department of Economics, Norwegian School of Economics,
> >>>> Helleveien 30, N-5045 Bergen, Norway.
> >>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
> >>>> https://orcid.org/0000-0003-2392-6140
> >>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> >>>>
> >>>
> >>>
> >>> --
> >>> Thomas E Adams, III
> >>> 1724 Sage Lane
> >>> Blacksburg, VA 24060
> >>> tea3rd using gmail.com (personal)
> >>> tea using terrapredictions.org (work)
> >>>
> >>> 1 (513) 739-9512 (cell)
> >>>
> >>>
> >>>
> >>
> >> --
> >> Thomas E Adams, III
> >> 1724 Sage Lane
> >> Blacksburg, VA 24060
> >> tea3rd using gmail.com (personal)
> >> tea using terrapredictions.org (work)
> >>
> >> 1 (513) 739-9512 (cell)
> >>
> >>
> >
> >
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
--
Thomas E Adams, III
1724 Sage Lane
Blacksburg, VA 24060
tea3rd using gmail.com (personal)
tea using terrapredictions.org (work)
1 (513) 739-9512 (cell)
[[alternative HTML version deleted]]
More information about the R-sig-Geo
mailing list