[R-sig-Geo] rgdal: problem reading a bigger raster dataset (R 4.0.0/3.6.3, Ubuntu 20.04)

[Thorsten Behrens]

Dear all,

my problem is that I want to read a big geotiff raster dataset into R 
and convert it to a matrix, which does not work.
The file is big but there is sufficient memory. I need all the data in 
the memory at the same time.

The error occurs under R 3.6.3 as well as 4.0.0 using Ubuntu 20.04 LTS 
with the latest version of the packages (see session info below) and 
256GB RAM installed.

When loading the raster dataset using rgdal (via readGDAL or 
raster::readAll) I get the follwoing error in R 4.0.0:

```
Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band 
= object using data@band) :
   long vectors not supported yet: memory.c:3782
```

In R 3.6.3 is is "... memory.c:3717"

However, I can load the same file with the tiff package and a file of 
the same size in the native raster package format (*.grd) with the 
raster package but again not with the rgdal package.

gdalinfo (gdalUtils) does not complain (see below). Hence, Even Rouault 
assumes the problem is related to rgdal and not gdal 
(https://github.com/OSGeo/gdal/issues/2442).

Below you find reproducible code, which generates a raster file, saves 
the two formats (.tiff and .grd) and tries to read them with the 
different packages.

Is this a known limitation? Any help is greatly appreciated!

Thanks a lot in advance!

Best wishes and stay healthy,
Thorsten



### Steps to reproduce the problem.

R code:

```
library(rgdal) # 1.4-8
library(raster) # 3.1-5
library(tiff) # 0.1-5

## generate and manipulate a big raster dataset
# - generate
rDemTest = raster(nrow = 48000, ncol = 72000, ext = extent(c(0, 72000, 
0, 48000))) # all fine

# - manipulate
values(rDemTest) = 1 # all fine

# - convert
mDemTest = raster::as.matrix(rDemTest) # all fine
str(mDemTest)

## save a big dataset

# - as raster/gdal
sFileNameTiff = "BigData.tif"
writeRaster(rDemTest, sFileNameTiff, "GTiff", overwrite = TRUE, NAflag = 
-9999) # all fine

# - as raster native
sFileNameNative = "BigData.grd"
writeRaster(rDemTest, sFileNameNative, "raster", overwrite = TRUE, 
NAflag = -9999) # all fine


## load the big raster datasets with different packages and options
# - load the tiff data with the gdal package via the raster package
rDem = raster(sFileNameTiff) # all fine
extent(rDem) # all fine
mDem = raster::as.matrix(rDem) # error
rDem = readAll(rDem) # error

# - load the native raster data with the raster package
rDem = raster(sFileNameNative) # all fine
extent(rDem) # all fine
mDem = raster::as.matrix(rDem) # all fine
str(mDem)

# - load the tiff data with the tiff package
mDem = readTIFF(sFileNameTiff) # all fine
str(mDem)

# - load the tiff data with the gdal package
sfDem = readGDAL(sFileNameTiff) # error

# - load the native raster data with the gdal package
sfDem = readGDAL(sFileNameNative) # error

```


### Startup messages when rgdal is attached (requested by Roger Bivand)
 > library(rgdal)
rgdal: version: 1.4-8, (SVN revision 845)
  Geospatial Data Abstraction Library extensions to R successfully loaded
  Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
  Path to GDAL shared files:
  GDAL binary built with GEOS: TRUE
  Loaded PROJ.4 runtime: Rel. 6.3.1, February 10th, 2020, [PJ_VERSION: 631]
  Path to PROJ.4 shared files: (autodetected)
  Linking to sp version: 1.4-1


### Session info
 > sessionInfo()
R version 4.0.0 (2020-04-24)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale:
  [1] LC_CTYPE=de_DE.UTF-8       LC_NUMERIC=C LC_TIME=de_DE.UTF-8
  [4] LC_COLLATE=de_DE.UTF-8     LC_MONETARY=de_DE.UTF-8 
LC_MESSAGES=de_DE.UTF-8
  [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C             LC_MEASUREMENT=de_DE.UTF-8 
LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods base

other attached packages:
[1] gdalUtils_2.0.3.2 rgdal_1.4-8       tiff_0.1-5 raster_3.1-5      
sp_1.4-1

loaded via a namespace (and not attached):
  [1] compiler_4.0.0    tools_4.0.0       Rcpp_1.0.4.6 R.methodsS3_1.8.0 
codetools_0.2-16
  [6] grid_4.0.0        iterators_1.0.12  foreach_1.5.0 
R.utils_2.9.2     R.oo_1.23.0
[11] lattice_0.20-41


### gdalInfo
 > gdalinfo(sFileNameTiff)
  [1] "Driver: GTiff/GeoTIFF"
  [2] "Files: BigData.tif"
  [3] "Size is 72000, 48000"
  [4] "Origin = (0.000000000000000,48000.000000000000000)"
  [5] "Pixel Size = (1.000000000000000,-1.000000000000000)"
  [6] "Image Structure Metadata:"
  [7] "  COMPRESSION=LZW"
  [8] "  INTERLEAVE=BAND"
  [9] "Corner Coordinates:"
[10] "Upper Left  (       0.000,   48000.000) "
[11] "Lower Left  (   0.0000000,   0.0000000) "
[12] "Upper Right (   72000.000,   48000.000) "
[13] "Lower Right (   72000.000,       0.000) "
[14] "Center      (   36000.000,   24000.000) "
[15] "Band 1 Block=72000x1 Type=Float32, ColorInterp=Gray"
[16] "  Min=1.000 Max=1.000 "
[17] "  Minimum=1.000, Maximum=1.000, Mean=nan, StdDev=nan"
[18] "  NoData Value=-9999"
[19] "  Metadata:"
[20] "    STATISTICS_MAXIMUM=1"
[21] "    STATISTICS_MEAN=nan"
[22] "    STATISTICS_MINIMUM=1"
[23] "    STATISTICS_STDDEV=nan"