[R-sig-Geo] Reporting an issue with rgdal in macOS BigSur
Roger Bivand
Roger@B|v@nd @end|ng |rom nhh@no
Mon Nov 30 15:46:18 CET 2020
On Sat, 28 Nov 2020, Gilberto Camara wrote:
> Dear Roger,
>
> Let me know if I can help you in any way.
Thanks. We'd need someone with access to a MacOS machine, and an
understanding of how recipes (https://github.com/R-macos/recipes) work, to
write recipes for the following libraries and headers, then to check that
a GDAL built via recipes does incorporate the drivers that are missing
in CRAN OSX binaries but are present in CRAN Windows binaries:
JP2OpenJPEG, all ODBC, WEBP and XLS
I'm posting this for reference with at least most of the source
information:
-------------------------------------
JP2OpenJPEG: openjpeg >= 2.1 https://github.com/uclouvain/openjpeg
https://github.com/uclouvain/openjpeg/archive/v2.3.1.tar.gz The copyright
in this software is being made available under the 2-clauses BSD License.
/* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS `AS
IS'
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE
* POSSIBILITY OF SUCH DAMAGE.
*/
Geomedia (ODBC)
MSSQLSpatial (ODBC)
ODBC (ODBC) http://www.unixodbc.org/download.html
http://www.unixodbc.org/unixODBC-2.3.9.tar.gz
http://www.unixodbc.org/unixODBC-2.3.9.tar.gz.md5 GNU LESSER GENERAL
PUBLIC LICENSE Version 2.1. ./configure --enable-gui=no
PGeo (ODBC)
Walk (ODBC)
XLS: libfreexl http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz FreeXL
is licensed under the MPL tri-license terms; you are free to choose the
best-fit license between:
the MPL 1.1
the GPL v2.0 or any subsequent version
the LGPL v2.1 or any subsequent version
WEBP: libwebp https://developers.google.com/speed/webp/download (can be
used by Rasterlite)
https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-2.1.0.tar.gz
Copyright (c) 2010, Google Inc. All rights reserved. Redistribution and
use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
* Neither the name of Google nor the names of its contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
----------------------------
Best wishes,
Roger
>
> Best
> Gilberto
>
>> On 28 Nov 2020, at 18:08, Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>
>> For completeness, I've put up a skeleton package on https://github.com/rsbivand/checkGDALDrivers <https://github.com/rsbivand/checkGDALDrivers>, with Github Actions to check the package on macOS-latest as well as ubuntu-20.04 (latest and devel) and Windows-latest. The MacOS platform opens a GTiff file, but does not open a JP2 file (using rgdal::GDAL.open()):
>>
>> https://github.com/rsbivand/checkGDALDrivers/runs/1467699832 <https://github.com/rsbivand/checkGDALDrivers/runs/1467699832>
>>
>> I've replicated this on R-hub for MacOS using CRAN and brew (the GDAL from brew in that workflow also misses openjpeg it seems):
>>
>> CRAN: https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-64d505798f4b251d1c801ea5dbe6aa33/ <https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-64d505798f4b251d1c801ea5dbe6aa33/>
>> brew
>> https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-5dfcf72469d1effc742710d53d167caf/ <https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-5dfcf72469d1effc742710d53d167caf/>
>>
>> On Winbuilder the GA results are replicated (success with both GTiff and JP2 on all platforms - there is no guarantee that Debian or Fedora packaged GDAL will include this or any other optional driver).
>>
>> The skeleton package might easily be forked, and could also simply check the driver listings rather than actually trying to read a GTiff and a JP2 file (PRs very welcome).
>>
>> So we have now a two-threaded way to check for JP2 absence, using this package's GA, and through R-hub.
>>
>> Next step to work out how to get openjpeg into the recipes listings so that GDAL there is built using it.
>>
>> On Sat, 28 Nov 2020, Gilberto Camara wrote:
>>
>>> Dear Roger
>>>
>>> Many thanks for your patience and your guidance.
>>>
>>> I installed the rgdal binary from CRAN and it gives errors in both commands, as follows:
>>>
>>>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)]
>>> character(0)
>>>
>>
>> Sorry, should have been:
>>
>> rgdal::rgdal_extSoftVersion()
>>
>> Best wishes,
>>
>> Roger
>>
>>
>>>> rgdal::extSoftVersion()
>>>
>>> Error: 'extSoftVersion' is not an exported object from 'namespace:rgdal’
>>>
>>> After installing rgdal from source the first command works:
>>>
>>>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)]
>>> “JP2OpenJPEG"
>>>
>>> but the second does not:
>>>
>>>> rgdal::extSoftVersion()
>>>
>>> Error: 'extSoftVersion' is not an exported object from 'namespace:rgdal’
>>>
>>> I am using R version 4.0.3 Patched (2020-11-25 r79505), the latest one available on CRAN. The rgdal version is 1.5-18 (SVN revision 1082).
>>>
>>> More details on rgdal:
>>>
>>>> library(rgdal)
>>> Loading required package: sp
>>> rgdal: version: 1.5-18, (SVN revision 1082)
>>> Geospatial Data Abstraction Library extensions to R successfully loaded
>>> Loaded GDAL runtime: GDAL 3.2.0, released 2020/10/26
>>> Path to GDAL shared files: /Users/gilbertocamara/Library/R/4.0/library/sf/gdal
>>> GDAL binary built with GEOS: TRUE
>>> Loaded PROJ runtime: Rel. 7.1.1, September 1st, 2020, [PJ_VERSION: 711]
>>> Path to PROJ shared files: /Users/gilbertocamara/Library/Application Support/proj:/usr/local/share/proj:/usr/local/share/proj
>>> PROJ CDN enabled: FALSE
>>> Linking to sp version:1.4-4
>>>
>>> As for sf, it also has to be installed from source. The binary version of sf also fails to recognise the JP2 driver, while the one built from source does.
>>>
>>>> sf::sf_extSoftVersion() works and gives:
>>> GEOS GDAL proj.4 GDAL_with_GEOS USE_PROJ_H
>>> "3.8.1" "3.2.0" "7.1.1" "true" "true”
>>>
>>> So, it appears that the automated recipes system missed the openjpeg library.
>>>
>>> Best regards
>>> Gilberto
>>> ===========================
>>> Prof Dr Gilberto Camara
>>> Secretariat Director
>>> GEO - Group on Earth Observations
>>> 7 bis, Avenue de La Paix
>>> CH-1211 Geneva - Switzerland
>>> Tel: +41227308480
>>> Web: www.earthobservations.org
>>>
>>>
>>>> On 27 Nov 2020, at 22:23, Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>>>
>>>> Dear Gilberto,
>>>>
>>>> The easiest way to deploy the JP2OpenJPEG driver is to check whether it is in the CRAN MacOS rgdal or sf binary packages, as I described before. If it is not there, then we need to post an issue or PR on https://github.com/R-macos/recipes <https://github.com/R-macos/recipes> <https://github.com/R-macos/recipes <https://github.com/R-macos/recipes>>, I think.
>>>>
>>>> Could any MacOS user with rgdal installed binary from CRAN please run:
>>>>
>>>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)]
>>>> rgdal::extSoftVersion()
>>>>
>>>> and the R and rgdal versions? The equivalent for sf is:
>>>>
>>>> drvs <- sf::st_drivers("raster")$name; drvs[grep("JP2", drvs)]
>>>> sf::sf_extSoftVersion()
>>>>
>>>> This is just to check whether the recent change to the automated recipes system perhaps missed the openjpeg library which is used automatically if present.
>>>>
>>>> If it went missing, we should be able to reinstate it (or if never available - provide it - the CRAN Windows binary does have this driver), then all users of CRAM MacOS binary packages built against GDAL will be able to use the driver without having to build from source.
>>>>
>>>> Best wishes,
>>>>
>>>> Roger
>>>>
>>>> On Fri, 27 Nov 2020, Gilberto Camara wrote:
>>>>
>>>>> Dear Roger
>>>>>
>>>>> Many thanks for your support. I built GDAL from scratch and now it supports the JPEG-2000 driver using the “openjpeg” library. The reported problems are now solved.
>>>>>
>>>>> The whole exercise gave me some lessons on RGDAL and GDAL for users of MacOS BigSur:
>>>>>
>>>>> 1. The usual ways of installing GDAL in macOS (using Homebrew or the
>>>>> Frameworks provided by KingChaos) do not provide support for
>>>>> JPEG-2000. So, one has to install GDAL and RGDAL from source.
>>>>>
>>>>> 2. One has to use Apple’s C/C++ clang compiler version 12.0, following
>>>>> the advice given in Stack Overflow:
>>>>>
>>>>> https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target <https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target>
>>>>>
>>>>> 3. For GDAL to compile properly, it is necessary to install first
>>>>> “pkg-config”, and then the libraries “proj”, “geo”, “tiff”, “jpeg”,
>>>>> “openjpeg”, and “zstd”.
>>>>>
>>>>> Thus, if other users of “gdal” find problems with BigSur, I can volunteer to support them.
>>>>>
>>>>> Best regards
>>>>> Gilberto
>>>>>
>>>>> ===========================
>>>>> Prof Dr Gilberto Camara
>>>>> Secretariat Director
>>>>> GEO - Group on Earth Observations
>>>>> 7 bis, Avenue de La Paix
>>>>> CH-1211 Geneva - Switzerland
>>>>> Tel: +41227308480
>>>>> Web: www.earthobservations.org
>>>>>
>>>>>
>>>>>> On 26 Nov 2020, at 15:40, Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>>>>>
>>>>>> Dear Gilberto,
>>>>>>
>>>>>> On Thu, 26 Nov 2020, GilbertoCamara wrote:
>>>>>>
>>>>>>> Dear R-SIG-GEO and Roger
>>>>>>>
>>>>>>> I am developing an R package (https://github.com/e-sensing/sits <https://github.com/e-sensing/sits>) to analyse satellite image time series (SITS) which is heavily dependent on “rgdal”.
>>>>>>>
>>>>>>> I would like to report an issue with rgdal in the new macOS BigSur. While I agree with Roger that it is not wise to install a new OS until it is stable, I hope the information below will be useful.
>>>>>>>
>>>>>>> For starters, let me praise Roger, Edzer and all those involved in “rgdal”. I did some debugging on the source code and was humbled by the effort involved in making the code an operational software many of us rely on. Chapeau bas!
>>>>>>>
>>>>>>> That said, please allow me to report on three issues:
>>>>>>>
>>>>>>> 1. Installation
>>>>>>>
>>>>>>> I have installed GDAL-3.2.0 using Homebrew on BigSur; it works fine. When installing the binary version of rgdal-1.5-18 (SVN revision 1082) from CRAN, recently released by Simon Urbanek, “rgdal" reports that is is using GDAL-3.1.1, which is not installed. The actual message is: "Loaded GDAL runtime: GDAL 3.1.1, released 2020/06/22”.
>>>>>>>
>>>>>>> When installing from source, rgdal recognises GDAL-3.2.0, if the Homebrew libraries are on the path. The message changes to: "Loaded GDAL runtime: GDAL 3.2.0, released 2020/10/26."
>>>>>>>
>>>>>>> It is odd that the binary version would report using a runtime that is not there.
>>>>>>
>>>>>> On MacOS and Windows, CRAN binary packages are built using static linking, for a number of reasons. One is to limit installation difficulties, another to make support easier, as all users of a updated package/R version will be using the same set of external software components (of course, locales will vary, but most variables are fixed).
>>>>>>
>>>>>>>
>>>>>>> 2. Errors in /vsis3 file access
>>>>>>>
>>>>>>> In earlier versions of macOS, rgdal had no problems with accessing files in AWS with commands such as:
>>>>>>>
>>>>>>>> s3_file <- "/vsis3/sentinel-s2-l2a/tiles/20/L/KP/2018/8/17/0/R20m/B02.jp2”
>>>>>>>> rgdal::GDALinfo(s3_file)
>>>>>>>
>>>>>>> In BigSur, it produces an error. Debugging the rgdal code, I found the error to arise from line 723 of the “gdal-bindings.cpp” file. This is one of the places where rgdal calls the GDAL function “GDALOpenEx”, as follows:
>>>>>>
>>>>>> Please first check on running systems with pre-Big Sur and Big Sur that the returned lists of drivers match. Crucially, jp2 (Jpeg 2000) needs GDAL built against openjpeg. If one of the systems has GDAL built with the driver, and the other without, this outcome would be expected. I suspect that the Big Sur GDAL (or the CRAN MacOS rgdal binary) has missing/defective drivers.
>>>>>>
>>>>>> For my Fedora 33 system with GDAL installed from source, I have:
>>>>>>
>>>>>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)]
>>>>>> ## [1] "JP2OpenJPEG"
>>>>>>
>>>>>> There are other JP2* drivers, but they need closed source libraries when GDAL itself is built, and cannot be redistributed from CRAN.
>>>>>>
>>>>>>>
>>>>>>> GDALDataset *pDataset = (GDALDataset *) GDALOpenEx(fn, RWFlag,
>>>>>>> papszAllowedDrivers, papszOpenOptions, NULL);
>>>>>>>
>>>>>>> The function call returns a NULL pointer. Since calling gdalinfo externally or using gdalUtils::gdalinfo works, it seems there is a problem with the return parameters which are provided to rgdal by GDAL.
>>>>>>>
>>>>>>> To reproduce the issue, one needs AWS credentials. If Roger and/or Edzer wish to have a go at this issue, I can provide a temporary credential for doing so.
>>>>>>
>>>>>> First, we need to be sure that the driver is present, then present and working.
>>>>>>
>>>>>>>
>>>>>>> 3. Errors in TIFF Decoding
>>>>>>>
>>>>>>> Running devtools::check() produces a strange error on TIFF decoding on a very small file. The error is odd:
>>>>>>>
>>>>>>> "TIFFReadDirectory: Warning, Unknown field with tag 42112 (0xa480) encountered."
>>>>>>>
>>>>>>> The call stack is as follows:
>>>>>>>
>>>>>>> 1: .Call("RGDAL_OpenDataset", .normalize_if_path(filename, mustWork = NA), TRUE, silent, allowedDrivers, options, PACKAGE = "rgdal")
>>>>>>> 2: .local(.Object, ...)
>>>>>>> 3: initialize(value, ...)
>>>>>>> 4: initialize(value, ...)
>>>>>>> 5: new("GDALReadOnlyDataset", filename, silent = silent, allowedDrivers = allowedDrivers, options = options)
>>>>>>> 6: GDAL.open(fname, silent = silent, allowedDrivers = allowedDrivers, options = options)
>>>>>>> 7: rgdal::GDALinfo(file, silent = FALSE)
>>>>>>>
>>>>>>> The error appears to be in the same place as the one reported above. However, it is hard to reproduce, because when running on the console, rgdal works.
>>>>>>
>>>>>> This feels again like a driver mismatch. Please try with the CRAN MacOS binary package, for which https://cran.r-project.org/web/checks/check_results_rgdal.html looks clean. The examples for rgdal::readGDAL() include reading TIFF files, so are run in CRAN daily checks.
>>>>>>
>>>>>> I have no access to any MacOS platform, so cannot do more than arm's length debug. My clear suspicion is that the GDAL you are using has drivers missing/defective. A small JP2 raster could be added to the examples to trap driver malfunction.
>>>>>>
>>>>>> Best wishes,
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>>
>>>>>>>> ndvi_file <- c(system.file("extdata/raster/mod13q1/sinop-ndvi-2014.tif", package = "sits”))
>>>>>>>> rgdal::GDALinfo(ndvi_file, silent = FALSE)
>>>>>>>
>>>>>>> However, given that the place on the “rgdal” code where the error occurs is the same, it might be worthwhile to look into the RGDAL-GDAL interface in more detail.
>>>>>>>
>>>>>>> I know it’s my responsibility to have moved to macOS BigSur too soon. Thus, I do not expect that you address the issue. That said, any help would be much appreciated.
>>>>>>>
>>>>>>> Best regards
>>>>>>> Gilberto
>>>>>>> ===========================
>>>>>>> Prof Dr Gilberto Camara
>>>>>>> Secretariat Director
>>>>>>> GEO - Group on Earth Observations
>>>>>>> 7 bis, Avenue de La Paix
>>>>>>> CH-1211 Geneva - Switzerland
>>>>>>> Tel: +41227308480
>>>>>>> Web: www.earthobservations.org <http://www.earthobservations.org/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Roger Bivand
>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>>
>>>>>
>>>>
>>>> --
>>>> Roger Bivand
>>>> Department of Economics, Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no <mailto:Roger.Bivand using nhh.no> <mailto:Roger.Bivand using nhh.no <mailto:Roger.Bivand using nhh.no>>
>>>> https://orcid.org/0000-0003-2392-6140 <https://orcid.org/0000-0003-2392-6140> <https://orcid.org/0000-0003-2392-6140 <https://orcid.org/0000-0003-2392-6140>>
>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________><https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________>>
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo using r-project.org <mailto:R-sig-Geo using r-project.org> <mailto:R-sig-Geo using r-project.org <mailto:R-sig-Geo using r-project.org>>
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo <https://stat.ethz.ch/mailman/listinfo/r-sig-geo> <https://stat.ethz.ch/mailman/listinfo/r-sig-geo <https://stat.ethz.ch/mailman/listinfo/r-sig-geo>>
>>>
>>
>> --
>> Roger Bivand
>> Department of Economics, Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>> https://orcid.org/0000-0003-2392-6140
>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
More information about the R-sig-Geo
mailing list