[R-sig-Geo] rgdal: About the new gdal3 and proj >=6 condition
Roy Mendelssohn - NOAA Federal
roy@mende|@@ohn @end|ng |rom no@@@gov
Fri May 29 23:15:58 CEST 2020
Hi All:
It appears that Anaconda has the requisite versions, as well as gdal-config and the appropriate include files etc etc, though I haven’t tried to compile yet, as I have to be on a different computer. So I have two questions for now:
1. Other than the rgdal source and setting everything to point to the correct locations, is there anything else I need to have installed (I have the compilers). If I can get it to work, this might be for R4.0 only, I don’t keep two versions around, and I don’t have Catalina.
2. I assume there are other packages on the Mac that are in a similar situation, would I be correct in assuming sp and sf?
Thanks,
-Roy
> On May 29, 2020, at 1:44 PM, Roger Bivand <Roger.Bivand using nhh.no> wrote:
>
> On Fri, 29 May 2020, Patrick Schratz wrote:
>
>> The URL returns a 404.
>
> Do you mean http://spatial.nhh.no/R/rgdal/? <http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your browser blocks http: rather than https:, that is your choice. I imagine that it silently inserts an "s".
>
>>
>> FWIW I’ve tried it locally using the latest svn revision but have no clue where I should find `proj_api.h` (I’ve looked in the complete proj installation directory).
>
> If PROJ 7.0.1 is properly installed, the header will be in $prefix/include. If your packager put all the headers somewhere else, use the command argument:
>
> $ ls /usr/local/include/proj*
> /usr/local/include/proj_api.h /usr/local/include/proj.h
> /usr/local/include/proj_constants.h /usr/local/include/proj_symbol_rename.h
> /usr/local/include/proj_experimental.h
>
> /usr/local/include/proj:
> common.hpp crs.hpp metadata.hpp
> coordinateoperation.hpp datum.hpp nn.hpp
> coordinatesystem.hpp io.hpp util.hpp
>
> You should not need to know. What does pkg-config proj --cflags say (once you've set PKG_CONFIG_PATH, of course.
>
>>
>> I see
>>
>> ```
>> ls /usr/local/Cellar/proj/7.0.1/lib/
>> libproj.19.dylib libproj.a libproj.dylib@ libproj.la <http://libproj.la/>* pkgconfig/
>> ```
>>
>
> In your earlier post, you negected to say that you used the necessary configure argument. The error message says that you did not use it. Please report back when you have done so. Futher, I quote from your message: "It further causes complete breakage for people relying on the homebrew package manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1." Please accept that I pay attention to what you write.
>
>> Even if this custom installation might work at some point, making such adjustments until gdal3 arrives in homebrew will be a major pain, also for CI builds.
>> It would still be highly appreciated if this internal assertion would be removed again.
>>
>
> Getting things right may be painful; GDAL 2 with PROJ 7 is wrong, because it mixes two incompatible metadata storage systems. Migrating all users to
> GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional error of about 125m, or about 4 satellite image cells. See the vignette here http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html <http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html <https://www.r-spatial.org/r/2020/03/17/wkt.html>.
>
> I am not willing to let silent error creep in because some people find it inconvenient to avoid the problem. Both sf and sp/rgdal are covering your backs even if it feels rough. We've already helped dozens of maintainers of reverse dependencies future-proof their users from the consequences of changes made in PROJ and GDAL, where we have also contributed to making the transition easier by exposing helper functions in their APIs, which we use.
>
> I expect you to be able to CMD check a tarball from the commmand line, no more than that, to report back, and to try to grasp that these external software components are really important and completely non-trivial.
>
> Roger
>
>> On 29 May 2020, at 18:37, Roger Bivand wrote:
>>
>>> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
>>> 7.0.1 that passes CMD check (with warnings for missing vignettes) on:
>>>
>>> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
>>>
>>> It involves code copying to provide duplicated functions in three
>>> settings:
>>>
>>> PROJ < 6 & GDAL < 3
>>> PROJ > = 6 & GDAL >= 3
>>> PROJ > = 6 & GDAL < 3 (your homebrew case)
>>>
>>> You need the configure argument:
>>> --configure-args=--with-proj_api=proj_api.h.
>>> Without the argument, you now get directed to use it.
>>>
>>> For now I've dropped the configure tests for sqlite3, curl and tiff; works
>>> for me but may not work for you.
>>>
>>> Please make the output of:
>>>
>>> R CMD check --install-args="--configure-args=--with-proj_api=proj_api.h"
>>> rgdal_1.5-9.tar.gz
>>>
>>> available ASAP. I count on an immediate response.
>>>
>>> Roger
>>>
>>> On Fri, 29 May 2020, Roger Bivand wrote:
>>>
>>>> On Fri, 29 May 2020, Patrick Schratz wrote:
>>>>
>>>>> The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>>>>> present. It does not even install the package if this condition is
>>>>> not met
>>>>> but errors during linking.
>>>>
>>>> Please document with the output of 00install.out from R CMD check
>>>> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo --version
>>>> with system details.
>>>>
>>>> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
>>>> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and the
>>>> old metadata structure with Proj4 string degradation; GDAL < 3 with PROJ
>>>>> = 6 must use the deprecated API and the new metadata structure.
>>>>>
>>>>> I could not find any sources in the web or in the announcement post of
>>>>> v1.5.8 why this condition was enforced so strictly.
>>>>> Does it cause unwanted results?
>>>>
>>>> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
>>>> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with PROJ > = 6) after GDAL barnraising. So:
>>>>
>>>> PROJ
>>>> < 6 >= 6 GDAL < 3 OK NO
>>>> > = 3 NO OK
>>>>
>>>>> If so, then there is a big issue since the “gdal2 and proj >= 6”
>>>>> combination has been used by many people in the past already.
>>>>
>>>> No good precedent, just binary packagers protecting slow downstream
>>>> adaptation.
>>>>
>>>>> I haven’t tracked down for how long this situation was on the market
>>>>> already.
>>>>> Also the {sf} package is still installable with this combination
>>>>> and I
>>>>> am not aware of any warnings/issues that came up due to this so far.
>>>>
>>>> rgdal is a much older package and has much more older code that needs
>>>> this protection. It is possible to accommodate the no-go areas, but I'd
>>>> really value patches to R-forge, as I cannot check multiple PROJ/GDAL
>>>> version combinations just because someone isn't up to speed.
>>>>
>>>>> It further causes complete breakage for people relying on the homebrew
>>>>> package manager on macOS since current versions are gdal v2.2.4 and proj
>>>>> 7.0.1.
>>>>
>>>> They should never have gone beyond PROJ 5 if they are stuck on GDAL. PROJ
>>>> 7 is a very different animal.
>>>>
>>>>> The update to gdal3 is blocked since months due to an incompatibility
>>>>> with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading the
>>>>> issue, it looks unclear when this issue will be resolved.
>>>>> The alternative osgeo4mac tap holds formula that is unstable and
>>>>> somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>>>>> from
>>>>> osgeo4mac.
>>>>>
>>>>> This incompatibility with homebrew-core formula leads to further issues
>>>>> for CI tests that rely on the spatial homebrew stack for macOS builds.
>>>>>
>>>>> All of this is really unfortunate and I am wondering if this was a known
>>>>> factor when introducing this condition in the new release.
>>>>> Unfortunately rgdal is not hosted on any Git* provider (there is
>>>>> just
>>>>> the CRAN mirror with not much information about the recent changes) and
>>>>> it is unclear/untransparent to me if there is a continuous integration
>>>>> setup at all for this package.
>>>>
>>>> Do not require me to leave the excellent SVN service on R-forge. Provide
>>>> patches there.
>>>>
>>>>> I hope this is not another case in the series “I do not like operating
>>>>> system XY and hence I do not test on it” which was seen recently in
>>>>> another package. That’s what CI is for and the responsibility of
>>>>> package authors.
>>>>
>>>> I run and develop on Fedora, I have no access to OSX, and the current
>>>> release was fully checked with ~ 900 revdeps repeatedly since November on
>>>> successive older and newer versions of PROJ and GDAL. I blocked new PROJ
>>>> with old GDAL recently to simplify development. CI is only as good as the
>>>> scripts, I am completely sure that repeated revdep testing and reading
>>>> the output is superior.
>>>>
>>>>> Also one should be aware that not every rgdal user is member of this
>>>>> mailing list (I guess not even 5%) and many people in production will
>>>>> face this error during install, most of them without any clue what to do
>>>>> or an idea why this was raised.
>>>>
>>>> There has been plenty of information given about the CRS changes. rgdal
>>>> could simply have been abandoned by me, and all those production
>>>> volunteers might have fixed things, but I never hear anything at all.
>>>>
>>>>> In summary it would be great if
>>>>>
>>>>> - There would be more detailed information on the introduction of the
>>>>> new condition
>>>>> - Awareness of state-of-the-art platform related library versions
>>>>> before
>>>>> making such a release
>>>>> - Transparency/Existence of a cross-platform build process
>>>>> - If the incompatibility is just due to some features not being
>>>>> accessible with this configuration, I suggest to raise a warning
>>>>> during
>>>>> package load but not prevent the package from installing in the first
>>>>> place
>>>>> - A version-based NEWS file would be available. Currently one only
>>>>> sees
>>>>> a revision-based Changelog:
>>>>> https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>>>>
>>>>> I am well aware that I am indirectly criticising (hopefully in a
>>>>> constructive way) the transparency and release process here, for
>>>>> which I
>>>>> will probably get a rough response.
>>>>> However, if that leads to more robustness and transparency in
>>>>> future
>>>>> release, I am fine with this.
>>>>>
>>>>> A quick patch release resolving the breakage would be highly
>>>>> appreciated.
>>>>
>>>> Only with community imput. what you ask is not needed, just extra
>>>> complexity. Please provide patches, or accept my invitation to join the
>>>> R-forge project and commit your fixes directly. I can see how to do it,
>>>> but I don't think it makes sense, and your messsage has not motivated me,
>>>> to be honest. I'm prioritizing working with CRAN to iron out reverse
>>>> dependency problems.
>>>>
>>>> Roger
>>>>>
>>>>> Best, Patrick
>>>>> [[alternative HTML version deleted]]
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-Geo mailing list
>>>>> R-sig-Geo using r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>> --
>>> Roger Bivand
>>> Department of Economics, Norwegian School of Economics,
>>> Helleveien 30, N-5045 Bergen, Norway.
>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>> https://orcid.org/0000-0003-2392-6140
>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>
>>
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no <mailto:Roger.Bivand using nhh.no>
> https://orcid.org/0000-0003-2392-6140 <https://orcid.org/0000-0003-2392-6140>
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________>
> R-sig-Geo mailing list
> R-sig-Geo using r-project.org <mailto:R-sig-Geo using r-project.org>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo <https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
[[alternative HTML version deleted]]
More information about the R-sig-Geo
mailing list