[R-sig-Geo] rgdal: About the new gdal3 and proj >=6 condition
Patrick Schratz
p@tr|ck@@chr@tz @end|ng |rom gm@||@com
Fri May 29 21:43:24 CEST 2020
It’s gdal 2.4.4 instead of 2.2.4 as written before. Should not make a
difference though.
```
pkg-config proj --modversion
7.0.1
gdalinfo --version
GDAL 2.4.4, released 2020/01/08
```
With revision 993 checked out from svn I see
``´
checking for unistd.h... yes
checking proj.h usability... yes
checking proj.h presence... yes
checking for proj.h... yes
checking for proj_context_create in -lproj... yes
checking Using GDAL < 3 with PROJ >= 6... yes
configure: error: GDAL >= 3 should be used with PROJ >= 6, use
--with-proj_api="proj_api.h" for deprecated API
```
> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the old metadata structure with Proj4 string degradation; GDAL < 3
> with PROJ
What does “makes no sense” mean here? I am interested whether this
combination produces significant bugs when being used together.
If its just “the user won’t be able to make use of proj7 features
without gdal3” but apart from this everything is fine, then I do not
see **any** reason for this strict installation blocker.
While both gdal and proj are coupled tightly, I would expect assertions
in the upstream libraries to account for such clashes.
It seems unlikely that client packages like rgdal need to prevent
installation if this undesired version mixture of both libraries is
found.
> No good precedent, just binary packagers protecting slow downstream
> adaptation.
Not sure what you mean here exactly. My point is that the gdal2 + proj7
combination exists since March 6 in homebrew. Meaning that since then
users on macOS used this combination.
commit 7234fc2073e5475931238281493ae3367b4dcdf6
Author: Stephan Hügel <shugel using tcd.ie>
Date: Fri Mar 6 12:56:51 2020 +0000
proj 7.0.0
> rgdal is a much older package and has much more older code that needs
> this protection. It is possible to accommodate the no-go areas, but
> I'd really value patches to R-forge, as I cannot check multiple
> PROJ/GDAL version combinations just because someone isn't up to speed.
I cannot argue about any needed protection here. However, if that is the
case then there was a danger zone of almost 3 months on macOS now.
IF that combination causes unwanted side-effects (which is still not
fully answered).
I am not saying that you should check every gdal/proj combination on
every platform/distribution, especially not on the niche ones.
However, there are three main platforms (with Ubuntu being the main
Linux one) and it is part of the game (imo) for client packages to check
valid installations at least on those three platforms, using their
default package manager.
If that would have happened, we would not be writing here probably
because you would have recognised the issue and maybe enforced the
update to GDAL3 on macOS or postponed the rgdal update.
It would be great if that kind of coverage could be done in the future.
> Do not require me to leave the excellent SVN service on R-forge.
> Provide patches there.
I am just sharing my user experience, which is probably the view of many
others.
More contributions would be made if contributing would be easier.
R-forge and svn do not make it attractive.
It will always be your choice, no question that.
But if you want to provide a service to the community, sometimes there
is more to that than just code/releases.
> CI is only as good as the scripts, I am completely sure that repeated
> revdep testing and reading the output is superior.
CI and revdep checks are two different things.
CI happens at every commit (in the best scenario) and on multiple
platforms. It aims at package AND platform integrity.
Revdepchecks are on the package side, tackling a completely different
layer of security.
As of today (but actually since at least 5 years) there are no excuses
for not being able to check on all main platforms despite arguing with
“I do not want to do it but rely on my very own local test setup”.
> There has been plenty of information given about the CRS changes.
> rgdal could simply have been abandoned by me, and all those production
> volunteers might have fixed things, but I never hear anything at all.
Not sure what this should tell me/us. If you need help maintaining the
package or porting its infrastructure to the 21st century then there
would be a lot of people willing to help.#
However, replying to (useful) criticism with “I could have simply
abandoned it, be thankful (to me), guys” (that’s my interpretation
of this sentence) does not help anyone.
No one questions your contributions to the r-spatial community.
However, adapting to new (development) changes and making it easier for
others to contribute is also a very valuable contribution that is not
really measurable in any unit.
> Only with community imput. what you ask is not needed, just extra
> complexity. Please provide patches, or accept my invitation to join
> the R-forge project and commit your fixes directly. I can see how to
> do it, but I don't think it makes sense, and your messsage has not
> motivated me, to be honest. I'm prioritizing working with CRAN to iron
> out reverse dependency problems.
Ok, I understood. Sadly I have not enough time fighting to break up old
structures and changing things to the better/making things easier if
there is no real support from the other side.
The r-spatial community is not really a “community” in my eyes but a
conglomerate of three (four) main persons doing it their way, each
completely different, with few motivation to change things.
Each semester students ask me how the relations between the packages and
their authors are and why things are so different.
I am not going to expand on what I tell them usually, but often enough
they come to me with their personal view after some time and all I need
to say is “yes” or “no” (sadly most often “yes”).
What this means is left as an exercise to the reader.
Regards, Patrick
On 29 May 2020, at 14:45, Roger Bivand wrote:
> On Fri, 29 May 2020, Patrick Schratz wrote:
>
>> The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
>> present. It does not even install the package if this condition is
>> not met but errors during linking.
>
> Please document with the output of 00install.out from R CMD check
> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> --version with system details.
>
> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the old metadata structure with Proj4 string degradation; GDAL < 3
> with PROJ
>> = 6 must use the deprecated API and the new metadata structure.
>
>>
>> I could not find any sources in the web or in the announcement post
>> of
>> v1.5.8 why this condition was enforced so strictly.
>> Does it cause unwanted results?
>
> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
> PROJ >= 6) after GDAL barnraising. So:
>
> PROJ
> < 6 >= 6 GDAL < 3 OK NO
> >= 3 NO OK
>
>
>
>> If so, then there is a big issue since the “gdal2 and proj >= 6”
>> combination has been used by many people in the past already.
>
> No good precedent, just binary packagers protecting slow downstream
> adaptation.
>
>> I haven’t tracked down for how long this situation was on the
>> market
>> already.
>> Also the {sf} package is still installable with this combination and
>> I
>> am not aware of any warnings/issues that came up due to this so far.
>>
>
> rgdal is a much older package and has much more older code that needs
> this protection. It is possible to accommodate the no-go areas, but
> I'd really value patches to R-forge, as I cannot check multiple
> PROJ/GDAL version combinations just because someone isn't up to speed.
>
>> It further causes complete breakage for people relying on the
>> homebrew
>> package manager on macOS since current versions are gdal v2.2.4 and
>> proj
>> 7.0.1.
>
> They should never have gone beyond PROJ 5 if they are stuck on GDAL.
> PROJ 7 is a very different animal.
>
>> The update to gdal3 is blocked since months due to an incompatibility
>> with `liblas` (https://github.com/libLAS/libLAS/issues/164). Reading
>> the
>> issue, it looks unclear when this issue will be resolved.
>> The alternative osgeo4mac tap holds formula that is unstable and
>> somewhat broken. One cannot link rgdal with the current gdal v3.0.1
>> from
>> osgeo4mac.
>>
>> This incompatibility with homebrew-core formula leads to further
>> issues
>> for CI tests that rely on the spatial homebrew stack for macOS
>> builds.
>>
>> All of this is really unfortunate and I am wondering if this was a
>> known
>> factor when introducing this condition in the new release.
>> Unfortunately rgdal is not hosted on any Git* provider (there is just
>> the CRAN mirror with not much information about the recent changes)
>> and
>> it is unclear/untransparent to me if there is a continuous
>> integration
>> setup at all for this package.
>
> Do not require me to leave the excellent SVN service on R-forge.
> Provide patches there.
>
>> I hope this is not another case in the series “I do not like
>> operating
>> system XY and hence I do not test on it” which was seen recently in
>> another package. That’s what CI is for and the responsibility of
>> package authors.
>>
>
> I run and develop on Fedora, I have no access to OSX, and the current
> release was fully checked with ~ 900 revdeps repeatedly since November
> on successive older and newer versions of PROJ and GDAL. I blocked new
> PROJ with old GDAL recently to simplify development. CI is only as
> good as the scripts, I am completely sure that repeated revdep testing
> and reading the output is superior.
>
>> Also one should be aware that not every rgdal user is member of this
>> mailing list (I guess not even 5%) and many people in production will
>> face this error during install, most of them without any clue what to
>> do
>> or an idea why this was raised.
>>
>
> There has been plenty of information given about the CRS changes.
> rgdal could simply have been abandoned by me, and all those
> production volunteers might have fixed things, but I never hear
> anything at all.
>
>> In summary it would be great if
>>
>> - There would be more detailed information on the introduction of the
>> new condition
>> - Awareness of state-of-the-art platform related library versions
>> before
>> making such a release
>> - Transparency/Existence of a cross-platform build process
>> - If the incompatibility is just due to some features not being
>> accessible with this configuration, I suggest to raise a warning
>> during
>> package load but not prevent the package from installing in the first
>> place
>> - A version-based NEWS file would be available. Currently one only
>> sees
>> a revision-based Changelog:
>> https://cran.r-project.org/web/packages/rgdal/ChangeLog
>>
>> I am well aware that I am indirectly criticising (hopefully in a
>> constructive way) the transparency and release process here, for
>> which I
>> will probably get a rough response.
>> However, if that leads to more robustness and transparency in future
>> release, I am fine with this.
>>
>> A quick patch release resolving the breakage would be highly
>> appreciated.
>
> Only with community imput. what you ask is not needed, just extra
> complexity. Please provide patches, or accept my invitation to join
> the R-forge project and commit your fixes directly. I can see how to
> do it, but I don't think it makes sense, and your messsage has not
> motivated me, to be honest. I'm prioritizing working with CRAN to iron
> out reverse dependency problems.
>
> Roger
>
>>
>> Best, Patrick
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>
> --
> Roger Bivand
> Department of Economics, Norwegian School of Economics,
> Helleveien 30, N-5045 Bergen, Norway.
> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
> https://orcid.org/0000-0003-2392-6140
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
[[alternative HTML version deleted]]
More information about the R-sig-Geo
mailing list