[R-sig-Geo] rgdal: About the new gdal3 and proj >=6 condition

chris english eng||@hchr|@topher@ @end|ng |rom gm@||@com
Sat May 30 02:22:12 CEST 2020 

rgdal-1.5-9 fixed all my install problems on ubuntu 20.04. My mother has a
needle point pillow at her summer home
which declares "Guests of Guests May Not Bring Guests". Analogously,
'dependencies of dependencies will not
check dependencies' is applicable here. rgdal-1.5-9 facilitated:

> library(rgdal)
Loading required package: sp
rgdal: version: 1.5-9, (SVN revision 991M)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.1.0, released 2020/05/03
Path to GDAL shared files: /usr/local/share/gdal
GDAL binary built with GEOS: TRUE
Loaded PROJ runtime: Rel. 7.0.1, May 1st, 2020, [PJ_VERSION: 701]
Path to PROJ shared files:
/root/.local/share/proj:/usr/local/share/proj:/usr/local/share/proj
PROJ CDN enabled:FALSE
Linking to sp version:1.4-1
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading
rgdal.

GEOS version is 3.9.0dev

I still have some work to do on the PostgreSQL end of things, but I am back
to working
spatial. I still received errors unless I started the Install session in
sudo R, which leads me
to think that my /usr/local/(bin,lib,include) are symlinks from
one/some/all installs of GEOS,
PROJ, GDAL, which are not the fault of R developers, but I'd forgotten this
issue and rediscovered
that I pestered Roger and Edzer about it in 2019. So, sudo R.

I'm back to having a working spatial environment. Thank you Roger!

Chris

On Fri, May 29, 2020 at 5:16 PM Roy Mendelssohn - NOAA Federal via
R-sig-Geo <r-sig-geo using r-project.org> wrote:

> Hi All:
>
> It appears that Anaconda has the requisite versions,  as well as
> gdal-config and the appropriate include files etc etc,  though I haven’t
> tried to compile yet,  as I have to be on a different computer. So I have
> two questions for now:
>
> 1.  Other than the rgdal source and setting everything to point to the
> correct locations,  is there anything else I need to have installed  (I
> have the compilers).  If I can get it to work,  this might be for R4.0
> only,  I don’t keep two versions around,  and I don’t have Catalina.
>
> 2.  I assume there are other packages on the Mac that are in a similar
> situation,  would I be correct in assuming sp and sf?
>
> Thanks,
>
> -Roy
>
>
> > On May 29, 2020, at 1:44 PM, Roger Bivand <Roger.Bivand using nhh.no> wrote:
> >
> > On Fri, 29 May 2020, Patrick Schratz wrote:
> >
> >> The URL returns a 404.
> >
> > Do you mean http://spatial.nhh.no/R/rgdal/? <
> http://spatial.nhh.no/R/rgdal/?> Please do say what you mean. If your
> browser blocks http: rather than https:, that is your choice. I imagine
> that it silently inserts an "s".
> >
> >>
> >> FWIW I’ve tried it locally using the latest svn revision but have no
> clue where I should find `proj_api.h` (I’ve looked in the complete proj
> installation directory).
> >
> > If PROJ 7.0.1 is properly installed, the header will be in
> $prefix/include. If your packager put all the headers somewhere else, use
> the command argument:
> >
> > $ ls /usr/local/include/proj*
> > /usr/local/include/proj_api.h           /usr/local/include/proj.h
> > /usr/local/include/proj_constants.h
> /usr/local/include/proj_symbol_rename.h
> > /usr/local/include/proj_experimental.h
> >
> > /usr/local/include/proj:
> > common.hpp               crs.hpp    metadata.hpp
> > coordinateoperation.hpp  datum.hpp  nn.hpp
> > coordinatesystem.hpp     io.hpp     util.hpp
> >
> > You should not need to know. What does pkg-config proj --cflags say
> (once you've set PKG_CONFIG_PATH, of course.
> >
> >>
> >> I see
> >>
> >> ```
> >> ls /usr/local/Cellar/proj/7.0.1/lib/
> >> libproj.19.dylib  libproj.a         libproj.dylib@    libproj.la <
> http://libproj.la/>* pkgconfig/
> >> ```
> >>
> >
> > In your earlier post, you negected to say that you used the necessary
> configure argument. The error message says that you did not use it. Please
> report back when you have done so. Futher, I quote from your message: "It
> further causes complete breakage for people relying on the homebrew package
> manager on macOS since current versions are gdal v2.2.4 and proj 7.0.1."
> Please accept that I pay attention to what you write.
> >
> >> Even if this custom installation might work at some point, making such
> adjustments until gdal3 arrives in homebrew will be a major pain, also for
> CI builds.
> >> It would still be highly appreciated if this internal assertion would
> be removed again.
> >>
> >
> > Getting things right may be painful; GDAL 2 with PROJ 7 is wrong,
> because it mixes two incompatible metadata storage systems. Migrating all
> users to
> > GDAL >= 3 & PROJ >= 6 is a matter of urgency to avoid silent positional
> error of about 125m, or about 4 satellite image cells. See the vignette
> here
> http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html
> <
> http://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html>
> and this r-spatial blog https://www.r-spatial.org/r/2020/03/17/wkt.html <
> https://www.r-spatial.org/r/2020/03/17/wkt.html>.
> >
> > I am not willing to let silent error creep in because some people find
> it inconvenient to avoid the problem. Both sf and sp/rgdal are covering
> your backs even if it feels rough. We've already helped dozens of
> maintainers of reverse dependencies future-proof their users from the
> consequences of changes made in PROJ and GDAL, where we have also
> contributed to making the transition easier by exposing helper functions in
> their APIs, which we use.
> >
> > I expect you to be able to CMD check a tarball from the commmand line,
> no more than that, to report back, and to try to grasp that these external
> software components are really important and completely non-trivial.
> >
> > Roger
> >
> >> On 29 May 2020, at 18:37, Roger Bivand wrote:
> >>
> >>> I have put a tarball, built without vignettes with GDAL 2.2.4 and PROJ
> >>> 7.0.1 that passes CMD check (with warnings for missing vignettes) on:
> >>>
> >>> http://spatial.nhh.no/R/rgdal/gdal_1.5-9.tar.gz
> >>>
> >>> It involves code copying to provide duplicated functions in three
> >>> settings:
> >>>
> >>> PROJ < 6 & GDAL < 3
> >>> PROJ > = 6 & GDAL >= 3
> >>> PROJ > = 6 & GDAL < 3 (your homebrew case)
> >>>
> >>> You need the configure argument:
> >>> --configure-args=--with-proj_api=proj_api.h.
> >>> Without the argument, you now get directed to use it.
> >>>
> >>> For now I've dropped the configure tests for sqlite3, curl and tiff;
> works
> >>> for me but may not work for you.
> >>>
> >>> Please make the output of:
> >>>
> >>> R CMD check
> --install-args="--configure-args=--with-proj_api=proj_api.h"
> >>> rgdal_1.5-9.tar.gz
> >>>
> >>> available ASAP. I count on an immediate response.
> >>>
> >>> Roger
> >>>
> >>> On Fri, 29 May 2020, Roger Bivand wrote:
> >>>
> >>>> On Fri, 29 May 2020, Patrick Schratz wrote:
> >>>>
> >>>>>  The just released rgdal v1.5.8 requires gdal >= 3 when proj >= 6 is
> >>>>>  present. It does not even install the package if this condition is
> >>>>> not met
> >>>>>  but errors during linking.
> >>>>
> >>>> Please document with the output of 00install.out from R CMD check
> >>>> rgdal_1.5-8.tar.gz, and pkg-config proj --modversion, gdalinfo
> --version
> >>>> with system details.
> >>>>
> >>>> Yes, PROJ >= 6 makes no sense without GDAL >= 3; GDAL 3 makes no sense
> >>>> with PROJ < 6. GDAL 3 with PROJ < 6 uses the old proj_api.h API and
> the
> >>>> old metadata structure with Proj4 string degradation; GDAL < 3 with
> PROJ
> >>>>> = 6 must use the deprecated API and the new metadata structure.
> >>>>>
> >>>>> I could not find any sources in the web or in the announcement post
> of
> >>>>>  v1.5.8 why this condition was enforced so strictly.
> >>>>>  Does it cause unwanted results?
> >>>>
> >>>> Yes, definitely. GDAL 3 was released as 3, not 2.5, to stress the
> >>>> complete break between GDAL 2 (with PROJ <= 5) and GDAL >= 3 (with
> PROJ > = 6) after GDAL barnraising. So:
> >>>>
> >>>>           PROJ
> >>>>           < 6 >= 6 GDAL < 3   OK   NO
> >>>>   >  = 3   NO   OK
> >>>>
> >>>>> If so, then there is a big issue since the “gdal2 and proj >= 6”
> >>>>>  combination has been used by many people in the past already.
> >>>>
> >>>> No good precedent, just binary packagers protecting slow downstream
> >>>> adaptation.
> >>>>
> >>>>> I haven’t tracked down for how long this situation was on the market
> >>>>>  already.
> >>>>>  Also the {sf} package is still installable with this combination
> >>>>> and I
> >>>>> am not aware of any warnings/issues that came up due to this so far.
> >>>>
> >>>> rgdal is a much older package and has much more older code that needs
> >>>> this protection. It is possible to accommodate the no-go areas, but
> I'd
> >>>> really value patches to R-forge, as I cannot check multiple PROJ/GDAL
> >>>> version combinations just because someone isn't up to speed.
> >>>>
> >>>>> It further causes complete breakage for people relying on the
> homebrew
> >>>>> package manager on macOS since current versions are gdal v2.2.4 and
> proj
> >>>>>  7.0.1.
> >>>>
> >>>> They should never have gone beyond PROJ 5 if they are stuck on GDAL.
> PROJ
> >>>> 7 is a very different animal.
> >>>>
> >>>>> The update to gdal3 is blocked since months due to an incompatibility
> >>>>> with `liblas` (https://github.com/libLAS/libLAS/issues/164).
> Reading the
> >>>>>  issue, it looks unclear when this issue will be resolved.
> >>>>>  The alternative osgeo4mac tap holds formula that is unstable and
> >>>>>  somewhat broken. One cannot link rgdal with the current gdal v3.0.1
> >>>>> from
> >>>>>  osgeo4mac.
> >>>>>
> >>>>> This incompatibility with homebrew-core formula leads to further
> issues
> >>>>> for CI tests that rely on the spatial homebrew stack for macOS
> builds.
> >>>>>
> >>>>> All of this is really unfortunate and I am wondering if this was a
> known
> >>>>>  factor when introducing this condition in the new release.
> >>>>>  Unfortunately rgdal is not hosted on any Git* provider (there is
> >>>>> just
> >>>>> the CRAN mirror with not much information about the recent changes)
> and
> >>>>> it is unclear/untransparent to me if there is a continuous
> integration
> >>>>>  setup at all for this package.
> >>>>
> >>>> Do not require me to leave the excellent SVN service on R-forge.
> Provide
> >>>> patches there.
> >>>>
> >>>>> I hope this is not another case in the series “I do not like
> operating
> >>>>> system XY and hence I do not test on it” which was seen recently in
> >>>>>  another package. That’s what CI is for and the responsibility of
> >>>>>  package authors.
> >>>>
> >>>> I run and develop on Fedora, I  have no access to OSX, and the current
> >>>> release was fully checked with ~ 900 revdeps repeatedly since
> November on
> >>>> successive older and newer versions of PROJ and GDAL. I blocked new
> PROJ
> >>>> with old GDAL recently to simplify development. CI is only as good as
> the
> >>>> scripts, I am completely sure that repeated revdep testing and reading
> >>>> the output is superior.
> >>>>
> >>>>> Also one should be aware that not every rgdal user is member of this
> >>>>> mailing list (I guess not even 5%) and many people in production will
> >>>>> face this error during install, most of them without any clue what
> to do
> >>>>>  or an idea why this was raised.
> >>>>
> >>>> There has been plenty of information given about the CRS changes.
> rgdal
> >>>> could simply have been abandoned by me, and all  those production
> >>>> volunteers might have fixed things, but I never hear anything at all.
> >>>>
> >>>>>  In summary it would be great if
> >>>>>
> >>>>> - There would be more detailed information on the introduction of the
> >>>>>  new condition
> >>>>>  - Awareness of state-of-the-art platform related library versions
> >>>>> before
> >>>>>  making such a release
> >>>>>  - Transparency/Existence of a cross-platform build process
> >>>>>  - If the incompatibility is just due to some features not being
> >>>>>  accessible with this configuration, I suggest to raise a warning
> >>>>> during
> >>>>> package load but not prevent the package from installing in the first
> >>>>>  place
> >>>>>  - A version-based NEWS file would be available. Currently one only
> >>>>> sees
> >>>>>  a revision-based Changelog:
> >>>>>  https://cran.r-project.org/web/packages/rgdal/ChangeLog
> >>>>>
> >>>>>  I am well aware that I am indirectly criticising (hopefully in a
> >>>>>  constructive way) the transparency and release process here, for
> >>>>> which I
> >>>>>  will probably get a rough response.
> >>>>>  However, if that leads to more robustness and transparency in
> >>>>> future
> >>>>>  release, I am fine with this.
> >>>>>
> >>>>>  A quick patch release resolving the breakage would be highly
> >>>>>  appreciated.
> >>>>
> >>>> Only with community imput. what you ask is not needed, just extra
> >>>> complexity. Please provide patches, or accept my invitation to join
> the
> >>>> R-forge project and commit your fixes directly. I can see how to do
> it,
> >>>> but I don't think it makes sense, and your messsage has not motivated
> me,
> >>>> to be honest. I'm prioritizing working with CRAN to iron out reverse
> >>>> dependency problems.
> >>>>
> >>>> Roger
> >>>>>
> >>>>>  Best, Patrick
> >>>>>   [[alternative HTML version deleted]]
> >>>>>
> >>>>>  _______________________________________________
> >>>>>  R-sig-Geo mailing list
> >>>>>  R-sig-Geo using r-project.org
> >>>>>  https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>>
> >>> --
> >>> Roger Bivand
> >>> Department of Economics, Norwegian School of Economics,
> >>> Helleveien 30, N-5045 Bergen, Norway.
> >>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
> >>> https://orcid.org/0000-0003-2392-6140
> >>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
> >>
> >>
> >
> > --
> > Roger Bivand
> > Department of Economics, Norwegian School of Economics,
> > Helleveien 30, N-5045 Bergen, Norway.
> > voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no <mailto:
> Roger.Bivand using nhh.no>
> > https://orcid.org/0000-0003-2392-6140 <
> https://orcid.org/0000-0003-2392-6140>
> >
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
> <
> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
> >
> > R-sig-Geo mailing list
> > R-sig-Geo using r-project.org <mailto:R-sig-Geo using r-project.org>
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo <
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

	[[alternative HTML version deleted]]

More information about the R-sig-Geo mailing list