[R-sig-Geo] rgdal release candidate 1.5-9 rev. 1000 ready for testing
Roger Bivand
Roger@B|v@nd @end|ng |rom nhh@no
Sat Jun 6 21:09:01 CEST 2020
Patrick,
Thanks for this ...
On Sat, 6 Jun 2020, Patrick Schratz wrote:
> Roger,
>
> thanks for your extensive answer, much appreciated.
>
> I won't commenting on every detail, this would make the discssion somewhat
> bloated.
>
> It's always the "cat and mouse game" between the libraries, packages
> managers and client packages. However, the chain is library > package
> manager > client package in my opinion, meaning the client package has
> the burden to at least check the first two and provide **at least**
> instructions for every OS how to get the package installable.
On Linux, I never myself use the package manager for critical libraries,
always installing R and software like PROJ/GDAL/GEOS from source. I'll
admit that it is years since, following Brian Ripley's advice, I installed
GCC from source to explore a particular wrinkle, but for me it has been
important to avoid the decisions made by package manager maintainers
interfering with my view of the relationships between the external
software and compiled code in packages for which I am responsible.
Recently, for example, it turned out that the R rpm package picked certain
values for LDFLAGS, which then got in the way of sf's configure script
https://github.com/r-spatial/sf/issues/1369.
For Windows, we know how to install, using a script in packages to
download the libraries and share/* files; it is robust and works.
For Linux, some of us install external software from source, it works, but
we need to handle mutual dependencies manually (say knowing which PROJ and
GEOS GDAL was built with). Some use Debian and Ubuntu package managers,
which are versioned to the level of the OS, and generally work; some repos
are more actively maintained than others. Some use ArchLinux and other
distributions, maybe source installs are sensible. For Centos/RHEL/Fedora,
the picture is rather mixed, with slowish tracking of the fast speed of
API and ABI change in PROJ and GDAL. Package managers also face
misunderstandings about when an ABI change needs to be flagged by changing
a minor version or patch number - this happened when GDAL 3.1.0 RCs were
being checked:
https://lists.osgeo.org/pipermail/gdal-dev/2020-May/052060.html. Here the
careful attention of a Debian packager working with us led to a
last-minute bump of the SONAME, also thanks to Even Rouault, who is
unfailingly helpful.
Just keeping things more or less functioning at this level is already
pretty absorbing.
Getting to CRAN binaries for macOS will take a little longer. From
R-sig-mac, we know that Simon is not convinced that package managers are
of use in getting to a general resolution, but he has published recipes:
Simon replied to me pointinng to them:
https://stat.ethz.ch/pipermail/r-sig-mac/2020-May/013537.html,
https://github.com/R-macos/recipes
Since he has asked for input, I think that this is likely to be a
fruitful way forward, but as noted in this thread, it is getting harder to
build PROJ and GDAL static, to get to static CRAN binary packages.
>
> The dev toolchain on macOS in general and how things are linked/build is a
> different topic.
> For this, I am somewhat missing (until recently) public accessibility of the
> build process/toolchain, even though things are moving forward with the
> creation of https://github.com/R-macos?type=source lately.
Exactly, it has been really difficult to make this kind of progress, so
positive contributions and patience seem most useful.
>
> Regarding GDAL2 and PROJ>=6: If there is such an offset in spatial
> projections using this combination, I welcome the blocker. However, this
> was not clear in any way before your last post a few hours ago. However,
> I would also welcome more information on this then, either during
> installation attempts or at some other place. Maybe even opening an
> issue at the respective package manager informing the people about the
> dangerous situation. The more attraction an issue gets, the earlier it
> gets fixed.
The package maintainers are also in a difficult position with regard to
packages/applications that still need GDAL 2. The offsets are not always
there, but until end users have had the opportunity to check their
workflows, we won't know. Certainly the announced degradation of GDAL's
OSR::exportToProj4() suprised a lot of us, even though we knew it was
coming. For me teaching the John Snow Soho Cholera story and finding the
Broad Street pump in Ingestre Place was a strong motivation to get users
to pay attention. Again, it will not affect everyone, but it may do so.
>
> Apologies if some of my recent messages were offensive to you or anyone else.
> However, sometimes discussions need "clear" statements with emotions
> excluded.
>
> In the end, we all aim for a smooth experience and aim to have everybody on
> board.
Certainly everybody on board, locations in data and the real world lined
up adequately, smooth at the moment is a big ask, but once we get the
community over to WKT2, PROJ >= 6 and GDAL >= 3, we'll be much, much
calmer.
Best wishes,
Roger
>
> Best, Patrick
>
> On 6 Jun 2020, at 18:09, Roger Bivand wrote:
>
>> On Sat, 6 Jun 2020, Patrick Schratz wrote:
>>
>>> Roger,
>>>
>>> I am sorry for arguing differently so often recently, but if I think that
>>> unfair arguing is going on, I have the feeling to correct this.
>>>
>>> First, again I think that stating "I do not have access to this system"
>>> is a weak reason in 2020.
>>
>> Your view, not mine. I cannot ask my employer to provide such resources.
>>
>>> I've said this previously but again: As a developer/maintainer, there is
>>> the implicit burden to set up a dev environment across different
>>> platforms. CI helps here.
>>
>> No, this is not the main burden. For forward-facing packages interfacing
>> external software, it is vastly more important to track developments in
>> that software, by reading development mailing lists, and checking against
>> alpha, beta and RC, as well as tracking master/trunk in periods between
>> releases or when things happening in the external software seems to have
>> the potential to impact existing R packages. This obviously absorbs a lot
>> of time and energy. https://github.com/r-spatial/sf/issues/545 documents
>> where we were in November 2017 with this; there are lots of other examples
>> in posts on the PROJ and GDAL lists, and in email exchanges with others.
>>
>>
>>> For more detailed testing, local emulation is possible via virtual
>>> machines which also applies to macOS systems (setting up is harder but
>>> not impossible).
>>
>>> Before switching back to macOS a few months ago, I had a virtual machine
>>> of macOS running and used it successfully for dev purposes.
>>>
>>
>> The problem with macOS compared to Windows has been that while providing
>> static builds for Windows for three iterations of the Windows RTools build
>> train, with help from CRAN and others, most recently Jeroen, has gone
>> fairly smoothly for more than ten years, macOS moves too quickly. For
>> Simon to keep R itself running has been an uphill struggle, with problems
>> with Fortran, versions of system software under clang (most recently the
>> archaic ssl shipped by Apple), so providing fresh builds of PROJ, GDAL and
>> GEOS has understandably not been a priority. We do not build Windows or
>> macOS PROJ, GDAL or GEOS libraries, we need others, probably close to
>> CRAN, to do this, following CRAN static linking policy.
>>
>> Note that Brian Ripley continues to monitor and check possibilities for
>> static builds of OSGeo software for static linking to CRAN binary packages
>> for macOS, but does not have breakthroughs to report for recent PROJ or
>> GDAL releases, unfortunately. We are very fortunate that he continues to
>> help us with this. rgdal 1.5-9 is partly to resolve your problem, but just
>> as much to resolve problems on Solaris and other corner-case CRAN check
>> platforms.
>>
>> Static linking isn't the only policy, but only Linux platforms provide
>> (fairly) mature dynamic build package managers. OSGeo4W - built on cygwin
>> - has been tried by some over the years, because CRAN static linked binary
>> packages may have different versions than say QGIS, GRASS or other Windows
>> binaries. We have touched on the idea of proposing R-spatial as an OSGeo
>> project, among other things to try to match the versions of key software
>> components better. But having many alternatives for source install on
>> macOS runs counter to this, we need to find a path that works - the
>> previous go-to was Kyngchaos, but that is no longer a viable solution.
>>
>>> Second: All package managers seek to provide stability and homebrew is
>>> one of the most sophisticated ones out there. I honestly do not
>>> understand the general bashing against package managers here.
>>
>> Package managers were debated on R-sig-mac (yes, I follow that too). If
>> you use a package manager, that is your choice, but having CI for macOS
>> (past and current releases) then gets multiplied by the number of package
>> managers and environments (conda), etc. For central resources, say curl,
>> package managers may be OK, but get stuck when some downstream packages
>> either get dropped because they are not up to speed with say PROJ/GDAL, or
>> PROJ/GDAL get held up because the packages using them are seen as
>> important. This happens in all package managers, and is frequently
>> discussed on PROJ and GDAL lists, amonng others by those responsible for
>> package manager releases. The decisions are not easy, but we cannot be
>> held back by package managers' policy choices. My guess is that some
>> package in homebrew depends on GDAL 2.* and has not yet upgraded to be
>> able to use GDAL 3.*; since GDAL 2.* can be built (although using the
>> deprecated API) with PROJ >= 6, they let things slide. They should have
>> chosen to stop PROJ at 5.2.*, avoiding the used of the deprecated API.
>>
>>>
>>> Third: What role does Apple play in all of this? I am not arguing that
>>> they made some decisions that did not necessarily enhance the dev
>>> experience on macOS.
>>> However, I do not see any of these having an effect on the spatial
>>> library stack, especially GDAL and PROJ.
>>>
>>> The current situation is that the **main** package manager on macOS,
>>> namely homebrew, has a temporary version situation of GDAL and PROJ that
>>> is (for no clear reason yet) blocked by the client package rgdal.
>>>
>>> Users on macOS can however use the formulas of the osgeo4mac
>>> (https://github.com/OSGeo/homebrew-osgeo4mac) tap which comes with PROJ7
>>> and GDAL3 to solve these issues.
>>>
>>> ```r
>>> brew tap osgeo/osgeo4mac
>>> brew unlink gdal
>>> brew unlink proj
>>>
>>> brew install osgeo-gdal
>>> brew install osgeo-proj
>>> brew link osgeo-gdal
>>> brew link osgeo-proj
>>> ```
>>>
>>> I am well aware that many users of spatial software are not developers in
>>> their every day life and should stick to binaries.
>>> However, there is a group that does source installs and there are CI
>>> checks that rely on proper source installations with the current stack of
>>> the main package managers on a platform.
>>> And exactly this group is blocked right now by blocking the rgdal
>>> installation at all, with a somewhat weak reasoning for this action.
>>>
>>
>> Block is your expression, what is being required is that users actively
>> opt in to using a very-short-life deprecated API. If I relax the
>> requirement (I have already opened up for using the deprecated API, which
>> I regard as potentially leading to positional accuracy loss, and certainly
>> encouraging business as usual rather than active migration of workflows to
>> WKT2 from Proj4 strings), I would be demonstrating crass irresponsibility
>> with regard to users of rgdal, who need to know that these changes (in
>> GDAL/PROJ) may impact their work.
>>
>>> In addition, arguing/ranting about specific platforms with points
>>> unrelated to the current issues is a thing that I absolutely dislike,
>>> getting me started arguing against it.
>>> I also do not like certain platforms and have personal favorites.
>>> However, I always check on all and make sure that everyone gets a
>>> pleasant experience on their platform, even if that's painful for me and
>>> costs a lot of time.
>>
>> Pleasant experience is not something that carries much weight. Things need
>> to work, and if user choices may lead to degradation through use of a
>> deprecated API, I feel that they should be warned, and that that is more
>> important than any pleaasant experience.
>>
>> For many years, OSX has been checked by CRAN, and this will continue. I do
>> not accept that package maintainers have to resolve these problems (source
>> installs of packages using external software), if they follow up CRAN
>> requirements and make sure that the source packages continue to install
>> and check cleanly with successive versions of the external software.
>>
>>>
>>> I am well aware that I might not be invited to the next imaginary party
>>> with my arguing here but maybe the discussion can make use of more real
>>> facts again, lower subjective views, and focus on providing support for
>>> everyone, on all platforms, without introducing something like a
>>> "platform racism" with semi-fake news.
>>
>> I'm uncertain what you are referring to. I do not have a problem with
>> platforms, I just see the role of package developers differently, as I
>> explained above. At some point this year or next, the CRAN Windows and
>> macOS binaries will have caught up (Windows already at PROJ 6.3.1 aand
>> GDAL 3.0.4), so that for all users and developers but those insisting on
>> source installs, the temporary difficulties now being experienced will
>> pass. For those installing from source, when package managers catch up
>> with the speed of change in PROJ and GDAL, things will calm down.
>>
>> Roger
>>
>>>
>>> Best, Patrick
>>>
>>> On 6 Jun 2020, at 15:45, Roger Bivand wrote:
>>>
>>>> On Sat, 6 Jun 2020, Ista Zahn wrote:
>>>>
>>>>> On Fri, Jun 5, 2020 at 7:47 PM Manuel Spínola <mspinola10 using gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Dear list members,
>>>>>>
>>>>>> Sorry for the confusion, but with all these suggestions, what is the
>>>>>> way
>>>>>> to
>>>>>> have the updated versions of the external
>>>>>> software GEOS, PROJ, and GDAL for macOS users.
>>>>>
>>>>> I think the current recommendation is "if you have to ask don't do
>>>>> it". Just wait for these to be updated in the OSX binary packages
>>>>> on
>>>>> CRAN.
>>>>
>>>> Thanks, a much better way of saying this!
>>>>
>>>> We really would like to be able to help macOS users who see "Install
>>>> from
>>>> source?" and are tempted to choose "yes", but not only do we not have
>>>> the
>>>> resources or access to running systems, but also, at the moment, things
>>>> seem very unpredictable. We do not think that environments are helpful,
>>>> and many package managers do not seem to have sufficient focussed
>>>> attention, which is understandable given Apple's gift for moving
>>>> the
>>>> goalposts.
>>>>
>>>> If users can install external software from source (macOS, Linux),
>>>> they/we
>>>> have a good deal of freedom. But this takes time, insight, and for many
>>>> is
>>>> problematic because their production system is blocked until the new
>>>> versions are ready (PROJ and GDAL are C++11 or more, and take an order
>>>> of
>>>> magnitude longer to compile than just a few years ago).
>>>>
>>>> So for Windows and macOS, waiting for the CRAN binaries is a reasonable
>>>> choice.
>>>>
>>>> Beyond this, we need to find ways of providing share/proj and share/gdal
>>>> metadata files for all of the packages now using the PROJ and GDAL
>>>> libraries, and of navigating the content download network for
>>>> geodetic
>>>> transformation grids available from PROJ 7. But that is another story
>>>> ...
>>>>
>>>> Roger
>>>>
>>>>>
>>>>> Best,
>>>>> Ista
>>>>>
>>>>>>
>>>>>> Manuel
>>>>>>
>>>>>> El vie., 5 jun. 2020 a las 14:31, Patrick Schratz (<
>>>>>> patrick.schratz using gmail.com>) escribió:
>>>>>>
>>>>>>> I am not sure if the part with
>>>>>>>
>>>>>>> use --with-proj_api="proj_api.h" for deprecated API
>>>>>>>
>>>>>>> Is of much help since c/p won’t work but the text let’s people
>>>>>>> assume that c/p could/should work.
>>>>>>> In fact, a full path to “proj_api.h” is required?
>>>>>>>
>>>>>>> I still do not like this blocker and I still do not know if this
>>>>>>> combination causes serious issues in production or just limits
>>>>>>> new
>>>>>>> features.
>>>>>>>
>>>>>>> For the time being, using and linking osgeo-gdal (3.0.1) and
>>>>>>> osgeo-proj
>>>>>>> (7.0.1) works and can be used as a workaround until homebrew-core
>>>>>>> formulas catch up.
>>>>>>>
>>>>>>>> checks OK on PROJ 7.0.1 and GDAL 2.2.4
>>>>>>>
>>>>>>> Again, since it was maybe caused by my typo a few mails ago: The
>>>>>>> homebred-core gdal version is at 2.4.4 and not 2.2.4.
>>>>>>>
>>>>>>> On 5 Jun 2020, at 13:29, Roger Bivand wrote:
>>>>>>>
>>>>>>>> The release candidate of rgdal_1.5-9 is ready for testing on
>>>>>>>> R-forge:
>>>>>>>>
>>>>>>>> https://r-forge.r-project.org/R/?group_id=884
>>>>>>>>
>>>>>>>> Those insisting on installing on PROJ >= 6 and GDAL < 3 must use
>>>>>>>> configure argument --with-proj_api="proj_api.h"; with this used,
>>>>>>>> this
>>>>>>>> version builds with --no-build-vignettes and installs and checks OK
>>>>>>>> on
>>>>>>>> PROJ 7.0.1 and GDAL 2.2.4 with --with-proj_api="proj_api.h".
>>>>>>>>
>>>>>>>> Otherwise checked OK with PROJ 4.8.0, 4.9.2, 4.9.3 and 5.2.0 with
>>>>>>>> GDAL
>>>>>>>> 1.11.4; with PROJ 5.2.0 and GDAL 2.2.4, 2.3.2 and 2.4.2; with PROJ
>>>>>>>> 6.3.2 and GDAL 3.0.4; with PROJ 7.0.1 and GDAL 3.0.4 and 3.1.0.
>>>>>>>>
>>>>>>>> All who have indicated issues with source installs are asked to try
>>>>>>>> the release candidate and to report back here by midnight CEST
>>>>>>>> Monday
>>>>>>>> 8 June. If no indications are forthcoming, I'll assume that problems
>>>>>>>> with 1.5-8 are resolved.
>>>>>>>>
>>>>>>>> Roger
>>>>>>>>
>>>>>>>> --
>>>>>>>> Roger Bivand
>>>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> R-sig-Geo mailing list
>>>>>>>> R-sig-Geo using r-project.org
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-Geo mailing list
>>>>>>> R-sig-Geo using r-project.org
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Manuel Spínola, Ph.D.*
>>>>>> Instituto Internacional en Conservación y Manejo de Vida
>>>>>> Silvestre
>>>>>> Universidad Nacional
>>>>>> Apartado 1350-3000
>>>>>> Heredia
>>>>>> COSTA RICA
>>>>>> mspinola using una.cr <mspinola using una.ac.cr>
>>>>>> mspinola10 using gmail.com
>>>>>> Teléfono: (506) 8706 - 4662
>>>>>> Personal website: Lobito de río
>>>>>> <https://sites.google.com/site/lobitoderio/>
>>>>>> Institutional website: ICOMVIS <http://www.icomvis.una.ac.cr/>
>>>>>>
>>>>>> [[alternative HTML version deleted]]
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-Geo mailing list
>>>>>> R-sig-Geo using r-project.org
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-Geo mailing list
>>>>> R-sig-Geo using r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>
>>>>
>>>> --
>>>> Roger Bivand
>>>> Department of Economics, Norwegian School of Economics,
>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>> https://orcid.org/0000-0003-2392-6140
>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo using r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>> --
>> Roger Bivand
>> Department of Economics, Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>> https://orcid.org/0000-0003-2392-6140
>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
>
>
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
More information about the R-sig-Geo
mailing list