[R-sig-Geo] rgdal 1.5-8 released on CRAN

Roger Bivand Roger@B|v@nd @end|ng |rom nhh@no
Tue Jun 2 17:33:23 CEST 2020 

On Sun, 31 May 2020, Simon Urbanek wrote:

> I believe you have to activate the anaconda environment to use it - but 
> that means you have to use R from Anaconda and it is incompatible with 
> anything else. All the package management systems are a horrible mess,

"All the package management systems are a horrible mess" is above the 
fortune threshold IMHO.

> because they now replace system libraries which makes them incompatible 
> with the system and each other. Is there a particular reason why you’re 
> not using the CRAN version? I know there was a discussion about how 
> broken some of the most recent versions of the spatial libraries are, 
> but I have not seen any requests yet...

There are several inter-twined problems here.

One is building rgdal from source on macOS using (I think from homebrew) 
mismatched versions of GDAL (<3) and PROJ (>= 6). rgdal 1.5-8 said no, 
1.5-9 says use the mismatched external software versions at your own 
responsibility (thread started here: 
https://stat.ethz.ch/pipermail/r-sig-geo/2020-May/028131.html).

Another is one about which I tried to elicit response earlier wih no 
success, but which affects Roy: with PROJ 7, CDN download of 
transformation grids becomes possible, but organizations may well want to 
make sure that all organizational users have the same geodetic 
transformation metadata. This extends across programs, because PROJ 
underlies QGIS, pyproj, GRASS, Saga, R spatial, and so on. The CRAN macOS 
binaries, like the CRAN Windows binaries, have bundled proj and gdal 
geodetic transformation metadata, which work, but may differ by version 
from the proj and gdal geodetic transformation metadata used by other 
programs.

We need to consider a) a separate (versioned) data-only CRAN package for 
Windows and macOS packages built against PROJ and GDAL to use, shipping 
once, not in each package built against PROJ and GDAL to use as at 
present. If the users can choose which version to install, they could 
match it to organization policy. b) is to re-think source installs for 
Windows (and macOS) to use resources like the fairly well-supported 
OSGeo4W for Windows. Maybe the organization data managers don't know or 
care that different versions of geodetic transformation metadata give 
different positions, but I think ongoing changes in many national mapping 
agencies suggest that, since 1984 is not recent and tectonic movement does 
happen, the versions of the metadata will matter more going forward.

Simon, I don't think we're ready to conclude about recipies for macOS; 
certainly I am not confident about suggesting anything I cannot test on 
hardware before committing to a choice. CRAN Windows binaries now ship 
with PROJ 6.3.1 and GDAL 3.0.4, but so far we haven't seen user reaction. 
sf has been using these for some weeks, rgdal was only published built on 
these very recently. Maybe we've brought the house down and nobody has 
emerged from the rubble yet ... Let's see user response to the CRAN 
Windows binaries and consider a next move then? Again, without access to 
hardware, I'm rather stuck.

Best wishes,

Roger

>
> Cheers,
> Simon
>
>
>> On May 31, 2020, at 3:06 PM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn using noaa.gov> wrote:
>>
>> They are all from Anaconda Python.  I used otool and that is indeed how 
>> they are set up (see below - and libgdal.26.dylib is the image it can't 
>> find) .  I tried DYLD_FALLBACK_LIBRARY_PATH also a little while ago and 
>> it didn't make any difference, and we are a the point where it is 
>> beyond where I really know things.
>>
>>> otool -D /Users/rmendels/anaconda3/lib/libgdal.26.dylib
>>> /Users/rmendels/anaconda3/lib/libgdal.26.dylib:
>>> @rpath/libgdal.26.dylib
>>
>>> otool -L /Users/rmendels/anaconda3/lib/libgdal.26.dylib
>>> /Users/rmendels/anaconda3/lib/libgdal.26.dylib:
>>> 	@rpath/libgdal.26.dylib (compatibility version 27.0.0, current version 27.4.0)
>>> 	@rpath/libtiledb.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> 	@rpath/libpoppler.78.dylib (compatibility version 78.0.0, current version 78.0.0)
>>> 	@rpath/libjson-c.4.dylib (compatibility version 5.0.0, current version 5.0.0)
>>> (and a bunch more)
>>
>> I only mention these because they seem to have the complete set of PROJ,  GEOS and GDAL at the newer versions,  as well as the headers and *-config files.  I have not been able to find another place that has everything  (well perhaps brew has been updated,  but brew has messed up my system in the past and last thing on this list is that it has not.)  Also my usually source,  Kyngchaos is still at gdal2.4
>>
>> Chris suggested it was a permissions problem,  but to use sudo I would have to get set up on a different machine,  because on this machine I can only sudo as a different user,  and that user doesn't have the necessary libraries and packages,  and to set that up would be more work than I want to do right now.
>>
>> So I was just hoping that if I could get it to compile,  I could post what I did  (I realized posting the package wouldn't work because you would need the dynamic libraries) and others could also do so if so desired.  But I am pretty much at the point where I don't really understand what the next steps would be,  and to find out would take more effort that I want to put into it right now. So I was hoping you (or someone else with a lot of knowledge about compiling on the Mac) might have suggestions.
>>
>> Thanks,
>>
>> -Roy
>>
>> PS - I have some packages that use sp and sf,  want to make certain I can test against the most recent versions using the newer libraries.  I can see if errors are generated using some of the test platforms,  but if there are errors they don't allow me to get in and debug.
>>
>>
>>
>>> On May 30, 2020, at 7:40 PM, Simon Urbanek <simon.urbanek using R-project.org> wrote:
>>>
>>> Roy,
>>>
>>> where did you get the libraries? If they use rpath it's quite tricky - that is not how native libraries work on macOS. Also the setting on macOS is not LD_LIBRARY_PATH but DYLD_FALLBACK_LIBRARY_PATH and is typically not needed unless the library is not built properly since macOS offers better options.
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>>
>>>> On May 31, 2020, at 6:11 AM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn using noaa.gov> wrote:
>>>>
>>>> Hi All:
>>>>
>>>> Okay after some fiddling I was able to get it to sort of successfully compile,  where I am having problems is in finding the dynamic libraries, such as I get this error:
>>>>
>>>>> dyld: Library not loaded: @rpath/libproj.19.dylib
>>>>
>>>> and
>>>>
>>>>> Error: package or namespace load failed for ‘rgdal’ in dyn.load(file, DLLpath = DLLpath, ...):
>>>>> unable to load shared object '/Users/rmendels/Library/R/4.0/library/00LOCK-rgdal/00new/rgdal/libs/rgdal.so':
>>>>> dlopen(/Users/rmendels/Library/R/4.0/library/00LOCK-rgdal/00new/rgdal/libs/rgdal.so, 6): Library not loaded: @rpath/libgdal.26.dylib
>>>>> Referenced from: /Users/rmendels/Library/R/4.0/library/00LOCK-rgdal/00new/rgdal/libs/rgdal.so
>>>>> Reason: image not found
>>>>> Error: loading failed
>>>>
>>>> It has to do with these are defined in terms of @rpath.  I thought defining this in $LD_LIBRARY_PATH would do it,  but I have:
>>>>
>>>>> echo $LD_LIBRARY_PATH
>>>>> /Users/rmendels/Library/R/4.0/library:/Users/rmendels/anaconda3/lib
>>>>
>>>> I also tried setting 	$DYLD_LIBRARY_PATH same problem.  If someone can tell me how to set it so @rpath finds the dynamic libraries I might be able to get this to work.
>>>>
>>>> Thanks,
>>>>
>>>> -Roy
>>>>
>>>>
>>>>
>>>>> On May 30, 2020, at 8:04 AM, chris english <englishchristophera using gmail.com> wrote:
>>>>>
>>>>> my setup - albeit on 20.04 as reported by sf
>>>>>
>>>>>> library(sf)
>>>>> Linking to GEOS 3.9.0dev, GDAL 3.1.0, PROJ 7.0.1
>>>>>
>>>>> Note I'm using release gdal, that I none the less built from source as I had to enable Mr. Sid for some particular
>>>>> data. Couldn't get GDAL 3.2.*dev working but still toying around with it. Roger's discussion of the history of all
>>>>> of this on r-spatial and the path forward is very much worth the read as and relates to his following message here.
>>>>>
>>>>>
>>>>>
>>>>> On Sat, May 30, 2020 at 10:33 AM Roy Mendelssohn - NOAA Federal <roy.mendelssohn using noaa.gov> wrote:
>>>>> Yes thanks.  As I said in reply to Roger I am far from an expert in these things,  but if I can get it to build,  then I will try to post somewhere for others to use.
>>>>>
>>>>> -Roy
>>>>>
>>>>>> On May 30, 2020, at 7:03 AM, chris english <englishchristophera using gmail.com> wrote:
>>>>>>
>>>>>> Roy,
>>>>>> From a related prior post Roger shared this link to rgdal-1.5-9,  http://spatial.nhh.no/R/rgdal/ . I followed discussions
>>>>>> from the 'sf' github readme and purged all prior proj/geos/gdal iterations that have lingered around over the past five
>>>>>> years and built them from source on ubuntu 20.04. This upgrade to 20.04 was perhaps my first mistake in a month
>>>>>> long effort to bring back my full spatial capabilities as it doesn't become the official LTS until sometime mid this month.
>>>>>> I don't know how Anaconda packages and may require sym links or copying files from down in the depths to visible,
>>>>>> but it worked before and can be made to work again.
>>>>>> HTH-
>>>>>> Chris
>>>>>>
>>>>>> On Sat, May 30, 2020 at 8:08 AM Roger Bivand <Roger.Bivand using nhh.no> wrote:
>>>>>> On Fri, 29 May 2020, Thiago V. dos Santos wrote:
>>>>>>
>>>>>>> Dear Roger,
>>>>>>>
>>>>>>> Many thanks for the effort keeping rgdal up-to-date with proj6.
>>>>>>>
>>>>>>> I'd like to report that I am unable to install rgdal 1.5.8 on my macOS
>>>>>>> system. I am reporting this error here on the list because I thought it
>>>>>>> would be the best channel in terms of reaching future users experiencing
>>>>>>> the same error. Please apologize if my rationale is not right, and
>>>>>>> ignore this message.
>>>>>>>
>>>>>>> GDAL was installed via MacPorts and is at its latest version, 3.1.0 (at
>>>>>>> the time of this writing). Proj6 has also been installed via Macports.
>>>>>>>
>>>>>>> This is how I am trying to install it:
>>>>>>>
>>>>>>> export PKG_CONFIG_PATH=/opt/local/lib/proj6/lib/pkgconfig
>>>>>>> R
>>>>>>> install.packages('rgdal', type="source", configure.args=c(
>>>>>>>   '--with-proj-include=/opt/local/lib/proj6/include',
>>>>>>>   '--with-proj-lib=/opt/local/lib/proj6/lib'))
>>>>>>>
>>>>>>> and this is the output I get:
>>>>>>>
>>>>>>> R version 4.0.0 (2020-04-24) -- "Arbor Day"
>>>>>>> Copyright (C) 2020 The R Foundation for Statistical Computing
>>>>>>> Platform: x86_64-apple-darwin17.0 (64-bit)
>>>>>>>
>>>>>>>
>>>>>>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>>>>>>> You are welcome to redistribute it under certain conditions.
>>>>>>> Type 'license()' or 'licence()' for distribution details.
>>>>>>>
>>>>>>>
>>>>>>> Natural language support but running in an English locale
>>>>>>>
>>>>>>>
>>>>>>> R is a collaborative project with many contributors.
>>>>>>> Type 'contributors()' for more information and
>>>>>>> 'citation()' on how to cite R or R packages in publications.
>>>>>>>
>>>>>>>
>>>>>>> Type 'demo()' for some demos, 'help()' for on-line help, or
>>>>>>> 'help.start()' for an HTML browser interface to help.
>>>>>>> Type 'q()' to quit R.
>>>>>>>
>>>>>>>
>>>>>>>> install.packages('rgdal', type="source", configure.args=c(
>>>>>>> +      '--with-proj-include=/opt/local/lib/proj6/include',
>>>>>>> +      '--with-proj-lib=/opt/local/lib/proj6/lib'))
>>>>>>> Installing package into ‘/Users/thiago/Documents/R-packages’
>>>>>>> (as ‘lib’ is unspecified)
>>>>>>>
>>>>>>>
>>>>>>>  trying URL 'https://cran.r-project.org/src/contrib/rgdal_1.5-8.tar.gz'
>>>>>>> Content type 'application/x-gzip' length 2299235 bytes (2.2 MB)
>>>>>>> ==================================================
>>>>>>> downloaded 2.2 MB
>>>>>>>
>>>>>>>
>>>>>>> * installing *source* package ‘rgdal’ ...
>>>>>>> ** package ‘rgdal’ successfully unpacked and MD5 sums checked
>>>>>>> ** using staged installation
>>>>>>> configure: R_HOME: /Library/Frameworks/R.framework/Resources
>>>>>>> configure: CC: /opt/local/bin/gcc
>>>>>>> configure: CXX: /opt/local/bin/g++
>>>>>>> configure: C++11 support available
>>>>>>> configure: rgdal: 1.5-8
>>>>>>> checking for /usr/bin/svnversion... yes
>>>>>>> configure: svn revision: 990
>>>>>>> checking for gdal-config... /opt/local/bin/gdal-config
>>>>>>> checking gdal-config usability... yes
>>>>>>> configure: GDAL: 3.1.0
>>>>>>> checking GDAL version >= 1.11.4... yes
>>>>>>> checking GDAL version <= 2.5 or >= 3.0... yes
>>>>>>> checking gdal: linking with --libs only... yes
>>>>>>> checking GDAL: gdal-config data directory readable... yes
>>>>>>> checking GDAL: /opt/local/share/gdal/stateplane.csv readable... yes
>>>>>>> configure: pkg-config proj exists, will use it
>>>>>>
>>>>>> So far so good, but you chose to override pkg-config proj by passing
>>>>>> configure arguments. You should let pkg-config proj provide those values.
>>>>>>
>>>>>>> configure: PROJ version: 6.3.2
>>>>>>> configure: proj CPP flags: -DPROJ_H_API -I/opt/local/lib/proj6/include
>>>>>>> configure: PROJ LIBS: -L/opt/local/lib/proj6/lib
>>>>>>> checking PROJ header API:... proj.h
>>>>>>> checking for gcc... /opt/local/bin/gcc
>>>>>>> checking whether the C compiler works... yes
>>>>>>> checking for C compiler default output file name... a.out
>>>>>>> checking for suffix of executables...
>>>>>>> checking whether we are cross compiling... no
>>>>>>> checking for suffix of object files... o
>>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>>> checking whether /opt/local/bin/gcc accepts -g... yes
>>>>>>> checking for /opt/local/bin/gcc option to accept ISO C89... none needed
>>>>>>> checking how to run the C preprocessor... /opt/local/bin/gcc -E
>>>>>>> checking for grep that handles long lines and -e... /usr/bin/grep
>>>>>>> checking for egrep... /usr/bin/grep -E
>>>>>>> checking for ANSI C header files... rm: conftest.dSYM: is a directory
>>>>>>> rm: conftest.dSYM: is a directory
>>>>>>> yes
>>>>>>> checking for sys/types.h... yes
>>>>>>> checking for sys/stat.h... yes
>>>>>>> checking for stdlib.h... yes
>>>>>>> checking for string.h... yes
>>>>>>> checking for memory.h... yes
>>>>>>> checking for strings.h... yes
>>>>>>> checking for inttypes.h... yes
>>>>>>> checking for stdint.h... yes
>>>>>>> checking for unistd.h... yes
>>>>>>> checking proj.h usability... yes
>>>>>>> checking proj.h presence... yes
>>>>>>> checking for proj.h... yes
>>>>>>
>>>>>> Also OK.
>>>>>>
>>>>>>> checking for proj_context_create in -lproj... no
>>>>>>> configure: error: proj_context_create not found in libproj.
>>>>>>
>>>>>> The directory where libproj should be does not contain it - whether you
>>>>>> got it wrong, pkg-config got it wrong, or for some other reason, the
>>>>>> library was not found.
>>>>>>
>>>>>>> ERROR: configuration failed for package ‘rgdal’
>>>>>>> * removing ‘/Users/thiago/Documents/R-packages/rgdal’
>>>>>>> * restoring previous ‘/Users/thiago/Documents/R-packages/rgdal’
>>>>>>>
>>>>>>>
>>>>>>> The downloaded source packages are in
>>>>>>>  ‘/private/var/folders/_z/01gg71zs19g816v6m2dddt8w0000gn/T/RtmpvZAChj/downloaded_packages’
>>>>>>> Warning message:
>>>>>>> In install.packages("rgdal", type = "source", configure.args = c("--with-proj-include=/opt/local/lib/proj6/include",  :
>>>>>>> installation of package ‘rgdal’ had non-zero exit status
>>>>>>>
>>>>>>> I found this error to be quite mysterious, and could not find any
>>>>>>> previous discussion about it.
>>>>>>
>>>>>> Try to avoid googling, it is usually misleading. Running ./configure in
>>>>>> the untarred source package directory, and running ./configure --help to
>>>>>> show the arguments to configure can be much more helpful. Crucially, they
>>>>>> are up to date, while most search hits are old, and they are ordered by
>>>>>> hit counts, which is certainly no guarantee of precision.
>>>>>>
>>>>>>>
>>>>>>> Are you familiar with it? Is there any other argument that I can pass to
>>>>>>> install.packages to solve it?
>>>>>>
>>>>>> Try not setting configure arguments and let pkg-config proj do the work if
>>>>>> you feel you can trust it. If, however, you have multiple installs of GDAL
>>>>>> and/or PROJ, the proj.pc file may point to a deleted install. Download and
>>>>>> untar the source package, and run just ./configure with its argumments, or
>>>>>> setting PKG_CONFIG_PATH= until you find a recipe that works that far.
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>>
>>>>>>> Greetings,
>>>>>>> -- Thiago V. dos Santos
>>>>>>>
>>>>>>> ThiagoDosSantos.com
>>>>>>> MudancasClimaticasBrasil.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thursday, May 28, 2020, 08:53:50 AM GMT-3, Roger Bivand <roger.bivand using nhh.no> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> With sp 1.4-2 and rgdal 1.5-8, those using GDAL 3 and PROJ 6 (including
>>>>>>> users of the CRAN Windows binary when it comes) will be part of the big
>>>>>>> migration to WKT2 from Proj4 strings for CRS representation. See
>>>>>>>
>>>>>>> https://cran.r-project.org/web/packages/rgdal/vignettes/CRS_projections_transformations.html
>>>>>>>
>>>>>>> or for a better rendering of "\phi" and bib-handling:
>>>>>>>
>>>>>>> https://rgdal.r-forge.r-project.org/articles/CRS_projections_transformations.html
>>>>>>>
>>>>>>> for background and https://www.r-spatial.org/r/2020/03/17/wkt.html for the
>>>>>>> full sf and sp story.
>>>>>>>
>>>>>>> You will see lots of warnings, they should grab your attention, that is
>>>>>>> what they are for. We need your attention to be as sure as we can be that
>>>>>>> your results - if affected by the transition - have been checked by you.
>>>>>>>
>>>>>>> I plan to flip the warnings to default off from rgdal 1.6-*, and sp 1.5-*.
>>>>>>> In some weeks but before flipping the defaults, I'll follow up showing
>>>>>>> ways of muting them, but we really need to be confident that this
>>>>>>> necessary and future-proofing change isn't silently destroying your work
>>>>>>> first. So please don't treat the many warnings as just pesky and nagging -
>>>>>>> they are priming your immune system to check for unwanted consequences of
>>>>>>> your legacy CRS representation choices.
>>>>>>>
>>>>>>> I'd also like to thank the maintainers of packages importing
>>>>>>> from/depending on sp/rgdal for largely positive responses to issues raised
>>>>>>> and emails sent, often less polite than they might have been, but grabbing
>>>>>>> attention was more important. From over 70 broken reverse dependency
>>>>>>> packages we are now well under 20, and I've contacted all of those, with
>>>>>>> only a couple not responding. Edzer has seen similar responsiveness for
>>>>>>> the parallel migration of sf.
>>>>>>>
>>>>>>> Enjoy!
>>>>>>>
>>>>>>> Roger
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Roger Bivand
>>>>>> Department of Economics, Norwegian School of Economics,
>>>>>> Helleveien 30, N-5045 Bergen, Norway.
>>>>>> voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
>>>>>> https://orcid.org/0000-0003-2392-6140
>>>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________
>>>>>> R-sig-Geo mailing list
>>>>>> R-sig-Geo using r-project.org
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>>
>>>>> **********************
>>>>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>>>>> **********************
>>>>> Roy Mendelssohn
>>>>> Supervisory Operations Research Analyst
>>>>> NOAA/NMFS
>>>>> Environmental Research Division
>>>>> Southwest Fisheries Science Center
>>>>> ***Note new street address***
>>>>> 110 McAllister Way
>>>>> Santa Cruz, CA 95060
>>>>> Phone: (831)-420-3666
>>>>> Fax: (831) 420-3980
>>>>> e-mail: Roy.Mendelssohn using noaa.gov www: https://www.pfeg.noaa.gov/
>>>>>
>>>>> "Old age and treachery will overcome youth and skill."
>>>>> "From those who have been given much, much will be expected"
>>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> ***Note new street address***
>>>> 110 McAllister Way
>>>> Santa Cruz, CA 95060
>>>> Phone: (831)-420-3666
>>>> Fax: (831) 420-3980
>>>> e-mail: Roy.Mendelssohn using noaa.gov www: https://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>>>
>>>
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new street address***
>> 110 McAllister Way
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: Roy.Mendelssohn using noaa.gov www: https://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: Roger.Bivand using nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en

More information about the R-sig-Geo mailing list