[R-sig-Geo] Pre-GDAL 2: rgdal changes - from all according to their means

[Roger Bivand]

On Tue, 2 Jun 2015, chris english wrote:

> Roger,
>
> I second Michael Sumner's excitement. I built a POSTGist SFCGAL stack based
> on GDAL 2.0.0beta, anticipating that I could seamlessly integrate with R
> and was disappointed that I couldn't use  2.0.0.beta  because at the point
> only sp-1.1-1 was supported.

Chris,

Excitement doesn't help. GDAL 2 is only an advance over GDAL 1 in that it 
makes the code base easier to maintain because only the GDAL driver 
manager is supported, and vector drivers have been folded into the same 
model. Beyond that, we have continuing feature creep (e.g. use of 
Integer64 to hold IDs that notionally should be strings (no arithmetic) 
and which always lose leading zeros, and others. The key challenge 
(threat) to GDAL is the very limited number of core developers, followed 
by the same danger affecting many of GDAL's external dependencies.

General comment: Could someone who knows how to do it please set up a 
searchable scribble sheet to which ordered examples, open issues, ideas, 
etc. could be contributed? This thread is now the prime location, but 
probably will become unwieldy.

You will never be able to switch easily between rgdal installed against 
GDAL 1 and rgdal installed against GDAL 2. Any possibilities are only 
available to those installing GDAL from source, then rgdal from source, as 
GDAL 2 overwrites GDAL 1 (same name for gdal-config, same app names) if 
--prefix= is the same. If --prefix= is different between 1 and 2 when 
installing GDAL, and you take care with LD path names and use of ldconfig, 
you can pass these different locations of GDAL 1 and GDAL 2 through when 
installing rgdal. Then you'd also have to remember to install rgdal for 
GDAL 1 in say one R library folder, for GDAL 2 in a different library 
folder, and add the appropriate one to .libPaths(). It would be messy.

Performance-wise, there should be no user-impacting difference between 
GDAL 1 and 2, or between rgdal opening for GDAL 1 or 2, and rgdal (<= 
0.9-3), which blocks GDAL 2.

The task now is to locate possible differences in behaviour between rgdal 
0.9-3 with GDAL 1, and rgdal 1.0-2 with either GDAL 1 or 2. If there are 
1|2 differences, are they because of changes in GDAL, or because rgdal was 
making inappropriate assumptions about GDAL?

So far, we really don't know the Integer64 field type in GDAL 2 vectors is 
going to play out - as of now rgdal 1.0-2 truncates to integer. We don't 
know how the changes in the driver manager for vector dsn and layers 
affect layer and dsn overwriting (I'm seeing very odd OGRErr numbers in 
GDAL 2, and apparently different behaviour for some drivers). Writing 
MapInfo File TAB files looks broken (maybe a dsn creation option needs 
setting, when it wasn't needed before), etc.

Some of this can be fixed by nasty kludges in rgdal, some should be fixed 
in GDAL 2, some are real misunderstandings. I don't have the time to try 
to make an rgdal Windows binary built against Windows GDAL 2 by 
cross-compiling for trying things out under Windows without user 
compilation. The transition will take time, but anyone who can install 
GDAL and rgdal from source, and who is interested and/or needs rgdal to 
work predictably, is welcome to run some standard procedures with rgdal 
0.9-3 and GDAL 1, storing a text version of key results; install rgdal 
1.0-2 (or later) and GDAL 1 (or later checked out from R-Forge and living 
under pkg/:

#cd rgdal
R CMD build pkg
R CMD check rgdal_1.0-2.tar.gz
# maybe --install-args='--configure.args=...'
R CMD INSTALL rgdal_1.0-2.tar.gz
# maybe --configure.args=...

rerun the procedure and compare the results to see whether changes in 
rgdal have led to changes in the results, the re-install rgdal 1.0-2 with 
GDAL 2, rerun the procedure and compare the results, and compare those for 
GDAL 1 and GDAL 2.

In particular, we need confirmation that the database access drivers work 
across these shifts.

Roger

>
> I have a couple of future projects that are going to fundamentally depend
> on GDAL and I hope to be able to address them from within R.  As an R
> beginner I'm happy I have in my notes on how to point to my /opt/gdal(
> please excuse if generalizing between an installed /opt/gdal(2.0,0.beta and
> /usr/local/gdal(earlier.build)  build, but many people perhaps have not
> been making a diary of R success(es).
>
> In the simplest case it could be a reminder to us beginners how we might
> load a library(rgdal("in this case 2.0.0.beta vs our normal
> library(rgdal)).  Well we are beginners, but we still want out stuff to
> work and we want to contribute to a successful and comprehensive 2.2.0
> release. So we'll run our particular research examples through 2.0.0 and
> report results, if we know how to load one vs. another library.
>
> If intermediate or expert, different examples would be deployed.
>
> I apologize in advance if these issues  or methods have already been
> addressed at SVN and I failed to notice them.  If that is the case I will
> dig further, and please ignore me.
>
> I think GDAL and rgdal are intrinsically important tools to this community
> and everyone wants them to work; please just tell us, by general example,
> how we might be best of service.
>
> I hope the foregoing is comprehensible. My thoughts, and looking forward to
> to contributing in a useful fashion to this important package (as a
> beginner).
>
> Cheers,
>
> Chris
>
>
> The second beta of GDAL 2 is now available, and as of revision 535 on
>> R-Forge, the legacy rgdal package passes R CMD check with either GDAL
>> 1.11.2 (or earlier) or GDAL 2.0.0 beta 2.
>>
>> One or two issues are known (Integer64 in vector fields and FIDs not
>> supported in R; gdalDrivers() reports both raster and vector drivers; the
>> MapInfo File TAB driver doesn't work for writing, ...), but others remain
>> to be discovered.
>>
>> For those who need rgdal in production, and can install the 2.0.0 beta, it
>> would be a really good use of time to identify issues now, rather than when
>> GDAL 2 starts to become the standard, stable release. Anyone else needing
>> an itch to scratch is also, of course, welcome to contribute.
>>
>> The rgdal package will continue to condition on GDAL 1 or 2, so hopefully
>> those users who do not need to move to GDAL 2 will not be affected.
>> However, it is worth noting that GDAL is maintained by very, very, few
>> volunteers (even plural is questionable here), and when they feel that
>> backporting fixed from GDAL 2 to GDAL 1 is taking time from more important
>> things, you will be stranded with EOL software.
>>
>> So please consider taking the time to contribute to the idenfication of
>> issues in the development version of rgdal built against GDAL 2 and/or 1,
>> available for anonymous SVN checkout at:
>>
>> svn checkout svn://scm.r-forge.r-project.org/svnroot/rgdal/trunk
>>
>> Enjoy!
>>
>> Roger
>>
>> --
>> Roger Bivand
>> Department of Economics, Norwegian School of Economics,
>> Helleveien 30, N-5045 Bergen, Norway.
>> voice: +47 55 95 93 55; fax +47 55 95 91 00
>> e-mail: Roger.Bivand at nhh.no
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no