[R-sig-Geo] rgdal_1.1-1: dsn path string problems

[Roger Bivand]

On Tue, 10 Nov 2015, Rainer Hurling wrote:

> Am 08.11.15 um 20:57 schrieb Roger Bivand:
>>  On Sun, 8 Nov 2015, Rainer Hurling wrote:
>> 
>> >  Hi Roger,
>> > 
>> >  many thanks for your quick answer.
>> > 
>> > 
>> >  Am 08.11.15 um 18:02 schrieb Roger Bivand:
>> > >  On Sun, 8 Nov 2015, Rainer Hurling wrote:
>> > > 
>> > > >  Dear Roger, dear list,
>> > > > 
>> > > >  many thanks for the newest update 1.1-1 of package rgdal. It is 
>> > > >  really
>> > > >  helpful.
>> > > > 
>> > > >  I am using rgdal with R-devel and GDAL 2.0.1, both on FreeBSD
>> > > >  11.0-CURRENT and Windows7. On both platforms, I have problems with 
>> > > >  path
>> > > >  strings in dsn.
>> > > > 
>> > > > 
>> > > > 
>> > > >  (a) The minor problem is on Windows. While a dsn="C:/some/path" 
>> > > >  works
>> > > >  fine, the same path with a slash at the end (dsn="C:/some/path/")
>> > > >  fails.
>> > > > 
>> > > >  On most other software it is conventional to allow a closing slash.
>> > > >  Wouldn't it be nice, if rgdal would be more tolerant about it?
>> > > > 
>> > > 
>> > >  R base::file.exists does include this in its help file: "However,
>> > >  directory names must not include a trailing backslash or slash on
>> > >  Windows".
>> > > 
>> > >  so this isn't just rgdal. The dsn= argument for many drivers is a file
>> > >  name, not a directory; a work-around seems like overkill.
>> > 
>> >  OK, that seems reasonable. Of course, rgdal should handle this like R in
>> >  total. Thanks you for clarification.
>> > 
>> > > > 
>> > > >  Now my main problem:
>> > > >  (b) On FreeBSD, for some time now, the dsn path can not be used 
>> > > >  anymore
>> > > >  to check for existence of a file.
>> > > > 
>> > > >  I want to use writeOGR to overwrite an existing shapefile. I use
>> > > >  something like dsn="/path/to/somewhere" and layer="layername".
>> > > > 
>> > > >  Within writeOGR(), R/ogr_write.R calls the C++ function
>> > > >  'ogrCheckExists'. ogrCheckExists is not able to return TRUE, if a
>> > > >  shapefile already exists. Every time its result is FALSE:
>> > > > 
>> > > >  R/ogr_write.R
>> > > >  61:        ogrI <- .Call("ogrCheckExists", as.character(dsn),
>> > > >  s.character(layer), PACKAGE = "rgdal")
>> > > >  [1] FALSE
>> > > > 
>> > > > 
>> > > >  As far, as I can see, 'ogrCheckExists' comes from src/ogrsource.cpp.
>> > > >  Something must go wrong in that C++ file, at least for FreeBSD :(
>> > > > 
>> > > >  Unfortunalety, I have no skills how to debug such a C++ code within
>> > > >  an R
>> > > >  package ...
>> > > > 
>> > > 
>> > >  I do not have access to such a system. As a first step, does
>> > >  rgdal::ogrListLayers work correctly?
>> > 
>> >  The syntax, shown in the helpfile, does not work for me:
>> > 
>> >  rgdal::ogrListLayers(dsn="/path", layer="layername")
>> >  Fehler in rgdal::ogrListLayers(dsn = "/path", layer = "layername") :
>> >   unbenutztes Argument (layer = "layername")    # unused argument
>> > 
>>
>>  The helpfile usage section says:
>>
>>  ogrListLayers(dsn)
>>
>>  which works, as you see below, there is no layer argument, as the
>>  function queries the dsn to ask what layers it contains (differs between
>>  drivers).
>> 
>
> Ok, as always, I should read documents more careful ;) Thanks for 
> clarification again. It seems, the GDAL2 driver for ESRI shapefiles is a very 
> special thing ...
>
>> > 
>> >  The same, with layer integrated in the dsn, works fine:
>> > 
>> >  rgdal::ogrListLayers(dsn="/path/layername.shp")
>> >  [1] "layername"
>> >  attr(,"driver")
>> >  [1] "ESRI Shapefile"
>> >  attr(,"nlayers")
>> >  [1] 1
>> > 
>> > 
>> >  This leads me to the idea, that the same could work for writeOGR or
>> >  ogrWrite.R. And yes, this works on FreeBSD:
>> > 
>> >  writeOGR(gis.layer, dsn="/path/layername.shp",
>> >          layer="layername", driver="ESRI Shapefile",
>> >          check_exists=TRUE, overwrite_layer=TRUE)
>> > 
>> >  So, I have to double 'layername' here, once in dsn and once in layer.
>> >  But it should be ok as a workaround until I found the real cause.
>> > 
>>
>>  I have seen this before in GDAL2; it also happened in GDAL1 where the
>>  dsn directory contained for example *.dbf without *.shp and *.shx. The
>>  way dsn is handled does vary a lot between drivers.
>
> I wonder, why it is necessary, to also use the layer="layername" (at least on 
> FreeBSD with GDAL2), if the full path and filename in 
> dsn="/path/layername.shp" is already given.
>
> For now, I will use this dsn and layer notation for shapefiles with GDAL2.

All drivers may be expected to differ in their treatments of the dsn= and 
layer= arguments - both within drivers between GDAL releases and between 
drivers. We benefit from GDAL, and are exposed to the choices changes that 
occur there. Things can be made too "easy" ...

>
>> 
>> > > 
>> > >  Beyond that, you'd need to insert Rprintf() statements into the C++ 
>> > >  file
>> > >  to see whether the FALSE result comes from a failure to open the dsn
>> > >  with a known driver, or from the fact that such a layer is not found 
>> > >  in
>> > >  that dsn.
>> > 
>> >  Thanks for the tip. I will investigate into it.
>> > 
>> >  A first question to this: If I want to integrate some Rprintf into
>> >  ogrsource.cpp in the section of ogrCheckExists (lines 832 ...), what is
>> >  a good syntax to output for example 'ogrSource' and 'Layer'?
>> > 
>> >  What I tried at ogrsource.cpp:l834 without success is:
>> > 
>> >  Rprintf( "ogrCheckExists: OGRSource %s %p\n", OGRLayer, (void *)
>> >  ogrSource) ;
>>
>>  You'd need simple "No dns found" after line 860 and "No layer found"
>>  after line 878 to separate the two FALSE returns, but I guess the cause
>>  may have been in GDAL2, the shapefile driver, and/or what was in your
>>  dsn directory.
>
> Ok, I was used to use print statements in debugging code also to get 
> variables content, pointers and so on ...
>
> Again, many many thanks for your quick and helpful comments. I now understand 
> the driver dependend requirements of GDAL2 a lot more :)

Many of the same types of things happened in < GDAL 2; this version is 
intended to try to be more systematic. Drivers are (were) contributed by 
developers outside GDAL (so not Even Roualt), and change sometimes 
rapidly.

Roger

>
> Greetings from Göttingen,
> Rainer
>
>>
>>  Roger
>> 
>> > 
>> >  It think, I have to wrap 'ogrSource' with some function here? (I am not
>> >  a C/C++ programmer)
>> > 
>> > > 
>> > >  Does using a different driver help? Does using the shapefile.shp as 
>> > >  the
>> > >  dsn help (I think this works better in GDAL2)?
>> > 
>> >  Because using the shapefile.shp as the dsn helps, I did not try a
>> >  different driver. If this is important for you, I can catch it up later.
>> > 
>> >  Thanks again for your help,
>> >  Rainer
>> > 
>> > > 
>> > >  Hope this helps,
>> > > 
>> > >  Roger
>> > > 
>> > > >  Any help is really appreciated. Please tell me, if I could try and 
>> > > >  test
>> > > >  something. My box has FreeBSD, R-devel, gcc-4.8.5, gdb-7.1.0 ...
>> > > > 
>> > > >  Greetings,
>> > > >  Rainer Hurling
>> > > > 
>> > > > 
>> > > 
>> > 
>> > 
>> 
>
>

-- 
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 91 00
e-mail: Roger.Bivand at nhh.no