[R-sig-Geo] readOGR or writeOGR not closing connections?

[Roger Bivand]

On Tue, 4 Dec 2012, MacQueen, Don wrote:

> I am using R to help me interactively create and manage GIS layers, stored
> as shapefiles, and I'm encountering a problem. From time to time I get an
> error:
>
> Error in writeOGR(tmpp, "shps", fn, "a.name", driver = "ESRI Shapefile", :
> Layer creation failed
>
> I can work around it by quitting and restarting R, but it would be nice if
> I didn't have to.
>
> Below is a little script that reproduces the behavior (on my machine).
> ('shps' is a sub-directory)
>
> The implication, as I see it, is that one or both of readOGR or writeOGR
> is not closing some connections. If I'm right about this, it would be nice
> to have a way to force-close them whenever necessary, but I haven't found
> a way to do that. Information for how, or any other suggestions, would be
> appreciated.
>

Don,

Thanks for a helpful report. Your diagnosis is, however, wrong. The files 
left open are a consequence of checking introduced in GDAL 1.8, meant by 
GDAL to prevent overwriting of layers. Because of this, the 
check_exists=NULL argument was introduced into writeOGR(), defaulting to 
FALSE before 1.8, TRUE >= 1.8. When TRUE, the C++ ogrInfo() was called to 
see whether the dsn and layer exist, and if so, and if 
overwrite_layer=TRUE, the layer is deleted. However, if the layer did not 
exist in the dsn, the C++ ogrInfo() error-exited, the error was trapped, 
and no deletion occurred.

In your case, with GDAL 1.9, check_exists is TRUE, the files did not 
exist, but the dsn was left open (and shapefile dsn open all the layers in 
the folder, then check to see if the required layer is present). The C++ 
ogrInfo() - committed to R-forge SVN, revision 395 - now closes the dsn on 
error, so your example runs fine. It would also have run (the first time) 
with check_exists=FALSE.

If you can install rgdal from source, I'd be grateful for confirmation 
that the bugfix works in your production setting.

Roger

>
>
> Note that in each iteration there are 4 files written and 4 files read.
>  2*4*14=112
> which is getting close to the number available according to ?open:
>
> -- quote --
>    A maximum of 128 connections can be allocated (not necessarily
>     open) at any one time.  Three of these are pre-allocated (see
>     'stdout').  The OS will impose limits on the numbers of
>     connections of various types, but these are usually larger than
>     125.
> -- end quote --
>
>
>
>
>
> #### begin script
> require(sp)
> require(rgdal)
>
> print(sessionInfo())
>
> data(meuse.grid)
> coordinates(meuse.grid) = ~x+y
> tmpp <- meuse.grid[1:10,]
>
> for(i in 1:100) {
>  fn <- paste('tmpp',i,sep='')
>  writeOGR(tmpp, 'shps',fn,'a.name',driver='ESRI Shapefile',
> overwrite_layer=TRUE)
>  foo <- readOGR('shps',fn)
> }
>
> print(sessionInfo())
> #### end script
>
>
> And here is the output when I source the script in a fresh R session
>
> -----------------------------
>> source('try-problem.r')
> Loading required package: sp
> Loading required package: rgdal
> rgdal: (SVN revision 360)
> Geospatial Data Abstraction Library extensions to R successfully loaded
> Loaded GDAL runtime: GDAL 1.9.0, released 2011/12/29
> Path to GDAL shared files:
> /Library/Frameworks/R.framework/Versions/2.15/Resources/library/rgdal/gdal
> Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
> Path to PROJ.4 shared files:
> /Library/Frameworks/R.framework/Versions/2.15/Resources/library/rgdal/proj
>
> R version 2.15.2 (2012-10-26)
> Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
>
> locale:
> [1] C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] rgdal_0.7-24 sp_1.0-2
>
> loaded via a namespace (and not attached):
> [1] grid_2.15.2     lattice_0.20-10
>
> OGR data source with driver: ESRI Shapefile
> Source: "shps", layer: "tmpp1"
> with 10 features and 5 fields
> Feature type: wkbPoint with 2 dimensions
> OGR data source with driver: ESRI Shapefile
> Source: "shps", layer: "tmpp2"
> with 10 features and 5 fields
> Feature type: wkbPoint with 2 dimensions
>
> --- output from iterations 2 through 12 omitted ---
>
> OGR data source with driver: ESRI Shapefile
> Source: "shps", layer: "tmpp13"
> with 10 features and 5 fields
> Feature type: wkbPoint with 2 dimensions
> OGR data source with driver: ESRI Shapefile
> Source: "shps", layer: "tmpp14"
> with 10 features and 5 fields
> Feature type: wkbPoint with 2 dimensions
> Error in writeOGR(tmpp, "shps", fn, "a.name", driver = "ESRI Shapefile",  :
>  Layer creation failed
>>
>
> ## then manually:
>
>> print(sessionInfo())
> Error in gzfile(file, "rb") : cannot open the connection
> In addition: Warning message:
> In gzfile(file, "rb") :
>  cannot open compressed file
> '/Library/Frameworks/R.framework/Versions/2.15/Resources/library/rgdal/Meta
> /package.rds', probable reason 'Too many open files'
>
>
>
> I'm beginning to suspect this has to do with "gzcon" or "gzfile"
> connections (not sure of correct terminology).
> ?open also says,
>
> -- quote --
>     'close' closes and destroys a connection.  This will happen
>     automatically in due course (with a warning) if there is no longer
>     an R object referring to the connection.
> -- end quote --
>
> But ?showConnections says
>
> -- quote --
> ... However, if there is no R level object referring to
>     the connection it will be closed automatically at the next garbage
>     collection (except for 'gzcon' connections).
> -- end quote --
>
>
> So if read/write OGR involve zipping and unzipping, I guess the
> connections don't get closed. (?)
>
> Anyway, it would be very nice if I didn't have to quit and restart my R
> sessions every so often when I'm working with my shapefiles.
>
>
> Thanks
> -Don
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no