Dear list,

We have a problem which appears to be a bug in either rgdal or raster, 
although it could also be a bug in base R or in our understanding of how 
to deal with connections.

We have a process which is writing a rather large (~10-20.000) number of 
geoTiffs via writeRaster. However, the process has frequently stopped 
with an error of the type:
Error in .local(.Object, ...) :
   TIFFOpen:/local0/skoiejo/hri/test.tif: Too many open files
The issue seems to be the creation of temp-files in the temp directory 
which is given by tempdir(), not by raster:::.tmpdir(). These temp-files 
seem to be created by the call
   transient <- new("GDALTransientDataset", driver=driver, rows=r@nrows, 
cols=r@ncols, bands=nbands, type=dataformat, fname=filename, 
options=options, handle=NULL)
from raster:::.getGDALtransient
The temp-files are deleted after writing the geoTiff, but are not 
removed from the list of open files in Linux, which on our system was 
limited to 1024 files (ulimit -n) per process. Below is a script which 
can replicate the issue (takes a few minutes to reach 1024) and 
sessionInfo().

Currently we are trying to solve the issue by increasing the limit of 
file connections, but we would prefer a solution where the connections 
are properly deleted, either before writeRaster finishes, or a command 
which we can include in our script, either R-code or a call to System(). 
The connections are not visible via showConnections(), and 
closeAllConnections() does not help.

Thanks,
Jon


r <- raster(system.file("external/test.grd", package="raster"))
for (ifile in 1:2000) {
   writeRaster(r, "test.tif", format = "GTiff", overwrite = TRUE)
   print(ifile)
}

After the script stops, I checked the open files from the process, and 
got the following:
  lsof -aPn -p 596 | more
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
R       596 skoiejo  cwd    DIR   8,17   139264  4213416 
/local0/skoiejo/Raster_temp
R       596 skoiejo  rtd    DIR  253,0     4096        2 /
R       596 skoiejo  txt    REG  253,0    14192  2500779 
/usr/lib64/R/bin/exec/R
R       596 skoiejo  mem    REG  253,0  2679280  2501177 
/usr/lib64/R/lib/libR.so
......................

R       596 skoiejo 1015u   REG  253,0      238  4983213 
/tmp/RtmpNyOKCk/toptest.tif (deleted)
R       596 skoiejo 1016u   REG  253,0      238  4983214 
/tmp/RtmpNyOKCk/qxdtest.tif (deleted)
R       596 skoiejo 1017u   REG  253,0      238  4983215 
/tmp/RtmpNyOKCk/zwotest.tif (deleted)
R       596 skoiejo 1018u   REG  253,0      238  4983216 
/tmp/RtmpNyOKCk/cnqtest.tif (deleted)
R       596 skoiejo 1019u   REG  253,0      238  4983217 
/tmp/RtmpNyOKCk/lottest.tif (deleted)
R       596 skoiejo 1020u   REG  253,0      238  4983218 
/tmp/RtmpNyOKCk/fartest.tif (deleted)
R       596 skoiejo 1021u   REG  253,0      238  4983219 
/tmp/RtmpNyOKCk/vsqtest.tif (deleted)
R       596 skoiejo 1022u   REG  253,0      238  4983220 
/tmp/RtmpNyOKCk/czptest.tif (deleted)

Even if tested by someone with a limit higher than 2000, it should still 
be possible to see the long list of open connections, as above.

 > sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-redhat-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=C                 LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rgdal_0.8-10  raster_2.1-49 sp_1.0-11

loaded via a namespace (and not attached):
[1] grid_3.0.1      lattice_0.20-15





-- 
Jon Olav Skøien
Joint Research Centre - European Commission
Institute for Environment and Sustainability (IES)
Land Resource Management Unit

Via Fermi 2749, TP 440,  I-21027 Ispra (VA), ITALY

jon.skoien@jrc.ec.europa.eu
Tel:  +39 0332 789206

Disclaimer: Views expressed in this email are those of the individual and do not necessarily represent official views of the European Commission.



	[[alternative HTML version deleted]]

