[R-sig-Geo] raster/rgdal- problem: Too many open files (Linux)

[Mauricio Zambrano-Bigiarini]

On 08/05/2013 04:37 PM, Jon Olav Skoien wrote:
> Dear list,
>
> We have a problem which appears to be a bug in either rgdal or raster,
> although it could also be a bug in base R or in our understanding of how
> to deal with connections.
>
> We have a process which is writing a rather large (~10-20.000) number of
> geoTiffs via writeRaster. However, the process has frequently stopped
> with an error of the type:
> Error in .local(.Object, ...) :
>     TIFFOpen:/local0/skoiejo/hri/test.tif: Too many open files
> The issue seems to be the creation of temp-files in the temp directory
> which is given by tempdir(), not by raster:::.tmpdir(). These temp-files
> seem to be created by the call
>     transient <- new("GDALTransientDataset", driver=driver, rows=r at nrows,
> cols=r at ncols, bands=nbands, type=dataformat, fname=filename,
> options=options, handle=NULL)
> from raster:::.getGDALtransient
> The temp-files are deleted after writing the geoTiff, but are not
> removed from the list of open files in Linux, which on our system was
> limited to 1024 files (ulimit -n) per process. Below is a script which
> can replicate the issue (takes a few minutes to reach 1024) and
> sessionInfo().
>
> Currently we are trying to solve the issue by increasing the limit of
> file connections, but we would prefer a solution where the connections
> are properly deleted, either before writeRaster finishes, or a command
> which we can include in our script, either R-code or a call to System().
> The connections are not visible via showConnections(), and
> closeAllConnections() does not help.
>
> Thanks,
> Jon

I stumbled across the same problem (with exactly the same configuration 
reported by Jon with 'sessionInfo()'), while trying to change the values 
of some pixels in more than 6000 maps.

Thank  you very much Jon for the detailed report about the problem, 
which helped me to find a workaround to this problem (so far, just to 
split the 6000 maps in smaller groups).

>
>
> r <- raster(system.file("external/test.grd", package="raster"))
> for (ifile in 1:2000) {
>     writeRaster(r, "test.tif", format = "GTiff", overwrite = TRUE)
>     print(ifile)
> }
>

After trying the previous reproducible code, I don't understand why I 
got the error when ifile=1019 and not 1024:

....
[1] 1018
[1] 1019
Error in .local(.Object, ...) :
   TIFFOpen:/home/hzambran/test.tif: Too many open files



Thanks again Jon for sharing your findings about this.

All the best,

Mauricio Zambrano-Bigiarini, Ph.D


-- 
=================================================
Water Resources Unit
Institute for Environment and Sustainability (IES)
Joint Research Centre (JRC), European Commission
webinfo    : http://floods.jrc.ec.europa.eu/
=================================================
DISCLAIMER:
"The views expressed are purely those of the writer
and may not in any circumstances be regarded as sta-
ting an official position of the European Commission"
=================================================
"Sometimes life's going to hit you in the head
with a brick. Don't lose faith" (Steve Jobs)


> After the script stops, I checked the open files from the process, and
> got the following:
>    lsof -aPn -p 596 | more
> COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
> R       596 skoiejo  cwd    DIR   8,17   139264  4213416
> /local0/skoiejo/Raster_temp
> R       596 skoiejo  rtd    DIR  253,0     4096        2 /
> R       596 skoiejo  txt    REG  253,0    14192  2500779
> /usr/lib64/R/bin/exec/R
> R       596 skoiejo  mem    REG  253,0  2679280  2501177
> /usr/lib64/R/lib/libR.so
> ......................
>
> R       596 skoiejo 1015u   REG  253,0      238  4983213
> /tmp/RtmpNyOKCk/toptest.tif (deleted)
> R       596 skoiejo 1016u   REG  253,0      238  4983214
> /tmp/RtmpNyOKCk/qxdtest.tif (deleted)
> R       596 skoiejo 1017u   REG  253,0      238  4983215
> /tmp/RtmpNyOKCk/zwotest.tif (deleted)
> R       596 skoiejo 1018u   REG  253,0      238  4983216
> /tmp/RtmpNyOKCk/cnqtest.tif (deleted)
> R       596 skoiejo 1019u   REG  253,0      238  4983217
> /tmp/RtmpNyOKCk/lottest.tif (deleted)
> R       596 skoiejo 1020u   REG  253,0      238  4983218
> /tmp/RtmpNyOKCk/fartest.tif (deleted)
> R       596 skoiejo 1021u   REG  253,0      238  4983219
> /tmp/RtmpNyOKCk/vsqtest.tif (deleted)
> R       596 skoiejo 1022u   REG  253,0      238  4983220
> /tmp/RtmpNyOKCk/czptest.tif (deleted)
>
> Even if tested by someone with a limit higher than 2000, it should still
> be possible to see the long list of open connections, as above.

>
>   > sessionInfo()
> R version 3.0.1 (2013-05-16)
> Platform: x86_64-redhat-linux-gnu (64-bit)
>
> locale:
>    [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
>    [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
>    [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
>    [7] LC_PAPER=C                 LC_NAME=C
>    [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] rgdal_0.8-10  raster_2.1-49 sp_1.0-11
>
> loaded via a namespace (and not attached):
> [1] grid_3.0.1      lattice_0.20-15
>
>
>
>
>
>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>