[R] Too many open files

Allan Engelhardt allane at cybaea.com
Tue Jul 21 11:00:35 CEST 2009


This seriously looks like a bug in load(): the gzcon() magic in the code 
leaves the connection open.  On Linux x86_64 I can do:

file.name <- "/tmp/foo.RData"
data <- runif(10)
save(data, file=file.name)
for ( i in 1:4000 ) { con <- gzfile(file.name, "rb"); load(file=con); 
close(con); }
## No problem!
for ( i in 1:4000 ) { con <- file(file.name, "rb"); load(file=con); 
close(con); }
# Error in load(file = con) : cannot open the connection
# In addition: Warning messages:
# 1: In gzcon(file) : cannot open file '/tmp/foo.RData': Too many open files
# 2: In load(file = con) :
#   cannot open file '/tmp/foo.RData': Too many open files
print(i)
# [1] 1021
showConnections()
#      description class mode text isopen can read can write

M&R: Use the gzfile() workaround above.

R-helpers: (a) is this a bug? (b) where does one report bugs in core R 
anyhow?

Allan


On 16/07/09 21:55, Marilyn & Rich Short wrote:
> Hello,
>
> I'm having a problem in R. I'm getting an error message that reads, 
> "Too many open files". I'm opening files and closing them (and 
> unlinking them), but when I go through that process 509 times, the 
> program halts and I get this error message: "cannot open the 
> connection" with warning messages: "Too many open files".  I've been 
> working on this problem for a couple of weeks and have gleaned a bit 
> of info from different internet threads, but no solutions yet.
>
> I'm using Windows XP, SP3 and R 2.9.1.
> Here is my session info:
>
>> sessionInfo()
> R version 2.9.1 (2009-06-26)
> i386-pc-mingw32
>
> locale:
> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United 
> States.1252;LC_MONETARY=English_United 
> States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> (The problem also occurs on my Vista machine as well.)
>
> There is some talk on the internet that some windows systems have a 
> limit of 512 files that can be open at one time. Even though I'm 
> closing my files each time, something is keeping track of how many 
> times I've opened and closed a file in a session. I've talked to 
> Microsoft and run a test program in Visual Studio C#, and, at the 
> moment, it looks like the problem does not lie in the Microsoft arena. 
> The C# program performed a similar task 10,000 times without a 
> problem. I'm not totally convinced, but the current evidence says to 
> look elsewhere.
>
> I've attached a script that will induce the problem. However, be 
> warned that, if you use it, you will have to get out of R after you 
> run it. R will no longer be able to access files such as help or 
> sessionInfo(). This can be solved by getting out of the R GUI and back 
> in. Also, I'm using my pathname, "MyPathnameA", so you'll probably 
> want to edit it with your own.
>
> R E-mails from as far back as 2006 ask for help on the issue. There 
> have been several suggestions, but no confirmed solution that I can 
> find. You will see my attempt at these suggestions in the script [ 
> rm(outMIA);rm(junk); and gc(); after close(outMIA); and 
> unlink(FileNameMIA);]. For me, this becomes important because it 
> limits the total number if iterations in nested do-loops. This is 
> where I ran into the problem. The program below is a distillation from 
> the original program.
>
> Any suggestion would be greatly appreciated.
>
> I'm a retired engineer and am picking up R to use on a genetic 
> algorithm I want to play with.
>
> Thanks for your attention,
> Rich Short
>
> # This script induces an error: "cannot open the connection" with
>
> # warnings: "Too many open files".
>
> ?
>
> # ---------------- Create a junk file and "save" it for use further 
> into the program ---
>
> junk <- 1
>
> MyPathnameA <- "C:\\Documents and Settings\\All 
> Users\\Documents\\SIREPO\\DBFS-0150"
>
> connectionX <- paste(MyPathnameA,"junk",sep="\\")
>
> outJunk <- file(connectionX, open="wb")
>
> save(junk, file=outJunk)
>
> close(outJunk)
>
>
>
>
>
>
> # The next two lines are a repeat from above. They will be useful if 
> you want to run
>
> # this script again after "junk" has been loaded.
>
> MyPathnameA <- "C:\\Documents and Settings\\All 
> Users\\Documents\\SIREPO\\DBFS-0150"
>
> connectionX <- paste(MyPathnameA,"junk",sep="\\")
>
> FileNameMIA <- connectionX
>
> for(i in 1:4000){
>
>   # ----------------- Load "junk" ---------------------------------
>
>   outMIA <- file(FileNameMIA, open="rb")
>
>   load(file=outMIA)                           # load a file
>
>   # ----------------- close "junk"; unlink; remove; ----------------
>
>   close(outMIA)                              # close the file
>
>   unlink(FileNameMIA)                    # Unlink
>
>   rm(outMIA)                          # This should be unnecessary. I 
> tried it just to be sure the retention of the variable was
>
>   rm(junk)                              # not causing the connection 
> to be kept open.
>
>   cat(" i = ",i,sep=" ")             # Show what iteration we are on.
>
>   gc()                                    # Garbage collection. This 
> should be unnecessary. Another failed attempt at a work-around.
>
>   zzz <- showConnections(all=FALSE)
>
>   cat(" zzz = ",zzz,"\n",sep=" ")  # This screen print shows that 
> there are no open files
>
> }
>
> ?
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.




More information about the R-help mailing list