[R] Too many open files
Duncan Murdoch
murdoch at stats.uwo.ca
Tue Jul 21 11:19:00 CEST 2009
On 21/07/2009 5:00 AM, Allan Engelhardt wrote:
> This seriously looks like a bug in load(): the gzcon() magic in the code
> leaves the connection open. On Linux x86_64 I can do:
It was a bug (in gzcon, not load), and has now been fixed.
>
> file.name <- "/tmp/foo.RData"
> data <- runif(10)
> save(data, file=file.name)
> for ( i in 1:4000 ) { con <- gzfile(file.name, "rb"); load(file=con);
> close(con); }
> ## No problem!
> for ( i in 1:4000 ) { con <- file(file.name, "rb"); load(file=con);
> close(con); }
> # Error in load(file = con) : cannot open the connection
> # In addition: Warning messages:
> # 1: In gzcon(file) : cannot open file '/tmp/foo.RData': Too many open files
> # 2: In load(file = con) :
> # cannot open file '/tmp/foo.RData': Too many open files
> print(i)
> # [1] 1021
> showConnections()
> # description class mode text isopen can read can write
>
> M&R: Use the gzfile() workaround above.
>
> R-helpers: (a) is this a bug? (b) where does one report bugs in core R
> anyhow?
See ?bug.report. Generally R-devel is the list for discussion of bugs.
You can follow changes to R-devel or R-patched on
http://developer.r-project.org/RSSfeeds.html, for example
http://developer.r-project.org/blosxom.cgi/R-2-9-branch/NEWS/2009/07/19#n2009-07-19
Duncan Murdoch
>
> Allan
>
>
> On 16/07/09 21:55, Marilyn & Rich Short wrote:
>> Hello,
>>
>> I'm having a problem in R. I'm getting an error message that reads,
>> "Too many open files". I'm opening files and closing them (and
>> unlinking them), but when I go through that process 509 times, the
>> program halts and I get this error message: "cannot open the
>> connection" with warning messages: "Too many open files". I've been
>> working on this problem for a couple of weeks and have gleaned a bit
>> of info from different internet threads, but no solutions yet.
>>
>> I'm using Windows XP, SP3 and R 2.9.1.
>> Here is my session info:
>>
>>> sessionInfo()
>> R version 2.9.1 (2009-06-26)
>> i386-pc-mingw32
>>
>> locale:
>> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
>> States.1252;LC_MONETARY=English_United
>> States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
>>
>> attached base packages:
>> [1] stats graphics grDevices utils datasets methods base
>>
>> (The problem also occurs on my Vista machine as well.)
>>
>> There is some talk on the internet that some windows systems have a
>> limit of 512 files that can be open at one time. Even though I'm
>> closing my files each time, something is keeping track of how many
>> times I've opened and closed a file in a session. I've talked to
>> Microsoft and run a test program in Visual Studio C#, and, at the
>> moment, it looks like the problem does not lie in the Microsoft arena.
>> The C# program performed a similar task 10,000 times without a
>> problem. I'm not totally convinced, but the current evidence says to
>> look elsewhere.
>>
>> I've attached a script that will induce the problem. However, be
>> warned that, if you use it, you will have to get out of R after you
>> run it. R will no longer be able to access files such as help or
>> sessionInfo(). This can be solved by getting out of the R GUI and back
>> in. Also, I'm using my pathname, "MyPathnameA", so you'll probably
>> want to edit it with your own.
>>
>> R E-mails from as far back as 2006 ask for help on the issue. There
>> have been several suggestions, but no confirmed solution that I can
>> find. You will see my attempt at these suggestions in the script [
>> rm(outMIA);rm(junk); and gc(); after close(outMIA); and
>> unlink(FileNameMIA);]. For me, this becomes important because it
>> limits the total number if iterations in nested do-loops. This is
>> where I ran into the problem. The program below is a distillation from
>> the original program.
>>
>> Any suggestion would be greatly appreciated.
>>
>> I'm a retired engineer and am picking up R to use on a genetic
>> algorithm I want to play with.
>>
>> Thanks for your attention,
>> Rich Short
>>
>> # This script induces an error: "cannot open the connection" with
>>
>> # warnings: "Too many open files".
>>
>> ?
>>
>> # ---------------- Create a junk file and "save" it for use further
>> into the program ---
>>
>> junk <- 1
>>
>> MyPathnameA <- "C:\\Documents and Settings\\All
>> Users\\Documents\\SIREPO\\DBFS-0150"
>>
>> connectionX <- paste(MyPathnameA,"junk",sep="\\")
>>
>> outJunk <- file(connectionX, open="wb")
>>
>> save(junk, file=outJunk)
>>
>> close(outJunk)
>>
>>
>>
>>
>>
>>
>> # The next two lines are a repeat from above. They will be useful if
>> you want to run
>>
>> # this script again after "junk" has been loaded.
>>
>> MyPathnameA <- "C:\\Documents and Settings\\All
>> Users\\Documents\\SIREPO\\DBFS-0150"
>>
>> connectionX <- paste(MyPathnameA,"junk",sep="\\")
>>
>> FileNameMIA <- connectionX
>>
>> for(i in 1:4000){
>>
>> # ----------------- Load "junk" ---------------------------------
>>
>> outMIA <- file(FileNameMIA, open="rb")
>>
>> load(file=outMIA) # load a file
>>
>> # ----------------- close "junk"; unlink; remove; ----------------
>>
>> close(outMIA) # close the file
>>
>> unlink(FileNameMIA) # Unlink
>>
>> rm(outMIA) # This should be unnecessary. I
>> tried it just to be sure the retention of the variable was
>>
>> rm(junk) # not causing the connection
>> to be kept open.
>>
>> cat(" i = ",i,sep=" ") # Show what iteration we are on.
>>
>> gc() # Garbage collection. This
>> should be unnecessary. Another failed attempt at a work-around.
>>
>> zzz <- showConnections(all=FALSE)
>>
>> cat(" zzz = ",zzz,"\n",sep=" ") # This screen print shows that
>> there are no open files
>>
>> }
>>
>> ?
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
More information about the R-help
mailing list