[Rd] [R] Problem With Repeated Use Of Load/Save/Close Commands (PR#13841)

[Duncan Murdoch]

On 18/07/2009 8:15 AM, murdoch at stats.uwo.ca wrote:
> On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote:
>> Hello,
>>
>> I'm having a problem in R with repeated use of the "load", "save", and 
>> "close" commands. I'm getting an error message that reads, "Too many 
>> open files". I'm opening files and closing them (and unlinking them), 
>> but when I go through that process 509 times, the program halts and I 
>> get this error message: "cannot open the connection" with warning 
>> messages: "Too many open files".  I've been working on this problem for 
>> a couple of weeks and have gleaned a bit of info from different internet 
>> threads, but no solutions yet.
>>
>> I'm using Windows XP, SP3 and R 2.9.1.
>> Here is my session info:
>>
>> R version 2.9.1 (2009-06-26)
>> i286-pc-mingw32
>>
>> locale:
>> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
>> States. 1252;LC_MONETARY=English_United
>> States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>
>> I'm using the vanilla load of R, with nothing added after booting up.
>> The problem also occurs on my Vista machine as well.
>>
>> The program below will induce the problem.
>> ------------------------------------------------------------
>>    junk       = 1
>>    PathA    = tempdir()
>>    conX     = paste(PathA,"junk",sep="\\")
>>    outJunk  = file(conX, open="wb")
>>    save(junk, file=outJunk)
>>    close(outJunk)
>>    for(i in 1:4000){
>>        outMIA = file(conX, open="rb")
>>        load(file=outMIA)
>>        close(outMIA)
>>        closeAllConnections()
>>        unlink(conX)
>>        rm(outMIA)
>>        rm(junk)
>>        cat(" i = ",i,sep=" ")
>>        gc()
>>        zzz = showConnections(all=FALSE)
>>        cat(" zzz = ",zzz,"\n",sep=" ")
>>    }
>> --------------------------------------------------------------------
>> There is some talk on the internet that some windows systems have a 
>> limit of 512 files that can be open at one time. Even though I'm closing 
>> my files each time, something is keeping track of how many times I've 
>> opened and closed a file in a session. I've talked to Microsoft and run 
>> a test program in Visual Studio C#, and, at the moment, it looks like 
>> the problem does not lie in the Microsoft arena. The C# program 
>> performed a similar task 10,000 times without a problem. I'm not totally 
>> convinced, but the current evidence says to look elsewhere.
>>
>> I've attached a script that will induce the problem. However, be warned 
>> that, if you use it, you will have to get out of R after you run it. R 
>> will no longer be able to access files such as help or sessionInfo(). 
>> This can be solved by getting out of the R GUI and back in.
>>
>> R E-mails from as far back as 2006 ask for help on the issue. There have 
>> been several suggestions, but no confirmed solution that I can find. You 
>> will see my attempt at these suggestions in the script [ rm(outMIA); 
>> rm(junk); closeAllConnections(); and gc(); after close(outMIA);  and 
>> unlink(conX);]. For me, this becomes important because it limits the 
>> total number if iterations in nested do-loops. This is where I ran into 
>> the problem. The program above will allow you to reproduce the problem.
>>
>> Any suggestion would be greatly appreciated.
> 
> This looks like a bug to me, and so I've posted it to the bugs list, but 
> your script doesn't make sense for normal use.  Before the loop you 
> create a file named according to conX containing the junk variable, and 
> then within the loop you load that file, *then delete it*, using 
> unlink().  However, if you examine the value of unlink(), you'll see 
> that it fails every time.
> 
> Moreover, the use of connections is unnecessary:  load() can work with a 
> filename.  So you can skip the file() and close() calls.
> 
> If I simplify your script by removing the unnecessary stuff, then it 
> works.  That is, this works:
> 
>      junk       = 1
>      PathA    = tempdir()
>      conX     = paste(PathA,"junk",sep="\\")
>      save(junk, file=conX)
>      for(i in 1:4000){
>          load(file=conX)
>          rm(junk)
>          cat(" i = ",i,sep=" ")
>          gc()
>          zzz = showConnections(all=FALSE)
>          cat(" zzz = ",zzz,"\n",sep=" ")
>      }
> 
> So a suggestion for a workaround is simply to follow the pattern above, 
> rather than that of your demonstration code.
> 
> I'll spend a bit more time and see if I can find what is causing the 
> open file leak.

Now fixed in R-devel, R-patched coming soon.

Duncan Murdoch