[R] Problem With Repeated Use Of Load/Save/Close Commands

[Duncan Murdoch]

On 18/07/2009 11:08 AM, Duncan Murdoch wrote:
> On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote:
>> Hello,
>>
>> I'm having a problem in R with repeated use of the "load", "save", and 
>> "close" commands. I'm getting an error message that reads, "Too many 
>> open files". I'm opening files and closing them (and unlinking them), 
>> but when I go through that process 509 times, the program halts and I 
>> get this error message: "cannot open the connection" with warning 
>> messages: "Too many open files".  I've been working on this problem for 
>> a couple of weeks and have gleaned a bit of info from different internet 
>> threads, but no solutions yet.
>>
>> I'm using Windows XP, SP3 and R 2.9.1.
>> Here is my session info:
>>
>> R version 2.9.1 (2009-06-26)
>> i286-pc-mingw32
>>
>> locale:
>> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
>> States. 1252;LC_MONETARY=English_United
>> States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>
>> I'm using the vanilla load of R, with nothing added after booting up.
>> The problem also occurs on my Vista machine as well.
>>
>> The program below will induce the problem.
>> ------------------------------------------------------------
>>    junk       = 1
>>    PathA    = tempdir()
>>    conX     = paste(PathA,"junk",sep="\\")
>>    outJunk  = file(conX, open="wb")
>>    save(junk, file=outJunk)
>>    close(outJunk)
>>    for(i in 1:4000){
>>        outMIA = file(conX, open="rb")
>>        load(file=outMIA)
>>        close(outMIA)
>>        closeAllConnections()
>>        unlink(conX)
>>        rm(outMIA)
>>        rm(junk)
>>        cat(" i = ",i,sep=" ")
>>        gc()
>>        zzz = showConnections(all=FALSE)
>>        cat(" zzz = ",zzz,"\n",sep=" ")
>>    }
>> --------------------------------------------------------------------
>> There is some talk on the internet that some windows systems have a 
>> limit of 512 files that can be open at one time. Even though I'm closing 
>> my files each time, something is keeping track of how many times I've 
>> opened and closed a file in a session. I've talked to Microsoft and run 
>> a test program in Visual Studio C#, and, at the moment, it looks like 
>> the problem does not lie in the Microsoft arena. The C# program 
>> performed a similar task 10,000 times without a problem. I'm not totally 
>> convinced, but the current evidence says to look elsewhere.
>>
>> I've attached a script that will induce the problem. However, be warned 
>> that, if you use it, you will have to get out of R after you run it. R 
>> will no longer be able to access files such as help or sessionInfo(). 
>> This can be solved by getting out of the R GUI and back in.
>>
>> R E-mails from as far back as 2006 ask for help on the issue. There have 
>> been several suggestions, but no confirmed solution that I can find. You 
>> will see my attempt at these suggestions in the script [ rm(outMIA); 
>> rm(junk); closeAllConnections(); and gc(); after close(outMIA);  and 
>> unlink(conX);]. For me, this becomes important because it limits the 
>> total number if iterations in nested do-loops. This is where I ran into 
>> the problem. The program above will allow you to reproduce the problem.
>>
>> Any suggestion would be greatly appreciated.
> 
> I've somewhat localized the problem.  load(file=outMIA) passes the 
> outMIA connection to gzcon(), which handles decompression of the data. 
> gzcon() re-opens the file, and it looks as though the original file 
> handle is lost, because closing either the result of gzcon() or the 
> original connection results in only the second file handle being closed.
> 
> So a second workaround besides the one I sent earlier is just not to 
> open or close outMIA.  That is, rewrite it as
> 
>     junk       = 1
>     PathA    = tempdir()
>     conX     = paste(PathA,"junk",sep="\\")
>     outJunk  = file(conX, open="wb")
>     save(junk, file=outJunk)
>     close(outJunk)
>     for(i in 1:4000){
>         outMIA = file(conX)
>         load(file=outMIA)
>         rm(outMIA)
>         rm(junk)
>         cat(" i = ",i,sep=" ")
>         gc()
>         zzz = showConnections(all=FALSE)
>         cat(" zzz = ",zzz,"\n",sep=" ")
>     }
> 
> so that load() does the open and close, and things are fine.

Oops, not quite fine.  There's a whole set of warnings...

Duncan Murdoch