[R] Parallel processing random 'save' error

[William Dunlap]

> Error in checkForRemoteErrors(val) :
>   one node produced an error: (converted from warning)
> 'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins' already exists

That warning looks like it comes from dir.create().  Do you have
code that looks like:
   if (!file.exists(tempDir)) {
      dir.create(tempDir)
   }
If so that could be the problem.  The directory may not exist when file.exists()
is called but by the time dir.create is called another process may have created
it.  Try replacing such code with
    suppressWarnings(dir.create(tempDir))
    if (!isTRUE(file.info(tempDir)$isdir)) {
        stop("Cannot create tempDir=", tempDir)
    }


Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com


> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf
> Of Rguy
> Sent: Monday, July 01, 2013 1:07 AM
> To: r-help at r-project.org
> Subject: [R] Parallel processing random 'save' error
> 
> Platform: Windows 7
> Package: parallel
> Function: parLapply
> 
> I am running a lengthy program with 8 parallel processes running in main
> memory.
> The processes save data using the 'save' function, to distinct files so
> that no conflicts writing to the same file are possible.
> I have been getting errors like the one shown below on a random basis,
> i.e., sometimes at one point in the execution, sometimes at another,
> sometimes no error at all.
> I should note that the directory referred to in the error message
> ( 'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins') contains, as I
> write, 124 files saved to it by the program without any error; which
> underscores the point that most of the time the saves occur with no problem.
> 
> Error in checkForRemoteErrors(val) :
>   one node produced an error: (converted from warning)
> 'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins' already exists
> 
> Enter a frame number, or 0 to exit
> 
>  1: main_top(9)
>  2: main_top.r#26: eval(call_me)
>  3: eval(expr, envir, enclos)
>  4: quantile_analysis(2)
>  5: quantile_analysis.r#69: run_all(layr, prjp, np, rules_tb, pctiles_tb,
> parx, logdir, logg)
>  6: run_all.r#73: parLapply(cl, ctrl_all$vn, qa1, prjp, dfr1, "iu__bool",
> parx, logdir, tstamp)
>  7: do.call(c, clusterApply(cl, x = splitList(X, length(cl)), fun = lapply,
> fun, ...), quote = TRUE)
>  8: clusterApply(cl, x = splitList(X, length(cl)), fun = lapply, fun, ...)
>  9: staticClusterApply(cl, fun, length(x), argfun)
> 10: checkForRemoteErrors(val)
> 11: stop("one node produced an error: ", firstmsg, domain = NA)
> 12: (function ()
> {
>     error()
>     utils::recover()
> })()
> 
> Following the latest error I checked the system's connections as follows:
> 
> Browse[1]> showConnections()
>    description             class      mode  text     isopen   can read can
> write
> 3  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 4  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 5  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 6  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 7  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 8  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 9  "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> 10 "<-LAPTOP_32G_01:11741" "sockconn" "a+b" "binary" "opened" "yes"
>  "yes"
> Browse[1]>
> 
> It seems that the parallel processes might be sharing the same
> connection--or is it that they are utilizing connections that have the same
> name but are actually distinct because they are running in parallel?
> If the connections are the problem, how can I force each parallel process
> to use a different connection?
> If the connections are not the problem, then can someone suggest a
> diagnostic I might apply to tease out what is going wrong? Or perhaps some
> program setting that I may have neglected to consider?
> 
> Thanks in advance for your help.
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.