[R-sig-hpc] problem with opening more than one SOCK cluster with package snow

[Stephen Weston]

You might want to talk to the sysadmins of your cluster to
see what they suggest.  If you start snow clusters in this way
using random ports, the workers that are started may trigger
a lot of error messages in the system log files which may
alarm a watchful sysadmin.

Personally, I would use the multicore package in this situation.

- Steve


On Fri, Dec 21, 2012 at 6:38 AM, Jannis <bt_jannis at yahoo.de> wrote:
> Hi Steve,
>
> thanks a lot for your reply. There may be an elegant and fail proof method
> for port determination but I learned during my work that sometimes the quick
> and dirty way has a much higher output/programming energy ratio ;-). So I
> would start with assigning random ports each time a cluster is started and
> restart the odd jobs with conflicting ports. I am no expert in ports and
> related things, but are there possible problems with other programs or even
> security that might occour if I use random port numbers between 1023 and
> 65535?
>
>
> Thanks a lot
> Jannis
>
>
> On 20.12.2012 16:00, Stephen Weston wrote:
>>
>> Hi Jannis,
>>
>> I think you are right as to the cause of the error.  I believe that
>> the problem occurs when two master processes start running
>> on the same machine at about the same time because they
>> will both try to bind to the same port by default.  Interestingly,
>> the master process only binds to that port temporarily, which
>> is why the problem is intermittent.  The more workers that you
>> start, the longer it takes to create the cluster, and the more
>> likely the error is to occur.
>>
>> I suggest that you use the "port" option when calling makeCluster,
>> as in:
>>
>>    cl <- makeCluster(8, type="SOCK", port=10188)
>>
>> where you specify a different value for every master on the
>> same machine.  I'm not aware of a fool-proof technique for
>> doing that, but you can probably come up with some method
>> that works pretty well for your situation.
>>
>> Of course, if you're not using Windows, you could try using
>> the "parallel" package, and create a "FORK" cluster.  That
>> doesn't use sockets at all, so multiple clusters on the same
>> machine isn't a problem.
>>
>> - Steve
>>
>>
>> On Thu, Dec 20, 2012 at 7:11 AM, Jannis <bt_jannis at yahoo.de> wrote:
>>>
>>> Der HPC R users,
>>>
>>>
>>> sorry for cross posting (to r-help) but I just realized after posting
>>> that
>>> the hpc list may be more suitable for my request. I will report solutions
>>> to
>>> both lists.
>>>
>>>
>>> Cheers
>>> Jannis
>>>
>>>
>>> Dear r-help list,
>>>
>>> i have some problems using the snow package to create a SOCK cluster. The
>>> errors just occour irregularly but it seems to me that they occour when I
>>> try to create more than one cluster on the same machine via different R
>>> instances started via submitting LSF jobs to a cluster. Does anyone have
>>> an
>>> idea how to solve this or where to start digging for solutions?
>>>
>>> The error messages are:
>>>
>>> library(snow)
>>> cl <- makeCluster(8, type = "SOCK")
>>>
>>>
>>> Error in socketConnection(port = port, server = TRUE, blocking = TRUE,  :
>>>    cannot open the connection
>>> Calls: run.call ... makeCluster -> makeSOCKcluster -> newSOCKnode ->
>>> socketConnection
>>> In addition: Warning message:
>>> In socketConnection(port = port, server = TRUE, blocking = TRUE,  :
>>>    port 10187 cannot be opened
>>>
>>>
>>> R version 2.15.1 (2012-06-22)
>>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>>
>>> locale:
>>>   [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
>>>   [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
>>>   [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
>>>   [7] LC_PAPER=C                 LC_NAME=C
>>>   [9] LC_ADDRESS=C               LC_TELEPHONE=C
>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>>
>>> attached base packages:
>>> [1] stats     graphics  grDevices datasets  utils     methods   base
>>>
>>> other attached packages:
>>> [1] snow_0.3-10
>>>
>>>
>>> Thanks a lot
>>> Jannis
>>>
>>> _______________________________________________
>>> R-sig-hpc mailing list
>>> R-sig-hpc at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc