[R-sig-hpc] SNOW Hybrid Cluster in R, Network problems

[Stephen Weston]

Hi Martin,

So if mpiexec works on your machines, why aren't you creating
a snow MPI cluster?

- Steve


On Tue, Jul 3, 2012 at 11:09 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
> Hi Steve,
>
> :) so you pointed out the most interesting problem in this challenge: "How
> to start (fast) a remote process on Windows from a console"
>
> The aim was to find a clean solution without mixing up Windows with Linux
> staff like ssh or PsTools or rsh or  vis versa. With MPICH2, smpd
> service/daemon running, it is possible so start processes on each worker.
> I am working on a Windows machine, so the R command to start a
> Windows-worker is as follows:
>     system(paste("mpiexec -hosts ",length(nodestostart),"
> ",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat
> ",hostname,sep=""),wait=F)
> "wait = F" important, because the next command should be
> makeCluster(listofnodes,type=SOCK)
> So I wrote a batch-file for windows and a shell script for Linux, which sets
> up each machine individually and which is started remote form MPICH2. This
> config script starts as many Rscripts as I had configured before. The Master
> on the other side knows how many workers are available an each machine.
>
> Important! It takes no configuration like rshcmd (because ssh is not
> available on Win). One must guarantee that "makeCluster()" comes after
> "starting Workers" thats the "trick". This works fine, because makeCluster()
> makes the master waiting for each worker calling back. If the master is not
> responding, the worker shuts down, so far in the Linux world.
>
> And finally: Yes Mpich2 is able to start every program / command on a remote
> machine. Thats a bit of a security problem, if you were very strong and
> restrictive to me.
>
> I hope I answered in your sense!
>
> Martin
>
>
>
> Dipl.-Ing. Martin Seilmayer
>
> Helmholtz-Zentrum Dresden-Rossendorf e. V.
>
> Institut fuer Fluiddynamik
> Abteilung Magnetohydrodynamik
> Bautzner Landstraße 400
> 01328 Dresden, Germany
>
> @fon: +49 351 260 3165
> @fax: +49 351 260 12969
> @web: www.hzdr.de
>
> Am 03.07.2012 16:44, schrieb Stephen Weston:
>
>> On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de>
>> wrote:
>>>
>>> Hi all of you,
>>>
>>> I successfully created a hybrid cluster of several Windows and Linux
>>> machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To
>>> start
>>> the Rscript processes on each machine MPICH2 comes in the game. Because
>>> it
>>
>> Are you saying that you're using MPICH2 to start your workers on the
>> remote machines when creating a SOCK cluster?  How are you doing that?
>> Are you setting the rshcmd option in some way?  Does MPICH2 include a
>> remote execution command that is ssh-like?
>>
>> - Steve Weston
>
>
>