[R-sig-hpc] SNOW Hybrid Cluster in R, Network problems

Martin Seilmayer M.Seilmayer at hzdr.de
Tue Jul 3 17:09:09 CEST 2012


Hi Steve,

:) so you pointed out the most interesting problem in this challenge: 
"How to start (fast) a remote process on Windows from a console"

The aim was to find a clean solution without mixing up Windows with 
Linux staff like ssh or PsTools or rsh or  vis versa. With MPICH2, smpd 
service/daemon running, it is possible so start processes on each worker.
I am working on a Windows machine, so the R command to start a 
Windows-worker is as follows:
     system(paste("mpiexec -hosts ",length(nodestostart)," 
",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat 
",hostname,sep=""),wait=F)
"wait = F" important, because the next command should be 
makeCluster(listofnodes,type=SOCK)
So I wrote a batch-file for windows and a shell script for Linux, which 
sets up each machine individually and which is started remote form 
MPICH2. This config script starts as many Rscripts as I had configured 
before. The Master on the other side knows how many workers are 
available an each machine.

Important! It takes no configuration like rshcmd (because ssh is not 
available on Win). One must guarantee that "makeCluster()" comes after 
"starting Workers" thats the "trick". This works fine, because 
makeCluster() makes the master waiting for each worker calling back. If 
the master is not responding, the worker shuts down, so far in the Linux 
world.

And finally: Yes Mpich2 is able to start every program / command on a 
remote machine. Thats a bit of a security problem, if you were very 
strong and restrictive to me.

I hope I answered in your sense!

Martin


Dipl.-Ing. Martin Seilmayer

Helmholtz-Zentrum Dresden-Rossendorf e. V.

Institut fuer Fluiddynamik
Abteilung Magnetohydrodynamik
Bautzner Landstraße 400
01328 Dresden, Germany

@fon: +49 351 260  3165
@fax: +49 351 260 12969
@web: www.hzdr.de

Am 03.07.2012 16:44, schrieb Stephen Weston:
> On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
>> Hi all of you,
>>
>> I successfully created a hybrid cluster of several Windows and Linux
>> machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To start
>> the Rscript processes on each machine MPICH2 comes in the game. Because it
> Are you saying that you're using MPICH2 to start your workers on the
> remote machines when creating a SOCK cluster?  How are you doing that?
> Are you setting the rshcmd option in some way?  Does MPICH2 include a
> remote execution command that is ssh-like?
>
> - Steve Weston



More information about the R-sig-hpc mailing list