[R-sig-hpc] SNOW Hybrid Cluster in R, Network problems
stephen.b.weston at gmail.com
Tue Jul 3 20:03:43 CEST 2012
So if mpiexec works on your machines, why aren't you creating
a snow MPI cluster?
On Tue, Jul 3, 2012 at 11:09 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
> Hi Steve,
> :) so you pointed out the most interesting problem in this challenge: "How
> to start (fast) a remote process on Windows from a console"
> The aim was to find a clean solution without mixing up Windows with Linux
> staff like ssh or PsTools or rsh or vis versa. With MPICH2, smpd
> service/daemon running, it is possible so start processes on each worker.
> I am working on a Windows machine, so the R command to start a
> Windows-worker is as follows:
> system(paste("mpiexec -hosts ",length(nodestostart),"
> ",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat
> "wait = F" important, because the next command should be
> So I wrote a batch-file for windows and a shell script for Linux, which sets
> up each machine individually and which is started remote form MPICH2. This
> config script starts as many Rscripts as I had configured before. The Master
> on the other side knows how many workers are available an each machine.
> Important! It takes no configuration like rshcmd (because ssh is not
> available on Win). One must guarantee that "makeCluster()" comes after
> "starting Workers" thats the "trick". This works fine, because makeCluster()
> makes the master waiting for each worker calling back. If the master is not
> responding, the worker shuts down, so far in the Linux world.
> And finally: Yes Mpich2 is able to start every program / command on a remote
> machine. Thats a bit of a security problem, if you were very strong and
> restrictive to me.
> I hope I answered in your sense!
> Dipl.-Ing. Martin Seilmayer
> Helmholtz-Zentrum Dresden-Rossendorf e. V.
> Institut fuer Fluiddynamik
> Abteilung Magnetohydrodynamik
> Bautzner Landstraße 400
> 01328 Dresden, Germany
> @fon: +49 351 260 3165
> @fax: +49 351 260 12969
> @web: www.hzdr.de
> Am 03.07.2012 16:44, schrieb Stephen Weston:
>> On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de>
>>> Hi all of you,
>>> I successfully created a hybrid cluster of several Windows and Linux
>>> machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To
>>> the Rscript processes on each machine MPICH2 comes in the game. Because
>> Are you saying that you're using MPICH2 to start your workers on the
>> remote machines when creating a SOCK cluster? How are you doing that?
>> Are you setting the rshcmd option in some way? Does MPICH2 include a
>> remote execution command that is ssh-like?
>> - Steve Weston
More information about the R-sig-hpc