[R-sig-hpc] SNOW Hybrid Cluster in R, Network problems

[Martin Seilmayer]

Hi Steve,

I was a little bit wrong! Rmpi is available for windows R, but it seems 
to be not compatible with MPICH2!

So whats next... I have the Network / Software problem!!

Cu
Martin

Dipl.-Ing. Martin Seilmayer

Helmholtz-Zentrum Dresden-Rossendorf e. V.

Institut fuer Fluiddynamik
Abteilung Magnetohydrodynamik
Bautzner Landstraße 400
01328 Dresden, Germany

@fon: +49 351 260  3165
@fax: +49 351 260 12969
@web: www.hzdr.de

Am 03.07.2012 20:03, schrieb Stephen Weston:
> Hi Martin,
>
> So if mpiexec works on your machines, why aren't you creating
> a snow MPI cluster?
>
> - Steve
>
>
> On Tue, Jul 3, 2012 at 11:09 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
>> Hi Steve,
>>
>> :) so you pointed out the most interesting problem in this challenge: "How
>> to start (fast) a remote process on Windows from a console"
>>
>> The aim was to find a clean solution without mixing up Windows with Linux
>> staff like ssh or PsTools or rsh or  vis versa. With MPICH2, smpd
>> service/daemon running, it is possible so start processes on each worker.
>> I am working on a Windows machine, so the R command to start a
>> Windows-worker is as follows:
>>      system(paste("mpiexec -hosts ",length(nodestostart),"
>> ",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat
>> ",hostname,sep=""),wait=F)
>> "wait = F" important, because the next command should be
>> makeCluster(listofnodes,type=SOCK)
>> So I wrote a batch-file for windows and a shell script for Linux, which sets
>> up each machine individually and which is started remote form MPICH2. This
>> config script starts as many Rscripts as I had configured before. The Master
>> on the other side knows how many workers are available an each machine.
>>
>> Important! It takes no configuration like rshcmd (because ssh is not
>> available on Win). One must guarantee that "makeCluster()" comes after
>> "starting Workers" thats the "trick". This works fine, because makeCluster()
>> makes the master waiting for each worker calling back. If the master is not
>> responding, the worker shuts down, so far in the Linux world.
>>
>> And finally: Yes Mpich2 is able to start every program / command on a remote
>> machine. Thats a bit of a security problem, if you were very strong and
>> restrictive to me.
>>
>> I hope I answered in your sense!
>>
>> Martin
>>
>>
>>
>> Dipl.-Ing. Martin Seilmayer
>>
>> Helmholtz-Zentrum Dresden-Rossendorf e. V.
>>
>> Institut fuer Fluiddynamik
>> Abteilung Magnetohydrodynamik
>> Bautzner Landstraße 400
>> 01328 Dresden, Germany
>>
>> @fon: +49 351 260 3165
>> @fax: +49 351 260 12969
>> @web: www.hzdr.de
>>
>> Am 03.07.2012 16:44, schrieb Stephen Weston:
>>
>>> On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de>
>>> wrote:
>>>> Hi all of you,
>>>>
>>>> I successfully created a hybrid cluster of several Windows and Linux
>>>> machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To
>>>> start
>>>> the Rscript processes on each machine MPICH2 comes in the game. Because
>>>> it
>>> Are you saying that you're using MPICH2 to start your workers on the
>>> remote machines when creating a SOCK cluster?  How are you doing that?
>>> Are you setting the rshcmd option in some way?  Does MPICH2 include a
>>> remote execution command that is ssh-like?
>>>
>>> - Steve Weston
>>
>>