Hi,

 

I am attempting to implement a mixed (windows/linux) snow sockets
parallelism in R, but am running into difficulties similar to a post made
Aug 31, 2006 under the same subject heading.  I feel like I may be one or
two non-obvious steps away from getting it all working, but I'm stuck.  If
anyone can shed some light on this (I believe Prof. Tierney stated that he
has successfully run a snow master on Windows with slave nodes on Linux
using ssh.exe through Cygwin, which is exactly what I am attempting), I'd be
most grateful.

 

SNOW master on Windows information:

WinXP 32bit

R-2.5.1

I built a windows package (.zip) version of snow-0.2-3 for R using Rtools
and installed it without trouble.

Cygwin including ssh as an exe and in the path

 

Linux slave nodes information:

ROCKS compute nodes, each 64bit (I mention this difference between the win
and linux platforms out of desperation but I don't think it's an issue since
snow should be agnostic).  (btw, the SOCK version of snow here is just for
me to get used to parallelism in R before attempting MPI).

R-2.5.1, local directory install (no su, global R install is older).

Snow installed to R-2.5.1 instance

Unattended ssh authentication through public/private key pairs.  Each node
allows since they are all NFS.

 

Rgui commands:

>library(snow)

>cl<-makeCluster(c("ip.path.to.node1","ip.path.to.node2"),type="SOCK")

Result:

R becomes unresponsive and has to be forcibly closed.  Sometimes before that
happens message stating env: C:\pathToR\library\snow\RSOCKnode.sh no file
found appears (if I make R redraw the interface screen)

Troubleshooting step1:

Interspersed print commands within C:\localRinstall\library\snow\R\snow
file.

Result of troubleshooting step1:

            It appears that this snow file hangs on line:
system(paste(rshcmd, "-l", user, machine, "env", env, script))

Resolution of troubleshooting step1:

      Manually attempted paste commands in R:

      >system("ssh -l me ip.path.to.node1 ls")

            Result: gets ls of remote machine; so the script path must be
incorrect and the offending command

      Located RSOCKnode.sh on remote machine (for all remote machines, since
it's NFS)

            Inserted line into C:\localRinstall\library\snow\R\snow file
just before the offending line system(paste(rshcmd, "-l", user, machine,
"env", env, script)).  That inserted line hard-codes the path of the script
and the script name like so:
script<-"/rootpath/home/cluster/me/R-2.5.1/library/snow/RSOCKnode.sh".  This
worked, and the next print statement following the system call in the snow
file now prints to the screen.  But again R hangs, this time at the line:
con <- socketConnection(port = port, server=TRUE, blocking=TRUE,
open="a+b").

Troubleshooting step2:

      Attempted manual socketConnection via R commands:

      > con<-socketConnection("ip.path.to.node1",port=22)

> showConnections()

            description         class    mode text   isopen   can read can
write

3  "->ip.path.to.node1" "socket" "a+" "text" "opened" "yes"    "yes"

*** so the socketConnection does work, but in SNOW it hangs at this last
command, and I'm completely stuck.

 

As a parallel test, I have installed SNOW on the R-2.5.1 instance on the
linux cluster.  If I access the head node and launch R, I can use SNOW with
sockets successfully (from the linux head node as master to the linux
compute nodes as slaves). However, running R more or less continually on the
head node of a cluster is bad form.  Ideally I would like to run a windows
snow master to the linux node slaves, but I can't seem to get past this
point.

 

For reference, I include the original post below.  I am stuck at one command
in the snow file past where this last post had problems.

 

I hope this is clear.  I am intent on solving this problem, so feel free to
ask questions if you have feedback but my description is not clear.  I
really appreciate any help!

 

Yours,

 

Michael Janis

UCLA Bioinformatics

 

*****transcript from original post entitles "makeSOCKcluster" follows*****

 

 

makeSOCKcluster

Click to flag this post

 

by Hrishikesh Rajpathak Aug 31, 2006; 10:39pm :: Rate this Message: - Use
ratings to moderate (?)

 

Reply | Reply to Author | View Threaded | Show Only this Message

Hi,

 

  I am a newbie to R and trying to implement parallelism in R. I am
currently using R-2.3.1, and Cygwin to run R on Windows xp.

 

  ssh and all are working fine,

 

  When I try to create a socket connection as

 

   makeSOCKcluster(c("localhost","localhost")),

 

  it just waits for the other prcess on localhost to get created and
respond. But this other process is not created.

 

  To debug, I put print statements in the "snow " file in library\snow\r  of
R after every statement that comes under Socket Implementation. I  realized
that it does the execution till

 

  system(a<-paste(rshcmd, "-l", user, machine, "env", env, script))

 

  part and then it goes in wait state. It cannot run the next command which
is

 

  con <- socketConnection(port = port, server=TRUE, blocking=TRUE,

                               open="a+b")

 

  can someone please tell me what exactly could be the problem?

 

  Thank you,

  Rishi

 

 

---------------------------------

 Here's a new way to find what you're looking for - Yahoo! Answers

 

        [[alternative HTML version deleted]]

 

______________________________________________

R-help@... mailing list

https://stat.ethz.ch/mailman/listinfo/r-help

PLEASE do read the posting guide http://www.R-project.org/posting-guide.html

and provide commented, minimal, self-contained, reproducible code.

 

      

Re: makeSOCKcluster

Click to flag this post

 

by Prof Brian Ripley Aug 31, 2006; 10:59pm :: Rate this Message: - Use
ratings to moderate (?)

 

Reply | Reply to Author | View Threaded | Show Only this Message

On Fri, 1 Sep 2006, Hrishikesh Rajpathak wrote:

 

> Hi,

>  

>   I am a newbie to R and trying to implement parallelism in R. I am

>   currently using R-2.3.1, and Cygwin to run R on Windows xp.

 

Did you build R under Cygwin (which is not a supported platform), or are

you running a native Windows build of R?  If the latter, you may be being

optimistic to expect R to run Unix commands in the same way as under Unix.

 

I presume you mean this line in package snow:

 

    system(paste(rshcmd, "-l", user, machine, "env", env, script))

 

which appears to be attempting to run a shell script: no shell is used by

system() on Windows R.  As the return value is not checked, this will not

fail.

 

R under Windows differs from R under Unix in a number of ways, and

system() is a major one.

 

 

>   ssh and all are working fine,

>  

>   When I try to create a socket connection as

>  

>    makeSOCKcluster(c("localhost","localhost")),

>  

>   it just waits for the other prcess on localhost to get created and
respond. But this other process is not created.

>  

>   To debug, I put print statements in the "snow " file in library\snow\r
of R after every statement that comes under Socket Implementation. I
realized that it does the execution till

>  

>   system(a<-paste(rshcmd, "-l", user, machine, "env", env, script))

>  

>   part and then it goes in wait state. It cannot run the next command
which is

>  

>   con <- socketConnection(port = port, server=TRUE, blocking=TRUE,

>                                open="a+b")

>  

>   can someone please tell me what exactly could be the problem?

>  

>   Thank you,

>   Rishi

 

-- 

Brian D. Ripley,                  ripley@...

Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/

University of Oxford,             Tel:  +44 1865 272861 (self)

1 South Parks Road,                     +44 1865 272866 (PA)

Oxford OX1 3TG, UK                Fax:  +44 1865 272595

 

______________________________________________

R-help@... mailing list

https://stat.ethz.ch/mailman/listinfo/r-help

PLEASE do read the posting guide http://www.R-project.org/posting-guide.html

and provide commented, minimal, self-contained, reproducible code.

 

      

Re: makeSOCKcluster

Click to flag this post

 

by Luke Tierney Sep 01, 2006; 04:33am :: Rate this Message: - Use ratings to
moderate (?)

 

Reply | Reply to Author | View Threaded | Show Only this Message

On Fri, 1 Sep 2006, Prof Brian Ripley wrote:

 

> On Fri, 1 Sep 2006, Hrishikesh Rajpathak wrote:

> 

>> Hi,

>> 

>>   I am a newbie to R and trying to implement parallelism in R. I am

>>   currently using R-2.3.1, and Cygwin to run R on Windows xp.

> 

> Did you build R under Cygwin (which is not a supported platform), or are

> you running a native Windows build of R?  If the latter, you may be being

> optimistic to expect R to run Unix commands in the same way as under Unix.

> 

> I presume you mean this line in package snow:

> 

>    system(paste(rshcmd, "-l", user, machine, "env", env, script))

> 

> which appears to be attempting to run a shell script: no shell is used by

> system() on Windows R.  As the return value is not checked, this will not

> fail.

> 

> R under Windows differs from R under Unix in a number of ways, and

> system() is a major one.

 

This runs a command of the form

 

  ssh -l user machine env X=x Y=z script

 

so if your ssh is a .exe this should be OK.  I have successfully run a

snow master on Windows with slave nodes on Linux with I believe the

ssh coming from cygwin, but that was a while ago.

 

My suggestion is that you look at the command that is generated by the

paste and try running it by hand and see if you can work out why it

hangs.

 

Best,

 

luke

 

 

> 

> 

>>   ssh and all are working fine,

>> 

>>   When I try to create a socket connection as

>> 

>>    makeSOCKcluster(c("localhost","localhost")),

>> 

>>   it just waits for the other prcess on localhost to get created and
respond. But this other process is not created.

>> 

>>   To debug, I put print statements in the "snow " file in library\snow\r
of R after every statement that comes under Socket Implementation. I
realized that it does the execution till

>> 

>>   system(a<-paste(rshcmd, "-l", user, machine, "env", env, script))

>> 

>>   part and then it goes in wait state. It cannot run the next command
which is

>> 

>>   con <- socketConnection(port = port, server=TRUE, blocking=TRUE,

>>                                open="a+b")

>> 

>>   can someone please tell me what exactly could be the problem?

>> 

>>   Thank you,

>>   Rishi

> 

> 

 

-- 

Luke Tierney

Chair, Statistics and Actuarial Science

Ralph E. Wareham Professor of Mathematical Sciences

University of Iowa                  Phone:             319-335-3386

Department of Statistics and        Fax:               319-335-3017

    Actuarial Science

241 Schaeffer Hall                  email:      luke@...

Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu

      

 

 

 


	[[alternative HTML version deleted]]

