[R] makeSOCKcluster
Luke Tierney
luke at stat.uiowa.edu
Tue Aug 14 13:28:15 CEST 2007
Try using the option homogenoeus=FALSE and make sure the appropriate
environment variables are set on the worker nodes.
Best,
luke
On Tue, 14 Aug 2007, Michael Janis wrote:
> Hi,
>
>
>
> I am attempting to implement a mixed (windows/linux) snow sockets
> parallelism in R, but am running into difficulties similar to a post made
> Aug 31, 2006 under the same subject heading. I feel like I may be one or
> two non-obvious steps away from getting it all working, but I'm stuck. If
> anyone can shed some light on this (I believe Prof. Tierney stated that he
> has successfully run a snow master on Windows with slave nodes on Linux
> using ssh.exe through Cygwin, which is exactly what I am attempting), I'd be
> most grateful.
>
>
>
> SNOW master on Windows information:
>
> WinXP 32bit
>
> R-2.5.1
>
> I built a windows package (.zip) version of snow-0.2-3 for R using Rtools
> and installed it without trouble.
>
> Cygwin including ssh as an exe and in the path
>
>
>
> Linux slave nodes information:
>
> ROCKS compute nodes, each 64bit (I mention this difference between the win
> and linux platforms out of desperation but I don't think it's an issue since
> snow should be agnostic). (btw, the SOCK version of snow here is just for
> me to get used to parallelism in R before attempting MPI).
>
> R-2.5.1, local directory install (no su, global R install is older).
>
> Snow installed to R-2.5.1 instance
>
> Unattended ssh authentication through public/private key pairs. Each node
> allows since they are all NFS.
>
>
>
> Rgui commands:
>
>> library(snow)
>
>> cl<-makeCluster(c("ip.path.to.node1","ip.path.to.node2"),type="SOCK")
>
> Result:
>
> R becomes unresponsive and has to be forcibly closed. Sometimes before that
> happens message stating env: C:\pathToR\library\snow\RSOCKnode.sh no file
> found appears (if I make R redraw the interface screen)
>
> Troubleshooting step1:
>
> Interspersed print commands within C:\localRinstall\library\snow\R\snow
> file.
>
> Result of troubleshooting step1:
>
> It appears that this snow file hangs on line:
> system(paste(rshcmd, "-l", user, machine, "env", env, script))
>
> Resolution of troubleshooting step1:
>
> Manually attempted paste commands in R:
>
> >system("ssh -l me ip.path.to.node1 ls")
>
> Result: gets ls of remote machine; so the script path must be
> incorrect and the offending command
>
> Located RSOCKnode.sh on remote machine (for all remote machines, since
> it's NFS)
>
> Inserted line into C:\localRinstall\library\snow\R\snow file
> just before the offending line system(paste(rshcmd, "-l", user, machine,
> "env", env, script)). That inserted line hard-codes the path of the script
> and the script name like so:
> script<-"/rootpath/home/cluster/me/R-2.5.1/library/snow/RSOCKnode.sh". This
> worked, and the next print statement following the system call in the snow
> file now prints to the screen. But again R hangs, this time at the line:
> con <- socketConnection(port = port, server=TRUE, blocking=TRUE,
> open="a+b").
>
> Troubleshooting step2:
>
> Attempted manual socketConnection via R commands:
>
> > con<-socketConnection("ip.path.to.node1",port=22)
>
>> showConnections()
>
> description class mode text isopen can read can
> write
>
> 3 "->ip.path.to.node1" "socket" "a+" "text" "opened" "yes" "yes"
>
> *** so the socketConnection does work, but in SNOW it hangs at this last
> command, and I'm completely stuck.
>
>
>
> As a parallel test, I have installed SNOW on the R-2.5.1 instance on the
> linux cluster. If I access the head node and launch R, I can use SNOW with
> sockets successfully (from the linux head node as master to the linux
> compute nodes as slaves). However, running R more or less continually on the
> head node of a cluster is bad form. Ideally I would like to run a windows
> snow master to the linux node slaves, but I can't seem to get past this
> point.
>
>
>
> For reference, I include the original post below. I am stuck at one command
> in the snow file past where this last post had problems.
>
>
>
> I hope this is clear. I am intent on solving this problem, so feel free to
> ask questions if you have feedback but my description is not clear. I
> really appreciate any help!
>
>
>
> Yours,
>
>
>
> Michael Janis
>
> UCLA Bioinformatics
>
>
>
> *****transcript from original post entitles "makeSOCKcluster" follows*****
>
>
>
>
>
> makeSOCKcluster
>
> Click to flag this post
>
>
>
> by Hrishikesh Rajpathak Aug 31, 2006; 10:39pm :: Rate this Message: - Use
> ratings to moderate (?)
>
>
>
> Reply | Reply to Author | View Threaded | Show Only this Message
>
> Hi,
>
>
>
> I am a newbie to R and trying to implement parallelism in R. I am
> currently using R-2.3.1, and Cygwin to run R on Windows xp.
>
>
>
> ssh and all are working fine,
>
>
>
> When I try to create a socket connection as
>
>
>
> makeSOCKcluster(c("localhost","localhost")),
>
>
>
> it just waits for the other prcess on localhost to get created and
> respond. But this other process is not created.
>
>
>
> To debug, I put print statements in the "snow " file in library\snow\r of
> R after every statement that comes under Socket Implementation. I realized
> that it does the execution till
>
>
>
> system(a<-paste(rshcmd, "-l", user, machine, "env", env, script))
>
>
>
> part and then it goes in wait state. It cannot run the next command which
> is
>
>
>
> con <- socketConnection(port = port, server=TRUE, blocking=TRUE,
>
> open="a+b")
>
>
>
> can someone please tell me what exactly could be the problem?
>
>
>
> Thank you,
>
> Rishi
>
>
>
>
>
> ---------------------------------
>
> Here's a new way to find what you're looking for - Yahoo! Answers
>
>
>
> [[alternative HTML version deleted]]
>
>
>
> ______________________________________________
>
> R-help at ... mailing list
>
> https://stat.ethz.ch/mailman/listinfo/r-help
>
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>
> and provide commented, minimal, self-contained, reproducible code.
>
>
>
>
>
> Re: makeSOCKcluster
>
> Click to flag this post
>
>
>
> by Prof Brian Ripley Aug 31, 2006; 10:59pm :: Rate this Message: - Use
> ratings to moderate (?)
>
>
>
> Reply | Reply to Author | View Threaded | Show Only this Message
>
> On Fri, 1 Sep 2006, Hrishikesh Rajpathak wrote:
>
>
>
>> Hi,
>
>>
>
>> I am a newbie to R and trying to implement parallelism in R. I am
>
>> currently using R-2.3.1, and Cygwin to run R on Windows xp.
>
>
>
> Did you build R under Cygwin (which is not a supported platform), or are
>
> you running a native Windows build of R? If the latter, you may be being
>
> optimistic to expect R to run Unix commands in the same way as under Unix.
>
>
>
> I presume you mean this line in package snow:
>
>
>
> system(paste(rshcmd, "-l", user, machine, "env", env, script))
>
>
>
> which appears to be attempting to run a shell script: no shell is used by
>
> system() on Windows R. As the return value is not checked, this will not
>
> fail.
>
>
>
> R under Windows differs from R under Unix in a number of ways, and
>
> system() is a major one.
>
>
>
>
>
>> ssh and all are working fine,
>
>>
>
>> When I try to create a socket connection as
>
>>
>
>> makeSOCKcluster(c("localhost","localhost")),
>
>>
>
>> it just waits for the other prcess on localhost to get created and
> respond. But this other process is not created.
>
>>
>
>> To debug, I put print statements in the "snow " file in library\snow\r
> of R after every statement that comes under Socket Implementation. I
> realized that it does the execution till
>
>>
>
>> system(a<-paste(rshcmd, "-l", user, machine, "env", env, script))
>
>>
>
>> part and then it goes in wait state. It cannot run the next command
> which is
>
>>
>
>> con <- socketConnection(port = port, server=TRUE, blocking=TRUE,
>
>> open="a+b")
>
>>
>
>> can someone please tell me what exactly could be the problem?
>
>>
>
>> Thank you,
>
>> Rishi
>
>
>
>
--
Luke Tierney
Chair, Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa Phone: 319-335-3386
Department of Statistics and Fax: 319-335-3017
Actuarial Science
241 Schaeffer Hall email: luke at stat.uiowa.edu
Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
More information about the R-help
mailing list