[R] parallel: socket connection behind a NAT router

[Jiefei Wang]

Thanks! This solution also looks promising. It should be more stable than
using ssh tunneling. I will also explore this method.

Best,
Jiefei

On Tue, Jan 19, 2021 at 3:11 PM Martin Morgan <mtmorgan.bioc using gmail.com>
wrote:

> A different approach uses doRedis
> https://CRAN.R-project.org/package=doRedis (currently archived, but
> actively developed) for use with the foreach package, or RedisParam
> https://github.com/mtmorgan/RedisParam (not released) for use with
> Bioconductor's BiocParallel package.
>
> These use a redis server https://redis.io/ to communicate -- the manager
> submits jobs / obtains results from the redis server, the workers retrieve
> jobs / submit results to the redis server. Manager and worker need to know
> the (http) address of the server, etc, but there are no other ports
> involved.
>
> Redis servers are easy to establish in a cloud environment, using e.g.,
> existing AWS or docker images. The README for doRedis
> https://github.com/bwlewis/doRedis probably provides the easiest
> introduction.
>
> The (not mature) k8sredis Kubernetes / helm chart
> https://github.com/Bioconductor/k8sredis illustrates a complete system
> using RedisParam, deploying manager and workers locally or in the google
> cloud; the app could be modified to only start the workers in the cloud,
> exposing the redis server for access by a local 'manager'; this would be
> cool.
>
> Martin
>
> On 1/19/21, 1:50 AM, "R-help on behalf of Henrik Bengtsson" <
> r-help-bounces using r-project.org on behalf of henrik.bengtsson using gmail.com>
> wrote:
>
>     On Mon, Jan 18, 2021 at 9:42 PM Jiefei Wang <szwjf08 using gmail.com> wrote:
>     >
>     > Thanks for introducing this interesting package to me! it is great
> to know a new powerful tool, but it seems like this method does not work in
> my environment. ` parallelly::makeClusterPSOCK` will hang until timeout.
>     >
>     > I checked the verbose output and it looks like the parallelly
> package also depends on `parallel:::.slaveRSOCK` on the remote instance to
> build the connection. This explains why it failed for the local machine
> does not have a public IP and the remote does not know how to build the
> connection.
>
>     It's correct that the worker does attempt to connect back to the
>     parent R process that runs on your local machine.  However, it does
>     *not* do so by your local machines public IP address but it does it by
>     connecting to a port on its own machine - a port that was set up by
>     SSH.  More specifically, when parallelly::makeClusterPSOCK() connects
>     to the remote machine over SSH it also sets up a so-called reverse SSH
>     tunnel with a certain port on your local machine and certain port of
>     your remote machine.  This is what happens:
>
>     > cl <- parallelly::makeClusterPSOCK("machine1.example.org",
> verbose=TRUE)
>     [local output] Workers: [n = 1] 'machine1.example.org'
>     [local output] Base port: 11019
>     ...
>     [local output] Starting worker #1 on 'machine1.example.org':
>     '/usr/bin/ssh' -R 11068:localhost:11068 machine1.example.org
>     "'Rscript'
> --default-packages=datasets,utils,grDevices,graphics,stats,methods
>     -e 'workRSOCK <- tryCatch(parallel:::.slaveRSOCK, error=function(e)
>     parallel:::.workRSOCK); workRSOCK()' MASTER=localhost PORT=11068
>     OUT=/dev/null TIMEOUT=2592000 XDR=FALSE"
>     [local output] - Exit code of system() call: 0
>     [local output] Waiting for worker #1 on 'machine1.example.org' to
>     connect back  '/usr/bin/ssh' -R 11019:localhost:11019
>     machine1.example.org "'Rscript'
>     --default-packages=datasets,utils,grDevices,graphics,stats,methods -e
>     'workRSOCK <- tryCatch(parallel:::.slaveRSOCK, error=function(e)
>     parallel:::.workRSOCK); workRSOCK()' MASTER=localhost PORT=11019
>     OUT=/dev/null TIMEOUT=2592000 XDR=FALSE"
>
>     All the magic is in that SSH option '-R 11068:localhost:11068' SSH
>     options, which allow the parent R process on your local machine to
>     communicate with the remote worker R process on its own port 11068,
>     and vice versa, the worker R process will communicate with the parent
>     R process as if it was running on MASTER=localhost PORT=11068.
>     Basically, for all that the worker R process' knows, the parent R
>     process runs on the same machine as itself.
>
>     You haven't said what operating system you're running on your local
>     machine, but if it's MS Windows, know that the 'ssh' client that comes
>     with Windows 10 has some bugs in its reverse tunneling.  See
>     ?parallelly::makeClusterPSOCK for lots of details.  You also haven't
>     said what OS the cloud workers run, but I assume it's Linux.
>
>     So, my guesses on your setup is, the above "should work" for you.  For
>     your troubleshooting, you can also set argument outfile=NULL.  Then
>     you'll also see output from the worker R process.  There are
>     additional troubleshooting suggestions in Section 'Failing to set up
>     remote workers' of ?parallelly::makeClusterPSOCK that will help you
>     figure out what the problem is.
>
>     >
>     > I see in README the package states it works with "remote clusters
> without knowing public IP". I think this might be where the confusion is,
> it may mean the remote machine does not have a public IP, but the server
> machine does. I'm in the opposite situation, the server does not have a
> public IP, but the remote does. I'm not sure if this package can handle my
> case, but it looks very powerful and I appreciate your help!
>
>     Thanks. I've updated the text to "remote clusters without knowing
>     [local] public IP".
>
>     /Henrik
>
>     >
>     > Best,
>     > Jiefei
>     >
>     >
>     >
>     >
>     >
>     > On Tue, Jan 19, 2021 at 1:22 AM Henrik Bengtsson <
> henrik.bengtsson using gmail.com> wrote:
>     >>
>     >> If you have SSH access to the workers, then
>     >>
>     >> workers <- c("machine1.example.org", "machine2.example.org")
>     >> cl <- parallelly::makeClusterPSOCK(workers)
>     >>
>     >> should do it.  It does this without admin rights and port
> forwarding.
>     >> See also the README in
> https://cran.r-project.org/package=parallelly.
>     >>
>     >> /Henrik
>     >>
>     >> On Mon, Jan 18, 2021 at 6:45 AM Jiefei Wang <szwjf08 using gmail.com>
> wrote:
>     >> >
>     >> > Hi all,
>     >> >
>     >> > I have a few cloud instances and I want to use them to do parallel
>     >> > computing. I would like to create a socket cluster on my local
> machine to
>     >> > control the remote instances. Here is my network setup:
>     >> >
>     >> > local machine -- NAT -- Internet -- cloud instances
>     >> >
>     >> > In the parallel package, the server needs to call `makeCluster()`
> and
>     >> > listens to the connection from the workers. In my case, the
> server is the
>     >> > local machine and the workers are the cloud instances. However,
> since the
>     >> > local machine is hidden behind the NAT, it does not have a public
> address
>     >> > and the worker cannot connect to it. Therefore, `makeCluster()`
> will never
>     >> > be able to see the connection from the workers and hang forever.
>     >> >
>     >> > One solution for letting the external machine to access the
> device inside
>     >> > the NAT is to use port forwarding. However, this would not work
> for my case
>     >> > as the NAT is set by the network provider(not my home router) so
> I do not
>     >> > have access to the router. As the cloud instances have public
> addresses,
>     >> > I'll wonder if there is any way to build the cluster by letting
> the server
>     >> > connect to the cloud? I have checked `?parallel::makeCluster` and
>     >> > `?snow::makeSOCKcluster` but I found no result. The only
> promising solution
>     >> > I can see now is to use TCP hole punching, but it is quite
> complicated and
>     >> > may not work for every case. Since building a connection from
> local to the
>     >> > remote is super easy, I would like to know if there exists any
> simple
>     >> > solution. I have searched it on Google for a week but find no
> answer. I'll
>     >> > appreciate it if you can provide me any suggestions!
>     >> >
>     >> > Best,
>     >> > Jiefei
>     >> >
>     >> >         [[alternative HTML version deleted]]
>     >> >
>     >> > ______________________________________________
>     >> > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>     >> > https://stat.ethz.ch/mailman/listinfo/r-help
>     >> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
>     >> > and provide commented, minimal, self-contained, reproducible code.
>
>     ______________________________________________
>     R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>     https://stat.ethz.ch/mailman/listinfo/r-help
>     PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
>     and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]