[R-sig-hpc] R parallel package <-> stopCluster method needed for mpi cluster

[Stefan Theussl]

Dear Paul,

I observe the some behavior with an almost identical setting (OpenMPI 
1.4.3, R 2.14.1, parallel, snow 0.3.8, Rmpi 0.5-9).

This seems to be a NAMESPACE issue in snow. It does not define any S3 
methods, it rather exports all functions "as is". This works just fine 
if snow is attached but fails for S3 methods if not.

I cc'ed the maintainer of snow, Luke Tierney, to let him know.

Best,
Stefan


On 01/11/2012 12:00 PM, r-sig-hpc-request at r-project.org wrote:
> Send R-sig-hpc mailing list submissions to
> 	r-sig-hpc at r-project.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> or, via email, send a message with subject or body 'help' to
> 	r-sig-hpc-request at r-project.org
>
> You can reach the person managing the list at
> 	r-sig-hpc-owner at r-project.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-hpc digest..."
>
>
> Today's Topics:
>
>     1. R parallel package<->  stopCluster method needed for	mpi
>        cluster (Paul Johnson)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 10 Jan 2012 22:21:30 -0600
> From: Paul Johnson<pauljohn32 at gmail.com>
> To: R SIG High Performance Computing<r-sig-hpc at r-project.org>
> Subject: [R-sig-hpc] R parallel package<->  stopCluster method needed
> 	for	mpi cluster
> Message-ID:
> 	<CAErODj9Bzt2ytn_maY0j4LwbzRU0ibtFQAAbu4q0HOodZSwgaQ at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> hi, everybody in r-sig-hpc land.  hopefully this year is bringing you
> good news.
>
> I'm trying to digest the fact that R now has a parallel package.  I'm
> pretty sure I've run into something that is on the TODO list. Briefly,
> the stopCluster method from R parallel doesn't lead back to the
> stopCluster methods from snow to shut down  an MPI cluster.
>
> Now, the long version:
>
> We have an OpenMPI based cluster, I've been working up examples that
> use Rmpi, snow, snowft, and now R parallel.  Here's my parallel
> effort.
>
> http://winstat.quant.ku.edu/svn/hpcexample/trunk/Ex65-R-parallel
>
> Its just a submission script and an example cluster usage.
>
> I'm stuck on (at least) one detail.  If you examine
>
> http://winstat.quant.ku.edu/svn/hpcexample/trunk/Ex65-R-parallel/parallel-hello.R
>
> you see the cluster is created like so
>
> library(parallel)
> ## Causes the snow package's MPI support to be invoked,
> cl<- makeCluster(10, type="MPI")
>
> That actually seems to be fine. the parallel makeCluster method refers
> the work to snow::makeCluster. Works fine.
>
> After that, how to close the thing down?   When I run
>
> stopCluster(cl)
>
> processing crashes, and the error output has this:
>
> Loading required package: Rmpi
> Error in UseMethod("stopCluster") :
>    no applicable method for 'stopCluster' applied to an object of class
> "c('spawnedMPIcluster', 'MPIcluster', 'cluster')"
> Calls:<Anonymous>
> Execution halted
> --------------------------------------------------------------------------
> orterun has exited due to process rank 0 with PID 28684 on
> node compute-2-11.local exiting without calling "finalize". This may
> have caused other processes in the application to be
> terminated by signals sent by orterun (as reported here).
> --------------------------------------------------------------------------
>
>
>
> I *think* I need to call the stopCluster method from snow.  It seems
> like this should have worked:
>
> snow::stopCluster(cl)
>
> or this:
>
> snow::stopCluster.MPIcluster(cl)
>
> but it gives the same error.
>
> But if I load the snow package, all is good.
>
> library(snow)
> stopCluster(cl)
> mpi.quit()
>
> The error log has a warning, but otherwise it appears OK.
>
> Attaching package: 'snow'
>
> The following object(s) are masked from 'package:parallel':
>
>      clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
>      clusterExport, clusterMap, clusterSplit, makeCluster, parApply,
>      parCapply, parLapply, parRapply, parSapply, splitIndices,
>      stopCluster
>
> What do you think.  This is a known issue?  Should I bother somebody
> about it? I mean somebody besides  you, of course.
>