[R-sig-hpc] parallel() requires snow package to create an MPI cluster?

[Bennet Fauber]

Thanks, Dirk.

This text,

It builds on the work done for CRAN packages multicore (Urbanek,
2009–present) and snow (Tierney et al., 2003–present) and provides
drop- in replacements for most of the functionality of those
packages...

at the beginning of the vignette, made more most hopeful, and I elided this





On Mon, Nov 10, 2014 at 7:39 AM, Dirk Eddelbuettel <edd at debian.org> wrote:
>
> On 8 November 2014 at 14:11, Bennet Fauber wrote:
> | I installed from source R 3.1.1 and Rmpi 0.6-5.  Both pass tests, and
> | both work.  In the past, I have always also installed multicore and
> | snow, but I was under the impression that parallel superceded those
> | two packages.  That seems to be at least partially incorrect...?
>
> From memory (and use a bucket of salt, I am at one cup of coffee so far
> today), it is as follows:
>
> i)   snow and multicore existed, and for a long time provided a decent-enough
>      basis for many tasks, and OS, at a possible cost of being spread over packages.
>
> ii)  at some point Brian Ripley wanted a common subset in R base, and wrote
>      parallel.  It does make multicore redundant (by coverting that aspect of
>      parallel computing), and as I recall multicore has now been archived on
>      CRAN. It also took the socket cluster creation from snow -- but nothing else.
>
>      Package parallel has a pretty excellent vignette delineating some of this.
>
> iii) snow still has its uses, particularly for things like the MPI cluster
>      creation. That code never went to parallel.
>
> | If I want to use the functions from parallel() to create a cluster
> | object using MPI, is snow a required package?  I was getting namespace
>
> Never was. You could always use Rmpi only, and then either do it by hand, or
> use something like snow (which came first) and doMPI / foreach (which came
> later) or any of the other package havng data travel over MPI.  Some do
> include a loop unroller etc pp -- but I, like others, got used to the early
> snow convenience functions.
>
> | errors while loading functions for snow when I tried it without snow
> | installed, and after installing snow, it all just seems to work.
>
> All is well.  Snow is an "enhancement" for Rmpi etc, and that never changed.
>
> | Is snow required for
> |
> | cl <- makeCluster(mpi.universe.size()-1, "MPI")
>
> makeCluster is a snow function, and the ability to work on MPI did not move
> to package parallel AFAIK.
>
> | to work?  I start R from an interactive PBS session with two nodes and
> | four processors using
> |
> | $ mpirun -np 1 R --no-save --no-restore
> |
> | Then I get,
> |
> | > mpi.universe.size()
> | mpi.universe.size()
> | [1] 4
> |
> | > sessionInfo()
> | sessionInfo()
> | R version 3.1.1 (2014-07-10)
> | Platform: x86_64-unknown-linux-gnu (64-bit)
> |
> | locale:
> |  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> |  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> |  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> |  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
> |  [9] LC_ADDRESS=C               LC_TELEPHONE=C
> | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> |
> | attached base packages:
> | [1] graphics  grDevices datasets  stats     utils     methods   base
> |
> | other attached packages:
> | [1] Rmpi_0.6-5
> |
> |
> | > library(parallel)
> | library(parallel)
> |
> | > cl <- makeCluster(mpi.universe.size()-1, "MPI")
> | cl <- makeCluster(mpi.universe.size()-1, "MPI")
> | Error in loadNamespace(name) : there is no package called ‘snow’
> | Calls: makeCluster ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
> |
> | With snow in the library path, I instead get as the response to the
> | last command,
> |
> | > cl <- makeCluster(mpi.universe.size()-1, "MPI")
> | cl <- makeCluster(mpi.universe.size()-1, "MPI")
> |     3 slaves are spawned successfully. 0 failed.
> |
> |
> | Is that correct and normal?  For MPI clusters, both Rmpi and snow are
> | rerequisites to prallel()'s makeCluster(..., "MPI")?  I was kind of
> | under the impression that parallel() was self-contained and superceded
> | both multicore and snow.
>
> I fear it's your impression which needs a slight tilt.
>
> | Or did I manage to somehow misconfigure or misinstall things?
>
> Nope.
>
> Compute away,  Dirk
>
> | Thanks,  -- bennet
> |
> | _______________________________________________
> | R-sig-hpc mailing list
> | R-sig-hpc at r-project.org
> | https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org