[R-sig-hpc] parallel() requires snow package to create an MPI cluster?
Dirk Eddelbuettel
edd at debian.org
Mon Nov 10 13:39:01 CET 2014
On 8 November 2014 at 14:11, Bennet Fauber wrote:
| I installed from source R 3.1.1 and Rmpi 0.6-5. Both pass tests, and
| both work. In the past, I have always also installed multicore and
| snow, but I was under the impression that parallel superceded those
| two packages. That seems to be at least partially incorrect...?
>From memory (and use a bucket of salt, I am at one cup of coffee so far
today), it is as follows:
i) snow and multicore existed, and for a long time provided a decent-enough
basis for many tasks, and OS, at a possible cost of being spread over packages.
ii) at some point Brian Ripley wanted a common subset in R base, and wrote
parallel. It does make multicore redundant (by coverting that aspect of
parallel computing), and as I recall multicore has now been archived on
CRAN. It also took the socket cluster creation from snow -- but nothing else.
Package parallel has a pretty excellent vignette delineating some of this.
iii) snow still has its uses, particularly for things like the MPI cluster
creation. That code never went to parallel.
| If I want to use the functions from parallel() to create a cluster
| object using MPI, is snow a required package? I was getting namespace
Never was. You could always use Rmpi only, and then either do it by hand, or
use something like snow (which came first) and doMPI / foreach (which came
later) or any of the other package havng data travel over MPI. Some do
include a loop unroller etc pp -- but I, like others, got used to the early
snow convenience functions.
| errors while loading functions for snow when I tried it without snow
| installed, and after installing snow, it all just seems to work.
All is well. Snow is an "enhancement" for Rmpi etc, and that never changed.
| Is snow required for
|
| cl <- makeCluster(mpi.universe.size()-1, "MPI")
makeCluster is a snow function, and the ability to work on MPI did not move
to package parallel AFAIK.
| to work? I start R from an interactive PBS session with two nodes and
| four processors using
|
| $ mpirun -np 1 R --no-save --no-restore
|
| Then I get,
|
| > mpi.universe.size()
| mpi.universe.size()
| [1] 4
|
| > sessionInfo()
| sessionInfo()
| R version 3.1.1 (2014-07-10)
| Platform: x86_64-unknown-linux-gnu (64-bit)
|
| locale:
| [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
| [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
| [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
| [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
| [9] LC_ADDRESS=C LC_TELEPHONE=C
| [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
|
| attached base packages:
| [1] graphics grDevices datasets stats utils methods base
|
| other attached packages:
| [1] Rmpi_0.6-5
|
|
| > library(parallel)
| library(parallel)
|
| > cl <- makeCluster(mpi.universe.size()-1, "MPI")
| cl <- makeCluster(mpi.universe.size()-1, "MPI")
| Error in loadNamespace(name) : there is no package called ‘snow’
| Calls: makeCluster ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
|
| With snow in the library path, I instead get as the response to the
| last command,
|
| > cl <- makeCluster(mpi.universe.size()-1, "MPI")
| cl <- makeCluster(mpi.universe.size()-1, "MPI")
| 3 slaves are spawned successfully. 0 failed.
|
|
| Is that correct and normal? For MPI clusters, both Rmpi and snow are
| rerequisites to prallel()'s makeCluster(..., "MPI")? I was kind of
| under the impression that parallel() was self-contained and superceded
| both multicore and snow.
I fear it's your impression which needs a slight tilt.
| Or did I manage to somehow misconfigure or misinstall things?
Nope.
Compute away, Dirk
| Thanks, -- bennet
|
| _______________________________________________
| R-sig-hpc mailing list
| R-sig-hpc at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
--
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
More information about the R-sig-hpc
mailing list