[R-sig-hpc] parallel() requires snow package to create an MPI cluster?

[Dirk Eddelbuettel]

On 8 November 2014 at 14:11, Bennet Fauber wrote:
| I installed from source R 3.1.1 and Rmpi 0.6-5.  Both pass tests, and
| both work.  In the past, I have always also installed multicore and
| snow, but I was under the impression that parallel superceded those
| two packages.  That seems to be at least partially incorrect...?

>From memory (and use a bucket of salt, I am at one cup of coffee so far
today), it is as follows:

i)   snow and multicore existed, and for a long time provided a decent-enough
     basis for many tasks, and OS, at a possible cost of being spread over packages.

ii)  at some point Brian Ripley wanted a common subset in R base, and wrote
     parallel.  It does make multicore redundant (by coverting that aspect of
     parallel computing), and as I recall multicore has now been archived on
     CRAN. It also took the socket cluster creation from snow -- but nothing else.

     Package parallel has a pretty excellent vignette delineating some of this.

iii) snow still has its uses, particularly for things like the MPI cluster
     creation. That code never went to parallel.
 
| If I want to use the functions from parallel() to create a cluster
| object using MPI, is snow a required package?  I was getting namespace

Never was. You could always use Rmpi only, and then either do it by hand, or
use something like snow (which came first) and doMPI / foreach (which came
later) or any of the other package havng data travel over MPI.  Some do
include a loop unroller etc pp -- but I, like others, got used to the early
snow convenience functions.

| errors while loading functions for snow when I tried it without snow
| installed, and after installing snow, it all just seems to work.

All is well.  Snow is an "enhancement" for Rmpi etc, and that never changed.
 
| Is snow required for
| 
| cl <- makeCluster(mpi.universe.size()-1, "MPI")

makeCluster is a snow function, and the ability to work on MPI did not move
to package parallel AFAIK.

| to work?  I start R from an interactive PBS session with two nodes and
| four processors using
| 
| $ mpirun -np 1 R --no-save --no-restore
| 
| Then I get,
| 
| > mpi.universe.size()
| mpi.universe.size()
| [1] 4
| 
| > sessionInfo()
| sessionInfo()
| R version 3.1.1 (2014-07-10)
| Platform: x86_64-unknown-linux-gnu (64-bit)
| 
| locale:
|  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
|  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
|  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
|  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
|  [9] LC_ADDRESS=C               LC_TELEPHONE=C
| [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
| 
| attached base packages:
| [1] graphics  grDevices datasets  stats     utils     methods   base
| 
| other attached packages:
| [1] Rmpi_0.6-5
| 
| 
| > library(parallel)
| library(parallel)
| 
| > cl <- makeCluster(mpi.universe.size()-1, "MPI")
| cl <- makeCluster(mpi.universe.size()-1, "MPI")
| Error in loadNamespace(name) : there is no package called ‘snow’
| Calls: makeCluster ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
| 
| With snow in the library path, I instead get as the response to the
| last command,
| 
| > cl <- makeCluster(mpi.universe.size()-1, "MPI")
| cl <- makeCluster(mpi.universe.size()-1, "MPI")
|     3 slaves are spawned successfully. 0 failed.
| 
| 
| Is that correct and normal?  For MPI clusters, both Rmpi and snow are
| rerequisites to prallel()'s makeCluster(..., "MPI")?  I was kind of
| under the impression that parallel() was self-contained and superceded
| both multicore and snow.

I fear it's your impression which needs a slight tilt.
 
| Or did I manage to somehow misconfigure or misinstall things?

Nope.

Compute away,  Dirk
 
| Thanks,  -- bennet
| 
| _______________________________________________
| R-sig-hpc mailing list
| R-sig-hpc at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org