[R-sig-hpc] Rmpi::mpi.spawn.Rslaves() stalls

[Henrik Bengtsson]

ISSUE:
I'm trying to set up Rmpi on Ubuntu 16.04, but I fail to get it
running.  Whenever I launch `R --vanilla` and do:

    > library("Rmpi")
    > mpi.spawn.Rslaves(nslaves = 2L)
2 slaves are spawned successfully. 0 failed.

it just stalls there.  I have to `SIG-QUIT` the R process there.


TROUBLESHOOTING:
I'm a bit puzzled on how to troubleshoot this one, especially from
R/Rmpi itself.  After calling `mpi.spawn.Rslaves()` I see the
following process tree:

    $ pstree -Aa 24442
    R --vanilla
     |-orted --hnp --set-sid --report-uri 13 --singleton-died-pipe 14
-mca state_novm_select 1 -mca ess_base_jobid 761266176
     |   |-Rslaves.sh /usr/lib/R/site-library/Rmpi/Rslaves.sh
/usr/lib/R/site-library/Rmpi/slavedaemon.R 24442+1 needlog /usr/lib/R
     |   |   `-R --no-init-file --slave --no-save -f
/usr/lib/R/site-library/Rmpi/slavedaemon.R
     |   |      `-{R}
     |   |-Rslaves.sh /usr/lib/R/site-library/Rmpi/Rslaves.sh
/usr/lib/R/site-library/Rmpi/slavedaemon.R 24442+1 needlog /usr/lib/R
     |   |   `-R --no-init-file --slave --no-save -f
/usr/lib/R/site-library/Rmpi/slavedaemon.R
     |   |      `-{R}
     |   `-{orted}
     `-{R}

and using `debug(Rmpi::mpi.spawn.Rslaves)` I can see that it appears
to stall in the `mpi.intercomm.merge(intercomm, 0, comm)` call;

    debug: if (realns == 0) stop("It seems no single slave spawned.")
    Browse[2]>
       debug: if (mpi.intercomm.merge(intercomm, 0, comm)) {
       mpi.comm.set.errhandler(comm)
       mpi.comm.disconnect(intercomm)
       mpi.bcast(nonblock, type = 1, rank = 0, comm = comm)
       mpi.bcast(sleep, type = 2, rank = 0, comm = comm)
       if (!quiet)
           slave.hostinfo(comm)
    } else stop("Fail to merge the comm for master and slaves.")

Other than that, I can confirm is that basic `mpicc` and `mpirun` work
as expected using one of the classic hello-world MPI examples in C:

    $ mpicc -o hello hello.c
    $ mpirun -n 2 hello
    Hello: rank 0, world: 2
    Hello: rank 1, world: 2

with

    mpirun --version
    mpirun (Open MPI) 1.10.2

    Report bugs to http://www.open-mpi.org/community/help/

Honestly, I'm not even sure that the above uses the same libraries as
`Rmpi` uses.

Also, I did notice that the Rmpi (Debian distro as frequently
recommended by Dirk E) was built for R 3.2.3 (see below) - is that
intended/outdated?

Any suggestions on how to proceed here?  Does anyone know of a Ubuntu
Docker container with a functional R and Rmpi that I can compare to?


SESSION INFORMATION:

I've installed R and Rmpi via the Debian distro;

    sudo apt install r-base
    sudo apt install r-cran-rmpi

The above problem occurred with R 3.4.1 as well as after updating to R
3.4.2.  I've also reinstalled site-wide Rmpi using `sudo apt install
--reinstall r-cran-rmpi`.

With the above I have:

    $ R --vanilla

    > library("Rmpi")
    > sessionInfo()
    R version 3.4.2 (2017-09-28)
    Platform: x86_64-pc-linux-gnu (64-bit)
    Running under: Ubuntu 16.04.3 LTS

    Matrix products: default
    BLAS: /usr/lib/atlas-base/atlas/libblas.so.3.0
    LAPACK: /usr/lib/atlas-base/atlas/liblapack.so.3.0

    locale:
     [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
     [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
     [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
     [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
     [9] LC_ADDRESS=C               LC_TELEPHONE=C
    [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base

    other attached packages:
    [1] Rmpi_0.6-5

    loaded via a namespace (and not attached):
    [1] compiler_3.4.2

    > packageDescription("Rmpi")
    Package: Rmpi
    Version: 0.6-5
    Date: 2014-05-20
    Title: Interface (Wrapper) to MPI (Message-Passing Interface)
    Author: Hao Yu
    Maintainer: Hao Yu <hyu at stats.uwo.ca>
    Depends: R (>= 2.15.1)
    Suggests: parallel
    Description: Rmpi provides an interface (wrapper) to MPI APIs. It also
            provides interactive R slave environment.
    License: GPL (>= 2)
    URL: http://www.stats.uwo.ca/faculty/yu/Rmpi
    Packaged: 2014-05-20 14:19:14 UTC; hyu
    NeedsCompilation: yes
    Repository: CRAN
    Date/Publication: 2014-05-21 08:20:01
    Built: R 3.2.3; x86_64-pc-linux-gnu; "Sat, 06 Feb 2016 19:52:50 +0000";
            unix

    -- File: /usr/lib/R/site-library/Rmpi/Meta/package.rds

I've also made sure that I have no `~/.Renviron` and `~/.Rprofile` file.


Thank you

Henrik