[R-sig-hpc] R, Nomad, HTCondor, etc... and future
georgeo@t @end|ng |rom gm@||@com
Fri May 22 03:19:21 CEST 2020
I live in a large HPC world, where distributed computing is inherently batch, so take my advice with that perspective. Large systems are mostly incompatible with the interactive concept of "backend" and instead support SPMD-style batch programming. SPMD is mostly MPI+X, meaning that the distributed aspect is handled by MPI and within node aspects can vary among several options including MPI, fork, OpenMP, and OpenACC. But even on a medium slurm-managed cluster (possibly in a corporate environment), for R I would recommend a combination of pbdR.org distributed packages and parallel package's mclapply components for within node parallelism.
From: R-sig-hpc <r-sig-hpc-bounces using r-project.org> on behalf of David Bellot <david.bellot using gmail.com>
Date: Thursday, May 21, 2020 at 8:26 PM
To: <r-sig-hpc using r-project.org>
Subject: [R-sig-hpc] R, Nomad, HTCondor, etc... and future
Hi R HPC,
I was wondering if anyone has ever used Nomad from Hashicorp as a backend
engine to run R distributed code.
Moreover, if you use it with *future, *I'd love to hear about your
experience and if you published a package for it, I'd love to use it too.
If not, which other distribution engine do you use? (apart from those
supported in *batchtools*)?
HTCondor, DockerSwarm, BOINC, etc... ?
I didn't make a final decision yet on which engine to use, but it has to be
versatile enough (I know it's not a lof of information but think about
"corporate environment" with various needs. My R need is just one among
Thanks for your help.
[[alternative HTML version deleted]]
R-sig-hpc mailing list
R-sig-hpc using r-project.org
More information about the R-sig-hpc