[R] Use BatchJobs package to submit Rmpi job

[Feng Li]

Hi,
BatchJobs package is popular to submit R batch jobs on various job scheduler. Now we are facing the question is to submit a Rmpi jobs. So we want to know how we can leverage BatchJobs to do so. After a bit investigation/enhancement, we tested below approach on OpenLava job scheduler. we want to get some inputs from the group to see how this makes sense to you. thanks,

1. R script that submit a Rmpi job using BatchJobs:
library(BatchJobs)
conf = BatchJobs:::getBatchJobsConf()
conf$cluster.functions = makeClusterFunctionsOpenLava("/home/clusteradmin/rmpi.tmpl")      <-- specify job template file that includes "mpirun" to start R job.
reg = makeRegistry(id = "RMPIJobsExample")
f = function(x) {                                                <-- RMPI algorithm function. This is the normal Rmpi R script.
                library("Rmpi")
                ...
                mpi.spawn.Rslaves(nslaves=slaveno)
                                                                ...
                mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size()))
                ...
}
batchMap(reg, f, 1)                                       <-- specify 1 Rmpi job
submitJobs(reg, np=2)                                  <-- specify number of processors to use
showStatus(reg)
waitForJobs(reg)
                                            showStatus(reg)

2. the job template: rmpi.tmpl
                                            #BSUB -q mpi_job_queue
                                            /opt/openlava/bin/openmpi-mpirun -np 1 R CMD BATCH "<%= rscript %>"


Please let us know your comments,

Thanks,
Feng Li
Teraproc Inc.


	[[alternative HTML version deleted]]