Hi all,

I'm having an odd problem where my MPI cluster is not distributing to the
correct number of slaves. I am using the stampede  xsede cluster (
https://www.xsede.org/resources/overview)

I am submitting a slurm script that looks like:

*
#!/bin/bash

#SBATCH -J DimDiv
#SBATCH -o /home1/02443/bw4sz/DimDiv/DimDivError.out
#SBATCH -p normal
#SBATCH -t 24:00:00
#SBATCH -A TG-DEB130023
#SBATCH -n 400 # Total cores

#SBATCH --mail-user=benweinstein2010@gmail.com
#SBATCH --mail-type=begin
#SBATCH --mail-type=end

module load R_mkl

#Get source file to install RMPI
source /work/01125/yye00/ParallelR/sourceme.sh

#echo "Dim Div 1"

export MV2_SUPPORT_DPM=1

ibrun RMPISNOW < /home1/02443/bw4sz/DimDiv/DimDivClusterAll.R >
/home1/02443/bw4sz/DimDiv/DimDiv.out*

Note that i specify 400 cores using the -n argument, but do not specify on
which nodes they need to be run, since the run is entirely embarrassingly
parallel.

i can see from my squeue call that i successfully received 25 nodes (16
proc per node)

Within the script I create a MPI cluster and ask it to report which nodes i
have using
* cluster <- getMPIcluster()

# Print the hostname for each cluster member
 sayhello <- function()
+ {
+   info <- Sys.info()[c("nodename", "machine")]
+   paste("Hello from", info[1], "with CPU type", info[2])
+ }

 names <- clusterCall(cluster, sayhello)
print(unlist(names))*

I can see 399 responses (# of nodes - 1 for master?) in the form of
:[1] "Hello from c415-603.stampede.tacc.utexas.edu with CPU type x86_64"
[2] "Hello from c415-603.stampede.tacc.utexas.edu with CPU type x86_64


I then export variables, and perform a function, that is currently working
successfully, by specifying

clusterApply(cluster,1:500,Phylo_TraitN)

Which means for the MPI cluster created (which has 399 slaves) perform the
function Phylo_TraitN 500 times, using a 1 for the variable on the first
run, 2 on the 2nd run ... 500 on the 500th run.

At the end of each loop i save my results to file with an iteration count,
ie which position in 1:500.
Each function takes about 30min to run (250 computer hours)

I'm finding instead of distributing to 399 slaves, the function is
distributing to 39 slaves ( 400/10 - 1 for the master). Every 30min, 39
outputs are saved into the folder, not 399. I have waited for 3 rounds, 39
then 78 then 117 results. It cannot be an error of output or overwrite, if
there are only 500 total, in two rounds it should have completed.

What might be causing this behavior, do i not properly understand the -n
argument in the slurm script? I have some more load balancing tests, and
i'm trying to create a reproducible example that might further highlight
these issues.

I appreciate your insight,

Ben Weinstein
-- 
Ben Weinstein
PhD Candidate
Ecology and Evolution
Stony Brook University

http://benweinstein.weebly.com/

	[[alternative HTML version deleted]]

