Hi all,

I have a bit of an ambiguous question, i'm still trying to understand the
exact nature of the error.

I am using the NSF Stampede Cluster, using Rmpi and snow to create
embarrassingly parallel structure.

I'm having a strange error that i hope someone can help push me in the
right direction.

I invoke a call using the form:

ibrun  RMPISNOW < nameoffile.R

#Within the script i call (generically)

#create clusters using:
getMPIcluster()

#Export objects to cluster:
clusterExport()

source functions on each cluster using:
clusterEvalQ()

Compute parallel functions using:
clusterApply()

Everything works beautifully the first time. When i go to create the 2nd
parallel loop.
It fails when i try to export the 2nd round of objects. Regardless of the
code, even dummy examples.

Here is the strange part: When i break the identical scripts into two
pieces, everything works fine.

I have 6 loops in total that needed to be parallelized. Could this be a
memory issue? Should i invoke a new MPI cluster for each parallelization in
a script, calling stopCluster() between each? Any and all suggestions
appreciated.

Thanks!
Ben

## head of the error readout below , doesn't appear to be informative

c520-001.stampede.tacc.utexas.edu:mpispawn_8][readline] Unexpected
End-Of-File on file descriptor 7. MPI process died?
[c520-001.stampede.tacc.utexas.edu:mpispawn_8][mtpmi_processops] Error
while reading PMI socket. MPI process died?
[c520-001.stampede.tacc.utexas.edu:mpispawn_8][child_handler] MPI process
(rank: 131, pid: 47649) exited with status 1
[c520-001.stampede.tacc.utexas.edu:mpispawn_8][child_handler] MPI process
(rank: 138, pid: 47656) exited with status 1
[c522-503.stampede.tacc.utexas.edu:mpispawn_10][readline] Unexpected
End-Of-File on file descriptor 13. MPI process died?
[c522-503.stampede.tacc.utexas.edu:mpispawn_10][mtpmi_processops] Error
while reading PMI socket. MPI process died?
[c522-503.stampede.tacc.utexas.edu:mpispawn_10][child_handler] MPI process
(rank: 163, pid: 9717) exited with status 1
[c523-904.stampede.tacc.utexas.edu:mpispawn_11][readline] Unexpected
End-Of-File on file descriptor 9. MPI process died?
[c523-904.stampede.tacc.utexas.edu:mpispawn_11][mtpmi_processops] Error
while reading PMI socket. MPI process died?
[c523-904.stampede.tacc.utexas.edu:mpispawn_11][child_handler] MPI process
(rank: 170, pid: 8190) exited with status 1
[c463-402.stampede.tacc.utexas.edu:mpispawn_1][read_size] Unexpected
End-Of-File on file descriptor 25. MPI process died?
[c463-402.stampede.tacc.utexas.edu:mpispawn_1][read_size] Unexpected
End-Of-File on file descriptor 25. MPI process died?
[c463-402.stampede.tacc.utexas.edu:mpispawn_1][handle_mt_peer] Error while
reading PMI socket. MPI process died?
[c464-203.stampede.tacc.utexas.edu:mpispawn_2][read_size] Unexpected
End-Of-File on file descriptor 23. MPI process died?
[c464-203.stampede.tacc.utexas.edu:mpispawn_2][read_size] Unexpected
End-Of-File on file descriptor 23. MPI process died?
[c464-203.stampede.tacc.utexas.edu:mpispawn_2][handle_mt_peer] Error while
reading PMI socket. MPI process died?
[c516-201.stampede.tacc.utexas.edu:mpispawn_6][error_sighandler] Caught
error: Segmentation fault (signal 11)
bash: line 1: 20180 Segmentation fault      /bin/env
LD_LIBRARY_PATH=/work/01125/yye00/ParallelR/lib64:/home1/apps/intel13/mvapich2/2.0a/lib:/home1/apps/intel13/mvapich2/2.0a/lib/shared:/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/apps/intel/13/composer_xe_2013.2.146/mpirt/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/ipp/../compiler/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/ipp/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/intel64
MPISPAWN_MPIRUN_MPD=0 USE_LINEAR_SSH=1 MPISPAWN_MPIRUN_HOST=
c430-102.stampede.tacc.utexas.edu MPISPAWN_GENERIC_VALUE_40="c430-102"
MPISPAWN_GENERIC_NAME_41=LMOD_DEFAULT_MODULEPATH
MPISPAWN_GENERIC_VALUE_41="/home1/apps/intel13/modulefiles:/opt/apps/xsede/modulefiles:/opt/apps/modulefiles:/opt/modulefiles"
MPISPAWN_GENERIC_NAME_42=ARCHIVE
MPISPAWN_GENERIC_VALUE_42="/home/02443/bw4sz" MPISPAWN_GENERIC_NAME_43=_
MPISPAWN_GENERIC_VALUE_43="/usr/local/bin/build_env.pl"
MPISPAWN_GENERIC_NAME_44=MV2_SUPPORT_DPM MPISPAWN_GENERIC_VALUE_44="1"
MPISPAWN_GENERIC_NAME_45=APPS MPISPAWN_GENERIC_VALUE_45="/opt/apps"
MPISPAWN_GENERIC_NAME_46=SHELL MPISPAWN_GENERIC_VALUE_46="/bin/bash"
MPISPAWN_GENERIC_NAME_47=ENVIRONMENT MPISPAWN_GENERIC_VALUE_47="BATCH"
MPISPAWN_GENERIC_NAME_48=TACC_FAMILY_MPI
MPISPAWN_GENERIC_VALUE_48="mvapich2" MPISPAWN_GENERIC_NAME_49=MV2_IBA_HCA
MPISPAWN_GENERIC_VALUE_49="mlx4_0"
MPISPAWN_GENERIC_NAME_50=SLURM_TACC_NODES MPISPAWN_GENERIC_VALUE_50="1"
MPISPAWN_GENERIC_NAME_51=_ModuleTable003_

Ecology and Evolution
Stony Brook University

http://benweinstein.weebly.com/

	[[alternative HTML version deleted]]

