I'm trying to debug a problem where jobs running R code that I submit via qsub are running very slow and causing significant traffic between the head node and the nodes that the jobs are running on.  The more instances of R I run, the more they collectively slow down.

Much of the documentation I've found on running R on an HPC recommends using "module load R".  Trying this led to the same problem as the R install referred to in the module is on the head node.  When I did "lsof -u MyUserName" on the head node, I could see that there were open files related to the R executable and the libraries.  These files were open multiple times, based on the number of running instances of R.

I then installed a new version of R in the /tmp directory of the head node, zipped it all up, and then tried running jobs that would copy this all to the /tmp directory of the nodes, unzip it, and run a version of R locally.  The same symptoms occurred.  This time, running lsof on the head node didn't show any files directly related to the R install, but I do see a reference to my $PBS_O_WORKDIR, and there's a mention of several *.so libraries located in lib directories like /.rootfs/el6.4-1/lib64/.  As I understand, these libraries are on the head node.  The output of lsof also shows that the R code isn't spending its time reading/writing to user-defined files on the head node.

The bash scripts I'm using to submit the jobs via qsub are slightly modified from ones that I used in the past to run java programs.  Instead of "java -jar...", I'm now doing "/tmp/R-3.0.2/bin/Rscript --vanilla MyCode.R $JOB_NUMBER".  The R code loads the necessary libraries on startup, and only reads and writes from the local node.  Processing time when things are working ok (tested on a single computer) are around several hours, and the code runs on a single CPU, so no parallel processing is involved.  In terms of libraries, I'm just using sqldf aside from the default R libraries, and as I understand from the sqldf documentation, the database is stored in memory, so there should be no network traffic associated with queries that are running.

Does anyone have ideas of how I can further debug this?  I'm not sure what else to check for and haven't been able to find a mention on mailing lists of people running into the same problems with network traffic slowing down running instances of R.

Regards,

Chris

	[[alternative HTML version deleted]]

