[R-sig-hpc] Running jobs on a linux cluster: ways to communicate finished jobs, so fill in with new jobs

[Paul Johnson]

On Tue, Aug 10, 2010 at 10:15 AM, Laura S <leslaura at gmail.com> wrote:
> Dear all:
>
> I would appreciate any help you are willing to offer. I have a simulation
> program that runs serially. However, I would like to run the jobs in such a
> way that when a simulation is finished another job can begin to run.  The
> simulations take different amounts of time, so it would be ideal to have a
> way to communicate that jobs are done, and to initiate new jobs. The linux
> cluster IT staff at my institution do not have much documentation or
> experience with running R jobs.  I am new to HPC, so my apologizes for this
> potentially very basic inquiry.
>
> Thank you for your time and consideration,
> Laura
>

You don't give us much to go on. What scheduler does your cluster use?
for example.

Here's what I'd do. Write a shell script that runs all of the programs
one after the other.  Without knowing more about the scheduling scheme
on your cluster, I can't say exactly how I would go about it.

If you have access to a BASH shell, for example, it should be as simple as

#!/bin/bash

R --vanilla -f yourRprogram1.R

R --vanilla -f yourRprogram2.R

=====================

and so forth. If you rewrite the first line of your R code to use
Rscript or littler, then you don't even need to bother with the "R
--vanilla -f" part, as each R program will become self aware (and take
over the world, like in Terminator).

If you run exactly the same R program over and over again, make a for loop.

As long as you have the details worked out on each individual run of
the model, the rest of it is not even really a "cluster" problem. You
have to run one after the other.

FYI, I've been uploading practical working examples for our Rocks
Linux cluster using the Torque/OpenPBS scheduling system.  Maybe some
will help you.

http://pj.freefaculty.org/cgi-bin/mw/index.php?title=Cluster:Main

I think I could work out an example of the sort you describe if you
tell us a bit more about how the separate simulation runs talk to each
other.

Or, I should add, if the runs go one after the other, why don't you
put them all in 1 R program.  ??

-- 
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas