[R-sig-hpc] Running jobs on a linux cluster: ways to communicate finished jobs, so fill in with new jobs
pauljohn32 at gmail.com
Tue Aug 10 18:57:50 CEST 2010
On Tue, Aug 10, 2010 at 10:15 AM, Laura S <leslaura at gmail.com> wrote:
> Dear all:
> I would appreciate any help you are willing to offer. I have a simulation
> program that runs serially. However, I would like to run the jobs in such a
> way that when a simulation is finished another job can begin to run. The
> simulations take different amounts of time, so it would be ideal to have a
> way to communicate that jobs are done, and to initiate new jobs. The linux
> cluster IT staff at my institution do not have much documentation or
> experience with running R jobs. I am new to HPC, so my apologizes for this
> potentially very basic inquiry.
> Thank you for your time and consideration,
You don't give us much to go on. What scheduler does your cluster use?
Here's what I'd do. Write a shell script that runs all of the programs
one after the other. Without knowing more about the scheduling scheme
on your cluster, I can't say exactly how I would go about it.
If you have access to a BASH shell, for example, it should be as simple as
R --vanilla -f yourRprogram1.R
R --vanilla -f yourRprogram2.R
and so forth. If you rewrite the first line of your R code to use
Rscript or littler, then you don't even need to bother with the "R
--vanilla -f" part, as each R program will become self aware (and take
over the world, like in Terminator).
If you run exactly the same R program over and over again, make a for loop.
As long as you have the details worked out on each individual run of
the model, the rest of it is not even really a "cluster" problem. You
have to run one after the other.
FYI, I've been uploading practical working examples for our Rocks
Linux cluster using the Torque/OpenPBS scheduling system. Maybe some
will help you.
I think I could work out an example of the sort you describe if you
tell us a bit more about how the separate simulation runs talk to each
Or, I should add, if the runs go one after the other, why don't you
put them all in 1 R program. ??
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas
More information about the R-sig-hpc