[R] cluster-lite

[Cedrick Johnson]

perhaps a long shot but maybe the package 'nws' could handle some of it. 
I've recently begun looking into something similar lately as well.

-c

ivo welch wrote:
> I am about to write a "cluster-lite" R solution for myself.  I wanted to
> know whether it already exists.  If not, I will probably write up how I do
> this, and I will make the code available.
>
> Background: we have various linux and OSX systems, which are networked, but
> not set up as a cluster.  I have no one here to set up a cluster, so I need
> a "hack" that facilitates parallel programming on standard networked
> machines.   I have accounts on all the machines, ssh access (of course
> password-less), and networked file directory access.
>
> what I am ultimately trying to accomplish is built around a "simple"
> function, that my master program would invoke:
>
> master.R:
>    multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40 file2.out", "ssh
> anotherhost R slv.R 41 80 file3.out"), announce=300)
>
> multisystem() should submit all jobs simultaneously and continue only after
> all are completed.  it should also tell me every 300 seconds what jobs it is
> still waiting for, and which have completed.
>
> with basically no logic in the cluster, my master and slv programs have to
> make up for it.  master.R must have the smarts to know where it can spawn
> jobs and how big each job should be.  slv.R must have the smarts to place
> its outputs into the marked files on the networked file directory.  master.R
> needs the smarts to combine the outputs of all jobs, and to resubmit jobs
> that did not complete successfully.  again, the main reason for doing all of
> this is to avoid setting up a cluster across OSX and linux system, and still
> to make parallel processing across linux/osx as easy as possible.  I don't
> think it gets much simpler than this.
>
> now, I know how to write the multisystem() in perl, but not in R.  so, if I
> roll it myself, I will probably rely on a mixed R/perl system here.  This is
> not desirable, but it is the only way I know how to do this.  if something
> like multisystem() already exists in R native, please let me know and save
> me from reinventing the wheel.  if it does not, some perl/R combo for this
> soon will.
>
> regards,
>
> /iaw
>
>
>