[R-sig-hpc] Sun Grid Engine (SGE) and Rsge package

[Rainer M Krug]

On Fri, Dec 5, 2008 at 7:42 PM, Dan Bode <dbode at univaud.com> wrote:
> I wrote the package for a customer that is just starting to build use cases around it.
>
> I will be updating to the latest and probably last planned version for now (.6) in the next week or so.
>
> The version available should be enough to get you started, but .6 contains some code changes that increase performance dramastically, it also provides more flexibility with the parallell distribution of local and global environments.

Sounds great - the performance increase will probably come from a
streamlined submitting (writing files,...) and getting results
(reading files)? Looking forward to that.
I actualy managed to use it in a few examples, and apart from the
overhang due to submitting, it really looks great.

>
> I would be more than willing to assist you with getting started with the package.

Great - thanks.

>
> I am assumming that you already have an SGE cluster installed?

Yes - I am just a normal user on the cluster and have (luckily?)
nothing to do with it's admin.

>
> We are working on open source packages around SGE for provisioning and configuration management with additional GLOBUS features.  http://www.grid.org/downloads
>
> all of the functions are documented here
>
> http://cran.r-project.org/web/packages/Rsge/Rsge.pdf, (although I am sure that you already found this)

Yes - but I was missing a simple example. I found one in the "test"
directory in the source package.

>
> In particular you should be interested in sge.parApply or sge.submit, depending on your use cases.

True.

>
> sge.apply(X, MARGIN, FUN, ..., njobs, packages, savelist)
>
> those are prob all of the params that you may need to get started.
>
> This will split the data structure defined by X into njobs data structures, FUN is applied against these data structures on the remote execution hosts, then the results will be merged togehther using the join.method.

OK - sounds straight forward.

>
> packages and savelist are both a vector of strings
> savelist is used to add objects from the global environment (this will change in the next version)

OK.

> packages lists the packages that need to be loaded in the remote execution environments

Nice.

>
> and X, MARGIN, FUN, and ... should be the same as regular apply.
>
> I hope this is enough to get you started.

Yes - thanks a lot for the additional info.

>
> Also, you are the first person to contact me, how did you find the package?

I am having tremendous problems getting parallel processing in R on a
cluster running opnmpi and SGE (mpich is also available). I tried to
use snow and Rmpi, but all the processes were spawned on the same node
on which the master session of R was running. Therefore I googled for
sge and R and found your package. It looks really promising.

Do you have any ideas, if I can use snow (through Rmpi) on a SGE -
openmpi cluster? if yes, how? I have to use qsub to start the job.

>
> Enjoy the package!

I already do,

Thanks,

Rainer

>
> Dan Bode
>
>
>
> -----Eredeti üzenet-----
> Feladó: Rainer M Krug [mailto:r.m.krug at gmail.com]
> Küldve: 12/4/2008, Cs 6:49 du.
> Címzett: r-sig-hpc at stat.math.ethz.ch
> Másolatot kap: Dan Bode
> Tárgy: Sun Grid Engine (SGE) and Rsge package
>
> Hi
>
> I just found the Rsge package which seems to suit my need to do
> parallel processing on a cluster with SGE. I did not manage to use
> Rmpi or snow (which would have been nice).
>
> My only problem is, that I can't find any documentation on how to use
> Rsge (apart from the one included in the source package) - it seems to
> utilise several options which might be useful to set.
> Does anybody has any experience with this package or can show me a way
> of using snow or Rmpi in an environment where I have to use SGE (qsub
> ...)?
>
> Thanks
>
> Rainer
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Faculty of Science
> Natural Sciences Building
> Private Bag X1
> University of Stellenbosch
> Matieland 7602
> South Africa
>
>
> ---------------------------------------------------------------------
>
> Notice from Univa UD Postmaster:
>
>
> This email message is for the sole use of the intended...{{dropped:25}}