[R-sig-hpc] New R packages BatchJobs and BatchExperiments for computing on HPCs

[Simon Urbanek]

Michael,

would you like to give Rserve clusters a shot? They offer state, pre-loading, parallelization and authentication (in particular the new version including multi-user) with very efficient R dispatch - much better than any generic systems. I was going to add something like a parallel/show class for them, but what you describe goes even further ..

Cheers,
Simon


On May 30, 2012, at 9:22 AM, Michel Lang wrote:

> The packages BatchJobs and BatchExperiments are now available on CRAN.
> 
> We believe some subscribers of this list may find the packages useful.
> Here is an excerpt of the abstract of our publication:
> 
> The package BatchJobs implements the basic objects and procedures to
> control a batch cluster within R. It is structured around cluster
> versions of the well-known higher order functions Map, Reduce and
> Filter from functional programming. An important feature is that the
> state of computation is persistently available in a database. The user
> can query the status of jobs and then continue working with a desired
> subset.
> The second package, BatchExperiments, is tailored for the still very
> general scenario of analyzing arbitrary algorithms on problem
> instances. It extends BatchJobs by letting the user define an array of
> jobs of the kind "apply algorithm A to problem instance P and store
> results". It is possible to associate statistical designs with
> parameters of algorithms and problems and therefore systematically
> study their influence on the results.
> 
> We currently support the following batch systems:
> * TORQUE/PBS based systems
> * LSF
> * Sun/Oracle Grid Engine (SGE)
> * Makeshift SSH clusters to operate on loosely connected boxes with a
> shared common file system
> * Local interactive execution, mainly for prototyping
> * Local multicore execution
> 
> Note that the adoption of other batch systems is in principle easily
> achieved, we simply had no access to them. If you want to see support
> for your batch system, please send us a mail.
> 
> Fore more information, see
> * Project site: <http://batchjobs.googlecode.com>
> * Technical report on the packages, also submitted to JSS:
> <http://sfb876.tu-dortmund.de/PublicPublicationFiles/bischl_etal_2012a.pdf>
> * BatchJobs CRAN page:
> <http://cran.r-project.org/web/packages/BatchJobs/index.html>
> * BatchExperiments CRAN page:
> <http://cran.r-project.org/web/packages/BatchExperiments/index.html>
> 
> Regards,
> Michel
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc