[R] torque/psb & snow library

[Luke Tierney]

You may or may not see speed substantial improvements using Rmpi with
(or without) snow but you might -- depends on how the cluster is
configured, what version of MPI, etc.

In my experience using snow with sockets, pvm, or LAM/MPI for typical
computations running an a multi-core machine vs remote machines
doesn't make much difference.  IF it does then you may have too much
communicationrelative to computation and that mey need to be resolved
no matter who you do things if you are to see good speedups.

The development version of snow on my website

     http://www.stat.uiowa.edu/~luke/R/cluster/snow_0.3-4.tar.gz

includes some experimental visualization tools that may help see what
is going on.  Do

    v <- snow.time(... your computation ...)
    plot(v)

and you get a Gantt chart of the computation.

Best,

luke


On Wed, 22 Oct 2008, jgarcia at ija.csic.es wrote:

> Thanks you! I'll look at this new list!
> Well, I'm not the system administrator, and my installation of Rmpi and/or
> pvm libraries for R crashes. As this is the first time I parallelize some
> jobs, snow appealed as a first approach because 1) it compiled correctly
> and 2) the use of the library is very easy.
>
> As you tell me MPI is faster, I'll retry Rmpi installation.
> B. Regards,
> Javier
> ......
>
>> Hi Javier,
>>
>> there is a new mailing list for R and HPC: r-sig-hpc at r-project.org
>> This is probably a better list for your question.
>>
>> I never tried torque with socket. We use torque and mpi or pvm (and R)
>> and it is working very well.
>> Why do you use socket as communication layer?
>> MPI was especially developed for communication between nodes in a
>> computer cluster. And there you can specify which nodes and the number
>> of processors per node you want use. Therfore I would strongly recommend
>> to use MPI. This will be faster in every condition!
>>
>> Best
>> Markus
>>
>>
>> jgarcia at ija.csic.es wrote:
>>> Hello all;
>>> I'm trying to execute parallel jobs trough library snow on a cluster
>>> built
>>> through torque/PSB. I'm succesfully obtaining the cluster with:
>>>
>>>
>>>> system("cat $PBS_NODEFILE > cluster.txt")
>>>> mycluster <- scan(file="cluster.txt",what="character")
>>>> cl <- makeSOCKcluster(mycluster)
>>>
>>> The only problem, at the moment, is that if I use processors in nodes
>>> other that the one in which I'm running R, the communication is
>>> extremely
>>> slow. If all processor are in the "master" computer there not seems ti
>>> be
>>> any problem.
>>>
>>> Has anyone got any experience with this and any advice? Perhaps snow() s
>>> not adequate for this kind of clusters?
>>>
>>> Thanks and best regards,
>>> Javier
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>>
>> --
>> Dipl.-Tech. Math. Markus Schmidberger
>>
>> Ludwig-Maximilians-Universität München
>> IBE - Institut für medizinische Informationsverarbeitung,
>> Biometrie und Epidemiologie
>> Marchioninistr. 15, D-81377 Muenchen
>> URL: http://www.ibe.med.uni-muenchen.de
>> Mail: Markus.Schmidberger [at] ibe.med.uni-muenchen.de
>> Tel: +49 (089) 7095 - 4599
>>
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Luke Tierney
Chair, Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:      luke at stat.uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu