[R-sig-hpc] Rsge combined with mclapply?

[Kasper Daniel Hansen]

On Thu, Jun 14, 2012 at 4:42 AM, Daniel Sabanés Bové
<daniel.sabanesbove at ifspm.uzh.ch> wrote:
> Hi Kasper,
>
> thank you for this hint! I was not aware of this possibility. Yes, indeed,
> then it should be trivial to let multiple jobs run on cores of the same
> node. In the meantime I have found a solution by tweaking the Rsge package
> to use mclapply() from the base R-package "parallel". This also works.

This might work in the sense that it produces workable code on your
system.  But are you sure that you do not over-use your resources.
For example, on our system, I can qsub a job that SGE thinks takes up
a single slot (= core, the default) and then still use mclapply over 8
cores.  In that situation, I have asked for just 1 core, but I am
really using 8.  This has clear implications for the other users on
the system as well as resource allocation.  This is because our SGE
system is not too tightly locked down - it may be possible for a
sysadmin to prevent this, but I know I can do it on the system I have
access to.

Kasper



>
> Best regards
> Daniel
>
>
> -------- Original-Nachricht --------
> Betreff: Re: [R-sig-hpc] Rsge combined with mclapply?
> Von: Kasper Daniel Hansen <kasperdanielhansen at gmail.com>
> An: Daniel Sabanés Bové <daniel.sabanesbove at ifspm.uzh.ch>
> Datum: Mit 13 Jun 2012 23:10:23 CEST
>
>
>> I am not too familiar with Rsge.  But on our SGE cluster, the sysadmin
>> has a specific type of parallel environment where you can do
>>   qsub -pe local 8-12
>> which asks for 8-12 cores on the same node.  My guess is that once
>> such a parallel resource has been set up, it should be trivial to use
>> with Rsge.
>>
>> Kasper
>>
>> On Tue, Jun 5, 2012 at 6:19 AM, Daniel Sabanés Bové
>> <daniel.sabanesbove at ifspm.uzh.ch>  wrote:
>>>
>>> Dear Dan Bode,
>>>
>>> first of all I would like to thank you very much for your R-package
>>> "Rsge".
>>> It looks very helpful, because I need to submit my parallel jobs on the
>>> local cluster using SGE.
>>>
>>> Actually I have a trivially parallel problem which could be sequentially
>>> solved by lapply(), and on a single machine with multiple cores by
>>> mclapply() from the base R-package "parallel". Because the local cluster
>>> has
>>> nodes with multiple cores, I would like to combine sge.parLapply (to
>>> distribute to nodes) with mclapply (not lapply, to distribute to cores in
>>> each node).
>>>
>>> Before doing some modifications to sge.parParApply like introducing
>>> another
>>> apply.method for mclapply, I would like to be sure that you have not
>>> already
>>> programmed a nice solution for this problem. So, did you already have the
>>> same problem as I am facing now? Would you think another apply.method for
>>> sge.parParApply would be a good way to go?
>>>
>>> Thank you very much in advance for any hints or suggestions.
>>>
>>> Best regards
>>> Daniel Sabanes
>>>
>>> --
>>> Daniel Sabanés Bové
>>> University of Zurich
>>> Institute of Social and Preventive Medicine
>>> Division of Biostatistics
>>> Hirschengraben 84
>>> CH-8001 Zurich
>>>
>>> Tel. +41 (0)44 634 49 73
>>> Fax: +41 (0)44 634 43 86
>>> URL : http://www.biostat.uzh.ch/aboutus/people/sabanes.html
>>>
>>> _______________________________________________
>>> R-sig-hpc mailing list
>>> R-sig-hpc at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>
>>
>