[R-sig-hpc] Rsge combined with mclapply?

[Daniel Sabanés Bové]

Hi Kasper,

OK, I understand. If the cluster is organised like that, you do not have 
to tinker with mclapply anyway.

But on our cluster the whole node is allocated for one job, so I do not 
have this problem. Or, put differently, if I just used the standard Rsge 
function sge.parLapply, I would have the problem that 8 cores are 
allocated for one single R job. This was my motivation for writing an 
sge.parMClapply function.

Thanks for the clarification!

Best
Daniel

-------- Original-Nachricht --------
Betreff: Re: [R-sig-hpc] Rsge combined with mclapply?
Von: Kasper Daniel Hansen <kasperdanielhansen at gmail.com>
An: Daniel Sabanés Bové <daniel.sabanesbove at ifspm.uzh.ch>
Datum: Don 14 Jun 2012 14:27:13 CEST

> On Thu, Jun 14, 2012 at 4:42 AM, Daniel Sabanés Bové
> <daniel.sabanesbove at ifspm.uzh.ch>  wrote:
>> Hi Kasper,
>>
>> thank you for this hint! I was not aware of this possibility. Yes, indeed,
>> then it should be trivial to let multiple jobs run on cores of the same
>> node. In the meantime I have found a solution by tweaking the Rsge package
>> to use mclapply() from the base R-package "parallel". This also works.
>
> This might work in the sense that it produces workable code on your
> system.  But are you sure that you do not over-use your resources.
> For example, on our system, I can qsub a job that SGE thinks takes up
> a single slot (= core, the default) and then still use mclapply over 8
> cores.  In that situation, I have asked for just 1 core, but I am
> really using 8.  This has clear implications for the other users on
> the system as well as resource allocation.  This is because our SGE
> system is not too tightly locked down - it may be possible for a
> sysadmin to prevent this, but I know I can do it on the system I have
> access to.
>
> Kasper
>
>
>
>>
>> Best regards
>> Daniel
>>
>>
>> -------- Original-Nachricht --------
>> Betreff: Re: [R-sig-hpc] Rsge combined with mclapply?
>> Von: Kasper Daniel Hansen<kasperdanielhansen at gmail.com>
>> An: Daniel Sabanés Bové<daniel.sabanesbove at ifspm.uzh.ch>
>> Datum: Mit 13 Jun 2012 23:10:23 CEST
>>
>>
>>> I am not too familiar with Rsge.  But on our SGE cluster, the sysadmin
>>> has a specific type of parallel environment where you can do
>>>    qsub -pe local 8-12
>>> which asks for 8-12 cores on the same node.  My guess is that once
>>> such a parallel resource has been set up, it should be trivial to use
>>> with Rsge.
>>>
>>> Kasper
>>>
>>> On Tue, Jun 5, 2012 at 6:19 AM, Daniel Sabanés Bové
>>> <daniel.sabanesbove at ifspm.uzh.ch>    wrote:
>>>>
>>>> Dear Dan Bode,
>>>>
>>>> first of all I would like to thank you very much for your R-package
>>>> "Rsge".
>>>> It looks very helpful, because I need to submit my parallel jobs on the
>>>> local cluster using SGE.
>>>>
>>>> Actually I have a trivially parallel problem which could be sequentially
>>>> solved by lapply(), and on a single machine with multiple cores by
>>>> mclapply() from the base R-package "parallel". Because the local cluster
>>>> has
>>>> nodes with multiple cores, I would like to combine sge.parLapply (to
>>>> distribute to nodes) with mclapply (not lapply, to distribute to cores in
>>>> each node).
>>>>
>>>> Before doing some modifications to sge.parParApply like introducing
>>>> another
>>>> apply.method for mclapply, I would like to be sure that you have not
>>>> already
>>>> programmed a nice solution for this problem. So, did you already have the
>>>> same problem as I am facing now? Would you think another apply.method for
>>>> sge.parParApply would be a good way to go?
>>>>
>>>> Thank you very much in advance for any hints or suggestions.
>>>>
>>>> Best regards
>>>> Daniel Sabanes
>>>>
>>>> --
>>>> Daniel Sabanés Bové
>>>> University of Zurich
>>>> Institute of Social and Preventive Medicine
>>>> Division of Biostatistics
>>>> Hirschengraben 84
>>>> CH-8001 Zurich
>>>>
>>>> Tel. +41 (0)44 634 49 73
>>>> Fax: +41 (0)44 634 43 86
>>>> URL : http://www.biostat.uzh.ch/aboutus/people/sabanes.html
>>>>
>>>> _______________________________________________
>>>> R-sig-hpc mailing list
>>>> R-sig-hpc at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>
>>>
>>
>