[R-sig-hpc] Shared Memory

[Markus Schmidberger]

Hi Nath,

have a look to the help files!

 > ?parallel
 >  p <- parallel(1:10)
 >  q <- parallel(1:20)
 >  collect(list(p, q))

$`8206`
 [1]  1  2  3  4  5  6  7  8  9 10

$`8207`
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20

 >


Best
Markus


Nathan S. Watson-Haigh schrieb:
> Hi Markus,
>
> Thanks for the link to multicore...definitely worth a look! As you've
> already used it, could you provide any examples of expr passed to
> parallel()? I'm using the following as the basis of my code and would
> like to try multicore:
> http://math.acadiau.ca/ACMMaC/Rmpi/task_pull.R
>
> Cheers,
> Nath
>
> Markus Schmidberger wrote:
>   
>> Nathan S. Watson-Haigh schrieb:
>>   
>>     
>>> I'm new to HPC and parallel programming but I've created my own R
>>> package which has a parallel (using Rmpi) and non-parallel
>>> implementation of the same algorithm. It works nicely, but I'm trying to
>>> better understand how/if Rmpi uses shared memory. Does/can Rmpi use
>>> shared memory? For instance, if each slave needs access to the same data
>>> matrix, does Rmpi create a copy of that data for each slave when I do:
>>> mpi.bcast.Robj2slave(myMatrix)
>>>   
>>>     
>>>       
>> Yes
>>   
>>     
>>> I think it does create a copy and I therefore currently pass a subset of
>>> the data matrix to each slave using:
>>> objList <- list(m=m[xMin:nrow(m), xMin:nrow(m)])
>>> mpi.send.Robj(objList, slave_id, 1)
>>>   
>>>     
>>>       
>> Yes, this works
>>   
>>     
>>> myMatrix can be up to 24k x 24k in size. That's > 4Gb of RAM just to
>>> hold it in memory! I suppose I'm wondering if memory requirements are
>>> proportional to the number of slaves requested - if each slave has to
>>> have it's own copy of the data and if I can reduce this requirement by
>>> utilising the shared memory!?
>>>   
>>>     
>>>       
>> Yes, you will get some memory problems. Using Rmpi I think there is no 
>> way to use shared memory.
>> There is a great new package for shared memory sytems: multicore
>> http://cran.r-project.org/web/packages/multicore/index.html
>> I already tested it with 500 cores and there is a great performance 
>> (nearly linear!)
>>
>> Best
>> Markus
>>   
>>     
>>> Cheers,
>>> Nath
>>>
>>> _______________________________________________
>>> R-sig-hpc mailing list
>>> R-sig-hpc at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>   
>>>     
>>>       
>>   
>>     
>
>   


-- 
Dipl.-Tech. Math. Markus Schmidberger

Ludwig-Maximilians-Universität München
IBE - Institut für medizinische Informationsverarbeitung,
Biometrie und Epidemiologie
Marchioninistr. 15, D-81377 Muenchen
URL: http://www.ibe.med.uni-muenchen.de 
Mail: Markus.Schmidberger [at] ibe.med.uni-muenchen.de
Tel: +49 (089) 7095 - 4497