[R-sig-hpc] Using rstream for reproducible parallel computations

[Renaud Gaujoux]

Hi Dirk,

I checked for the possibility of me using snow instead of multicore.
Currently I make use of mutexes (package synchronicity) to modify an 
object in shared memory (package bigmemory), and this mechanism might 
not work with snow and doSNOW. I will investigate this with the the 
packages' authors.

Renaud


On 08/02/2011 17:01, Dirk Eddelbuettel wrote:
> On 8 February 2011 at 16:13, Renaud Gaujoux wrote:
> | Hi Dirk,
> |
> | I had first chosen the multicore package for simplicity (though
> | reproducibility was an issue), but I might switch to snow and doSNOW as
> | you suggest.
> | I would also make multicore computation possible under Windows, am I
> | correct?
>
> Nope. multicore uses facilities not exposed by Windoze.
>
> Revolution helped by writing and even open-sourcing doSMP.
>
> snow and sockets is probably still the best bet for "simple and cross-platform".
>
> Dirk
>
> | Thank you.
> |
> | Renaud
> |
> | On 08/02/2011 16:02, Dirk Eddelbuettel wrote:
> |>  On 8 February 2011 at 15:28, Renaud Gaujoux wrote:
> |>  | Hi,
> |>  |
> |>  | I want to use the package rstream in order to ensure reproducibility of
> |>  | parallel independent computations.
> |>  |
> |>  | To implement the computation I use a foreach loop, which should work
> |>  | with both parallel backends doMC and doMPI. The idea I follow to ensure
> |>  | reproducibility and flexibility is to create all the required random
> |>  | streams before starting loop and pass them to the workers, one by one as
> |>  | these complete their tasks. I saw pieces of code around the web that
> |>  | does this using the package rlecuyer. I like the unified approach of
> |>  | rstream though, so I would like to use this package (that also provides
> |>  | L'Ecuyer's RNGstream generator). Parallel stuff can get tricky with
> |>  | hidden side effects, that's why I'd like to have some comments on the
> |>  | following code.
> |>
> |>  It's tricky and important to get this stuff right: The foreach and do*
> |>  families do not support it out of the box.
> |>
> |>  You could try to make your life easier by switching to the snow package. With
> |>  the doSNOW plugin and by using the MC and/or MPI backends supported by snow
> |>  you would get the snow integration to rlecuyer and rsprng for free.
> |>
> |>  That said, it would be awfully nice if someone could work similar logic into
> |>  do* as a general solution.
> |>
> |>  Dirk
> |>
> |>  | Thank you.
> |>  |
> |>  | Renaud
> |>  |
> |>  | # this function creates n streams, starting with a given seed (<=>
> |>  | rlecuyer::.lec.CreateStream)
> |>  | createStream<- function(n, seed){
> |>  |
> |>  |      # check parameters
> |>  |      if( n<= 0 )
> |>  |          stop("NMF::createStream - invalid value for 'n' [positive value
> |>  | expected]")
> |>  |
> |>  |      s<- new('rstream.mrg32k3a', seed=seed, force.seed=TRUE)
> |>  |      rstream.packed(s)<- TRUE
> |>  |      s<- list(s)
> |>  |      if( n>   1 )
> |>  |          s<- c(s, replicate(n-1,{ s<- new('rstream.mrg32k3a');
> |>  | rstream.packed(s)<- TRUE; s} ))
> |>  |
> |>  |      invisible(s)
> |>  | }
> |>  |
> |>  | library(rstream)
> |>  | library(doMC)
> |>  |
> |>  | registerDoMC()
> |>  | dummy<- foreach(i=1:10, s=createStream(10, 1:6)) %dopar% {
> |>  |      print(s)
> |>  |      # do stuff
> |>  | }
> |>  |
> |>  |
> |>  |
> |>  |
> |>  |
> |>  | ###
> |>  | UNIVERSITY OF CAPE TOWN
> |>  |
> |>  | This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}
> |>  |
> |>  | _______________________________________________
> |>  | R-sig-hpc mailing list
> |>  | R-sig-hpc at r-project.org
> |>  | https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> |>
> |
> |
> |
> | ###
> | UNIVERSITY OF CAPE TOWN
> |
> | This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
> |
> | ###
> |
> |
>

 

###
UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}