[R-sig-dyn-mod] Parallel deSolve on AWS

[John Harrold]

Hey Folks,

So I'm slowly working my way through this. I've decided to try to get
things going using multiple cores on the same computer first, work out
the bugs there then try to move this over to multiple computers. I'm
using foreach/%dopar%  (see the example below). When testing this on
different computers it seems that the environment gets transferred in
an inconsistent way :). I think I've managed to fix this with respect
to packages using the .packages option. However I've got functions
that I wrote myself and I've also run the model compiled. What I've
found that seems to work (more or less consistently) is the following:

n = 100
cores=detectCores()
cl <- makeCluster(cores[1]-1)
registerDoParallel(cl)
xall_p <- foreach(i=1:n,.packages=c("deSolve")) %dopar% {
   source('myfunc.r')
   dyn.load(paste("ode_model", .Platform$dynlib.ext, sep = ""))
   x = run_simulation(args)

   x }
stopCluster(cl)

You can see that within the loop I'm sourcing myfunc.r and I'm loading
the dynamic C library as well. Now my question :).

Is this the most appropriate way to do this? It seems, and this may be
my lack of understanding of the foreach function, that I'm trying to
load things at every iteration. Is this correct?

Thanks
John

Thanks
John

On Mon, Mar 27, 2017 at 7:45 AM, Tim Keitt <tkeitt at utexas.edu> wrote:
> http://www.keittlab.org/
>
> On Sat, Mar 25, 2017 at 9:53 AM, Daniel Kaschek <
> daniel.kaschek at physik.uni-freiburg.de> wrote:
>
>> Dear John,
>>
>> I wrote a function for my dMod package to automatize the scp and ssh
>> processes. You can have a look at the runbg() function in
>>
>> https://github.com/dkaschek/dMod/blob/development/R/remoteComputing.R
>>
>> The approach might not be the most elegant. Furthermore, it is written
>> with Linux on both the origin and target machine in mind. The function
>> is documented and you might just have a look at it. Perhaps you need to
>> modify it to your needs. If you have questions, let me know.
>>
>> Best regards,
>> Daniel
>>
>> On Mon, 2017-03-20 at 13:02 +0530, John Harrold wrote:
>> > Howdy Folks,
>> >
>> > I'm looking at parallelizing some of my code. Mostly I'm thinking
>> > about
>> > spreading the individual simulations in a Monte Carlo across a bunch
>> > of
>>
>
> That sounds like a task for one of the many parallel apply cluster
> functions in R. Create a vector of rng seeds and run one parallel job per
> seed? I would start with the parallel package.
>
> THK
>
>
>> > computers.  I'd like to spread this out across computers in a more
>> > automated fashion as opposed to just adding cores or ssh-ing to
>> > computers
>> > is a somewhat manual fashion.  I was going to get a free account on
>> > AWS to
>> > play around with, so I started looking through different tutorials
>> > online.
>> > There seems to be quite a few options, and I thought it was likely
>> > someone
>> > on this forum has some direct experience with this. And I'd like to
>> > hear
>> > about what folks are using in terms of packages, instances they are
>> > using
>> > on AWS (or other services if thats the case), etc.
>> >
>> > Thanks
>> > John
>> >
>> >       [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > R-sig-dynamic-models mailing list
>> > R-sig-dynamic-models at r-project.org
>> > https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models
>>
>> _______________________________________________
>> R-sig-dynamic-models mailing list
>> R-sig-dynamic-models at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-dynamic-models mailing list
> R-sig-dynamic-models at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models



-- 
-------------------------------------
John M. Harrold
john.m.harrold _at_gmail