[R-sig-hpc] an opinion question

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/02/12 00:56, Joshua Wiley wrote:
> Hi Erin,
> 
> Okay so our question is about parallelization?  There are packages 
> designed for this that handle distributing to the nodes nicely.
> I'm not sure if they are the best but snow and snowfall come to
> mind. Recently I just use the new parallel package but that only
> came out in R 2.14.0.
> 
> Basically you jut setup the 10000 loops or implicit loops and the 
> package distributes to nodes as they become available (or you
> could pressing ate certain nodes to do x number of runs but I don't
> see a compelling reason why not to just send as they are
> available).

OK - Salami taktik: I would NOT paralelize the whole problem, but
rather split it in two parts:

1) 10000 embasassengly parallel simulation of one AR model and store
the result. For this you do not have anything to do, apart from
possibly take care of the random seeds (if you use random numbers)

2) calcluate the measures you want from the saved results (in your
case the mean of phi and the error variance)

By doing this, you leave the whole paralelisatuion to the cluster
software, which usually saves quite some development time.

This, by the way, is my odus operandi: I am running regularly
sensitivity analysis of monte carlo simulation models, and I use array
jobs to simulate these. The calculations for the SA are either done in
another job afterwards on the cluster on a desktop computer.

Hope this helps and don't hesitate to ask further questions - clusters
and paralelization can be quite daunting initially.

Cheers,

Rainer


> 
> Cheers,
> 
> Josh
> 
> On Feb 4, 2012, at 15:41, "Hodgess, Erin" <HodgessE at uhd.edu>
> wrote:
> 
>> Hi Josh:
>> 
>> Thanks for your reply.....the answer to my original question is,
>> of course, "it depends".
>> 
>> Anyhow, suppose I have something like this:
>> 
>> I want to run a simulation of 10,000 AR models and get an
>> average of the phi coefficient and the error variance.
>> 
>> I have 12 nodes.
>> 
>> I was thinking of how best to ship out the reps to minimize the 
>> time.
>> 
>> Does this help, please?
>> 
>> Thanks again, Erin
>> 
>> 
>> Erin M. Hodgess, PhD Associate Professor Department of Computer
>> and Mathematical Sciences University of Houston - Downtown
>> mailto: hodgesse at uhd.edu
>> 
>> 
>> 
>> -----Original Message----- From: Joshua Wiley 
>> [mailto:jwiley.psych at gmail.com] Sent: Sat 2/4/2012 5:34 PM To: 
>> Hodgess, Erin Cc: r-sig-hpc at r-project.org Subject: Re:
>> [R-sig-hpc] an opinion question
>> 
>> Hi Erin,
>> 
>> What do you mean by 'start'?  That is much too vague to get a
>> good reply.  We need information about what you are trying to do
>> and what your situation is.  I typically ssh in, try simple stuff
>> / verify everything works in a small case in interactive mode
>> and then submit a batch job.
>> 
>> Josh
>> 
>> On Sat, Feb 4, 2012 at 3:25 PM, Hodgess, Erin <HodgessE at uhd.edu> 
>> wrote:
>>> Hi everyone!
>>> 
>>> Here is an opinion question please:  when using R on a
>>> cluster, what is the best way to start please?
>>> 
>>> I learned a little bit a couple of years ago but have
>>> forgotten most of it.
>>> 
>>> Thanks, Erin
>>> 
>>> 
>>> Erin M. Hodgess, PhD Associate Professor Department of
>>> Computer and Mathematical Sciences University of Houston -
>>> Downtown mailto: hodgesse at uhd.edu
>>> 
>>> 
>>> [[alternative HTML version deleted]]
>>> 
>>> _______________________________________________ R-sig-hpc
>>> mailing list R-sig-hpc at r-project.org 
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>> 
>> 
>> 
>> -- Joshua Wiley Ph.D. Student, Health Psychology Programmer
>> Analyst II, Statistical Consulting Group University of
>> California, Los Angeles https://joshuawiley.com/
>> 
> 
> [[alternative HTML version deleted]]
> 
> _______________________________________________ R-sig-hpc mailing 
> list R-sig-hpc at r-project.org 
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8vk8wACgkQoYgNqgF2egq7MACghH32dTijXGbJZALrhY2Ye3Xc
aX4An0CTtjHpaf9jjiBRx3gbxO0NRdkz
=4oIJ
-----END PGP SIGNATURE-----