[R] using multiple cpu's - scaling in processing power

[soren wilkening]

Hi All

In the past I have worked with parallel processing in R where a function F
is applied to the elements of a list L. The more cpu cores one has, the
faster the process will run. At the time of launching the process for (F,L)
I will have a certain fixed number of cpu's that I can use. I have tested
this approach and it works fine (i.e. package 'multicore' , using 'mapply' )

But now I am encountering a slightly different situation.

I have a task (F,L) that will run for a *long* time (a week) even if I have
N cpu's processing it. N is the maximum possible number cpus that I can use,
however they will not all be available when I start the process.  
So the problem is that, when I start the process, I may have only
n1 < N cpu's at my disposal. AFter some time, I then have
n1 < n2 < N cpu's at my disposal. After some more time, I have 
n2 < n3 < N cpu's  and finally, at one point, I will have
N cpu's that I can work with. I "scale in" cpu power over the duration of
the process. Why this is the case does not matter. Essentially I cannot
control when new cpu's become available nor how many of them will become
available at that point.

With this I cannot use the standard approach above, where all the cpu cores
have to be available before I launch the process !!

It would help me if someone knew if R offered a solution for this type of
processing. But I would also be happy for pointers to non-R resources that
could deal with this.

Thanks

Soren


-----
http://censix.com
--
View this message in context: http://r.789695.n4.nabble.com/using-multiple-cpu-s-scaling-in-processing-power-tp4634405.html
Sent from the R help mailing list archive at Nabble.com.