[R] Resources for utilizing multiple processors

[Martin Morgan]

On 06/08/2011 08:54 PM, Robin Jeffries wrote:
> Hello,
>
> I know of some various methods out there to utilize multiple processors but
> am not sure what the best solution would be. First some things to note:
> I'm running dependent simulations, so direct parallel coding is out
> (multicore, doSnow, etc).
> I'm on Windows, and don't know C. I don't plan on learning C or any of the
> *nix languages.
>
> My main concern deals with Multiple analyses on large data sets. By large I
> mean that when I'm done running 2 simulations R is using ~3G of RAM, the
> remaining ~3G is chewed up when I try to create the Gelman-Rubin statistic
> to compare the two resulting samples, grinding the process to a halt. I'd
> like to have separate cores simultaneously run each analysis. That will save
> on time and I'll have to ponder the BGR calculation problem another way. Can
> R temporarily use HD space to write calculations to instead of RAM?
>
> The second concern boils down to whether or not there is a way to split up
> dependent simulations. For example at iteration (t) I feed a(t-2) into FUN1
> to generate a(t), then feed a(t), b(t-1) and c(t-1) into FUN2 to simulate
> b(t) and c(t). I'd love to have one core run FUN1 and another run FUN2, and
> better yet, a third to run all the pre-and post- processing tidbits!

If FUN1 is independent of b() and c(), perhaps the example at the bottom 
of ?socketConnection points in a useful direction -- start one R to 
calculate a(t) and send the result to a socket connection, then move on 
to a(t+1). Start a second R to read from the socket connection and do 
FUN2(t), . You'll be able to overlap the computations and double 
throughput; the 'pipeline' could be extended with pre- and 
post-processing workers, too, though one would want to watch out for the 
complexity of managing this.

Martin
>
>
> So if anyone has any suggestions as to a direction I can look into, it would
> be appreciated.
>
>
> Robin Jeffries
> MS, DrPH Candidate
> Department of Biostatistics
> UCLA
> 530-633-STAT(7828)
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
Computational Biology
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109

Location: M1-B861
Telephone: 206 667-2793