[R] Parallel processing question ...

[Steve Lianoglou]

Hi,

On Tue, Feb 8, 2011 at 6:18 PM, Robinson, David G <drobin at sandia.gov> wrote:
> I am experimenting with parallel processing using foreach and seem to be
> missing something fundamental. Cool stuff. I've gone through the list and
> seen a couple of closely related issues, but nothing I've tried seems to
> work.
>
> I know that the results from foreach are combined, but what if there is more
> than one variable within the loop?  Below is a snippet (non-functioning) of
> code that I hope provides enough insight into what I am trying to do.  The
> commented out lines are what I would be doing (successfully) if I wasn't
> trying to implement the %dopar% . The goal is to do statistics on the
> sequence of lambda vectors that were originally accumulated in the matrix
> lambdas using cbind.
>
> Thanks in advance for any suggestions,
> Dave
>
> ---------------snip
> update_N <- function(sets, indexes, lam) {
>    n <- length(indexes)-1    # count of events
>    N <- rep(0, K) # count of failures per node
>    for (i in 1:n) {
>        nodes <- as.numeric(sets[indexes[i]:(indexes[i+1]-1)])
>        node <- resample(nodes, 1, prob=lam[nodes]/sum(lam[nodes]))
>        N[node] = N[node] + 1
>    }
>    N
> }
>
> lambdas<- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
>    N <- update_N(min_sets, min_sets_indexes, lambda)
>    lambda <- rgamma(K, shape=a+N, rate=bT)
>    lambda
>    if (j%%100==0) { print(j); print(lambda); print(N)}
> #    if (j > burn_in) {
> #        lambdas <- cbind(lambdas, lambda)
> #    }
> }
>
> ---------------snip

Sorry -- I don't get what you're asking/trying to do.

Is it a coincidence that your commented block uses the same variable
name as the one you are assigning the result of foreach() to?

Essentially, foreach will work just like an lapply ... if you changed
foreach to lapply here, what do you expect that %dopar% {} block to
return after each iteration?

I'm not sure if this is what you're asking, but if you want to return
two elements per iteration in your loop, just return a list with two
elements, and post process it later.

I'd start by trying to remove your .combine=cbind param/argument from
the foreach() function and get your code running so you get the right
"things" returned as a normal list (or list of lists, if you want to
return > 1 thing per foreach iteration). Once that's done, you can try
to auto 'cbind' your things if you think it's necessary.

Sorry if this isn't helpful .. it's not clear to me what you're trying
to do, so I'm kind of stabbing at the dark here.

-- 
Steve Lianoglou
Graduate Student: Computational Systems Biology
 | Memorial Sloan-Kettering Cancer Center
 | Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact