[R-sig-hpc] indexing outputs from foreach / doMPI

[Stephen Weston]

For this kind of problem, you need to use the foreach .init argument,
which allows you to specify the first argument to the .combine function
the first time it is called.  On subsequent invocations of the .combine
function, the first argument is the return value from the previous
iteration.  That allows you to use the first argument as an accumulator
(at least, that's the way I think of it).  In this case, the first argument
is a list of lists of matrices, while the remain arguments to the .combine
function are lists of matrices.  Here's an example:

  comb <- function(accum, x) {
    list(a=c(accum$a, list(x$a)), b=c(accum$b, list(x$b)))
  }
  init <- list(a=list(), b=list())
  actual <- foreach(i=icount(length(a)), .combine=comb, .init=init) %dopar% {
    list(a=a[[i]], b=b[[i]])
  }

That should work, but unfortunately, it falls victim to one of the classic
R blunders: building up a data structure an element at a time!
However, the .combine function can be called with multiple results at
a time, by using the foreach .multicombine option.  Then, we can
rewrite the .combine function as:

  comb <- function(accum, ...) {
    s <- seq(along=accum)
    names(s) <- names(accum)
    lapply(s, function(i) c(accum[[i]], lapply(list(...), function(a) a[[i]])))
  }

This is also more general, since it doesn't assume either the
number or names of the elements in the result lists.

Here is a complete, self-contained example of this .combine
function in action:

library(foreach)

n <- 10000
s <- seq(length=n)
a <- lapply(s, function(i) matrix(rnorm(16), 4))
b <- lapply(s, function(i) matrix(rnorm(16), 4))
d <- lapply(s, function(i) matrix(rnorm(16), 4))
e <- lapply(s, function(i) matrix(rnorm(16), 4))

expected <- list(a=a, b=b, d=d, e=e)

comb <- function(accum, ...) {
  s <- seq(along=accum)
  names(s) <- names(accum)
  lapply(s, function(i) c(accum[[i]], lapply(list(...), function(a) a[[i]])))
}

init <- list(a=list(), b=list(), d=list(), e=list())

actual <- foreach(i=icount(length(a)), .combine=comb, .init=init,
                  .multicombine=TRUE) %dopar% {
  list(a=a[[i]], b=b[[i]], d=d[[i]], e=e[[i]])
}

print(identical(actual, expected))


- Steve


On Sun, Jan 23, 2011 at 12:57 PM, Maas James Dr (MED) <J.Maas at uea.ac.uk> wrote:
> Really struggling with this one,  in a toy example I use the following code.  When I run it a straight R script it works fine, but when  running it within a foreach loop and doMPI backend it re-arranges the outputs.   Is there a way to force foreach to keep all "a" matrices together ... and all the "b" matrices.  I've tried many permutations, no luck yet.
>
> Thanks
>
> J
>
> =========================
>
>
>
>  a <-  apply(results$OR, c(1,2), mean)
>  b <- apply(results$OR, c(1,2), sd)
>
>  output <- append (list(a=a),list(b=b))
>
>  return(output)
> }
>
> ==========================
>
>
>
> I get this
>
> =======================
>> stuff
> [[1]]
> [[1]]$a
>         [,1]      [,2]      [,3]
> [1,] 1.000000 0.8118515 0.6072900
> [2,] 1.236972 1.0000000 0.7499372
> [3,] 1.654514 1.3409720 1.0000000
>
> [[1]]$b
>           [,1]       [,2]       [,3]
> [1,] 0.00000000 0.05265696 0.04191226
> [2,] 0.08099146 0.00000000 0.05635320
> [3,] 0.11458134 0.10078931 0.00000000
>
>
> [[2]]
> [[2]]$a
>         [,1]      [,2]      [,3]
> [1,] 1.000000 0.7320218 0.5796935
> [2,] 1.371648 1.0000000 0.7942211
> [3,] 1.734505 1.2681720 1.0000000
>
> [[2]]$b
>           [,1]      [,2]       [,3]
> [1,] 0.00000000 0.0462359 0.04265879
> [2,] 0.08866844 0.0000000 0.06835189
> [3,] 0.12906653 0.1063840 0.00000000
>
> ==============================
>
>
> but I want this ....
>
>
>>stuff
> [[1]]
> [[1]]$a
>         [,1]      [,2]      [,3]
> [1,] 1.000000 0.8118515 0.6072900
> [2,] 1.236972 1.0000000 0.7499372
> [3,] 1.654514 1.3409720 1.0000000
>
> [[2]]$a
>         [,1]      [,2]      [,3]
> [1,] 1.000000 0.7320218 0.5796935
> [2,] 1.371648 1.0000000 0.7942211
> [3,] 1.734505 1.2681720 1.0000000
>
> [[2]]
> [[1]]$b
>           [,1]       [,2]       [,3]
> [1,] 0.00000000 0.05265696 0.04191226
> [2,] 0.08099146 0.00000000 0.05635320
> [3,] 0.11458134 0.10078931 0.00000000
>
>
> [[2]]$b
>           [,1]      [,2]       [,3]
> [1,] 0.00000000 0.0462359 0.04265879
> [2,] 0.08866844 0.0000000 0.06835189
> [3,] 0.12906653 0.1063840 0.00000000
>
>
>
>
> ===============================
> Dr. Jim Maas
> University of East Anglia
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>