[R-sig-hpc] foreach package - Parallel Apply (Section 5.2 in Vignettes foreach Manual)

[Stephen Weston]

You can iterate by rows using the "by" argument
as follows:

> library(iterators)
> x <- matrix(rnorm(16), 4, 4)
> it <- iter(x, by='row', chunksize=2)
> print(as.list(it))
[[1]]
          [,1]      [,2]      [,3]       [,4]
[1,] 0.0856324  1.669167 -1.002161 -0.5635069
[2,] 0.5784011 -2.306515 -1.335513  0.3340934

[[2]]
          [,1]       [,2]       [,3]       [,4]
[1,] -1.548765  0.7862464 -0.5746842 -1.6977693
[2,]  0.707212 -0.4903425  1.7908369 -0.9125944

- Steve


On Wed, Dec 9, 2009 at 6:24 PM, Debabrata Midya
<Debabrata.Midya at services.nsw.gov.au> wrote:
> Stephen,
>
> Thanks for such a quick response. Your advice will assist me a lot.
>
> One clarification please.
>
> "... allows you to iterate over multiple columns at a time. ...".
>
> Is it possible to do over multiple rows at a time?
>
> Once again, thank you very much for the time you have given.
> I am looking forward for your reply.
>
> Regards,
> Deb
>
>>>> Stephen Weston <stephen.b.weston at gmail.com> 10/12/2009 10:07 am >>>
> It's worth noting that the iter method for matrices also allows you to
> iterate
> over multiple columns at a time.  But instead of specifying the number
> of blocks, you have to specify the number of columns in each block.
>
> Here's an example:
>
>> library(iterators)
>> x <- matrix(rnorm(16), 4, 4)
>> it <- iter(x, chunksize=2)
>> print(as.list(it))
> [[1]]
>            [,1]       [,2]
> [1,]  0.1811217  1.7126129
> [2,]  0.1415969  0.6980985
> [3,] -1.3921071 -0.9700261
> [4,] -1.2338734 -0.5530707
>
> [[2]]
>            [,1]        [,2]
> [1,] -1.1162171  1.09920729
> [2,]  0.9893546 -0.37677605
> [3,] -0.5508948 -0.02057267
> [4,] -0.3817569 -1.53422901
>
> The other method is more convenient when you want to
> split up the work evenly between all of the parallel workers,
> however.
>
> - Steve
>
>
> On Wed, Dec 9, 2009 at 5:38 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>>
>> On 10 December 2009 at 09:12, Debabrata Midya wrote:
>> | Dear hpc users,
>> |
>> | Thanks in advance.
>> |
>> | I am using R 2.10.0 on Windows XP, Intel(R) Core(TM) 2 Duo CPU E8400
>> @3.00GHz 2.99GHz, 1.96 of RAM.
>> |
>> | I need you assistance to solve the problem below:
>> |
>> | foreach package - Parallel Apply (Section 5.2 in Vignettes foreach
>> Manual)
>> |
>> | "Now we're only sending any given column of the matrix to one parallel
>> execution worker. But it
>> | would be even more efficient if we sent the matrix in bigger chunks. To
>> do that, we use a function
>> | called iblkcol that returns an iterator that will return multiple
>> columns of the original matrix.
>> | That means that the R expression will need to execute the user's
>> function once for every column in
>> | its submatrix".
>> |
>> | The function below is from the above Vigenette:
>> |
>> | applyKernel <- function(newX, FUN, d2, d.call, dn.call = NULL, ...) {
>> |  foreach(x = iblkcol(newX, 3), .combine = "c", .packages = "foreach")
>> %dopar%
>> |  {
>> |     foreach(i = 1:ncol(x)) %do% FUN(array(x[, i], d.call, dn.call), ...)
>> |  }
>> | }
>> |
>> | > applyKernel(matrix(1:16, 4), mean, 4, 4)
>> |
>> | Error in eval(expr, envir, enclos) : could not find function "iblkcol"
>> |
>> | How can I get this function?
>>
>> I am not sure how familiar you are with vignettes but the key is that they
>> contain all the code.  So here it is, straight from the package sources.
>> The
>> <<...>> and @ lines are the Sweave markers, R code is in-between.
>>
>> <<ex13.iter, results=hide, echo=FALSE>>=
>> iblkcol <- function(a, chunks) {
>>  n <- ncol(a)
>>  i <- 1
>>
>>  nextElem <- function() {
>>    if (chunks <= 0 || n <= 0) stop('StopIteration')
>>    m <- ceiling(n / chunks)
>>    r <- seq(i, length=m)
>>    i <<- i + m
>>    n <<- n - m
>>    chunks <<- chunks - 1
>>    a[,r, drop=FALSE]
>>  }
>>
>>  structure(list(nextElem=nextElem), class=c('iblkcol', 'iter'))
>> }
>> nextElem.iblkcol <- function(obj) obj$nextElem()
>> @
>>
>>
>> Use the sources for R and its packages. They are a much-underused
>> treasure.
>>
>> Dirk
>>
>> |
>> | Once again, thank you very much for the time you have given.
>> |
>> |
>> | I am looking forward for your reply.
>> |
>> | Regards,
>> |
>> | Deb
>> |
>> |       [[alternative HTML version deleted]]
>> |
>> | _______________________________________________
>> | R-sig-hpc mailing list
>> | R-sig-hpc at r-project.org
>> | https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>
>> --
>> Three out of two people have difficulties with fractions.
>>
>> _______________________________________________
>> R-sig-hpc mailing list
>> R-sig-hpc at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>
>