[R] Can't reliably use RefClass methods in Snowfall

[Martin Morgan]

On 09/24/2011 08:55 AM, henry wrote:
> Greetings,
>
> I am trying to use Reference Class methods in Snowfall, using R 2.12.1
> on Ubuntu Natty. Using then directly seems to work (stanza 2 below), but
> using them indirectly does not (stanza 3 below). I get an "attempt to
> apply non-function" error.
>
> In addition to exporting the instance of the object I created to the
> Snowfall slaves, I also made several attempts to export the method
> itself (e.g. sfExport("instance$method") or some such), with no success.
> Very strangely, if I run Stanza 2 below, then stanza 3 will work.
>
> I tried emailing the Snowfall author, but have not heard back. Any help
> is greatly appreciated!
>
> Henry Bryant
> Texas A&M University
>
>
>
> library("snowfall")
> library("methods")
>
> # set up a simple reference class
> calculator <- setRefClass("calculator")
> calculator$methods(do_calc = function(x) {print(x*x)})
> my_calc <- calculator$new()
> wrapper <- function(x) {my_calc$do_calc(x)}
>
> # STANZA 2: use snowfall without wrapper -- WORKING
> #sfInit(parallel=TRUE, cpus=2, type="SOCK")
> #sfExport("calculator")
> #sfExport("my_calc")
> #results <- sfLapply(1:10,my_calc$do_calc)
> #sfStop()
> #print(results)
>
> # STANZA 3: use snowfall with wrapper -- NOT WORKING
> # (not working by itself, but does work if the previous stanza is run
> first!!)
> sfInit(parallel=TRUE, cpus=2, type="SOCK")
> sfExport("calculator")
> sfExport("my_calc")
> sfExport("wrapper")
> last_results <- sfLapply(1:10,wrapper)

Hi Henry --

The problem here is that references classes implement their methods in a 
kind of 'lazy' way -- my_calc$do_calc doesn't exist until it's actually 
invoked for the first time, so when you

   sfExport("my_calc")

you export the object but "do_calc" is defined as NULL

 > sfClusterEval(my_calc$do_calc)
[[1]]
NULL

[[2]]
NULL

This seems to be a variant of the problem experienced elsewhere

https://stat.ethz.ch/pipermail/r-devel/2011-June/061260.html
http://stackoverflow.com/questions/7331518/method-initialisation-in-r-reference-classes

The solutions seem mostly ugly, e.g., to evaluate locally

 > my_calc$do_calc

and then sfExport, or maybe a bit better as

calculator <- setRefClass("calculator")
calculator$methods(
     initialize = function(...) {
         callSuper(...)
         .self$do_calc                   # force definition
         .self
     },
     do_calc = function(x) x*x)

I'm unclear about whether there are other aspects of the reference class 
implementation that tie the instance to the class definition in a way 
that would undermine parallel evaluation.

Also in general it seems better to write truly 'functional programming' 
style functions that don't rely on aspects of the environment to work, 
e.g.,

   wrapper <- function(x, calc) calc$do_calc(x)
   sfLapply(1:10, wrapper, my_calc)

Martin

> sfStop()
> cat("\nRESULTS FROM SECOND TRY:\n")
> print(last_results)
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
Computational Biology
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109

Location: M1-B861
Telephone: 206 667-2793