[R] Snow Package and R: Exported Variable Problem

[Ramon Diaz-Uriarte]

Dear Robert,


On 2/2/07, robert.robinson at maine.edu <robert.robinson at maine.edu> wrote:
> Hello and thanks in advance for your time.
>
> I've created a simulation on my cluster which uses a custom package
> developed by me for different functions and also the snow package.
> Right now I'm using LAM to communicate between nodes and am currently
> only testing my code on 3 nodes for simplicity, though I plan on
> expanding to 16 later.  My problem is this error:
>
> "Error in fn(par, ..) : object \"x1\" not found \n"
> attr(,"class")
> "try-error"
>
> In my simulation I need to run a function several times with an
> different variable each time.  All the invocations on the functions
> are independent of the others.  I start the simulation on one node,
> create a cluster of several nodes, load my custom package and snow on
> all of them, use  clusterExport(cl, "x1") to export the variable
> x1(among other variable I need), then I call my simulation on the
> cluster using clusterApplyLB(cl, 2:S, simClust)  where cl is the
> cluster and S is a constant defined above as 500.  Using print
> statements (since snow, or R for that matter, has next to no ability
> to debug) I found that the error cropped up in this statement:
>
> theta6 = optim(c(0,0,0,0,0,0,.2), loglikelihood, scrore6, method =
> "CG", control=list(fnscale=-1,reltol=1e-8,maxit=2000))$par
>
> Both the functions loglikelelihood and score6 use x1, but I know that
> it is getting exported to the node correctly since it gets assigned
> earlier in the simulation:
>
> x1 = rep(0,n1)
>
> The error I stated above happens fo every itteration of the simulation
> (499 times) and I'm really at a loss as to why its happening and what
> I can do to determine what it is.  I'm wondering at this point if
> exporting the variable makes it unavailable to certain other packages,
> though that doesn't really make any sense.
>

>From reading quickly through your description, I do not see anything
obviously wrong.


> If anyone can help me with this problem, or let me know how I can
> debug this, or even a clue as to why it might be happening I would
> greatly appreciate it.  I've been wrestling with this for some time
> and no online documentation can help.  Thank you for your time and help.
>

When I was feeling really lost, I've resorted to assigning
intermediate output from commands such as ls, search, etc, to
variables (i.e., something like "this.ls <<- ls()" from inside you
function call, e.g., simClust) and then, e.g., from mpi.remote.exec,
looking at the value of those variables.

And, for over a year now, I've been doing most of my MPI stuff with
papply; the one nice thing of papply is that, if you have no LAM/MPI
universe, it will use a serial (not a parallel) version, so it is
much, much, much easier to debug, because you see the warnings and the
errors. So most of the frustration of things like launching something
and seeing it never return, etc, is gone.



Best,

R.

> Just so you know I'm a Computer Scientist not a Statistician, though I
> will be able to give any information about the statistics involved in
> this program.  I am reluctant to give away all source code since it is
> not my work but rather code I'm converting from standard code to
> parallelized code for a professor of mine.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz