[BioC] snow library, question on clusterExport
Martin Morgan
mtmorgan at fhcrc.org
Mon Apr 12 19:44:29 CEST 2010
On 04/12/2010 09:25 AM, mattia pelizzola wrote:
> thanks Martin for the explanations and thanks Norman for pointing out
> that error in the example,
>
> unfortunately I am still stuck with the main problem:
> I have to use clusterExport to export an object to the cluster nodes.
It's hard to know without a (simple) example why you have to use
clusterExport. If the object is not defined in the environment of the
function, then a sure-fire way of getting it to your cluster nodes is to
explicitly include it in the clusterApplyLB call
fun4=function(startInd, endInd=3, data) data[startInd:endInd,]
clusterApplyLB(cl, 1:3, fun4, data=Mat)
Martin
> clusterExport only seems to export objects from the GlobalEnv,
> unfortunately. In my case this object is created within a function and
> clusterExport is called within the same function, so the object is not
> available in the GlobalEnv and I get error ..
>
> I'll try writing to the other mailing list,
> thanks
>
> mattia
>
> On Sun, Apr 11, 2010 at 8:49 AM, Pavelka, Norman <NXP at stowers.org> wrote:
>> Hi Mattia,
>>
>> Maybe I'm not getting what you're trying to do, but shouldn't your fun3 be using object 'data' rather than 'Mat' internally?
>>
>> HTH ;-)
>> Norman
>>
>> # Hi,
>> #
>> # I have a simple function:
>> #
>> # > library(snow)
>> # > fun2=function() {
>> # + cl=makeCluster(3)
>> # + Mat=matrix(2:10,3,3)
>> # + fun3=function(startInd, endInd=3, data=Mat) {Mat[startInd:endInd,]}
>> # + print(clusterApplyLB(cl, 1:3, fun3))
>> # + stopCluster(cl)
>> # + }
>> #
>> # that is working fine:
>> #
>> # > fun2()
>> # [[1]]
>> # [,1] [,2] [,3]
>> # [1,] 2 5 8
>> # [2,] 3 6 9
>> # [3,] 4 7 10
>> #
>> # [[2]]
>> # [,1] [,2] [,3]
>> # [1,] 3 6 9
>> # [2,] 4 7 10
>> #
>> # [[3]]
>> # [1] 4 7 10
>> #
>> # now, if I run the same commands outside the function:
>> #
>> # > cl=makeCluster(3)
>> # > Mat=matrix(2:10,3,3)
>> # > fun3=function(startInd, endInd=3, data=Mat) {Mat[startInd:endInd,]}
>> # > print(clusterApplyLB(cl, 1:3, fun3))
>> # Error in checkForRemoteErrors(val) :
>> # 3 nodes produced errors; first error: object 'Mat' not found
>> #
>> # so I figured out I have to export 'Mat' on the cluster nodes:
>> #
>> # > clusterExport(cl, 'Mat')
>> # > print(clusterApplyLB(cl, 1:3, fun3))
>> # [[1]]
>> # [,1] [,2] [,3]
>> # [1,] 2 5 8
>> # [2,] 3 6 9
>> # [3,] 4 7 10
>> #
>> # [[2]]
>> # [,1] [,2] [,3]
>> # [1,] 3 6 9
>> # [2,] 4 7 10
>> #
>> # [[3]]
>> # [1] 4 7 10
>> #
>> # I still do not understand why clusterExport is NOT necessary within
>> # the function 'fun2' and actually it would give an error:
>> #
>> # > rm(Mat)
>> # > fun2=function() {
>> # + cl=makeCluster(3)
>> # + Mat=matrix(2:10,3,3)
>> # + clusterExport(cl, 'Mat')
>> # + fun3=function(startInd, endInd=3, data=Mat) {Mat[startInd:endInd,]}
>> # + print(clusterApplyLB(cl, 1:3, fun3))
>> # + stopCluster(cl)
>> # + }
>> # > fun2()
>> # Error in get(name, env = .GlobalEnv) : object 'Mat' not found
>> #
>> #
>> # I found clusterExport to be the solution for a more complex example,
>> # can I can't make it working within a function.
>> # What is it happening here with clusterExport? and how can I export an
>> # object that is not on my globalEnv but rather is created within a
>> # function?
>> #
>> # many thanks!
>> #
>> # mattia
>> #
>> # > sessionInfo()
>> # R version 2.10.1 (2009-12-14)
>> # x86_64-unknown-linux-gnu
>> #
>> # locale:
>> # [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
>> # [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
>> # [5] LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
>> # [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
>> # [9] LC_ADDRESS=C LC_TELEPHONE=C
>> # [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>> #
>> # attached base packages:
>> # [1] stats graphics grDevices utils datasets methods base
>> #
>> # other attached packages:
>> # [1] snow_0.3-3
--
Martin Morgan
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109
Location: Arnold Building M1 B861
Phone: (206) 667-2793
More information about the Bioconductor
mailing list