[R-sig-hpc] Parallelizing function with subfunctions using snowfall

[Joakim Ringblom]

Hi

I just found the answer myself, I can´t believe I missed it before. Thanks anyone who has read my post, sorry for taking your time.

Regards

Joakim Ringblom
PhD-student
Karolinska Institutet, Institutet för Miljömedicin
Nobels väg, 13
Box 210 17177 Stockholm
Sweden
+46- (0)733965553

________________________________________
Från: Stephen Weston [stephen.b.weston at gmail.com]
Skickat: den 25 augusti 2011 15:22
Till: Joakim Ringblom
Kopia: r-sig-hpc at r-project.org
Ämne: Re: [R-sig-hpc] Parallelizing function with subfunctions using snowfall

If the functions that jr.parsimest calls are all defined in a package,
then just load that package on each your workers, and they should be
available to jr.parsimest.  Of course, that means that the package has
to be installed on all of the worker machines.

- Steve


On Thu, Aug 25, 2011 at 4:39 AM, Joakim Ringblom <joakim.ringblom at ki.se> wrote:
> Hi
>
> I have a function (jr.parsimest) which is rather time consuming and since I want to do use it several times with different input I tried to parallelize the process using snowfall. The code works fine when running in serial with sfInit(parallel=FALSE) but I have a problem getting it to work with sfInit(parallel=TRUE, cpus=8).
>
> The problem is that the function jr.parsimest contains several subfunctions that in turn have several more subfunctions and so on and it seems that I have to export all of them with sfExport or sfExportall. This will be rather tricky and tedious at best, since I don´t really know some of the subfunctions with subfunctions since they come from a non-Cran package not written by me.  I also get a feeling that this is not the way to do things but I am not able confirm it anywhere.
>
> Is my approach fundamentally wrong? Is there a better way of doing things?
>
> Parallelizing a time-consuming subfunction is of course an idea but I rather parallelize the "topfunction" if possible.
>
> I don´t think my code is important for the questions but I post it here anyhow just in case. I run the program on windows 7 computer at the moment but I would like to run it on unix later on if that matters.
>
> Any help would be greatly appreciated.
>
> /
> Thanks in advance
>
> Joakim Ringblom
> PhD-student
> Karolinska Institutet
> Stockholm, Sweden
> joakim.ringblom at ki.se
>
> paramvect<-c(1,0.52, 1.3,4)
> paramsd<-c(1,0.52, 1.3,4)
> paramsd<-c(0,0,0,0)
> ressd<-c(0.01)
> doser<-c(0,0.381,0.475,0.564)
> gruppstorlek<-c(10,10,10,10)
> n<-1
> nrdiff<-4
> cef<-0.05
> opiter<-2
> searchnr<-1
> bestnr<-gruppstorlek
> outtable<-matrix(0,opiter,27+2*length(doser))
> colnames(outtable)<-c("Medel-BMD","Medel-BMDL","Medel-BMDU","Medel-BMDL/BMD","Medel-BMDtrue", "Medel-SquaredError", "Medel-APE", "BMD-Varians","BMDL-Varians","BMDU-Varians","BMDL/BMD-Varians","BMDtrue-Varians", "SquaredError-Varians", "APE-Varians", "Antal BMD-NAs", "Parameter","Parameter","Parameter","Parameter", "SD","SD","SD","SD", "Ressd", 1:length(doser), 1:length(doser),"n", "CES", "searchnr")
> for (k in 1:opiter){
>  iternr<-1
>  grupplist<-list()
>  grupplist[[iternr]]<-gruppstorlek
>  for (i in 1:length(doser)){
>  arbnr<-gruppstorlek
>  if (arbnr[i]>=nrdiff){
>   arbnr[i]<-arbnr[i]-nrdiff
>   for (j in 1:length(doser)){
>    tempnr<-arbnr
>    if (!(i==j)){
>     searchnr<-searchnr+1
>     iternr<-iternr+1
>     tempnr[j]=tempnr[j]+nrdiff
>     grupplist[[iternr]]<-tempnr
>    }
>   }
>  }
>  }
>  sfInit(parallel=TRUE, cpus=8)
>  svar<-sfLapply(grupplist, jr.parsimest, paramvect=paramvect, paramsd=paramsd, ressd=ressd, doser=doser, n=n, CES=cef, outtable=outtable,searchnr=searchnr)
>  sfStop()
>
>  besteffect<-svar[[1]]
>  for (i in 2:length(svar)){
>  if (svar[[i]][7]<besteffect[7]){
>   besteffect<-svar[[i]]
>     }
>  }
>  bestnr<-c(besteffect[29],besteffect[30],besteffect[31],besteffect[32])
>  if(all(gruppstorlek==bestnr & nrdiff>1)){nrdiff<-nrdiff/2}
>  gruppstorlek<-bestnr
>  outtable[k,]<-besteffect
>  write.table(outtable, file="utskrift.txt")
> }
> return(bestnr)
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>