[R-sig-hpc] Parallelizing function with subfunctions using snowfall

Stephen Weston stephen.b.weston at gmail.com
Thu Aug 25 15:22:54 CEST 2011


If the functions that jr.parsimest calls are all defined in a package,
then just load that package on each your workers, and they should be
available to jr.parsimest.  Of course, that means that the package has
to be installed on all of the worker machines.

- Steve


On Thu, Aug 25, 2011 at 4:39 AM, Joakim Ringblom <joakim.ringblom at ki.se> wrote:
> Hi
>
> I have a function (jr.parsimest) which is rather time consuming and since I want to do use it several times with different input I tried to parallelize the process using snowfall. The code works fine when running in serial with sfInit(parallel=FALSE) but I have a problem getting it to work with sfInit(parallel=TRUE, cpus=8).
>
> The problem is that the function jr.parsimest contains several subfunctions that in turn have several more subfunctions and so on and it seems that I have to export all of them with sfExport or sfExportall. This will be rather tricky and tedious at best, since I don´t really know some of the subfunctions with subfunctions since they come from a non-Cran package not written by me.  I also get a feeling that this is not the way to do things but I am not able confirm it anywhere.
>
> Is my approach fundamentally wrong? Is there a better way of doing things?
>
> Parallelizing a time-consuming subfunction is of course an idea but I rather parallelize the "topfunction" if possible.
>
> I don´t think my code is important for the questions but I post it here anyhow just in case. I run the program on windows 7 computer at the moment but I would like to run it on unix later on if that matters.
>
> Any help would be greatly appreciated.
>
> /
> Thanks in advance
>
> Joakim Ringblom
> PhD-student
> Karolinska Institutet
> Stockholm, Sweden
> joakim.ringblom at ki.se
>
> paramvect<-c(1,0.52, 1.3,4)
> paramsd<-c(1,0.52, 1.3,4)
> paramsd<-c(0,0,0,0)
> ressd<-c(0.01)
> doser<-c(0,0.381,0.475,0.564)
> gruppstorlek<-c(10,10,10,10)
> n<-1
> nrdiff<-4
> cef<-0.05
> opiter<-2
> searchnr<-1
> bestnr<-gruppstorlek
> outtable<-matrix(0,opiter,27+2*length(doser))
> colnames(outtable)<-c("Medel-BMD","Medel-BMDL","Medel-BMDU","Medel-BMDL/BMD","Medel-BMDtrue", "Medel-SquaredError", "Medel-APE", "BMD-Varians","BMDL-Varians","BMDU-Varians","BMDL/BMD-Varians","BMDtrue-Varians", "SquaredError-Varians", "APE-Varians", "Antal BMD-NAs", "Parameter","Parameter","Parameter","Parameter", "SD","SD","SD","SD", "Ressd", 1:length(doser), 1:length(doser),"n", "CES", "searchnr")
> for (k in 1:opiter){
>  iternr<-1
>  grupplist<-list()
>  grupplist[[iternr]]<-gruppstorlek
>  for (i in 1:length(doser)){
>  arbnr<-gruppstorlek
>  if (arbnr[i]>=nrdiff){
>   arbnr[i]<-arbnr[i]-nrdiff
>   for (j in 1:length(doser)){
>    tempnr<-arbnr
>    if (!(i==j)){
>     searchnr<-searchnr+1
>     iternr<-iternr+1
>     tempnr[j]=tempnr[j]+nrdiff
>     grupplist[[iternr]]<-tempnr
>    }
>   }
>  }
>  }
>  sfInit(parallel=TRUE, cpus=8)
>  svar<-sfLapply(grupplist, jr.parsimest, paramvect=paramvect, paramsd=paramsd, ressd=ressd, doser=doser, n=n, CES=cef, outtable=outtable,searchnr=searchnr)
>  sfStop()
>
>  besteffect<-svar[[1]]
>  for (i in 2:length(svar)){
>  if (svar[[i]][7]<besteffect[7]){
>   besteffect<-svar[[i]]
>     }
>  }
>  bestnr<-c(besteffect[29],besteffect[30],besteffect[31],besteffect[32])
>  if(all(gruppstorlek==bestnr & nrdiff>1)){nrdiff<-nrdiff/2}
>  gruppstorlek<-bestnr
>  outtable[k,]<-besteffect
>  write.table(outtable, file="utskrift.txt")
> }
> return(bestnr)
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>



More information about the R-sig-hpc mailing list