[R-sig-hpc] Parallelizing function with subfunctions using snowfall

[Joakim Ringblom]

Hi

I have a function (jr.parsimest) which is rather time consuming and since I want to do use it several times with different input I tried to parallelize the process using snowfall. The code works fine when running in serial with sfInit(parallel=FALSE) but I have a problem getting it to work with sfInit(parallel=TRUE, cpus=8).

The problem is that the function jr.parsimest contains several subfunctions that in turn have several more subfunctions and so on and it seems that I have to export all of them with sfExport or sfExportall. This will be rather tricky and tedious at best, since I don´t really know some of the subfunctions with subfunctions since they come from a non-Cran package not written by me.  I also get a feeling that this is not the way to do things but I am not able confirm it anywhere.

Is my approach fundamentally wrong? Is there a better way of doing things?

Parallelizing a time-consuming subfunction is of course an idea but I rather parallelize the "topfunction" if possible.

I don´t think my code is important for the questions but I post it here anyhow just in case. I run the program on windows 7 computer at the moment but I would like to run it on unix later on if that matters.

Any help would be greatly appreciated.

/
Thanks in advance

Joakim Ringblom
PhD-student
Karolinska Institutet
Stockholm, Sweden
joakim.ringblom at ki.se

paramvect<-c(1,0.52, 1.3,4)
paramsd<-c(1,0.52, 1.3,4)
paramsd<-c(0,0,0,0)
ressd<-c(0.01)
doser<-c(0,0.381,0.475,0.564)
gruppstorlek<-c(10,10,10,10)
n<-1
nrdiff<-4
cef<-0.05
opiter<-2
searchnr<-1
bestnr<-gruppstorlek
outtable<-matrix(0,opiter,27+2*length(doser))
colnames(outtable)<-c("Medel-BMD","Medel-BMDL","Medel-BMDU","Medel-BMDL/BMD","Medel-BMDtrue", "Medel-SquaredError", "Medel-APE", "BMD-Varians","BMDL-Varians","BMDU-Varians","BMDL/BMD-Varians","BMDtrue-Varians", "SquaredError-Varians", "APE-Varians", "Antal BMD-NAs", "Parameter","Parameter","Parameter","Parameter", "SD","SD","SD","SD", "Ressd", 1:length(doser), 1:length(doser),"n", "CES", "searchnr")
for (k in 1:opiter){
 iternr<-1
 grupplist<-list()
 grupplist[[iternr]]<-gruppstorlek
 for (i in 1:length(doser)){
  arbnr<-gruppstorlek
  if (arbnr[i]>=nrdiff){
   arbnr[i]<-arbnr[i]-nrdiff
   for (j in 1:length(doser)){
    tempnr<-arbnr
    if (!(i==j)){
     searchnr<-searchnr+1
     iternr<-iternr+1     
     tempnr[j]=tempnr[j]+nrdiff 
     grupplist[[iternr]]<-tempnr  
    }
   }
  }
 }
 sfInit(parallel=TRUE, cpus=8)
 svar<-sfLapply(grupplist, jr.parsimest, paramvect=paramvect, paramsd=paramsd, ressd=ressd, doser=doser, n=n, CES=cef, outtable=outtable,searchnr=searchnr)
 sfStop()
 
 besteffect<-svar[[1]]
 for (i in 2:length(svar)){
  if (svar[[i]][7]<besteffect[7]){
   besteffect<-svar[[i]]
     }
 }
 bestnr<-c(besteffect[29],besteffect[30],besteffect[31],besteffect[32])
 if(all(gruppstorlek==bestnr & nrdiff>1)){nrdiff<-nrdiff/2}
 gruppstorlek<-bestnr
 outtable[k,]<-besteffect
 write.table(outtable, file="utskrift.txt")
}
return(bestnr)