[R-pkg-devel] Too many processes spawned on Windows and Debian, although only 2 cores should be used

[Riko Kelter]

Hi Ivan,

thanks for the info, I changed the check as you pointed out and it 
worked. R CMD build and R CMD check --as-cran run without errors or 
warnings on Linux + MacOS. However, I uploaded the package again at the 
WINBUILDER service and obtained the following weird error:

* checking re-building of vignette outputs ... ERROR
Check process probably crashed or hung up for 20 minutes ... killed
Most likely this happened in the example checks (?),
if not, ignore the following last lines of example output:

======== End of example output (where/before crash/hang up occured ?) ========

Strangely, there are no examples included in any .Rd file. Also, I 
checked whether a piece of code spawns new clusters. However, the 
critical lines are inside a function which is repeatedly called in the 
vignettes. The parallelized part looks as copied below. After the code 
is executed the cluster is stopped. I use registerDoSNOW(cl) because 
otherwise my progress bar does not work.


Code:

############################### CHECK CORES

chk <- tolower(Sys.getenv("_R_CHECK_LIMIT_CORES_", ""))
   if (nzchar(chk) && (chk != "false")){  # then limit the workers
     num_workers <- 2L
   } else {
     # use all cores
     num_workers <- parallel::detectCores()
   }

   chk <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "")

   cl <- parallel::makeCluster(num_workers[1]-1) # not to overload your 
computer
   #doParallel::registerDoParallel(cl)
   doSNOW::registerDoSNOW(cl)

############################### SET UP PROGRESS BAR

pb <- progress_bar$new(
     format = "Iteration = :letter [:bar] :elapsed | expected time till 
finish: :eta",
     total = nsim,    # 100
     width = 120)

   progress_letter <- seq(1,nsim)  # token reported in progress bar

   # allowing progress bar to be used in foreach 
-----------------------------
   progress <- function(n){
     pb$tick(tokens = list(letter = progress_letter[n]))
   }

   opts <- list(progress = progress)

############################### MAIN SIMULATION

if(method=="PP"){
     finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, .packages 
= c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
       tempMatrix = singleTrial_PP(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0)

       tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
     }
   }

   if(method=="PPe"){
     refFunc = refFunc
     nu = nu
     shape1 = shape1
     shape2 = shape2
     if(refFunc == "flat"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "flat")

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
     if(refFunc == "beta"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "beta",
                                      shape1 = shape1, shape2 = shape2)

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
     if(refFunc == "binaryStep"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "binaryStep",
                                      shape1 = shape1, shape2 = shape2, 
truncation = truncation)

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
     if(refFunc == "relu"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "relu",
                                      shape1 = shape1, shape2 = shape2, 
truncation = truncation)

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
     if(refFunc == "palu"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "palu",
                                      shape1 = shape1, shape2 = shape2, 
truncation = truncation)

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
     if(refFunc == "lolu"){
       finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
         tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "lolu",
                                      shape1 = shape1, shape2 = shape2, 
truncation = truncation)

         tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
       }
     }
   }

############################### STOP CLUSTER

parallel::stopCluster(cl) #stop cluster



Kind regards,

Riko


Am 16.11.22 um 08:29 schrieb Ivan Krylov:
> В Wed, 16 Nov 2022 07:29:25 +0100
> Riko Kelter<riko.kelter using uni-siegen.de>  пишет:
>
>> if (nzchar(chk) && chk == "TRUE") {
>>   # use 2 cores in CRAN/Travis/AppVeyor
>>   num_workers <- 2L
>> }
> The check in parallel:::.check_ncores is a bit different:
>
> chk <- tolower(Sys.getenv("_R_CHECK_LIMIT_CORES_", ""))
> if (nzchar(chk) && (chk != "false")) # then limit the workers
>
> Unless you actually set _R_CHECK_LIMIT_CORES_=FALSE on your machine
> when running the checks, I would perform a more pessimistic check of
> nzchar(chk) (without additionally checking whether it's TRUE or not
> FALSE), though copy-pasting the check from parallel:::.check_ncores
> should also work.
>
> Can we see the rest of the vignette? Perhaps the problem is not with
> the check. For example, a piece of code might be implicitly spawning a
> new cluster, defaulting to all of the cores instead of num_workers.
>
>> 	[[alternative HTML version deleted]]
> Unfortunately, the plain text version of your message prepared by your
> mailer has all the code samples mangled:
> https://stat.ethz.ch/pipermail/r-package-devel/2022q4/008647.html
>
> Please compose your messages to R mailing lists in plain text.
>
	[[alternative HTML version deleted]]