[Bioc-devel] bplapply Processes Sometimes Stall

[Dario Strbenac]

Good day,

I haven't been able to make a small and reproducible example, but I am using bpstart and bpstop to run a loop with 25 workers multiple times on a large bioinformatics dataset. After a few times of running the loop successfully, a small number of the R workers use 100% CPU endlessly :

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                               
 3300 dario     20   0 1190832 837212  17988 R 100.0  0.2   3848:00 R                                                     
 5014 dario     20   0 1194528 829084   8224 R  99.8  0.2   3843:44 R                                                     
 5015 dario     20   0 1194532 829088   8224 R  99.8  0.2   3843:44 R

There are also three connections belonging to the R processes waiting to close :

~$ lsof -i | grep CLOSE
R          3300 dario 1025u  IPv4 160778259      0t0  TCP localhost:11881->localhost:49379 (CLOSE_WAIT)
R          5014 dario 1025u  IPv4 160778259      0t0  TCP localhost:11881->localhost:49379 (CLOSE_WAIT)
R          5015 dario 1025u  IPv4 160778259      0t0  TCP localhost:11881->localhost:49379 (CLOSE_WAIT)

~$ lsof -i | grep -c R
256

I use :

R version 3.2.3 (2015-12-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 8 (jessie)

with BiocParallel 1.5.12

--------------------------------------
Dario Strbenac
PhD Student
University of Sydney
Camperdown NSW 2050
Australia