[R] inconsistency in mclapply.....

[Ivan Krylov]

On Sat, 10 Jun 2023 08:42:37 +0000
akshay kulkarni <akshay_e4 using hotmail.com> wrote:

> This is the output of top when I run function LOWp() with mclapply
> also in it. it hangs:

> %Cpu0  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0
> si,  0.0 st
> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa, 0.0 hi,  0.0
> si,  0.0 st

I see. The system is idle.

> TRaceback of after interrupting LOWp:
> 
> 
> > LOWp(OHLCDataEP[[63]])  
> ^C
> There were 50 or more warnings (use warnings() to see the first 50)

What are the warnings()?

> > traceback()  
> 3: selectChildren(jobs[!is.na(jobsp)], -1)
> 2: mclapply(LYGH, FUN = arfima, mc.cores = 2, mc.preschedule = FALSE)
> at <tmp>#26
> 1: LOWp(OHLCDataEP[[63]])

...and the parent is stuck waiting for at least some of the children.
The problem must be inside the child processes.

> BLAS/LAPACK: /usr/lib64/libopenblaso-r0.3.15.so

If you start R like this:

OPENBLAS_NUM_THREADS=1 R

...and confirm that Sys.getenv('OPENBLAS_NUM_THREADS') returns 1, then
try running your analysis, does it still get stuck?

There are reports complaining about OpenBLAS failing fork() tests when
built with OpenMP support, and this 'o' at the end of the shared
library name looks suspicious.

> You had asked me to attach a debugger to the child processes. How do
> you get the child processes spawned by mclapply? For example, how do
> i identify, among the listed R processes above, the child processes?

Make sure that there's only one R session running. Stray R processes
can be cleaned up by running killall R or rebooting the AWS virtual
machine.

Start R and run Sys.getpid(). This is the PID of the parent process.
Start your analysis and get it stuck. Do not interrupt the function,
let it keep waiting for the child process.

In a separate ssh connection or virtual terminal, run top. It should be
possible to type lowercase v to display a process tree, then navigate
it using arrow keys. Confirm that there is a parent R process with the
PID you saw above and a number of child processes branching from it with
different PIDs. If there's more than one child, try them both and see
if the results are different (hopefully not).

Run gdb and enter the following commands (substitute the value of the
PID for PID_OF_THE_CHILD_PROCESS):

attach PID_OF_THE_CHILD_PROCESS
thread apply all backtrace

If there's too much text and you can't copy it out of the terminal, use
the commands "set logging file gdb.log" and "set logging on", then get
the text output from the file "gdb.log" that will be created in the
current directory.

-- 
Best regards,
Ivan