[R-sig-hpc] mclapply() hangs when keras-based neural networks are involved

[Marius Hofert]

Hi,

Below is an example where mclapply() 'hangs' after starting the work
on two cores.
This happens on macOS and Ubuntu (sessionInfo() below). I also see no activity
on 'htop'. lapply() works, though. What is the cause of this behavior?

Cheers,
M

library(tensorflow)
library(keras)
library(parallel)
## TensorFlow also needs to be installed, which can be done via
install_tensorflow() from R

## 1) Setup
in.lay <- layer_input(shape = 2)
hid.lay <- layer_dense(in.lay,  units = 300, activation = "relu")
out.lay <- layer_dense(hid.lay, units = 2,   activation = "sigmoid")
NN <- keras_model(in.lay, out.lay)
loss_fn <- function(x, y = out.lay) loss_mean_squared_error(x, y)
NN %>% compile(optimizer = "adam", loss = loss_fn)

## 2) Training
NN %>% fit(x = matrix(runif(10000 * 2), ncol = 2), # prior data
           y = matrix(rnorm(10000 * 2), ncol = 2), # training data
           batch_size = 5000, epochs = 1)

## 3) Generate samples by evaluating the NN on a prior sample
aux <- function(b) {
    cat(paste("Working on case",b,"\n"))
    Sys.sleep(2)
    predict(NN, x = matrix(runif(100 * 2), ncol = 2)) # mclapply()
hangs here (on macOS and Ubuntu)
}

## 4) Call that hangs after the two processes are started
res.serial   <-   lapply(1:5, function(b) aux(b)) # works
res.parallel <- mclapply(1:5, function(b) aux(b), mc.cores = 2) #
hangs once both cores are used

## Output:
## For lapply():
Working on case 1
Working on case 2
Working on case 3
Working on case 4
Working on case 5
## For mclapply():
Working on case 1
Working on case 2

## sessionInfo() on macOS:
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin18.7.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /usr/local/R/R-3.6.1_build/lib/libRblas.dylib
LAPACK: /usr/local/R/R-3.6.1_build/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1

## sessionInfo() on Ubuntu:
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /u/mhofert/soft/R/R-3.6.0_build/lib/libRblas.so
LAPACK: /u/mhofert/soft/R/R-3.6.0_build/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.0