[R-sig-hpc] mclapply() hangs when keras-based neural networks are involved

[Simon Urbanek]

Marius,

Tensorflow doesn’t support any parallel computing including forking. It is assumed that all parallelization is done by TF itself and it takes over all resources in a way such that they cannot be shared across processes. Hence you cannot combine TF and parallel (and hence by induction Keras).

Cheers,
Simon


> On Aug 28, 2019, at 3:32 AM, Marius Hofert <marius.hofert using uwaterloo.ca> wrote:
> 
> Hi,
> 
> Below is an example where mclapply() 'hangs' after starting the work
> on two cores.
> This happens on macOS and Ubuntu (sessionInfo() below). I also see no activity
> on 'htop'. lapply() works, though. What is the cause of this behavior?
> 
> Cheers,
> M
> 
> library(tensorflow)
> library(keras)
> library(parallel)
> ## TensorFlow also needs to be installed, which can be done via
> install_tensorflow() from R
> 
> ## 1) Setup
> in.lay <- layer_input(shape = 2)
> hid.lay <- layer_dense(in.lay,  units = 300, activation = "relu")
> out.lay <- layer_dense(hid.lay, units = 2,   activation = "sigmoid")
> NN <- keras_model(in.lay, out.lay)
> loss_fn <- function(x, y = out.lay) loss_mean_squared_error(x, y)
> NN %>% compile(optimizer = "adam", loss = loss_fn)
> 
> ## 2) Training
> NN %>% fit(x = matrix(runif(10000 * 2), ncol = 2), # prior data
>           y = matrix(rnorm(10000 * 2), ncol = 2), # training data
>           batch_size = 5000, epochs = 1)
> 
> ## 3) Generate samples by evaluating the NN on a prior sample
> aux <- function(b) {
>    cat(paste("Working on case",b,"\n"))
>    Sys.sleep(2)
>    predict(NN, x = matrix(runif(100 * 2), ncol = 2)) # mclapply()
> hangs here (on macOS and Ubuntu)
> }
> 
> ## 4) Call that hangs after the two processes are started
> res.serial   <-   lapply(1:5, function(b) aux(b)) # works
> res.parallel <- mclapply(1:5, function(b) aux(b), mc.cores = 2) #
> hangs once both cores are used
> 
> ## Output:
> ## For lapply():
> Working on case 1
> Working on case 2
> Working on case 3
> Working on case 4
> Working on case 5
> ## For mclapply():
> Working on case 1
> Working on case 2
> 
> ## sessionInfo() on macOS:
> R version 3.6.1 (2019-07-05)
> Platform: x86_64-apple-darwin18.7.0 (64-bit)
> Running under: macOS Mojave 10.14.6
> 
> Matrix products: default
> BLAS:   /usr/local/R/R-3.6.1_build/lib/libRblas.dylib
> LAPACK: /usr/local/R/R-3.6.1_build/lib/libRlapack.dylib
> 
> locale:
> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
> 
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
> 
> loaded via a namespace (and not attached):
> [1] compiler_3.6.1 tools_3.6.1
> 
> ## sessionInfo() on Ubuntu:
> R version 3.6.0 (2019-04-26)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 18.04.3 LTS
> 
> Matrix products: default
> BLAS:   /u/mhofert/soft/R/R-3.6.0_build/lib/libRblas.so
> LAPACK: /u/mhofert/soft/R/R-3.6.0_build/lib/libRlapack.so
> 
> locale:
> [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
> [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> 
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
> 
> loaded via a namespace (and not attached):
> [1] compiler_3.6.0
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>