[R-sig-hpc] mclapply() hangs when keras-based neural networks are involved
Simon Urbanek
@|mon@urb@nek @end|ng |rom R-project@org
Fri Aug 30 10:39:38 CEST 2019
Marius,
Tensorflow doesn’t support any parallel computing including forking. It is assumed that all parallelization is done by TF itself and it takes over all resources in a way such that they cannot be shared across processes. Hence you cannot combine TF and parallel (and hence by induction Keras).
Cheers,
Simon
> On Aug 28, 2019, at 3:32 AM, Marius Hofert <marius.hofert using uwaterloo.ca> wrote:
>
> Hi,
>
> Below is an example where mclapply() 'hangs' after starting the work
> on two cores.
> This happens on macOS and Ubuntu (sessionInfo() below). I also see no activity
> on 'htop'. lapply() works, though. What is the cause of this behavior?
>
> Cheers,
> M
>
> library(tensorflow)
> library(keras)
> library(parallel)
> ## TensorFlow also needs to be installed, which can be done via
> install_tensorflow() from R
>
> ## 1) Setup
> in.lay <- layer_input(shape = 2)
> hid.lay <- layer_dense(in.lay, units = 300, activation = "relu")
> out.lay <- layer_dense(hid.lay, units = 2, activation = "sigmoid")
> NN <- keras_model(in.lay, out.lay)
> loss_fn <- function(x, y = out.lay) loss_mean_squared_error(x, y)
> NN %>% compile(optimizer = "adam", loss = loss_fn)
>
> ## 2) Training
> NN %>% fit(x = matrix(runif(10000 * 2), ncol = 2), # prior data
> y = matrix(rnorm(10000 * 2), ncol = 2), # training data
> batch_size = 5000, epochs = 1)
>
> ## 3) Generate samples by evaluating the NN on a prior sample
> aux <- function(b) {
> cat(paste("Working on case",b,"\n"))
> Sys.sleep(2)
> predict(NN, x = matrix(runif(100 * 2), ncol = 2)) # mclapply()
> hangs here (on macOS and Ubuntu)
> }
>
> ## 4) Call that hangs after the two processes are started
> res.serial <- lapply(1:5, function(b) aux(b)) # works
> res.parallel <- mclapply(1:5, function(b) aux(b), mc.cores = 2) #
> hangs once both cores are used
>
> ## Output:
> ## For lapply():
> Working on case 1
> Working on case 2
> Working on case 3
> Working on case 4
> Working on case 5
> ## For mclapply():
> Working on case 1
> Working on case 2
>
> ## sessionInfo() on macOS:
> R version 3.6.1 (2019-07-05)
> Platform: x86_64-apple-darwin18.7.0 (64-bit)
> Running under: macOS Mojave 10.14.6
>
> Matrix products: default
> BLAS: /usr/local/R/R-3.6.1_build/lib/libRblas.dylib
> LAPACK: /usr/local/R/R-3.6.1_build/lib/libRlapack.dylib
>
> locale:
> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.6.1 tools_3.6.1
>
> ## sessionInfo() on Ubuntu:
> R version 3.6.0 (2019-04-26)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 18.04.3 LTS
>
> Matrix products: default
> BLAS: /u/mhofert/soft/R/R-3.6.0_build/lib/libRblas.so
> LAPACK: /u/mhofert/soft/R/R-3.6.0_build/lib/libRlapack.so
>
> locale:
> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.6.0
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
More information about the R-sig-hpc
mailing list