[R] parallel::makeCluster does not finish

[Hille, Roman (STEAG Energy Services extern)]

Question to parallel and cluster package developers,

we have a serious problem with the cluster usage.

OS Windows Server 2016 64Bit
R-Version 3.6.1
parallel 3.6.1
Cluster 2.1.0

Sometimes, the cluster operation does not finish, one cluster Rscript.exe will never finish,
(or wait until timeout of 2592000...???)

In more than 90% all works fine,
but with some more parallel R-Sessions on the same machine and heavy load,
one cluster process does not finish sometimes
(I suppose, the final "ready" message of the process will be sent to early, the cluster "server" is not ready to receive....?)

Our R-Code:
   cl <- parallel::makeCluster(8, outfile = "", port = 64414)
    registerDoParallel(cl)
    Cluster.Res <- foreach(i = 1:Error.Dim.Selected[2] , combine=cbind) %dopar% {
....
    }

    stopCluster(cl)

parallel Cluster-process was started with:

C:\PROGRA~1\R\bin\x64\Rscript.exe --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=64414 OUT="" SETUPTIMEOUT=120 TIMEOUT=2592000 XDR=TRUE

Any idea or help

Thanks
Roman


Sysinternals is showing the following stack for the "hanging" Rscript.exe cluster process
(this is every time for this problem the same stack)

ntoskrnl.exe!KeSynchronizeExecution+0x5bf6
ntoskrnl.exe!KeWaitForMultipleObjects+0x109d
ntoskrnl.exe!KeWaitForMultipleObjects+0xb3f
ntoskrnl.exe!KeWaitForSingleObject+0x377
ntoskrnl.exe!KeQuerySystemTimePrecise+0x881
ntoskrnl.exe!ObDereferenceObjectDeferDelete+0x28a
ntoskrnl.exe!KeWaitForMultipleObjects+0x1284
ntoskrnl.exe!KeWaitForMultipleObjects+0xb3f
ntoskrnl.exe!KeWaitForSingleObject+0x377
ntoskrnl.exe!NtWaitForSingleObject+0xf8
ntoskrnl.exe!setjmpex+0x68b3
ntdll.dll!ZwWaitForSingleObject+0x14
mswsock.dll!Tcpip6_WSHOpenSocket2+0x1cf
mswsock.dll!Tcpip6_WSHGetSocketInformation+0x14f4
WS2_32.dll!select+0x1d3
internet.dll!R_init_internet+0xae8a
internet.dll!R_init_internet+0x6abe
R.dll!R_gzclose+0x29d6
R.dll!R_initAssignSymbols+0x7969
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x734
R.dll!R_has_methods_attached+0xeb9
R.dll!R_initAssignSymbols+0xa1a4
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!Rf_eval+0x9dc
R.dll!Rf_eval+0x127a
R.dll!R_initAssignSymbols+0xae53
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f
R.dll!R_initAssignSymbols+0xd2ff
R.dll!Rf_eval+0x331
R.dll!R_cmpfun1+0x508
R.dll!Rf_applyClosure+0x16f



	[[alternative HTML version deleted]]