[Rd] Get Logical processor count correctly whether NUMA is enabled or disabled

[Tomas Kalibera]

Dear Arun,

thank you for checking the workaround scripts.

I've modified detectCores() to use GetLogicalProcessorInformationEx. It 
is in revision 75198 of R-devel, could you please test it on your 
machines? For a binary, you can wait until the R-devel snapshot build 
gets to at least this svn revision.

Thanks for the link to the processor groups documentation. I don't have 
a machine to test this on, but I would hope that snow clusters (e.g. 
PSOCK) should work fine on systems with >64 logical processors as they 
spawn new processes (not just threads). Note that FORK clusters are not 
supported on Windows.

Thanks
Tomas

On 08/21/2018 02:53 PM, Srinivasan, Arunkumar wrote:
> Dear Tomas, thank you for looking into this. Here's the output:
>
> # number of logical processors - what detectCores() should return
> out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE)
> [1] "NumberOfLogicalProcessors  \r" "22                         \r" "22                         \r"
> [4] "20                         \r" "22                         \r" "\r"
> sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
> # [1] 86
>
> [I've asked the IT team to understand why one of the values is 20 instead of 22].
>
> # number of cores - what detectCores(FALSE) should return
> out <- system("wmic cpu get numberofcores", intern=TRUE)
> [1] "NumberOfCores  \r" "22             \r" "22             \r" "20             \r" "22             \r"
> [6] "\r"
> sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
> # [1] 86
>
> [Currently hyperthreading is disabled. So this output being identical to the previous output makes sense].
>
> system("wmic computersystem get numberofprocessors")
> NumberOfProcessors
> 4
>
> In addition, I'd also bring to your attention this documentation: https://docs.microsoft.com/en-us/windows/desktop/ProcThread/processor-groups on processor groups which explain how one should go about running a process ro run on multiple groups (which seems to be different to NUMA). All this seems overly complicated to allow a process to use all cores by default TBH.
>
> Here's a project on Github 'fio' where the issue of running a process on more than 1 processor group has come up -  https://github.com/axboe/fio/issues/527 and is addressed - https://github.com/axboe/fio/blob/c479640d6208236744f0562b1e79535eec290e2b/os/os-windows-7.h . I am not sure though if this is entirely relevant since we would be forking new processes in R instead of allowing a single process to use all cores. Apologies if this is utterly irrelevant.
>
> Thank you,
> Arun.
>
> From: Tomas Kalibera <tomas.kalibera using gmail.com>
> Sent: 21 August 2018 11:50
> To: Srinivasan, Arunkumar <Arunkumar.Srinivasan using uk.mlp.com>; r-devel using r-project.org
> Subject: Re: [Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
>
> Dear Arun,
>
> thank you for the report. I agree with the analysis, detectCores() will only report logical processors in the NUMA group in which R is running. I don't have a system to test on, could you please check these workarounds for me on your systems?
>
> # number of logical processors - what detectCores() should return
> out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE)
> sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
>
> # number of cores - what detectCores(FALSE) should return
> out <- system("wmic cpu get numberofcores", intern=TRUE)
> sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
>
> # number of physical processors - as a sanity check
>
> system("wmic computersystem get numberofprocessors")
>
> Thanks,
> Tomas
>
> On 08/17/2018 05:11 PM, Srinivasan, Arunkumar wrote:
> Dear R-devel list,
>
> R's detectCores() function internally calls "ncpus" function to get the total number of logical processors. However, this doesnot seem to take NUMA into account on Windows machines.
>
> On a machine having 48 processors (24 cores) in total and windows server 2012 installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If NUMA is disabled, detectCores() returns 48.
>
> Similarly, on a machine with 88 cores (176 processors) and windows server 2012, detectCores() with NUMA disabled only returns the maximum value of 64. If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will only return 44. This is particularly limiting since we cannot get to use all processors by enabling/disabling NUMA in this case.
>
> We think this is because R's ncpus.c file uses "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). Specifically, quoting from the first link:
>
> "On systems with more than 64 logical processors, the GetLogicalProcessorInformation function retrieves logical processor information about processors in the https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx to which the calling thread is currently assigned. Use the https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx function to retrieve information about processors in all processor groups on the system."
>
> Therefore, it might be possible to get the right count of total processors even with NUMA enabled by using "GetLogicalProcessorInformationEX".  It'd be nice to know what you think.
>
> Thank you very much,
> Arun.
>
> --
> Arun Srinivasan
> Analyst, Millennium Management LLC
> 50 Berkeley Street | London, W1J 8HD
>