[Rd] Get Logical processor count correctly whether NUMA is enabled or disabled

[Tomas Kalibera]

Dear Arun,

thank you for the report. I agree with the analysis, detectCores() will 
only report logical processors in the NUMA group in which R is running. 
I don't have a system to test on, could you please check these 
workarounds for me on your systems?

# number of logical processors - what detectCores() should return

out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE)
sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, 
value=TRUE))))

# number of cores - what detectCores(FALSE) should return

out <- system("wmic cpu get numberofcores", intern=TRUE)
sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, 
value=TRUE))))

# number of physical processors - as a sanity check

system("wmic computersystem get numberofprocessors")

Thanks,
Tomas

On 08/17/2018 05:11 PM, Srinivasan, Arunkumar wrote:
> Dear R-devel list,
>
> R's detectCores() function internally calls "ncpus" function to get the total number of logical processors. However, this doesnot seem to take NUMA into account on Windows machines.
>
> On a machine having 48 processors (24 cores) in total and windows server 2012 installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If NUMA is disabled, detectCores() returns 48.
>
> Similarly, on a machine with 88 cores (176 processors) and windows server 2012, detectCores() with NUMA disabled only returns the maximum value of 64. If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will only return 44. This is particularly limiting since we cannot get to use all processors by enabling/disabling NUMA in this case.
>
> We think this is because R's ncpus.c file uses "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). Specifically, quoting from the first link:
>
> "On systems with more than 64 logical processors, the GetLogicalProcessorInformation function retrieves logical processor information about processors in the https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx to which the calling thread is currently assigned. Use the https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx function to retrieve information about processors in all processor groups on the system."
>
> Therefore, it might be possible to get the right count of total processors even with NUMA enabled by using "GetLogicalProcessorInformationEX".  It'd be nice to know what you think.
>
> Thank you very much,
> Arun.
>
> --
> Arun Srinivasan
> Analyst, Millennium Management LLC
> 50 Berkeley Street | London, W1J 8HD
>
>
> ######################################################################
>
> The information contained in this communication is con...{{dropped:11}}