[R-SIG-Mac] R-SIG-Mac] Multi-thread R processes performance on 8-core

[Bill Northcott]

On 18/04/2007, at 5:52 PM, Marra, David wrote:
> I am running R 2.4.1 on the new 8-core Mac Pro with the parSapply
> function from the Snow package. Tests using 2,4, and 8 threads with
> makeCluster() yield somewhat disappointing results. The 4 thread  
> process
> is fastest. With 8 threads, all the cores max out at about 70% power,
> and even then it is slower than the 4 thread process which maxes  
> out the
> 4 threads at about 90-95%. This suggests the additional 4 cores on the
> Mac Pro do not improve performance in an embarrassingly parallel R/ 
> Snow
> environment...

This was brought up recently on an Apple developer mail list in the  
context of running multiple compilation threads in Xcode.

There was a reply from Chris Espinoza at Apple explaining that the  
Tiger kernel is not optimised for large numbers of CPUs.  In general  
trying to use more than 4 CPUs for one process will cause performance  
to drop just as you have found.

That is the bad news.  The good news is that the kernel in Leopard  
has been substantially rewritten and does not have these  
limitations.  However, you will have to wait until October.  :-(

Bill Northcott