[Rd] Parallel makes

[Prof Brian Ripley]

Now multi-core machines are more widely available, we have gotten to 
stress-test the parallel building capabilities of R and of packages. 
The current Windows and Mac build machines are both 8-core and I test 
on an 8-core machine.  These are all fairly recent changes of hardware 
and the following applies only to R-devel, the version to become 2.9.0 
next month.

Parallel builds of R under Unix-alikes have long been supported, and 
now allow rather more to be done in parallel.  Using 'make -j' will 
work on a machine with enough resources, and gives something like a 3x 
speed up.  THe main limiting factor is converting help, which is done 
serially and is likely remain so until we move to R-based conversion. 
New for this version is the ability to install in parallel (e.g. 'make 
-j install install-pdf').  It is also possible to check the R build in 
parallel, but the output is so intermingled that it is hard to see any 
discrepancies.  However, few people build R from scratch every day and 
for those with such powerful machines building R is probably already 
fast enough (ca 3 mins).

When installing or checking a single package, the only thing done in 
parallel is making any compiled code (2.8.x ran tests in the 'tests' 
directory in parallel, but this is not currently done).  The standard 
procedures work safely with parallel make, but users who write 
Makevars or Makefile files need to take this into account.  For 
example, yesterday's Rcsdp/src/Makevars has

PHONY: all

all: before $(SHLIB)

before: Csdp.ts

but this 'before' target has to be completed before $(SHLIB) can be 
built (and it failed to install for me).  I am aware that the 
documentation did not stress this sufficiently in the past, and 
'Writing R Extensions' has been revised to do so.  (And the package 
has been updated with alacrity, thank you.)

Because Windows users are much less likely to be aware of these issues 
and because of the last para below, Uwe and I tweaked the procedures 
for the Windows build machine so that packages are always 
installed/checked with a non-paralle make.

Installing/updating packages in parallel can help a lot, and we've 
made two changes to facilitate that.  First, there is a new option for 
R CMD INSTALL, --pkglock.  This uses locks on a per-package basis so 
prevents more than one process trying to install a package at the same 
time, but allows several packages to be installed to the same library 
simultaneously.  This places the onus on the caller to ensure that 
dependencies are installed first, and the 'Ncpus' option to 
install.packages() provides a way to marshall package installation to 
make best use of multiple CPUs.

Under Windows 'make all' and 'make recommended; (but not 'make 
distribution') can each be done in parallel.  There are some question 
marks over how well the 'make; used on Windows works in parallel (we 
found one case where it worked incorrectly and had to rethink 
share/make/winshlib.mk) so it should be used with caution.

Please give these new facilties a go and report (here) how you get 
on.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595