[R] parallel processing an lme model

[Douglas Bates]

On 5/28/07, toby909 at gmail.com <toby909 at gmail.com> wrote:
> Hi All

> Had anyone of you seen if it is possible to split a large lme() job to be
> processed by multiple cpus/computers?

> I am just at the very beginning to understand related things, but does the lme()
> use solution finding functions like nlm() mle(), and I-dont-know-what-else of
> the standard R pagage or does lme come with its own? If the former, has onyone
> seen how to split an mle() function call to be processed by multiple cpus?

First, if you want speed and your model can be fit by lmer I would
recommend using lmer or lmer2 from the lme4 package.  These functions
can fit models with crossed or partially crossed random effects which
is often the case for models in very large data sets.  However, they
do not provide the facility for specifying correlation structures or
variance functions in addition to those implied by the random effects.

Both lme and lmer end up calling nlminb to do the optimization of the
log-likelihood or the REML criterion.  The lmer2 function does not
call nlminb explicitly but does use the underlying code from nlminb.

None of these operations are easily parallelizable.  The only hope for
getting a speed boost from multiple CPU cores or multiple processors
is by using a multithreaded accelerated BLAS (Basic linear algebra
subroutines) library (see the R Installation and Administration manual
for details).  However, in some cases we have observed that
multithreaded BLAS actually slow down the computation.  To check if
this is the case for you try the following both with and without
multithreaded BLAS.

library(lme4)
data(star, package = "mlmRev")
system.time(fm1 <- lmer(math ~ sx*eth+ses+gr+cltype
   +(yrs|id)+(1|tch)+(yrs|sch),
    star, control = list(grad = 0, nit = 0, msV = 1))
system.time(m1 <- lmer2(math ~ sx*eth+ses+gr+cltype
   +(yrs|id)+(1|tch)+(yrs|sch),
    star, control = list(msV = 1))

> In the case if, I would very much appreicate your hint or point to a source
> where I can read about.