[R-sig-hpc] Problems parallelizing glmnet

[Peter Langfelder]

On Thu, Sep 6, 2012 at 1:58 PM, Zachary Mayer <zach.mayer at gmail.com> wrote:
> In this case, each iteration of the function is very quick:
>> system.time(summary(lm(y ~ x[,1]))$coefficients[2,4])
>    user  system elapsed
>    0.01    0.00    0.02
>
> And you are doing 10,000 iterations, so overhead matters a lot.  In the
> glmnet problem, each iteration of the function is very slow, and you are
> doing 8 iterations, so overhead doesn't matter at all.
>
> Finally, I suspect that using the doMC foreach backend will improve things
> considerably, but I can't currently test that.

FWIW, the foreach construct itself (without any parallel backend) is
quite slow and I would not use to loop over a large number of quick
calculations.

Peter