[R] Need some advice on optimization

[Ben Bolker]

Wijffels, Jan <jan.wijffels <at> thomascook.be> writes:

> 
> Hi
> 
> I'm quite new to optimization algorithms and I could use some advice or
> pointers. I'm using ?optim (method L-BFGS-B) to optimize a function over
> a 60-dimensional parameter space. The function itself takes about 1 to 6
> minutes to compute. It finds an optimum after 6 tot 24 hours, depending
> on the problem I need to solve. For me, speed is an issue and I would
> like to reduce this to 1 hour. 
> 
> Hence my question:: are there optimization algorithms that are
> particularly useful for settings where the evaluation of the function
> takes a lot of time to compute?

  Others may have more specific advice, but here is some
general information:

 * the more information from the function you can use, the
better.  In this respect the derivative-based methods (BFGS,
L-BFGS-B) will be more efficient than Nelder-Mead.  If there
is any way you can get analytic derivatives of your function,
that will help a lot.

 * Your best bet is probably to look at your objective function
and see if you can speed it up.  Can you vectorize or code the
time-consuming bits of it in C/FORTRAN?

 * For large-scale optimization, you can potentially get a very
large improvement in speed by using automatic differentiation,
which exists in R only in a rudimentary form. Depending on the
resources you have available, I would seriously consider
AD Model Builder from Otter Research.

  * I haven't tried it, but you might take a quick look at
http://www.milbo.users.sonic.net/ra/index.html, particularly
if your code involves non-vectorizable loops.

  good luck,
   Ben Bolker