[R] Confindence interval for Levenberg-Marquardt fit

[Douglas Bates]

On 2/21/07, joerg van den hoff <j.van_den_hoff at fzd.de> wrote:
> On Wed, Feb 21, 2007 at 11:09:52AM +0000, Prof Brian Ripley wrote:
> > Well, the algorithm used does not affect the confidence interval (provided
> > it works correctly), but what is nls.ml (presumably in some package you
> > have not mentioned) and why would I want to use an old-fashioned
> > algorithm?

> is'nt this a bit strong? in what respect do you consider levenberg-marquardt
> (going back to the 19-forties, I think) as old-fashioned (especially
> in comparsion to the `nls' standard gauss-newton approach (both gentlemen
> seem to have done their major work a bit longer ago :-))?
> AFAICS levenberg-marquardt is generally appreciated for it's rapid
> convergence achieved by a smooth transition from an
> inverse-hessian approach to steepest descent. my own experience
> with non-linear least squares minimization using this algorithm
> are positive as well, but
> I have not tried out the levenberg-marquardt
> implementation in package `minpack.lm' (which originates from netlib.org)
> and don't know if it's good. but in any case there sure are implementations
> around (e.g. in the CERN MINUIT library) which have proven to be
> of high quality.

The nls function provides implementations of the Gauss-Newton
algorithm and the Golub-Pereyra algorithm and the NL2SOL algorithm.
Check the possible values of the optional argument "algorithm" to
nls().

If you will allow me to wax philosophical for a moment, I submit that
there are three stages to any iterative optimization:
  - formulating initial values of the parameters
  - determining what step to take from iteration i to iteration i+1
  - determining when to declare convergence

For the most part optimization research focuses on the second step.
The first and third steps are also important.  One of the advantages
of the implementation of the Gauss-Newton and Golub-Pereyra algorithms
in nls is that they use a convergence criterion (based only on the
current parameter values) not a termination criterion (based on the
changes in the parameter values or the changes in the objective
function or both).  See chapter 2 of Bates and Watts, "Nonlinear
Regression Analysis and Its Applications" (Wiley, 1988) for a
discussion of the particular convergence criterion used.

Another point that is often lost in a discussion of various
optimization algorithms is that one does not compare algorithms - one
compares implementations of algorithms which, at the very least, must
involve all of the above steps.

For many people in the nonlinear least squares field the NL2SOL
algorithm by Dennis, Gay and Welch was considered the "gold standard"
and David Gay's implementation is now available in nls.  This allows
those who are interested to compare implementations of these
algorithms.  Also, with R being an Open Source system anyone can add
their own implementation or modify an existing implementation to try
things out.




> `nls' sure is a _very_ valuable function, but not necessarily the
> "last word" with regards to the chosen algorithm(s).
>
>
> >
> > You could start nls at the solution you got from nls.ml and use confint()
> > on that.
> maybe one should look at profile.nls and confint.nls and see what information
> of the usual `nls' object is actually used for the confidence intervall
> computation and mimick this for the `nls.lm' output? at a (admittedly)
> quick glance it seems that only parameters, std.errs. and the fitted/residual
> values are needed which should all be provided by nls.lm as well.
> maybe one could even try to map the nls.lm results into a structure of class
> `nls' (although this would not be a clean solution, probably) in order
> to use `confint.nls'?

Not really.  The profile.nls function requires, not surprisingly, the
ability to profile the objective function.  (See the discussion of the
profile sum of squares and the profile-t transformations in chapter 6
of Bates and Watts).  Such a capability would need to be added to a
nonlinear least squares implementation before methods for those
generics could be developed.

> >
> > On Wed, 21 Feb 2007, Michael Dondrup wrote:
> >
> > > Dear all,
> > > I would like to use the Levenberg-Marquardt algorithm for non-linear
> > > least-squares regression using function nls.lm. Can anybody help  me to
> > >   find a a way to compute confidence intervals on the  fitted
> > > parameters as it is possible for nls (using confint.nls, which does not
> > > work for nls.lm)?
> > >
> > > Thank you for your help
> > > Michael
> >
> >
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>