[R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Douglas Bates
bates at stat.wisc.edu
Sat Nov 25 20:18:58 CET 2006
On 11/24/06, dave fournier <otter at otter-rsch.com> wrote:
>
>
> Dave
> > Did you try supplying gradient information to nlminb? (I note that
> nlminb is used for the optimization, but I don't see any gradient
> information supplied to it.) I would suspect that supplying gradient
> information would greatly speed up the computation (as you note in
> comments at http://otter-rsch.ca/tresults.htm.)
> Actually you should probably ask Norm Olsen these questions.
> I am not proficient in R and am just using his code.
Don't you find it somewhat disingenuous that you publish a comparison
between the AD Model Builder software that you sell and R - a
comparison that shows a tremendous advantage for your software - and
then you write "I am not proficient in R"?
Had you been proficient in R you might have known about the symbolic
differentiation capabilities, specifically the deriv function, that
have been part of the S language since the late 1980s. I believe that
the 'AD' in "AD model builder" stands for automatic differentiation,
which is actually something that John Chambers and I discussed at
length when we were developing nonlinear modeling methods for S. In
the end we went with symbolic differentiation rather than automatic
differentiation because we felt that symbolic was more flexible.
This is not to say that automatic differentiation isn't a perfectly
legitimate technique. However, my recollection is that it would have
required extensive revisions to the arithmetic expression evaluator,
which is already very tricky code because of the "recycling rule" and
the desire to shield users from knowledge of the internal
representations and such details as whether you are using logical or
integer or double precision operands or a combination.
If you want to see these details you can, of course, examine the
source code. I don't believe we would have the opportunity to examine
how you implemented automatic differentiation.
I must also agree with Spencer Graves that when I start reading a
description of a nonlinear model with over 100 parameters, the example
that you chose, I immediately start thinking of nonlinear mixed
effects models. In my experience the only way in which a nonlinear
model ends up with that number of parameters is through applying an
underlying model with a low number of parameters to various groups
within the data. Table 2 in the Schnute et al. paper to which you
make reference states that the number of parameters in the model is T
+ A + 5 where T is the number of years of data and A is the number of
age classes. To me that looks a lot like a nonlinear mixed effects
model.
Also, your choice of subject heading for your message seems
deliberatively provocative. You seem to be implying that you are
discussing a comparisons of AD Model Builder and R on all aspects of
nonlinear statistical modeling but you are only discussing one
comparison on simulated data using a model from the applications area
for which you wrote AD Model Builder. Then you follow up by saying "I
am not proficient in R" and your results for R are from applying code
that someone else gave you.
It seems that ADMB had a bit of a "home-field advantage" in this
particular comparison.
I view nonlinear statistical modeling differently. I have had a bit
of experience in the area and I find that I want to examine the data
carefully, usually through plots, before I embark on fitting
complicated models. I like to have some assurance that the model
makes sense in the context of the data. (In your example you don't
need to worry about appropriateness of the model because the data were
simulated.) I would never try to fit a nonlinear model with 100
parameters to data without carefully examining the data, and
especially selected subsets of the data, first. For this the
flexibility of the S language and tools like lattice graphics that
were developed in this language are invaluable to me. The flexibility
of data manipulation and graphics for interactive exploration of data
is what attracted me to S in the first place.
I realize that for many people the area of nonlinear statistical
modeling is reduced to "Fit this model to these data and don't ask any
questions. Just give me parameter estimates and p-values." If that
is your situation then it would make sense to use software that gets
you those estimates as quickly as possible with a minimum of effort.
I'm just happy that I get to turn down people who ask me to do that.
I like that fact that I can spend my time asking questions about the
data and of the data.
> However I can say that providing derivatives for such a model is a
> highly nontrivial exercise. As I said in my posting, the R script and
> data are available to anyone who feels that the exercise was not carried
> out properly and would like to improve on it. Also one does not need
> to provide derivatives to the AD Model Builder program.
>
> Finally suppose that you are very good at calculating derivatives and
> manage to get them right. Then someone else comes along who wants to
> modify the model. Unless they are also very good at calculating
> derivatives there will be trouble.
>
> >
> > I'm curious -- when you say "R may not be a suitable platform for
> development for such models", what aspect of R do you feel is lacking?
> Is it the specific optimization routines available, or is it some other
> more general aspect?
>
> 2 seconds vs 90 minutes. For a real problem of tihs type the timings
> would probably be something like 10 minutes vs more than 2,700 minutes.
>
> >
> > Also, another optimization algorithm available in R is the "L-BFGS-B"
> method for optim() in the MASS package. I've had extremely good
> experiences with using this code in S-PLUS. It can take box
> constraints, and can use gradient information. It is my first choice
> for most optimization problems, and I believe it is very widely used.
> Did you try using that optimization routine with this problem?
> >
> > -- Tony Plate
> >
> > dave fournier wrote:
> >> There has recently been some discussion on the list about
> >> AD Model builder and the suitability of R for constructing the
> >> types of models used in fisheries management.
> >>
> >> https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html
> >>
> >> https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html
> >>
> >> I think that many R users understimate the numerical challenges
> >> that some of the typical nonlinear statistical model used in different
> >> fields present. R may not be a suitable platform for development for
> >> such models.
> >>
> >> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
> >> with Canadian federal fisheries undertook an investigation
> >> comparing various statistical modeling packages for a simple
> >> age-structured statistical model of the type commonly used in
> >> fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
> >> Splus. Unfortunately a working model could not be produced with Splus
> >> so its times could not be included in the comparison. It is possible
> >> to produce a working model with the present day version of R so that
> >> R can now be directly compared with AD Model Builder for this type
> of model.
> >>
> >> I have put the results of the test together with the original
> >> Schnute and Richards paper and the working R and AD Model Builder
> >> codes on Otter's web site
> >>
> >> http://otter-rsch.ca/tresults.htm
> >>
> >> The results are that AD Model builder is roughly 1000 times faster than
> >> R for this problem. ADMB takes about 2 seconds to converge while
> >> R takes over 90 minutes.
> >>
> >> This is a simple toy example. Real fisheries models are often hundred of
> >> times more computationally intensive as this one.
> >>
> >> Cheers,
> >>
> >> Dave
> >> ~
> >
>
>
> --
> David A. Fournier
> P.O. Box 2040,
> Sidney, B.C. V8l 3S3
> Canada
> Phone/FAX 250-655-3364
> http://otter-rsch.com
>
> --
> David A. Fournier
> P.O. Box 2040,
> Sidney, B.C. V8l 3S3
> Canada
> Phone/FAX 250-655-3364
> http://otter-rsch.com
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
More information about the R-help
mailing list