[Rd] Automatic Differentiation for R

[Martin Maechler]

[MM stumbling over on old thread ... he'd be interested]

>>>>> "GaGr" == Gabor Grothendieck <ggrothendieck at gmail.com>
>>>>>     on Wed, 15 Apr 2009 09:53:18 -0400 writes:

    GaGr> Not sure if this is sufficient for your needs but R does include symbolic
    GaGr> differentiation, see ?D, and the Ryacas and rSymPy
    GaGr> packages interface R to the yacas and sympy computer algebra
    GaGr> systems (CAS) and those system include symbolic differentiation.

No, symbolic differentiation is not enough.
Automatic Differentiation (AD) is something much more general (in one
way) and much less mathematical from  a classical view point:
But then, AD is much more generally useful for minimization as, basically,
the input is an R function 
    f(x)     	       {with x multidimensional}
or  f(x1,x2, ..., xp)  {with scalar x1, x2, ..}
and the output is again an R function
which computes f() and all {or just selected} partial
derivatives  d f / d{xi}.

Now consider that the function f()  can contain if() and while()
clauses and conceptually ever language feature of R.
In practice, I'm pretty sure the list of features would have to
be restricted, similarly as they'd have to for an R compiler to
be feasible.

I agree that  AD for R would be very nice and could be very
useful.
I'd also be interested to help AD people learn the S4 classes
and methods (hoping that it's close enough to what they call
"operator overloading" something I'd presume to be less general
than the powerful S4 class/methods system).

Martin Maechler, ETH Zurich

    GaGr> http://ryacas.googlecode.com
    GaGr> http://rsympy.googlecode.com
    
    [.............]

    GaGr> On Wed, Apr 15, 2009 at 9:30 AM, John C Nash <nashjc at uottawa.ca> wrote:
    >> In efforts to improve optimization tools for R, one of my
    >> interests has been getting automatic differentiation capabilities
    >> so that analytic rather than numerical derivatives can be used. They
    >> would be helpful in several other areas besides optimization, My timings
    >> show
    >> factors of the order of 1000s in time improvements by avoiding
    >> numerical derivatives in some cases.
    >> 
    >> There has been some work in this e.g.,
    >> http://code.google.com/p/pbs-software/
    >> is an R interface to ADMB (Automatic Differentiation Model Builder).
    >> However,
    >> as far as I can see, this is directed essentially to nonlinear least squares
    >> modelling,
    >> an important but not general problem.
    >> 
    >> Tom Coleman of Waterloo responded favourably with some advice, but the most
    >> enthusiastic answer came from Shaun Forth, which I have included below. I
    >> read
    >> this as an opportunity to develop what could be a profitable collaboration
    >> with
    >> the AD community. Unfortunately, I cannot take up the invitation to join the
    >> AD
    >> folk in Oxford due to a pre-existing obligation. Nor am I more than a
    >> complete
    >> novice with S3 and S4 classes etc. I am, nevertheless, willing to help
    >> organize
    >> the effort e.g., do some of the communications, chasing grant money, getting
    >> Google Summer of Code applications filled in etc.
    >> 
    >> Can the R community come up with a few people who can provide the AD
    >> workers with appropriate information? If so, is there a reasonable chance to
    >> generate sufficient funding for a student? I suspect the answer in both
    >> cases
    >> is yes, but that we need some form of "booster cables" to get things going.
    >> (In Canada, booster cables are used to get cars started in winter by
    >> connecting
    >> a running vehicle's battery to that of a dead one.)
    >> 
    >> I suggest communications off-list until there is progress to report.
    >> Possibly
    >> there is a better forum for this -- suggestions welcome.
    >> 
    >> John Nash
    >> 
    >> ---- included msg from Shaun Forth ---
    >> 
    >> Hi John,
    >> My computational statistics colleague Trevor Ringrose has asked me to
    >> consider AD in R in the past. As you may or may not be aware AD is
    >> implemented in one of two ways: overloading using OO features of the
    >> target language, or source transformation using compiler tools (after
    >> several man years of development) to read in the target code and spit
    >> out the differentiated code. Last time I looked I didn't think the
    >> object oriented features of R were up to overloading but on checking
    >> today I can see that this might now be possible (I can see overloading
    >> of arithmetic operators and functions for example now which I didn't see
    >> last time).
    >> I'd certainly be interested in following this up particularly on the
    >> overloading side but would need to get funding for a PhD student to do
    >> the graft. It would be particularly interesting doing this in an open
    >> source language because we could then perhaps tweak some of the core
    >> language features if they didn't seamlessly support the AD (we can't do
    >> this in Matlab and that is a pain!).
    >> 
    >> My immediate suggestion is that you, or some other more local (to UK) R
    >> expert talks at the next European AD workshop in Oxford
    >> http://www.autodiff.org/?module=Workshops&submenu=EuroAD/8/main
    >> We're a very friendly group and I'm sure there are others who might like
    >> to tackle R or perhaps we could put together a multigroup project. If
    >> someone could give a talk on R, its language features including the OO
    >> aspects, and some optimisation examples with associated code, the group
    >> there would be able to give you the best feedback on the planet on the
    >> possibilities.
    >> 
    >> Please do treat this as a positive response and let's keep in touch on
    >> this.
    >> 
    >> Regards
    >> 
    >> Shaun
    >> 
    >> 
    >> ####################################################################
    >> Dr Shaun Forth
    >> Applied Mathematics & Scientific Computation
    >> Cranfield Defence and Security Cranfield University, Shrivenham Campus
    >> Swindon SN6 8LA, England
    >> tel: +44 (0)1793 785311
    >> fax: +44 (0)1793 784196
    >> email: S.A.Forth at cranfield.ac.uk
    >> http://www.amorg.co.uk
    >> #####################################################################
    >>