[Rd] S3 lookup rules changed in R 3.6.1

[Duncan Murdoch]

On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:
> tl;dr: S3 lookup no longer works in custom non-namespace environments as of
> R 3.6.1. Is this a bug?

I don't know whether this was intentional or not, but a binary search 
through the svn commits finds that the errors started in this one:

------------------------------------------------------------------------
r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines
Changed paths:
    M /trunk/src/main/objects.c
    M /trunk/tests/reg-tests-1a.R

Have S3 methods lookup by default look for the S3 registry in the topenv
of the generic.
------------------------------------------------------------------------

Duncan Murdoch

> 
> I am implementing S3 dispatch for generic methods in environments that are
> not
> packages. I am trying to emulate the R package namespace mechanism by
> having a
> “namespace” environment that defines generics and methods, but only exposes
> the
> generics themselves, not the methods.
> 
> To make S3 lookup work when using the generics, I am using
> `registerS3method`.
> While this method itself has no extensive documentation, the documentation
> of
> `UseMethod` contains this relevant passage:
> 
>> Namespaces can register methods for generic functions. To support this,
>> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
>> environment in which the generic function is called, and in the
> registration
>> data base for the environment in which the generic is defined (typically a
>> namespace). So methods for a generic function need to be available in the
>> environment of the call to the generic, or they must be registered. (It
> does
>> not matter whether they are visible in the environment in which the
> generic is
>> defined.) As from R 3.5.0, the registration data base is searched after
> the
>> top level environment (see ‘topenv’) of the calling environment (but
> before
>> the parents of the top level environment).
> 
> This used to work but it stopped working in R 3.6.1 and I cannot figure out
> (a)
> why, and (b) how to fix it. Unfortunately I am unable to find the relevant
> information by reading the R source code, even when “diff”ing what seem to
> be
> the only even remotely relevant changes [1].
> 
> The R NEWS merely list the following change for R 3.6.0:
> 
>> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
> delayed
>>    S3 method registration.
>> […]
>> * Method dispatch uses more relevant environments when looking up class
>>    definitions.
> 
> Unfortunately it is not clear to me what exactly this means.
> 
> Here’s a minimal example code that works under R 3.5.3 but breaks under
> R 3.6.1
> (I don’t know about 3.6.0).
> 
> ```
> # Define “package namespace”:
> ns = new.env(parent = .BaseNamespaceEnv)
> local(envir = ns, {
>      test = function (x) UseMethod('test')
>      test.default = function (x) message('test.default')
>      test.foo = function (x) message('test.foo')
> 
>      .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
>      .__S3MethodsTable__.$test.default = test.default
>      .__S3MethodsTable__.$test.foo = test.foo
> 
>      # Or, equivalently:
>      # registerS3method('test', 'default', test.default)
>      # registerS3method('test', 'foo', test.foo)
> })
> 
> # Expose generic publicly:
> test = ns$test
> 
> # Usage:
> test(1)
> test(structure(1, class = 'foo'))
> ```
> 
> Output in R up to 3.5.3:
> 
> ```
> test.default
> test.foo
> ```
> 
> Output in R 3.6.1:
> 
> ```
> Error in UseMethod("test") :
>    no applicable method for 'test' applied to an object of class
> "c('double', 'numeric')"
> ```
> 
> It’s worth noting that the output of `.S3methods` is the same for all R
> versions, and from my understanding of its output, this *should* indicate
> that
> S3 lookup should behave identically, too. Furthermore, lookup via
> `getS3method`
> succeeds in all R versions, and (again, in my understanding) the logic of
> this
> function should be identical to the logic of R’s internal S3 dispatch:
> 
> ```
> getS3method('test', 'default')(1)
> getS3method('test', 'foo')(1)
> ```
> 
> Conversely, specialising an existing generic from a loaded package works.
> E.g.:
> 
> ```
> local(envir = ns, {
>      print.foo = function (x) message('print.foo')
>      registerS3method('print', 'foo', print.foo)
> })
> 
> print(structure(1, class = 'foo'))
> ```
> 
> This prints “print.foo” in all R versions as expected.
> 
> So my question is: Why do the `test(…)` calls in R 3.6.1 no longer trigger
> S3
> method lookup in the generic function’s environment? Is this behaviour by
> design
> or is it a bug? If it’s by design, why does `getS3method` still use the old
> behaviour? And, most importantly, how can I fix my definition of `ns` to
> make
> S3 dispatch for non-exposed methods work again?
> 
> … actually I just found a workaround:
> 
> ```
> ns$.packageName = 'not important'
> ```
> 
> This marks `ns` as a package namespace. To me, the documentation seems to
> imply
> that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
> the
> code for `registerS3method` explicitly supports non-package namespace
> environments. Unfortunately this workaround is not satisfactory because
> pretending that the environment is a package namespace, when it really
> isn’t,
> might break other things.
> 
> [1] See r75273; there’s also r74625, which changes the actual lookup
> mechanism
>      used by `UseMethod`, but that seems even less relevant, because it is
>      disabled unless a specific environment variable is set.
>