[Rd] S3 lookup rules changed in R 3.6.1

[Konrad Rudolph]

tl;dr: S3 lookup no longer works in custom non-namespace environments as of
R 3.6.1. Is this a bug?

I am implementing S3 dispatch for generic methods in environments that are
not
packages. I am trying to emulate the R package namespace mechanism by
having a
“namespace” environment that defines generics and methods, but only exposes
the
generics themselves, not the methods.

To make S3 lookup work when using the generics, I am using
`registerS3method`.
While this method itself has no extensive documentation, the documentation
of
`UseMethod` contains this relevant passage:

> Namespaces can register methods for generic functions. To support this,
> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
> environment in which the generic function is called, and in the
registration
> data base for the environment in which the generic is defined (typically a
> namespace). So methods for a generic function need to be available in the
> environment of the call to the generic, or they must be registered. (It
does
> not matter whether they are visible in the environment in which the
generic is
> defined.) As from R 3.5.0, the registration data base is searched after
the
> top level environment (see ‘topenv’) of the calling environment (but
before
> the parents of the top level environment).

This used to work but it stopped working in R 3.6.1 and I cannot figure out
(a)
why, and (b) how to fix it. Unfortunately I am unable to find the relevant
information by reading the R source code, even when “diff”ing what seem to
be
the only even remotely relevant changes [1].

The R NEWS merely list the following change for R 3.6.0:

> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
delayed
>   S3 method registration.
> […]
> * Method dispatch uses more relevant environments when looking up class
>   definitions.

Unfortunately it is not clear to me what exactly this means.

Here’s a minimal example code that works under R 3.5.3 but breaks under
R 3.6.1
(I don’t know about 3.6.0).

```
# Define “package namespace”:
ns = new.env(parent = .BaseNamespaceEnv)
local(envir = ns, {
    test = function (x) UseMethod('test')
    test.default = function (x) message('test.default')
    test.foo = function (x) message('test.foo')

    .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
    .__S3MethodsTable__.$test.default = test.default
    .__S3MethodsTable__.$test.foo = test.foo

    # Or, equivalently:
    # registerS3method('test', 'default', test.default)
    # registerS3method('test', 'foo', test.foo)
})

# Expose generic publicly:
test = ns$test

# Usage:
test(1)
test(structure(1, class = 'foo'))
```

Output in R up to 3.5.3:

```
test.default
test.foo
```

Output in R 3.6.1:

```
Error in UseMethod("test") :
  no applicable method for 'test' applied to an object of class
"c('double', 'numeric')"
```

It’s worth noting that the output of `.S3methods` is the same for all R
versions, and from my understanding of its output, this *should* indicate
that
S3 lookup should behave identically, too. Furthermore, lookup via
`getS3method`
succeeds in all R versions, and (again, in my understanding) the logic of
this
function should be identical to the logic of R’s internal S3 dispatch:

```
getS3method('test', 'default')(1)
getS3method('test', 'foo')(1)
```

Conversely, specialising an existing generic from a loaded package works.
E.g.:

```
local(envir = ns, {
    print.foo = function (x) message('print.foo')
    registerS3method('print', 'foo', print.foo)
})

print(structure(1, class = 'foo'))
```

This prints “print.foo” in all R versions as expected.

So my question is: Why do the `test(…)` calls in R 3.6.1 no longer trigger
S3
method lookup in the generic function’s environment? Is this behaviour by
design
or is it a bug? If it’s by design, why does `getS3method` still use the old
behaviour? And, most importantly, how can I fix my definition of `ns` to
make
S3 dispatch for non-exposed methods work again?

… actually I just found a workaround:

```
ns$.packageName = 'not important'
```

This marks `ns` as a package namespace. To me, the documentation seems to
imply
that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
the
code for `registerS3method` explicitly supports non-package namespace
environments. Unfortunately this workaround is not satisfactory because
pretending that the environment is a package namespace, when it really
isn’t,
might break other things.

[1] See r75273; there’s also r74625, which changes the actual lookup
mechanism
    used by `UseMethod`, but that seems even less relevant, because it is
    disabled unless a specific environment variable is set.

-- 
Konrad Rudolph

	[[alternative HTML version deleted]]