[Rd] Should package version requirements assume installation from sources?

[Duncan Murdoch]

This issue may be the culprit in an obscure bug that's been reported on 
R-package-devel:  see 
https://stat.ethz.ch/pipermail/r-package-devel/2022q3/008481.html.  It 
appears that some ggplot2 version 3.4.0 code is being run even though 
3.3.6 is the version on CRAN, and 3.3.6 is *also* being run in the same 
check session.

Duncan Murdoch

On 14/09/2022 6:04 a.m., Duncan Murdoch wrote:
> On 13/09/2022 5:45 p.m., Mikael Jagan wrote:
>> [Arguably also appropriate for R-package-devel, but posted to R-devel
>>        as the discussion is aimed primarily at "experts" ... ]
>>
>> We, the authors of Matrix, have encountered a somewhat subtle issue
>> induced by caching of S4 classes and methods in package namespaces.
>>
>> The namespaces of three reverse dependent packages (SeuratObject, conText,
>> mcmcsae) cache the formal definition of our virtual class Matrix (and some
>> subclasses).  For example:
>>
>>    > ns <- asNamespace("SeuratObject")
>>    > grep("^[.]__C__.*Matrix$", names(ns), value = TRUE)
>> [1] ".__C__dMatrix"       ".__C__compMatrix"    ".__C__AnyMatrix"
>> [4] ".__C__generalMatrix" ".__C__CsparseMatrix" ".__C__sparseMatrix"
>> [7] ".__C__dsparseMatrix" ".__C__Matrix"
>>
>> The cached definition (which includes a _validity method_) is obtained from
>> the version of Matrix available when the reverse dependent package was built
>> from sources.  For example, if SeuratObject was built under Matrix 1.4-1,
>> then we get:
>>
>>    > getValidity(ns$.__C__Matrix)
>> function (object)
>> {
>>        if (!isTRUE(r <- .Call(Dim_validate, object, "Matrix")))
>>            r
>>        else .Call(dimNames_validate, object)
>> }
>> <bytecode: 0x11e7ca508>
>> <environment: namespace:Matrix>
>>
>> whereas if SeuratObject was built under Matrix >= 1.5-0, then we get:
>>
>>    > getValidity(ns$.__C__Matrix)
>> function (object)
>> .Call(Matrix_validate, object)
>> <bytecode: 0x107dc1698>
>> <environment: namespace:Matrix>
>>
>> There are two "questions" here:
>>
>> 1.  The symbol 'Matrix_validate' is not defined until Matrix 1.5-0.
>>        Is it necessary, for this reason alone, for SeuratObject to have
>>        'Imports: Matrix (>= 1.5-0)'?  Or can SeuratObject continue using
>>        'Imports: Matrix (>= 1.3-3)', at the risk of errors like
>>
>>        > Error: object 'Matrix_validate' not found
>>
>>        (as already seen here: https://stackoverflow.com/questions/73700130)?
>>
>>        Note that this error would not occur for anyone installing SeuratObject
>>        from sources, unless they decide to _downgrade_ Matrix after doing so.
>>        Hence this primarily concerns Windows and macOS where R users would
>>        typically install a binary built by CRAN (i.e., not on their system).
>>
>>        We are aware that package TMB tests in .onLoad() that the current Matrix
>>        version is equal to or greater than the version available at build time,
>>        thus avoiding a "strict" version requirement, but do not want this practice
>>        to spread ...
>>
>> 2.  For how long should Matrix retain the superceded 'Dim_validate' and
>>        'dimNames_validate', in order to ensure that "stale" cached validity
>>        methods continue to work?
>>
>> We hope that this discussion will highlight the potential ramifications
>> of importing classes and methods from other packages, and having one's
>> classes and methods imported _by_ other packages, especially for version
>> requirements.
> 
> This sounds like a bug or bad design in the S4 system, i.e. caching
> things without a way to detect or update when the cache becomes stale.
> 
> Is it really necessary to cache things as part of the binary package, or
> could they be put in place when needed using lazy loading, getting a
> copy from the loaded copy of Matrix?
> 
> Duncan Murdoch