[Rd] R-devel Digest, Vol 246, Issue 6

[Mikael Jagan]

Two quite related and recent threads on R-devel:

[1] proposal for WRE: clarify that use of S4 classes implies use of superclasses
https://stat.ethz.ch/pipermail/r-devel/2023-July/082739.html
[2] Should package version requirements assume installation from sources?
https://stat.ethz.ch/pipermail/r-devel/2022-September/081971.html

Notably, if SeuratObject 4.1.3, the current version, had

     importClassesFrom(Matrix, dgCMatrix,
                       ## and the exported superclasses:
                       CsparseMatrix, Matrix, compMatrix, dMatrix,
                       dsparseMatrix, generalMatrix, sparseMatrix)

and not just

     importClassesFrom(Matrix, dgCMatrix)

then the silent breakage due to missing symbols would not have occurred.
That is because _imported_ class definitions are retrieved at load time,
not at install time.  SeuratObject 4.1.3 retrieves the CsparseMatrix
definition at install time, so indeed it is liable to become stale with
Matrix updates ...

Not to derail Dirk's proposal for "Breaks", which could certainly have
uses beyond this particular example where a "correct" NAMESPACE directive
would solve the problems ...

Mikael

On 2023-08-07 6:00 am, r-devel-request using r-project.org wrote:
> I would support this suggestion. There is a similar binary dependency chain from 
> Matrix → TMB → glmmTMB; we have implemented various checks to make users aware 
> that they need to reinstall from source, and to some extent we've tried to push 
> out synchronous updates (i.e., push an update of TMB to CRAN every time Matrix 
> changes, and an update of glmmTMB after that), but centralized machinery for 
> this would certainly be nice. FWIW some of the machinery is here: 
> https://github.com/glmmTMB/glmmTMB/blob/d9ee7b043281341429381faa19b5e53cb5a378c3/glmmTMB/R/utils.R#L209-L295 -- it relies on a Makefile rule that caches the current installed version of TMB: https://github.com/glmmTMB/glmmTMB/blob/d9ee7b043281341429381faa19b5e53cb5a378c3/glmmTMB/R/utils.R#L209-L295 cheers Ben Bolker On 2023-08-06 5:05 p.m., Dirk Eddelbuettel wrote:
>> CRAN, by relying on the powerful package management system that is part of R,
>> provides an unparalleled framework for extending R with nearly 20k packages.
>>
>> We recently encountered an issue that highlights a missing element in the
>> otherwise outstanding package management system. So we would like to start a
>> discussion about enhancing its feature set. As shown below, a mechanism to
>> force reinstallation of packages may be needed.
>>
>> A demo is included below, it is reproducible in a container. We find the
>> easiest/fastest reproduction is by saving the code snippet below in the
>> current directory as eg 'matrixIssue.R' and have it run in a container as
>>
>>      docker run --rm -ti -v `pwd`:/mnt rocker/r2u Rscript /mnt/matrixIssue.R
>>     
>> This runs in under two minutes, first installing the older Matrix, next
>> installs SeuratObject, and then by removing the older Matrix making the
>> (already installed) current Matrix version the default. This simulates a
>> package update for Matrix. Which, as the final snippet demonstrates, silently
>> breaks SeuratObject as the cached S4 method Csparse_validate is now missing.
>> So when SeuratObject was installed under Matrix 1.5.1, it becomes unuseable
>> under Matrix 1.6.0.
>>
>> What this shows is that a call to update.packages() will silently corrupt an
>> existing installation.  We understand that this was known and addressed at
>> CRAN by rebuilding all binary packages (for macOS and Windows).
>>
>> But it leaves both users relying on source installation as well as
>> distributors of source packages in a dire situation. It hurt me three times:
>> my default R installation was affected with unit tests (involving
>> SeuratObject) silently failing. It similarly broke our CI setup at work.  And
>> it created a fairly bad headache for the Debian packaging I am involved with
>> (and I surmise it affects other distro similarly).
>>
>> It would be good to have a mechanism where a package, when being upgraded,
>> could flag that 'more actions are required' by the system (administrator).
>> We think this example demonstrates that we need such a mechanism to avoid
>> (silently !!) breaking existing installations, possibly by forcing
>> reinstallation of other packages.  R knows the package dependency graph and
>> could trigger this, possibly after an 'opt-in' variable the user / admin
>> sets.
>>
>> One possibility may be to add a new (versioned) field 'Breaks:'. Matrix could
>> then have added 'Breaks: SeuratObject (<= 4.1.3)' preventing an installation
>> of Matrix 1.6.0 when SeuratObject 4.1.3 (or earlier) is present, but
>> permitting an update to Matrix 1.6.0 alongside a new version, say, 4.1.4 of
>> SeuratObject which could itself have a versioned Depends: Matrix (>= 1.6.0).
>>
>> Regards,  Dirk
>>
>>
>> ## Code example follows. Recommended to run the rocker/r2u container.
>> ## Could also run 'apt update -qq; apt upgrade -y' but not required
>> ## Thanks to my colleague Paul Hoffman for the core of this example
>>
>> ## now have Matrix 1.6.0 because r2u and CRAN remain current but we can install an older Matrix
>> remotes::install_version('Matrix', '1.5.1')
>>
>> ## we can confirm that we have Matrix 1.5.1
>> packageVersion("Matrix")
>>
>> ## we now install SeuratObject from source and to speed things up we first install the binary
>> install.packages("SeuratObject")   # in this container via bspm/r2u as binary
>> ## and then force a source installation (turning bspm off) _while Matrix is at 1.5.1_
>> if (requireNamespace("bspm", quietly=TRUE) bspm::disable()
>> Sys.setenv(PKG_CXXFLAGS='-Wno-ignored-attributes') 	# Eigen compilation noise silencer
>> install.packages('SeuratObject')
>>
>> ## we now remove the Matrix package version 1.5.1 we installed into /usr/local leaving 1.6.0
>> remove.packages("Matrix")
>> packageVersion("Matrix")
>>
>> ## and we now run a bit of SeuratObject code that is now broken as Csparse_validate is gone
>> suppressMessages(library(SeuratObject))
>> data('pbmc_small')
>> graph <- pbmc_small[['RNA_snn']]
>> class(graph)
>> getClass('Graph')
>> show(graph) # this fails
>>
>>