[Rd] A demonstrated shortcoming of the R package management system

[Hadley Wickham]

Hi Dirk,

Do you think it's worth also/instead considering a fix to S4 to avoid
this caching issue in future R versions?

(This is top of my for me as we consider the design of S7, and I
recently made a note to ensure we avoid similar problems there:
https://github.com/RConsortium/OOP-WG/issues/317)

Hadley

On Sun, Aug 6, 2023 at 4:05 PM Dirk Eddelbuettel <edd using debian.org> wrote:
>
>
> CRAN, by relying on the powerful package management system that is part of R,
> provides an unparalleled framework for extending R with nearly 20k packages.
>
> We recently encountered an issue that highlights a missing element in the
> otherwise outstanding package management system. So we would like to start a
> discussion about enhancing its feature set. As shown below, a mechanism to
> force reinstallation of packages may be needed.
>
> A demo is included below, it is reproducible in a container. We find the
> easiest/fastest reproduction is by saving the code snippet below in the
> current directory as eg 'matrixIssue.R' and have it run in a container as
>
>    docker run --rm -ti -v `pwd`:/mnt rocker/r2u Rscript /mnt/matrixIssue.R
>
> This runs in under two minutes, first installing the older Matrix, next
> installs SeuratObject, and then by removing the older Matrix making the
> (already installed) current Matrix version the default. This simulates a
> package update for Matrix. Which, as the final snippet demonstrates, silently
> breaks SeuratObject as the cached S4 method Csparse_validate is now missing.
> So when SeuratObject was installed under Matrix 1.5.1, it becomes unuseable
> under Matrix 1.6.0.
>
> What this shows is that a call to update.packages() will silently corrupt an
> existing installation.  We understand that this was known and addressed at
> CRAN by rebuilding all binary packages (for macOS and Windows).
>
> But it leaves both users relying on source installation as well as
> distributors of source packages in a dire situation. It hurt me three times:
> my default R installation was affected with unit tests (involving
> SeuratObject) silently failing. It similarly broke our CI setup at work.  And
> it created a fairly bad headache for the Debian packaging I am involved with
> (and I surmise it affects other distro similarly).
>
> It would be good to have a mechanism where a package, when being upgraded,
> could flag that 'more actions are required' by the system (administrator).
> We think this example demonstrates that we need such a mechanism to avoid
> (silently !!) breaking existing installations, possibly by forcing
> reinstallation of other packages.  R knows the package dependency graph and
> could trigger this, possibly after an 'opt-in' variable the user / admin
> sets.
>
> One possibility may be to add a new (versioned) field 'Breaks:'. Matrix could
> then have added 'Breaks: SeuratObject (<= 4.1.3)' preventing an installation
> of Matrix 1.6.0 when SeuratObject 4.1.3 (or earlier) is present, but
> permitting an update to Matrix 1.6.0 alongside a new version, say, 4.1.4 of
> SeuratObject which could itself have a versioned Depends: Matrix (>= 1.6.0).
>
> Regards,  Dirk
>
>
> ## Code example follows. Recommended to run the rocker/r2u container.
> ## Could also run 'apt update -qq; apt upgrade -y' but not required
> ## Thanks to my colleague Paul Hoffman for the core of this example
>
> ## now have Matrix 1.6.0 because r2u and CRAN remain current but we can install an older Matrix
> remotes::install_version('Matrix', '1.5.1')
>
> ## we can confirm that we have Matrix 1.5.1
> packageVersion("Matrix")
>
> ## we now install SeuratObject from source and to speed things up we first install the binary
> install.packages("SeuratObject")   # in this container via bspm/r2u as binary
> ## and then force a source installation (turning bspm off) _while Matrix is at 1.5.1_
> if (requireNamespace("bspm", quietly=TRUE) bspm::disable()
> Sys.setenv(PKG_CXXFLAGS='-Wno-ignored-attributes')      # Eigen compilation noise silencer
> install.packages('SeuratObject')
>
> ## we now remove the Matrix package version 1.5.1 we installed into /usr/local leaving 1.6.0
> remove.packages("Matrix")
> packageVersion("Matrix")
>
> ## and we now run a bit of SeuratObject code that is now broken as Csparse_validate is gone
> suppressMessages(library(SeuratObject))
> data('pbmc_small')
> graph <- pbmc_small[['RNA_snn']]
> class(graph)
> getClass('Graph')
> show(graph) # this fails
>
>
> --
> dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org
>
> ______________________________________________
> R-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



-- 
http://hadley.nz