[R-pkg-devel] Error when install binary from CRAN but not if install from source - Macintosh, lme4, Matrix, buildmer

[Martin Maechler]

>>>>> Duncan Murdoch 
>>>>>     on Wed, 19 Oct 2022 14:55:24 -0400 writes:

    > It seems that this could be mostly avoided if instead of caching copies 
    > of Matrix methods when buildmer was installed, R would cache promises to 
    > get those methods.  Then if the method for some particular signature 
    > changed, on first use buildmer would retrieve the current version.

    > There would still be a problem if Matrix completely dropped a method or 
    > it migrated to a different package, but I think that's less frequent 
    > than a change to the internal implementation.

    > Has any thought been given to making this change for R 4.3.0?

    > Duncan Murdoch

Not that I know of.

It sounds like a very promising (pun! ;-) idea, that could solve
many subtle  inter-package installation/update  problems.

Note that the methods *and* classes  caching had been introduced
a longer while ago as part of a concerted effort to make the
loading of packages with (many) S4 methods and classes considerably faster.

Before that effort, all these class and method "tables"/"lists"
(environments effectively) were (re-)computed at package load
time which was quite costly [time consuming].
I was not very much involved in that effort and I don't know if
using promises instead had been considered at the time.

(I've added Michael and the other Martin M to the CC , hoping
 they will know / remember more )

Martin Maechler


    > On 19/10/2022 3:39 a.m., Martin Maechler wrote:
    >>>>>>> Duncan Murdoch
    >>>>>>> on Tue, 18 Oct 2022 15:17:33 -0400 writes:
    >> 
    >> > I see the same thing.  This sounds like a problem in the handling of
    >> > methods that has been discussed somewhat recently:
    >> 
    >> > https://stat.ethz.ch/pipermail/r-devel/2022-September/081971.html
    >> 
    >> > The problem is that when the binary is built, some code from other
    >> > packages is kept as part of it.  When that other package is updated, you
    >> > need a new source install of your own package (or a rebuild at CRAN to
    >> > replace the binary) to cache the new code.
    >> 
    >> > This can also be done explicitly by package startup code; I think this
    >> > thread
    >> 
    >> > https://stat.ethz.ch/pipermail/r-package-devel/2022q3/008481.html
    >> 
    >> > turned out to be a case where rstan was caching something, and an update
    >> > to ggplot2 didn't work with the cached data.  Given the error message,
    >> > your issue sounds more like the first one.
    >> 
    >> > I don't recall if there was a resolution.  Maybe you can ask the CRAN
    >> > maintainers to rebuild buildmer.
    >> 
    >> > Duncan Murdoch
    >> 
    >> Yes, thank you, Duncan!
    >> 
    >> The binary version of 'buildmer' (or also lme4 ?) must have been created
    >> with an older version of Matrix  (which did not have the
    >> dgeMatrix_getDiag C code)  and the respective maintainer of the
    >> binary builds (*) must rebuild the binary versions of those
    >> Matrix-reverse-dependant packages,
    >> ---
    >> *) in this case the macOS ones (both for Intel and ARM Macs)
    >> 
    >> Martin Maechler
    >> (maintainer of Matrix)
    >> 
    >> 
    >> 
    >> 
    >> 
    >> 
    >> 
    >> 
    >> > On 18/10/2022 2:51 p.m., Carl Schwarz wrote:
    >> >> I've run into a problem where if you install the lme4, Matrix, and buildmer
    >> >> packages using the binaries from CRAN on a Mac, I get an error message
    >> >> about a missing method, but if I install the same packages from SOURCE, the
    >> >> code runs fine.
    >> >>
    >> >> I would have thought that installing from source or using the binary
    >> >> should look the same.
    >> >>
    >> >> Any suggestions on how to proceed to resolve this issue?
    >> >>
    >> >> The maintainer of buildmer is also puzzled.
    >> >> You can follow the issue in more detail at:
    >> >> https://github.com/cvoeten/buildmer/issues/20
    >> >>
    >> >> This is way above my paygrade...
    >> >>
    >> >> Carl Schwarz
    >> >>
    >> >> ------------------------------------------------------
    >> >>
    >> >> Tried this on an intel-Mac and arm-Mac with the same result.
    >> >> Works fine on Windows machines under both scenarios.
    >> >>
    >> >> Here is a test example
    >> >>
    >> >> library(buildmer)
    >> >> library(lme4)
    >> >>
    >> >> nrow <- 100
    >> >>
    >> >> test <- data.frame(x01=runif(nrow),
    >> >> y=runif(nrow)<.1, block=as.factor(floor((1:nrow)/50)))
    >> >> head(test)
    >> >>
    >> >>
    >> >> fit.model <- lme4::glmer(y~x01 + (1|block), data=test,
    >> >> family=binomial(link="logit"))
    >> >> fit.model # this works
    >> >>
    >> >> class(fit.model)
    >> >>
    >> >> summary(fit.model)
    >> >> # gives the following error message
    >> >> Error in diag(from, names = FALSE) : object 'dgeMatrix_getDiag' not found
    >> >>
    >> >> A pdf document showing output is attached (shows the sessionInfo etc).
    >> >>
    >> >> When you install the buildmer and Matrix packages from SOURCE, it runs fine.
    >> >>
    >> >> I've tried all combinations of installing binary/source and only if both
    >> >> packages (Matrix and buidmer) are installed from source, does the code run.
    >> >>
    >> >> Same issue on an intel-Mac.
    >> >> Same issue when running under the R directly rather than Rstudio on a Mac.