[R-SIG-Mac] utils:::.available.both() breaks install.packages() on Mac sonoma-arm64
Simon Urbanek
@|mon@urb@nek @end|ng |rom R-project@org
Thu Apr 23 01:58:22 CEST 2026
Hervé,
"both" has been always based on the (bad) assumption that binaries are a subset of sources. Before the update, it has simply ignored binaries entirely when determining the set of packages (i.e. both old.packages and available.packages were always wrong for "both" as they simply ran on "source"). It causes a lot of problems which is why my preferred solution would be to remove "both" entirely as the concept is fundamentally broken (see PR#18396), but since the horse bolted we have to deal with it for now to make it at least seem like it's working. As you noted, In practice, it is close to impossible to maintain that assumption, because those are two entirely independent processes, so archiving etc. would have to be done in a different way by removing binaries first and waiting for it to propagate - which won't happen anytime soon.
> On 23 Apr 2026, at 04:18, Hervé Pagès <hpages.on.github using gmail.com> wrote:
>
> FWIW more than 17 hrs after my initial report, the CRAN mirrors still have hundreds of sonoma-arm64 packages with no corresponding sources:
>
Your report was about the code - it was appreciated and fixed by Peter if I see correctly. As noted above, perfect match is not expected. However, the macOS repo issue is entirely different - it turns out the PACKAGES* metadata files were (incorrectly) listing orphans which are not in that directory, that's why there were so many, now fixed:
repos <- "https://cran.r-project.org"
av.src <- available.packages(contrib.url(repos, "source"))
bin_types <- c(Windows="win.binary", Mac="mac.binary.sonoma-arm64")
master <- c(repos, "https://mac.r-project.org")
for (i in seq_along(bin_types)) {
av.bin <- available.packages(contrib.url(master[i], bin_types[[i]]))
bin_pkgs <- setdiff(rownames(av.bin), rownames(av.src))
cat(names(bin_types)[[i]], "packages with no source:", length(bin_pkgs), "\n")
}
Windows packages with no source: 14
Mac packages with no source: 0
(and it's only 0 because I just synced after the build run while it's night in Europe and the northern CRAN team is sleeping :P).
Cheers,
Simon
> bin_pkgs_with_no_src <- function(repos)
> {
> av.src <- available.packages(contrib.url(repos, "source"))
> bin_types <- c(Windows="win.binary", Mac="mac.binary.sonoma-arm64")
> for (i in seq_along(bin_types)) {
> av.bin <- available.packages(contrib.url(repos, bin_types[[i]]))
> bin_pkgs <- setdiff(rownames(av.bin), rownames(av.src))
> cat(names(bin_types)[[i]], "packages with no source:", length(bin_pkgs), "\n")
> }
> }
>
> > bin_pkgs_with_no_src("https://cran.r-project.org")
> Windows packages with no source: 14
> Mac packages with no source: 817
>
> > bin_pkgs_with_no_src("https://cran.rstudio.com")
> Windows packages with no source: 14
> Mac packages with no source: 817
>
> So assuming that in normal operation the source dir should have a superset of the binary dir doesn't seem realistic.
>
> Best,
> H.
>
> On 22/04/2026 07:09, Uwe Ligges wrote:
>>
>>
>> On 22.04.2026 13:30, Peter Dalgaard wrote:
>>> Thanks for spotting this, Hervé.
>>>
>>> You are probably right that this condition should be checked for in the code.
>>>
>>> However, we are in code freeze and the direct cause of the issue is that the PACKAGES file in the binaries directory on CRAN is out of sync with it contents (AFAICS some Archived pakages had their binaries removed, but they remained listed in PACKAGES).
>>>
>>> In normal operation, the source dir should have a superset of the binary dir and the PACKAGES file likewise and the problem shouldn't appear.
>>
>>
>> Peter,
>>
>> there are separate PACKAGES files for the source repository and each of the binary repositories.
>> A PACKAGES file should always be consistent with the contents in the corresponding repository, of course. It must not contain entries for packages that are unavailable in that repo.
>> Technically, in order to avoid race conditions in the source repo, the entry in PACKAGES is even removed even before the package (or at least a link to it) is moved away from the source repository.
>>
>> For binary repositories: Binaries are not instantly removed from binary repositories if corresponding sources are moved to the archives. So you cannot expect that the source repo is always a superset of packages of a binary repo.
>>
>> Best,
>> Uwe
>>
>>>
>>> - Peter D.
>>>
>>>> On 22 Apr 2026, at 00.35, Hervé Pagès <hpages.on.github using gmail.com> wrote:
>>>>
>>>> With R 4.6 >= r89829 we're seeing this install.packages() error on Mac sonoma-arm64 in various places:
>>>>
>>>> Error in av.src[pkg.bin, "Version"] : subscript out of bounds
>>>>
>>>> This happens for example with install.packages("OSTA.data", repos=BiocManager::repositories()).
>>>>
>>>> Also started to happen in this GitHub action a few days ago: https://github.com/HyrienLab/igblastr/actions/runs/24495733061/job/71590105822
>>>>
>>>> The error occurs in the new utils:::.available.both() function introduced at r89827 in trunk and r89829 in R-4-6-branch (file src/library/utils/R/packages.R). It can simply be reproduced with:
>>>>
>>>> utils:::.available.both("https://cran.r-project.org")
>>>>
>>>> # Error in av.src[pkg.bin, "Version"] : subscript out of bounds
>>>>
>>>> It seems that the function should be able to handle the situation where some binary packages in the repo don't have a corresponding source package. Something like this maybe:
>>>>
>>>> .available.both <- function(repos, method, ...) {
>>>> ## "both" gets complicated as we have to use available.packages twice
>>>> fields <- c(tools:::.get_standard_repository_db_fields(), "File", "Built")
>>>> av.src <- available.packages(contriburl = contrib.url(repos, "source"), method = method, fields = fields, ...)
>>>> av.bin <- available.packages(contriburl = contrib.url(repos, "binary"), method = method, fields = fields, ...)
>>>> ## we keep things simple in that we use source unless there is the same binary version
>>>> ## so that we can get at the Built field. This is more simple than in install.packages
>>>> ## since we only care about determining whether a package is new, but not which type is
>>>> ## preferred - install.packages will sort that out.
>>>> all_src_pkgs <- row.names(av.src)
>>>> all_bin_pkgs <- row.names(av.bin)
>>>> dual_pkgs <- intersect(all_src_pkgs, all_bin_pkgs)
>>>> src_ver <- av.src[dual_pkgs, "Version"]
>>>> bin_ver <- av.bin[dual_pkgs, "Version"]
>>>> use_bin <- as.numeric_version(bin_ver) >= as.numeric_version(src_ver)
>>>> ## If for some reason 'src_ver' or 'bin_ver' contain NAs, they would propagate
>>>> ## to 'use_bin', but can this ever happen?
>>>> use_bin[is.na(use_bin)] <- TRUE
>>>> src_pkgs <- c(setdiff(all_src_pkgs, dual_pkgs), dual_pkgs[!use_bin])
>>>> bin_pkgs <- c(setdiff(all_bin_pkgs, dual_pkgs), dual_pkgs[use_bin])
>>>> ## Returned matrix has 1 row per package in 'union(all_src_pkgs, all_bin_pkgs)'.
>>>> rbind(av.src[src_pkgs, ], av.bin[bin_pkgs, ])
>>>> }
>>>>
>>>> Best,
>>>>
>>>> H.
>>>>
>>>> --
>>>> Hervé Pagès
>>>>
>>>> Bioconductor Core Team
>>>> hpages.on.github using gmail.com
>>>>
>>>> _______________________________________________
>>>> R-SIG-Mac mailing list
>>>> R-SIG-Mac using r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>>
>>
> --
> Hervé Pagès
>
> Bioconductor Core Team
> hpages.on.github using gmail.com
>
> _______________________________________________
> R-SIG-Mac mailing list
> R-SIG-Mac using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>
More information about the R-SIG-Mac
mailing list