[R-SIG-Mac] utils:::.available.both() breaks install.packages() on Mac sonoma-arm64

[Hervé Pagès]

FWIW more than 17 hrs after my initial report, the CRAN mirrors still 
have hundreds of sonoma-arm64 packages with no corresponding sources:

bin_pkgs_with_no_src <- function(repos)
{
     av.src <- available.packages(contrib.url(repos, "source"))
     bin_types <- c(Windows="win.binary", Mac="mac.binary.sonoma-arm64")
     for (i in seq_along(bin_types)) {
         av.bin <- available.packages(contrib.url(repos, bin_types[[i]]))
         bin_pkgs <- setdiff(rownames(av.bin), rownames(av.src))
         cat(names(bin_types)[[i]], "packages with no source:", 
length(bin_pkgs), "\n")
     }
}

 > bin_pkgs_with_no_src("https://cran.r-project.org")
Windows packages with no source: 14
Mac packages with no source: 817

 > bin_pkgs_with_no_src("https://cran.rstudio.com")
Windows packages with no source: 14
Mac packages with no source: 817

So assuming that in normal operation the source dir should have a 
superset of the binary dir doesn't seem realistic.

Best,
H.

On 22/04/2026 07:09, Uwe Ligges wrote:
>
>
> On 22.04.2026 13:30, Peter Dalgaard wrote:
>> Thanks for spotting this, Hervé.
>>
>> You are probably right that this condition should be checked for in 
>> the code.
>>
>> However, we are in code freeze and the direct cause of the issue is 
>> that the PACKAGES file in the binaries directory on CRAN is out of 
>> sync with it contents (AFAICS some Archived pakages had their 
>> binaries removed, but they remained listed in PACKAGES).
>>
>> In normal operation, the source dir should have a superset of the 
>> binary dir and the PACKAGES file likewise and the problem shouldn't 
>> appear.
>
>
> Peter,
>
> there are separate PACKAGES files for the source repository and each 
> of the binary repositories.
> A PACKAGES file should always be consistent with the contents in the 
> corresponding repository, of course. It must not contain entries for 
> packages that are unavailable in that repo.
> Technically, in order to avoid race conditions in the source repo, the 
> entry in PACKAGES is even removed even before the package (or at least 
> a link to it) is moved  away from the source repository.
>
> For binary repositories: Binaries are not instantly removed from 
> binary repositories if corresponding sources are moved to the 
> archives. So you cannot expect that the source repo is always a 
> superset of packages of a binary repo.
>
> Best,
> Uwe
>
>>
>> - Peter D.
>>
>>> On 22 Apr 2026, at 00.35, Hervé Pagès <hpages.on.github using gmail.com> 
>>> wrote:
>>>
>>> With R 4.6 >= r89829  we're seeing this install.packages() error on 
>>> Mac sonoma-arm64 in various places:
>>>
>>>      Error in av.src[pkg.bin, "Version"] : subscript out of bounds
>>>
>>> This happens for example with install.packages("OSTA.data", 
>>> repos=BiocManager::repositories()).
>>>
>>> Also started to happen in this GitHub action a few days ago: 
>>> https://github.com/HyrienLab/igblastr/actions/runs/24495733061/job/71590105822
>>>
>>> The error occurs in the new utils:::.available.both() function 
>>> introduced at r89827 in trunk and r89829 in R-4-6-branch (file 
>>> src/library/utils/R/packages.R). It can simply be reproduced with:
>>>
>>> utils:::.available.both("https://cran.r-project.org")
>>>
>>>      # Error in av.src[pkg.bin, "Version"] : subscript out of bounds
>>>
>>> It seems that the function should be able to handle the situation 
>>> where some binary packages in the repo don't have a corresponding 
>>> source package. Something like this maybe:
>>>
>>> .available.both <- function(repos, method, ...) {
>>>      ## "both" gets complicated as we have to use available.packages 
>>> twice
>>>      fields <- c(tools:::.get_standard_repository_db_fields(), 
>>> "File", "Built")
>>>      av.src <- available.packages(contriburl = contrib.url(repos, 
>>> "source"), method = method, fields = fields, ...)
>>>      av.bin <- available.packages(contriburl = contrib.url(repos, 
>>> "binary"), method = method, fields = fields, ...)
>>>      ## we keep things simple in that we use source unless there is 
>>> the same binary version
>>>      ## so that we can get at the Built field. This is more simple 
>>> than in install.packages
>>>      ## since we only care about determining whether a package is 
>>> new, but not which type is
>>>      ## preferred - install.packages will sort that out.
>>>      all_src_pkgs <- row.names(av.src)
>>>      all_bin_pkgs <- row.names(av.bin)
>>>      dual_pkgs <- intersect(all_src_pkgs, all_bin_pkgs)
>>>      src_ver <- av.src[dual_pkgs, "Version"]
>>>      bin_ver <- av.bin[dual_pkgs, "Version"]
>>>      use_bin <- as.numeric_version(bin_ver) >= 
>>> as.numeric_version(src_ver)
>>>      ## If for some reason 'src_ver' or 'bin_ver' contain NAs, they 
>>> would propagate
>>>      ## to 'use_bin', but can this ever happen?
>>>      use_bin[is.na(use_bin)] <- TRUE
>>>      src_pkgs <- c(setdiff(all_src_pkgs, dual_pkgs), 
>>> dual_pkgs[!use_bin])
>>>      bin_pkgs <- c(setdiff(all_bin_pkgs, dual_pkgs), 
>>> dual_pkgs[use_bin])
>>>      ## Returned matrix has 1 row per package in 
>>> 'union(all_src_pkgs, all_bin_pkgs)'.
>>>      rbind(av.src[src_pkgs, ], av.bin[bin_pkgs, ])
>>> }
>>>
>>> Best,
>>>
>>> H.
>>>
>>> -- 
>>> Hervé Pagès
>>>
>>> Bioconductor Core Team
>>> hpages.on.github using gmail.com
>>>
>>> _______________________________________________
>>> R-SIG-Mac mailing list
>>> R-SIG-Mac using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>
>
-- 
Hervé Pagès

Bioconductor Core Team
hpages.on.github using gmail.com