[R-pkg-devel] Order of repo access from options("repos")

Uwe Ligges ||gge@ @end|ng |rom @t@t|@t|k@tu-dortmund@de
Mon Apr 1 17:44:43 CEST 2024 

Untested:

install.packages() calls available.packages() to find out which packages 
are available - and passes a "filters" argument if supplied.
That can be a user defined filter. It should be possible to write a user 
defined filter which prefers the packages in your local repo.

Best,
Uwe Ligges




On 01.04.2024 02:07, Greg Hunt wrote:
> Martin, Dirk, Kevin,
> Thanks for your help.  To summarise: the order of access is undefined, and
> every repo URL is accessed.   I'm working in an environment
> where "known-good" is more important than "latest", so what follows is an
> explanation of the problem space from my perspective.
> 
> What I am experimenting with is pinning down the versions of the packages
> that a moderately complex solution is built against using a combination of
> an internal repository of cached packages (internally written packages, our
> own hopefully transient copies of packages archived from CRAN,
> packages live on CRAN, and packages present in both Github and CRAN which
> we build and cache locally) and a proxy that separately populates that
> cache in specific build processes by intercepting requests to CRAN.  I'd
> like to use the base R function if possible and I want to let the version
> numbers in the dependencies float because a) we do need to maintain
> approximate currency in what versions of packages we use and b) I have no
> business monkeying around with third party's dependencies.  Renv looks
> helpful but has some assumptions about disk access to its cache that I'd
> rather avoid by running an internal repo.  The team is spread around the
> world, so shared cache volumes are not a great idea.
> 
> The business with the multiple repo addresses is one approach to working
> around Docker's inability to understand that people need to access the
> Docker host's ports from inside a container or a build, and that the
> current Docker treatment of the host's internal IP is far from transparent
> (I have scripts that run both inside and outside of Docker containers and
> they used to be able to work out for themselves what environment they run
> in, thats got harder lately).  That led down a path in which one set of
> addresses did not reject connection attempts, making each package
> installation (and there are hundreds) take some number of minutes for the
> connections to time out.  Thankfully I don't actually have to deal with
> that.
> 
> We have had a few cases where our dependencies have been archived from CRAN
> and we have maintained our own copy for a period of days to months, a
> period in which we do not know what the next package version number is.  It
> would be convenient to not have to think about that - a deterministic,
> terminating search of a sequence of repos looked like a nice idea for that,
> but I may have to do something different.
> 
> There was a recent case where a package made a breaking change in its
> interface in a release (not version) update that broke another package we
> depend on.  It would be nice to be able to temporarily pin that package at
> its previous version (without updating the source of the third party
> package that depends on it) to preserve our own build-ability while those
> packages sort themselves out.
> 
> There is one case where a pull request for a CRAN-hosted package was
> verbally accepted but never actioned so we have our own forked version of a
> CRAN-hosted package which I need to decide what to do with one day soon.
> Another case where the package version number is different in CRAN from the
> one we want.
> 
> We have a dependency on a package that we build from a Git repo but which
> is also present in CRAN.  I don't want to be dependent on the maintainers
> keeping the package version in the Git copy of the DESCRIPTION file higher
> than the version in CRAN.  Ideally I'd like to build and push to the
> internal repo and not have to think about it after that. Same issue as
> before arises, as it stands today I have to either worry about, and
> probably edit, the version number in the build or manage the cache
> population process so the internal package instance is added after any
> CRAN-sourced dependencies and make sure that the public CRAN instances are
> not accessed in the build.
> 
> All of these problems are soluble by special-casing the affected installs,
> specifically managing the cache population (with a requirement that the
> cache and CRAN not be searched at the same time), or editing version
> numbers whose next values I do not control, but I would like to try for the
> simplest approach first. I know I'm not going to get a clean solution here,
> the relative weights of "known-good" and "latest" are different
> depending on where you stand.
> 
> 
> Greg
> 
> On Sun, 31 Mar 2024 at 22:43, Martin Morgan <mtmorgan.xyz using gmail.com> wrote:
> 
>> available.packages indicates that
>>
>>
>>
>>       By default, the return value includes only packages whose version
>>
>>       and OS requirements are met by the running version of R, and only
>>
>>       gives information on the latest versions of packages.
>>
>>
>>
>> So all repositories are consulted and then the result filtered to contain
>> just the most recent version of each. Does it matter then what order the
>> repositories are visited?
>>
>>
>>
>> Martin Morgan
>>
>>
>>
>> *From: *R-package-devel <r-package-devel-bounces using r-project.org> on behalf
>> of Greg Hunt <greg using firmansyah.com>
>> *Date: *Sunday, March 31, 2024 at 7:35 AM
>> *To: *Dirk Eddelbuettel <edd using debian.org>
>> *Cc: *List r-package-devel <r-package-devel using r-project.org>
>> *Subject: *Re: [R-pkg-devel] Order of repo access from options("repos")
>>
>> Dirk,
>> Sadly I can't use localhost for all of those.  172.17.0.1 is an internal
>> Docker IP, not the localhost address (127.0.0.1), they are there to handle
>> two different scenarios and different ones will fail to resolve in
>> different scenarios.  Are you saying that the DNS lookup adds a timing
>> issue to the search order?  Isn't the list deterministically ordered?
>>
>>
>> Greg
>>
>> On Sun, 31 Mar 2024 at 22:15, Dirk Eddelbuettel <edd using debian.org> wrote:
>>
>>>
>>> Greg,
>>>
>>> There are AFAICT two issues here: how R unrolls the named vector that is
>>> the
>>> 'repos' element in the list 'options', and how your computer resolves DNS
>>> for
>>> localhost vs 172.17.0.1.  I would try something like
>>>
>>>     options(repos = c(CRAN = "http://localhost:3001/proxy",
>>>                       C = "http://localhost:3002",
>>>                       B = "http://localhost:3003/proxy",
>>>                       A = "http://localhost:3004"))
>>>
>>> or the equivalent with 172.17.0.1. When I do that here I get errors from
>>> first to last as we expect:
>>>
>>>     > options(repos = c(CRAN = "http://localhost:3001/proxy",
>>>                       C = "http://localhost:3002",
>>>                       B = "http://localhost:3003/proxy",
>>>                       A = "http://localhost:3004"))
>>>     > available.packages()
>>>     Warning: unable to access index for repository
>>> http://localhost:3001/proxy/src/contrib:
>>>       cannot open URL 'http://localhost:3001/proxy/src/contrib/PACKAGES'
>>>     Warning: unable to access index for repository
>>> http://localhost:3002/src/contrib:
>>>       cannot open URL 'http://localhost:3002/src/contrib/PACKAGES'
>>>     Warning: unable to access index for repository
>>> http://localhost:3003/proxy/src/contrib:
>>>       cannot open URL 'http://localhost:3003/proxy/src/contrib/PACKAGES'
>>>     Warning: unable to access index for repository
>>> http://localhost:3004/src/contrib:
>>>       cannot open URL 'http://localhost:3004/src/contrib/PACKAGES'
>>>          Package Version Priority Depends Imports LinkingTo Suggests
>>> Enhances License License_is_FOSS License_restricts_use OS_type Archs
>> MD5sum
>>> NeedsCompilation File Repository
>>>     >
>>>
>>> Dirk
>>>
>>> --
>>> dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org
>>>
>>
>>          [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-package-devel using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>>
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel

More information about the R-package-devel mailing list