[R-SIG-Mac] System-wide site library [Was: CRAN installer for macOS - directory permissions]

Thu Jun 9 03:28:57 CEST 2022

We could re-design the layout of the framework and site locations to be more in line with the modern Apple standards. Splitting the library into system (R itself only) and site library could make the framework more correctly self sufficient. The site library would then go into "/Library/Application Support/org.R-project.R/library” and user library in the equally named subdirectory of $HOME. That would directly correspond to the Apple designations of NSApplicationSupportDirectory in NSLocalDomainMask and NSUserDomainMask domains. The drawback would be that none of this is versioned any longer, so we probably would have to rely on different bundle IDs (e.g. to distinguish big-sur builds from high-sierra builds) and possibly add versioned subdirectories inside our realm. Also this would make it impossible to make self-contained R apps, because the packages are outside of the framework path structure, but I have not seen anyone using that feature in a long time.

Obviously, this would be a major breaking change so for R-devel and R.app would have to be updated to use the corresponding paths for the package management, but that’s not too hard. I wouldn’t put this on top of my list given that the effect is mostly cosmetic, but by using the Apple API it would allow the framework to be used in a container if anyone cared (there are bigger issues if anyone wanted to create an iOS version, though ;)).

Cheers,
Simon

> On 9/06/2022, at 06:56, Duncan Murdoch <murdoch.duncan using gmail.com> wrote:
> 
> Henrik, you posted this a couple of days ago and I didn't address the _R_CHECK_DEPENDS_ONLY_ point you raised.
> 
> You're right that the current implementation of _R_CHECK_DEPENDS_ONLY_ doesn't work if all packages are installed in one lib.  This is a flaw, with one fix being to never put contributed packages into the system lib.  (I haven't done a Linux install in a long time; don't they by default put recommended packages there?  They can also be Suggested packages, so if they're in the system lib, that's a bug.)
> 
> Another possible fix is to change how _R_CHECK_DEPENDS_ONLY_ works, so that it affects package loading directly, by allowing the user to specify a whitelist of packages (e.g. based on the dependencies in the DESCRIPTION file) and having the package loader refuse to load any package unless it's in there.  I think I like the current implementation better.
> 
> So I'd change my recommendation for single-user systems:  they should have two libs.  One contains base packages and nothing else, the other contains all contributed packages, including recommended ones.  Assuming the single user is in the admin group, they could modify the second lib, but only reinstalls of R would change the first one.
> 
> On a multi-user system there would typically be another lib in the user account.
> 
> Duncan Murdoch
> 
> On 03/06/2022 12:45 p.m., Henrik Bengtsson wrote:
>> I see two fairly big problems with users installing R packages to
>> .Library by default.  One is related to package checking and CRAN, and
>> one is related to translation of expectations when moving between
>> operating systems (as Patrick already pointed out).  At the end, I'll
>> also argue that R_LIBS_SITE exists for those who wish to maintain
>> site-wide R package libraries to be shared among users, which is
>> better than using .Library for this.
>> # R CMD check
>> When you check a package with 'R CMD check --as-cran', or, with
>> environment variable `_R_CHECK_DEPENDS_ONLY_` set to true, the checks
>> are run in a sandbox where only declared package dependencies and any
>> packages in the system package library (= .Library) are on the library
>> path (= .libPaths()), e.g.
>> print(.libPaths())
>> [1] "/tmp/alice/RtmpYDq3KF/RLIBS_2410b74eb16752"
>> [2] "/path/to/R-4.2.0/lib/R/library"
>> What's in the user's library (= R_LIBS_USER) or in the site library (=
>> .Library.site/R_LIBS_SITE) is not part of the testing.  This mechanism
>> is very valuable since it helps to identify undeclared package
>> dependencies.
>> **The default behavior on macOS discussed here, where R packages are
>> installed to .Library, breaks this.**  Developers with non-base R
>> packages in .Library will not benefit from the 'R CMD check --as-cran'
>> checks for undeclared packages. This increases the risk of them not
>> being aware of the problem of undeclared packages, which is a
>> discussion we see from time to time on R-devel and R-pkg-devel, e.g.
>> when it comes to what should be listed under Suggests: or not.
>> BTW, this makes me wonder how many macOS developers notice this
>> problem only as they submit to CRAN, and have to resubmit. Also, this
>> issue might add extra work to the CRAN Team, e.g. spending time
>> locking at and responding to possible false positives, handling more
>> emails, and handling more re-submissions.
>> # Social expectations
>> The second problem with the current default macOS behavior is when
>> people hop between systems and operating systems.  Particularly, a
>> macOS user coming to Unix or Windows does not immediately understand
>> how and where R packages are installed.  They get a prompt about a
>> "personal library" and might choose to decline because it's not what
>> they're used to seeing.  Then they might end up in the Stack Overflow
>> cut'n'paste rabbit hole, where they find some instructions on setting
>> 'R_LIBS_USER=$HOME/R-lib' without version specifiers.  Works fine
>> until they upgrade R next year, when they start getting weird warnings
>> or errors of some packages not working that they slowly start to
>> accept as the normal behavior of R. I see this problem on large HPC
>> environments where I help out thousands of HPC users. Also, reading
>> various support forums out there, I think this is a real problem. It's
>> only recently, thanks to Patrick, I learned about this rather odd
>> macOS behavior, and I do think it is a cause for confusion and
>> miscommunication.  Another problem with different OS behaviors is that
>> it complicated documentation and instructions.  I strongly believe, it
>> would be beneficial to the R community if we all have the same
>> experience and expectations regardless of OS.
>> I believe the above problems are best addressed by changing the
>> *default* settings on macOS so that it is *not* possible to install to
>> .Library, and instead require a user to install to their personal
>> package library.  Advanced users who prefer to install to .Library,
>> can still configure R, or .Library, to do so.
>> As Patrick suggests, defaulting .Library to 755, instead of 775, or
>> avoid setting the "admin" group, seems like a simple solution that
>> would help harmonize the user experience of R.
>> # Maintaining a site-wide package library
>>> I would argue that the current setup tends to be a lot safer than the alternatives, because it allows commonly used packages to be installed at the system level and private packages to be installed at user level. This is also the design typically used on shared machines, where you separate local packages from user packages where local ones are installed by administrators - so exactly the same setup. Moreover R upgrades are a lot cleaner, since you can easily upgrade all system packages at once so you don't have to worry about individual users having stale packages - the biggest problem for admins.
>> I believe .Library.site/R_LIBS_SITE exists to address this purpose,
>> and I argue it is better suited for this than using .Library.  It also
>> has the benefit of *not* conflicting with the
>> _R_CHECK_DEPENDS_ONLY_=true checks (Problem #1 above).  It also allows
>> users to opt out from centrally installed packages (by setting
>> R_LIBS_SITE=""), which they cannot do if they're installed in the
>> system library.
>> All the best,
>> Henrik
>> PS. FWIW, for all of the above reasons, I always set 755 on .Library,
>> even if I install R to my personal home folder; it simply lowers the
>> risk for side effects when developing and troubleshooting.
>> On Thu, Jun 2, 2022 at 11:50 PM Patrick Schratz
>> <patrick.schratz using gmail.com> wrote:
>>> 
>>> I’d like to bump this thread again, especially since I am interested in a reply from Simon (or any other core member) on the arguments raised in the most recent messages of this thread.
>>> 
>>> What would be the best way for this idea/proposal going forward?
>>> 
>>> I’ve found https://contributor.r-project.org/, which links to a public Slack channel. Would this be a better place to discuss this matter?
>>> Or https://github.com/r-devel/rcontribution?
>>> 
>>> Best,
>>> Patrick_______________________________________________
>>> R-SIG-Mac mailing list
>>> R-SIG-Mac using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>> _______________________________________________
>> R-SIG-Mac mailing list
>> R-SIG-Mac using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 
> _______________________________________________
> R-SIG-Mac mailing list
> R-SIG-Mac using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> 