[Rd] Compiling R-devel on older Linux distributions, e.g. RHEL / CentOS 7

[Henrik Bengtsson]

On Wed, Feb 8, 2023 at 12:22 PM Iñaki Ucar <iucar using fedoraproject.org> wrote:
>
> On Wed, 8 Feb 2023 at 19:59, Henrik Bengtsson
> <henrik.bengtsson using gmail.com> wrote:
> >
> > I just want to add a few reasons that I know of for why users are
> > still on Red Hat/CentOS 7 and learned from being deeply involved with
> > big academic and research high-performance compute (HPC) environments.
> > These systems are not like your regular sailing boat, but more like a
> > giant container ship; much harder to navigate, slower to react, you
> > cannot just cruise around and pop into any harbor you like to, or when
> > you like to. It takes much more efforts, more logistics, and more
> > people to operate them. If you mess up, the damage is much bigger.
>
> I'm fully aware of, and I understand, all the technical and
> organizational reasons why there are CentOS 7 systems out there. I
> only challenge a single point (cherry-picked from your list):
>
> > * The majority of users and sysadmins prefer stability over the being
> > able to run the latest tools.
>
> This is simply not true. In general, sysadmins do prefer stability,
> but users want the latest tools (otherwise, this very thread would not
> exist, QED). And the first thing is hardly compatible with the second
> one. That is, without containers, which brings us to the next point.

We might operate in different environments, but there are lots of labs
that keep the exact same pipeline for years (5-10 years), because "it
works", and because if they change anything, they might have to
re-analyze all their old data to avoid batch effects purely from
different versions of algorithms. I can agree with this strategy too,
especially if your data are huge and staging them back on the compute
environment from cold storage can be a huge task in itself.  Then
there are reasons such as being less savvy, and bad memories from last
time they tried this (e.g. years ago), everything broke, and it took
them weeks and months to sort it out.  I'm not trying to make fun of
anyone here - it's just that on big clusters with many users, the
skill-level spectrum varies a lot.

>
> > * Although you might want to tell everyone to just run a new version
> > via Linux containers, it's not the magic sauce for all of the above.
> > Savvy users might be able to do it, but not your average users. Also,
> > this basically puts the common sysadmin burden on the end-user, who
> > now have to keep their container stacks up-to-date and in sync.  In
> > contrast to a homogeneous environment, this strategy increases the
> > support burden on sysadms, because they will get much more questions
> > and request for troubleshooting on very specific setups.
>
> How is that so? Let's say a user wants the latest version of R.
> Nothing prevents a sysadmin to set up a script called "R" in the PATH
> that runs e.g. the r2u container [1] with the proper mounts. And
> that's it: the user runs "R" and receives the latest version (and even
> package installations seem to be blazing fast now!) without even
> knowing that it's running inside a container.
>
> I know, you are thinking "security", "permissions"...

I'm actually thinking maintenance and support. When you bring in Linux
containers, you basically introduce a bunch of new compute
environments in addition to your host system. So, instead of the
support team (often same as the sysadm) having to understand and
answer questions for a homogeneous environment, they now have to be
up-to-date with different versions of CentOS/Rocky, Ubuntu, Debian,
... and different container images. In R we often have a hard time to
even get users to report their sessionInfo() - now imagine their
container details.  If admins start providing one-off container
images, that becomes an added maintenance load. But, I agree, Linux
containers are great and makes it possible for a lot of users to run
analyzes that they otherwise would not be able to do on the host
system.

>
> $ yum install podman
>
> Drop-in replacement for docker, but rootless, daemonless. Also there's
> a similar thing called Apptainer [1], formerly Singularity, that was
> specifically designed with HPC in mind, now part of the Linux
> Foundation.
>
> [1] https://github.com/eddelbuettel/r2u
> [2] https://apptainer.org/

Yes, Singularity/Apptainer is awesome, especially since Docker is
mostly considered a no-no in HPC environments. The minimal, or even
zero use of SUID, these days, is great. That it runs as a regular
process as the users itself with good default file mounts is also
neat.  These things get even better with newer Linux kernels, which,
by the way, is another motivation for upgrading the OS.

That said, with Apptainer and likes, the user might run into conflicts
here, similar to what we see when users install software via Conda,
which often generals a parallel software stack to that of the host
system.  Taking R as an example, when a user installs packages, they
end up in R_LIBS_USER=~/R/%p-library/%v (*).  This is the same
directory regardless of running R on the host system, in a Linux
container, and in Linux containers based on different OSes.  So, if
they end up running a little bit here and there, which is not
unreasonable to expect if they work on different projects, then there
will a mishmash of R package binaries that are not compatible with
each other.  This happens a ton when people use Conda.  Of course, a
savvy user will at some point figure this out, and configure their
R_LIBS_USER to be agile to the environment they run, but the majority
won't notice this until it's too late.  And, boom, now you're adding
lots of load on the support team, and troubleshooting and undoing
these conflicts consumes a lot of wasted efforts.  In the worst case,
the user does not reach out for help, but in stead struggle in silence
and might work with something half broken.  From my experience at UCSF
(~2,000 users on two big clusters), this is unfortunately not that
uncommon.

(*) My wish would be if R could to include also the OS name and the OS
version in the default R_LIBS_USER, something like
R_LIBS_USER=~/R/%O-%p-library/%v, where %O would be a new
specification that expands to, say, "centos-7", "ubuntu-22.04".  That
would mitigate lots of these issues automatically.

Thanks for the feedback and questions,

Henrik

> > What R Core, Gabor, and many others are doing, often silently in the
> > background, is to make sure R works smoothly for many R users out
> > there, whatever operating system and version they may be on. This is
> > so essential to R's success, and, more importantly, for research and
> > science to be able to move forward.
>
> +1000
>
> --
> Iñaki Úcar