[Rd] Compiling R-devel on older Linux distributions, e.g. RHEL / CentOS 7

[Henrik Bengtsson]

I just want to add a few reasons that I know of for why users are
still on Red Hat/CentOS 7 and learned from being deeply involved with
big academic and research high-performance compute (HPC) environments.
These systems are not like your regular sailing boat, but more like a
giant container ship; much harder to navigate, slower to react, you
cannot just cruise around and pop into any harbor you like to, or when
you like to. It takes much more efforts, more logistics, and more
people to operate them. If you mess up, the damage is much bigger.

Reasons:

* Users don't have many options, but have to use what is available.

* Red Hat/CentOS is designed for long term stability and backward
compatibility. They've only done major upgrades every 3-4 years.

* Red Hat backports security fixes to old versions of common software,
which is why you see, for instance, Python 3.6 still being the
provided version, although Python made that End-of-Life in December
2021.

* HPC environments (aka "compute cluster") often have 100s to 1000s of
users. Imagine the amount of software tools and difference versions
installed in such environments.

* Upgrading an HPC environments is a major disruption for users who
rely on it in their work and research, e.g. some software stacks,
pipelines, and scripts have to reinstalled and re-coded.

* The majority of users and sysadmins prefer stability over the being
able to run the latest tools.

* Over years, stability increases the technical debt, to a point where
it is cheaper to upgrade than to stay behind.

* Even if sysadmins want to upgrade to a newer release, their hands
are often also tied, because of external factors.  For example, the
global parallel file system or the backup system you rely on has not
yet be validated for the next version of OS you want to upgrade too.
There might also be research critical scientific pipelines that does
not yet support the new version, which can be because the maintainers
of those tools don't have access to a new version to test on. GPU
drives much not be available. This is also the case for commercial
tools. Sometimes IT security requirements cannot be met on the new
version, because security scanning tools are not yet up-to-date. There
can also be hardware limitations, e.g. you might even have to replace
some central server for the whole cluster to be able to upgrade.

* Although you might want to tell everyone to just run a new version
via Linux containers, it's not the magic sauce for all of the above.
Savvy users might be able to do it, but not your average users. Also,
this basically puts the common sysadmin burden on the end-user, who
now have to keep their container stacks up-to-date and in sync.  In
contrast to a homogeneous environment, this strategy increases the
support burden on sysadms, because they will get much more questions
and request for troubleshooting on very specific setups.

Specifically to Red Hat/CentOS 7: When sites started to think about
migrating to CentOS 8, Red Hat decided to pull the plug and change
their long-term business plan.  This itself was a disruptive event,
because any plans to do a "regular" distro upgrade had to be flushed
down the toilet.  The community waited to see what would happen and
what the options would be.  A lot of sites now plan on migrating to
Rocky 8, which (AFAIU) tries to stay true to the original CentOS
mission.  This means they are waiting for third-party hardware and
software providers to validate their products for Rocky 8, e.g.
parallel file systems, backup software, software stacks, etc.

What R Core, Gabor, and many others are doing, often silently in the
background, is to make sure R works smoothly for many R users out
there, whatever operating system and version they may be on. This is
so essential to R's success, and, more importantly, for research and
science to be able to move forward.  Those endless hours spend on
trying to support some OS, even obscure ones, to pay off many times,
especially on common OSes such as Red Hat and CentOS.  You spare lots
of users and sysadmins lots of pain when you put those hours in.  So,
thank you for doing all that.

/Henrik

On Wed, Feb 8, 2023 at 2:24 AM Iñaki Ucar <iucar using fedoraproject.org> wrote:
>
> On Wed, 8 Feb 2023 at 07:05, Prof Brian Ripley <ripley using stats.ox.ac.uk> wrote:
> >
> > On 08/02/2023 00:13, Gábor Csárdi wrote:
> > > As preparation for the next release, I am trying to compile R devel on
> > > RHEL / CentOS 7, which is still supported by RedHat until 2024 June.
>
> True, but with a big asterisk. Full updates ended on 2020-08-06, and
> it's been in maintenance mode since then, meaning that only security
> and critical fixes are released until EOL to facilitate a transition
> to a newer version. So CentOS 7 users shouldn't expect new releases of
> software to be available.
>
> > > There are two issues.
> > >
> > > One is that the libcurl version in CentOS 7 is quite old, 7.29.0, and
> > > R devel now requires 7.32.0, since 83715 about a week ago. This
> > > requirement is here to stay for R 4.3.0, right?
>
> I suppose that if R-devel doesn't use any API endpoint not available
> in 7.29, you could just patch out that requirement. Otherwise, you
> would need to build your own.
>
> > Unless we revert it.  The comment in the manual says
> >
> > @c libcurl 7.32.0 was released in Aug 2013
> >
> > and Centos 7 was released in 2014-07-07, 11 months later.  Do they
> > really never security-patch libcurl?
>
> Oh, they do port all security fixes, but without changing the version,
> which is the whole point of LTS. In fact, current version is
> 7.29.0-59, and there are probably a hundred patches on top of those 59
> builds.
>
> > > The second is that the recommended packages are now installed with R
> > > CMD INSTALL --use-C17, which fails on CentOS 7 with
> > >
> > > begin installing recommended package MASS
> > > * installing *source* package 'MASS' ...
> > > ** package 'MASS' successfully unpacked and MD5 sums checked
> > > ** using non-staged installation
> > > ** libs
> > > Error: C17 standard requested but CC17 is not defined
> > > * removing '/root/R-devel/library/MASS'
> > >
> > > CentOS 7 has GCC 4.8.5, which does not have a -std=gnu17 option.
> > > However the commit message of this change in commit 83566 hints that
> > > this requirement might be temporary. Hence my questions.
> >
> > It is temporary -- needed for survival (now updated) and mgcv (awaited).
> >   However,
> >
> > 1) You should be able to set
> >
> > CC17="gcc -std=gnu11"
> >
> > in config.site, as C17 is a bug-fixed C11.
> >
> > 2) Centos 7 has later compilers available, and people are going to need
> > them for C++.  The manual says
> >
> >      ... later compilers are available: for RHEL/Centos 7 look for
> > ‘devtoolset’.
>
> Exactly, here is the reference:
> https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/
>
> R 3.6.0 is the last version we support in EPEL, because EPEL is not
> allowed to build on top of SCL. But you can enable SCL and install the
> devtoolset available, which contains gcc version 7.3.1.
>
> But anyway, I don't think that staying in an almost 10-year old distro
> in maintenance mode and at the same time expecting a cutting-edge
> version of R (or any software) is reasonable.
>
> Iñaki
>
> > > Is the C17 requirement temporary or it will be a requirement for R 4.3.0?
> > > Should I expect any problems if I remove the --use-C17 flag for
> > > installing the recommended packages?
> >
> > Not with that compiler.
> >
> > >
> > > There are a lot of R users still on RHEL 7, so it would be great to
> > > know what to expect for the next release.
> > an D. Ripley,                  ripley using stats.ox.ac.uk
> > Emeritus Professor of Applied Statistics, University of Oxford
> >
> > ______________________________________________
> > R-devel using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
>
> --
> Iñaki Úcar
>
> ______________________________________________
> R-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel