[Rd] [External] Re: Update on rtools4 and ucrt support

[Tomas Kalibera]

On 8/24/21 12:03 AM, Avraham Adler wrote:
> On Tue, Aug 24, 2021 at 12:47 AM Simon Urbanek <simon.urbanek using r-project.org>
> wrote:
>
>> Avi,
>>
>> please see the announcement:
>>
>>
>> https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html
>>
>> the documentation is in
>>
>>
>> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html
>>
>> Cheers,
>> Simon
>>
>>
>>
>>
>> On Aug 24, 2021, at 8:34 AM, Avraham Adler <avraham.adler using gmail.com>
>> wrote:
>>
>> On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney using uiowa.edu> wrote:
>>
>> On Mon, 23 Aug 2021, Duncan Murdoch wrote:
>>
>> On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote:
>>
>> Hi Jeroen,
>>
>> I mostly lurk on this list, but I was struck by your combative tone.
>>
>> To pick on two random bits:
>>
>> … a 6gb tarball with manually built things on his personal machine…
>>
>>
>> … a black-box system that is so opaque and complex that only one person
>> knows how it works, and would make it much more difficult for
>> students, universities, and other organisations to build R packages
>> and libraries on Windows…
>>
>>
>>
>> Tomas’ tool chain isn't a blackbox, it has copious documentation (see
>>
>> [1])
>>
>> and builds on any machine thanks to the provided docker container.
>>
>> This is not to criticise your work which has its unique strengths, but
>>
>> to
>>
>> state the obvious: these strengths are best discussed without passion
>> based on factually accurate descriptions.
>>
>>
>> I agree with Jan.  I'm not sure a discussion in this forum would be
>>
>> fruitful,
>>
>> but I really wish Jeroen and Tomas would get together, aiming to merge
>>
>> their
>>
>> toolchains, keeping the best aspects of both.
>>
>> I haven't been involved in the development of either one, but have been
>>
>> a
>>
>> "victim" of the two chain rivalry, because the rgl package is not easy
>>
>> to
>>
>> build.  I get instructions from each of them on how to do the build, and
>> those instructions for one toolchain generally break the build on the
>>
>> other
>>
>> one.  While it is probably possible to detect the toolchain and have the
>> build adapt to whichever one is in use, it would be a lot easier for me
>>
>> (and
>>
>> I imagine every other maintainer of a package using external libs) if I
>>
>> just
>>
>> had to follow one set of instructions.
>>
>> Duncan Murdoch
>>
>>
>> Here are just a few comments from my perspective (I am an R-core
>> member, but am not part of the CRAN team and do only very limited work
>> on Windows). Other R-core members may have different perspectives and
>> insights.
>>
>> One bit of background: dealing with encoding issues on Windows has
>> been taking an unsustainable amount of R-core resources for some time
>> now. Tomas Kalibera has been taking the lead on trying to address
>> these issues in the existing framework, but this means he has not had
>> the time to make any of the many other valuable and important
>> contributions he could make. The only viable way forward is to move to
>> a Windows tool chain that supports UTF-8 as the C library current
>> encoding via the Windows UCRT framework.
>>
>> Tomas Kalibera has, on behalf of all of R core and in
>> coordination with CRAN, been looking for a way forward for some
>> time and has reported on the progress in several blog posts at
>> https://developer.r-project.org/Blog/public/. This has lead to
>> the development of the MXE-based UCRT tool chain, which is now
>> well tested and ready for deployment.  Checks using the UCRT tool
>> chain have been part of the CRAN check process for a while. I
>> believe CRAN plans to switch R-devel checks and builds to the
>> UCRT tool chain during the upcoming CRAN downtime. I expect there
>> will be some communication from CRAN on this soon, including on
>> any issues in supporting binaries for both R-devel and R-patched.
>>
>> In putting together something as large as a tool chain there will
>> always be many choices, each with advantages and disadvantages.  Some
>> things may be advantages in some settings and not others. Taking just
>> one case in point: Cross compilation. This is likely to be a better
>> approach for CRAN in the future and is supported by the MXE framework
>> on which the new tool chain is based.
>>
>> The much more recent changes in rtools4 to support UCRT are at this
>> point not yet as well tested as the new tool chain. Once these changes
>> to rtools4 mature, and if binary compatibility can be assured, then
>> having a second tool chain may be useful in some cases.  But if there
>> are incompatibilities then it will be up to rtools4 to keep up with
>> the tool chain used by CRAN. On the other, contributing to improving
>> the MXE-based tool chain may be a better investment of time.
>>
>> Best,
>>
>> luke
>>
>>
>> ______________________________________________
>> R-devel using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>>
>> --
>> Luke Tierney
>> Ralph E. Wareham Professor of Mathematical Sciences
>> University of Iowa                  Phone:             319-335-3386
>> Department of Statistics and        Fax:               319-335-3017
>>     Actuarial Science
>> 241 Schaeffer Hall                  email:   luke-tierney using uiowa.edu
>> <luke-tierney using uiowa.edu>
>> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>>
>>
>> Thank you, Dr. Tierney. However, I am concerned about the not-insignificant
>> number of us who for various reasons can only do our development on
>> Windows. Rtools has been the official tool chain with which to build
>> windows for the at least 20 years I have been using R (yes, a babe in the
>> woods compared to most, but not a complete neophyte). Duncan and Jereoen
>> have each done yeoman’s jobs in ensuring that R can be built from source on
>> Windows and that packages can be developed for all OSS on Windows—even
>> Solaris SPARC.
>>
>> I am much less aware of Thomas’s work, and I’ll gladly take the blame for
>> it, but I haven’t seen an accessible tool chain built by him which would
>> allow me, the Windows developer, to build R and all packages from source on
>> a native Windows box. Have I just missed it? If not, is that planned? If
>> R-core switches the official Windows toolchain, where does that leave us?
>>
>> Thank you,
>>
>> Avi
>>
>> --
>> Sent from Gmail Mobile
>>
>>
> Thank you, Simon. That was valuable. Skimming that quickly, I get a bit
> concerned. I’ve been building from source and then using OpenBLAS in my R
> source for many, many years now, and it looks like its support is tenuous
> in the experimental chain. Similarly with packages like nloptr, where I
> build NLOPT 2.6+ and have adjusted Jelmer’s code for it to work in R for
> Windows. I maintain packages with Fortran/OpenMP and Rcpp(Parallel). I hope
> that should the decision be made to switch, it will be done when the build
> process is more streamlined, especially for some fundamental packages.

Hi Avi,

if your R package includes source code (C/C++/Fortran), there is no 
fundamental difference, just the version of the compilers and/or 
external libraries may have changed.

nloptr checks are fine, see 
https://cran.r-project.org/web/checks/check_results_nloptr.html

The architecture is r-devel-windows-x86_64-gcc10-UCRT.

Best
Tomas

>
> That being said, I must take the opportunity again to thank R-core, Tomas,
> Jeroen, Duncan among many others who have built an infrastructure that
> allows amateur programmers to contribute to the statistical infrastructure.
>
> Thanks again,
>
> Avi