[R-SIG-Win] Rtools44
Avraham Adler
@vr@h@m@@d|er @end|ng |rom gm@||@com
Tue Mar 19 18:32:57 CET 2024
Thank you very much, Tomas.
I guess when the variables were recast to long long, the compiler
tried to use AVX2 commands. This is rather disappointing from GCC's
end; there isn't much we in R can do. As this bug has been around for
over a dozen years, and the last comment is two years ago, I do not
have hope that anything will be fixed in my lifetime.
Being restricted to Windows, are the only two options to disable all
AVX2 or use "-Wa,-muse-unaligned-vector-move"? There is no alternative
easy-to-drop-in toolchain to build R from source on Windows, is there?
As someone who develops packages requiring source code but is
restricted to Windows, I am loath to reduce the power of my code,
unless I must. Any advice or suggestions you may have would be greatly
appreciated.
Thank you for figuring this out,
Avi
On Tue, Mar 19, 2024 at 12:15 PM Tomas Kalibera
<tomas.kalibera using gmail.com> wrote:
>
>
> On 3/19/24 17:05, Tomas Kalibera wrote:
> > On 3/19/24 15:33, Avraham Adler wrote:
> >> For completeness, I compiled R from source using the native BLAS, and
> >> make check fails again, this time on "aregexec". Running "agrep" from
> >> within Rgui causes it to crash to desktop. Could it be something to do
> >> with this:
> >> https://github.com/wch/r-source/commit/ba486d898c2698c2afb678abdab807510327541e?
> >> But that didn't affect aregexc or adist? This is beyond my
> >> understanding, but I'm happy to do any tests you need to help identify
> >> what is going on.
> >
> > I can reproduce the problem (without LTO, without openblas), just by
> > "-O3 -march=native" and running
> >
> > agrep("lasy", "1 lazy 2")
> >
> > which crashes R.
> >
> > The crash happens in do_agrep from agrep.c, when calling tre_regaexec:
> >
> > } else {
> > const char *s = translateChar(STRING_ELT(vec, i));
> > if(mbcslocale && !mbcsValid(s))
> > error(_("input string %lld is invalid in this locale"),
> > (long long)i+1);
> > rc = tre_regaexec(®, s, &match, params, 0); // <===
> > here (line 242)
> > vmaxset(vmax);
> > }
> >
> > 0x00007fff79c642da <+1290>: mov 0x50(%rsp),%rdx
> > 0x00007fff79c642df <+1295>: test %eax,%eax
> > 0x00007fff79c642e1 <+1297>: je 0x7fff79c6522a <do_agrep+5210>
> > 0x00007fff79c642e7 <+1303>: movl $0x0,0x20(%rsp)
> > 0x00007fff79c642ef <+1311>: mov 0x38(%rsp),%rcx
> > 0x00007fff79c642f4 <+1316>: mov %r15,%r8
> > 0x00007fff79c642f7 <+1319>: lea 0x80(%rsp),%r9
> > 0x00007fff79c642ff <+1327>: vmovdqu 0xb0(%rsp),%ymm4
> > => 0x00007fff79c64308 <+1336>: vmovdqa %ymm4,0x80(%rsp)
> > 0x00007fff79c64311 <+1345>: vzeroupper
> > 0x00007fff79c64314 <+1348>: call 0x7fff79d183d0 <tre_regaexec>
> >
> > And it happens because of incorrect alignment in an AVX2 instruction.
> > In the above, vmovdqa is used as a second step of copying the
> > regparams_t structure to the stack (to be passed as 4th argument of
> > tre_regaexec). But GCC uses 0x80(%rsp) for that, which happens only to
> > be 16-byte aligned, but not 32-byte aligned. AVX2 requires this to be
> > 32-byte aligned in vmovdqa instruction, and hence the program segfaults.
> >
> > This is a long standing bug in GCC that has been reported long time
> > ago: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
> >
> > It must be a coincidence that you haven't run into this with GCC 12.3
> > and earlier (so earlier versions of Rtools for Windows).
> >
> > I think the simplest and most reliable work-around is to not use AVX2
> > with GCC on Windows.
>
> You can also use "-Wa,-muse-unaligned-vector-move" option for the GNU
> assembler (add to optimization options), which will make the assembler
> use instructions that don't require memory to be aligned. Whether it
> would still be beneficial for performance to use AVX2 in the end or not
> probably depends on the code.
>
> Best
> Tomas
>
> >
> > Best
> > Tomas
> >
> >>
> >> Thank you,
> >>
> >> Avi
> >>
> >>
> >>
> >> On Mon, Mar 18, 2024 at 11:58 PM Avraham Adler
> >> <avraham.adler using gmail.com> wrote:
> >>> I ran "tools::testInstalledBasic("both")" from within Rterm, and it
> >>> failed on reg-tests-1a, and doesn't that last call in the FAIL look
> >>> familiar? Three times isn;t coincidence. What could be causing
> >>> agrep/adist to fail?
> >>>
> >>> Avi
> >>>
> >>>> a <- c("NA", NA, "BANANA")
> >>>> na <- NA_character_
> >>>> a1 <- substr(a,1,1)
> >>>> stopifnot(is.na(a1)==is.na(a))
> >>>> a2 <- substring(a,1,1)
> >>>> stopifnot(is.na(a2)==is.na(a))
> >>>> a3 <- sub("NA","na",a)
> >>>> stopifnot(is.na(a3)==is.na(a))
> >>>> a3 <- gsub("NA","na",a)
> >>>> stopifnot(is.na(a3)==is.na(a))
> >>>> substr(a3, 1, 2) <- "na"
> >>>> stopifnot(is.na(a3)==is.na(a))
> >>>> substr(a3, 1, 2) <- na
> >>>> stopifnot(all(is.na(a3)))
> >>>> stopifnot(agrep("NA", a) == c(1, 3))
> >>> On Mon, Mar 18, 2024 at 11:52 PM Avraham Adler
> >>> <avraham.adler using gmail.com> wrote:
> >>>> Hello, Tomas.
> >>>>
> >>>> I ran check again and this time it failed on base. Base failed at
> >>>> "agrep" and utils failed at "adist" if that triggers any ideas.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Avi
> >>>>
> >>>> On Mon, Mar 18, 2024 at 11:49 PM Avraham Adler
> >>>> <avraham.adler using gmail.com> wrote:
> >>>>> Hello, Tomas.
> >>>>>
> >>>>> I had hoped that my problem was that I was linking to OpenBLAS built
> >>>>> under Rtools43, but sadly that is not the case, as I built OpenBLAS
> >>>>> using Rtools44 and I still get the error in 'utils'. I do have
> >>>>> Rtools43 installed on this machine, but I am doing eveything from the
> >>>>> Rtools44 bash. My MkRules.local follows:
> >>>>>
> >>>>> USE_ATLAS = YES
> >>>>> ATLAS_PATH = C:/R/OPB/OPB_03.26_1T_44
> >>>>> EOPTS = -march=native -pipe -mno-rtm
> >>>>> LTO = -flto=1 -fuse-linker-plugin
> >>>>> LTO_OPT = -flto=1 -fuse-linker-plugin
> >>>>> LTO_FC = -flto=1 -fuse-linker-plugin
> >>>>> LTO_FC_OPT = -flto=1 -fuse-linker-plugin
> >>>>> QPDF = C:/R/qpdf-11.9.0-msvc64
> >>>>> OPENMP = -fopenmp
> >>>>>
> >>>>> I also make the following change to Makefile.win in
> >>>>> /src/extra/blas as
> >>>>> I have been doing for more than a decade:
> >>>>>
> >>>>> --- /c/r/trunk/src/extra/blas/Makefile.win 2024-01-24
> >>>>> 18:34:42.755255900 +0000
> >>>>> +++ /c/r/Makefile.win 2024-01-24 18:39:39.716458000 +0000
> >>>>> @@ -12,7 +12,7 @@
> >>>>> ../../../$(BINDIR)/Rblas.dll: blas00.o ../../gnuwin32/dllversion.o
> >>>>> @$(ECHO) -------- Building $@ --------
> >>>>> $(DLL) -s -shared $(DLLFLAGS) -o $@ $^ Rblas.def \
> >>>>> - -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lf77blas -latlas
> >>>>> + -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -fopenmp
> >>>>> -lopenblas
> >>>>> else
> >>>>> ../../../$(BINDIR)/Rblas.dll: blas.o blas2.o cmplxblas.o
> >>>>> cmplxblas2.o
> >>>>> ../../gnuwin32/dllversion.o
> >>>>> @$(ECHO) -------- Building $@ --------
> >>>>>
> >>>>> The utils-Ex.Rout.fail is 1188 lines long and I don't see any obvious
> >>>>> point of failure. The last few lines are:
> >>>>>> cleanEx()
> >>>>>> nameEx("adist")
> >>>>>> ### * adist
> >>>>>>
> >>>>>> flush(stderr()); flush(stdout())
> >>>>>>
> >>>>>> ### Name: adist
> >>>>>> ### Title: Approximate String Distances
> >>>>>> ### Aliases: adist
> >>>>>> ### Keywords: character
> >>>>>>
> >>>>>> ### ** Examples
> >>>>>>
> >>>>>> ## Cf. https://en.wikipedia.org/wiki/Levenshtein_distance
> >>>>>> adist("kitten", "sitting")
> >>>>> [,1]
> >>>>> [1,] 3
> >>>>>> ## To see the transformation counts for the Levenshtein distance:
> >>>>>> drop(attr(adist("kitten", "sitting", counts = TRUE), "counts"))
> >>>>> ins del sub
> >>>>> 1 0 2
> >>>>>> ## To see the transformation sequences:
> >>>>>> attr(adist(c("kitten", "sitting"), counts = TRUE), "trafos")
> >>>>> [,1] [,2]
> >>>>> [1,] "MMMMMM" "SMMMSMI"
> >>>>> [2,] "SMMMSMD" "MMMMMMM"
> >>>>>> ## Cf. the examples for agrep:
> >>>>>> adist("lasy", "1 lazy 2")
> >>>>> [,1]
> >>>>> [1,] 5
> >>>>>> ## For a "partial approximate match" (as used for agrep):
> >>>>>> adist("lasy", "1 lazy 2", partial = TRUE)
> >>>>> The build works under Rtools43. Should I uninstall both versions of
> >>>>> Rtools, my current R installation, and its library and try again? I'm
> >>>>> doubtful that will help as my "active" R installation is in a
> >>>>> completely different directory, but I am willing to try. If there is
> >>>>> any output or other tests I can do, please let me know. Or, if you
> >>>>> think I should raise this on r-devel.
> >>>>>
> >>>>> Thank you,
> >>>>>
> >>>>> Avi
> >>>>>
> >>>>> On Mon, Mar 18, 2024 at 4:08 AM Tomas Kalibera
> >>>>> <tomas.kalibera using gmail.com> wrote:
> >>>>>> Hello Avi,
> >>>>>>
> >>>>>> On 3/17/24 17:53, Avraham Adler wrote:
> >>>>>>> Hello, Tomas.
> >>>>>>>
> >>>>>>> As always, thank you for your incessant hard work. I have compiled
> >>>>>>> R-devel 86144 using Rtools44 and it completes normally. However, it
> >>>>>>> fails very early in make check-devel. Specifically, I get the
> >>>>>>> output
> >>>>>>> below, and have received it more than once. Is this something to
> >>>>>>> raise
> >>>>>>> on R-devel or is it an Rtools44-specific issue?
> >>>>>> I can't tell from this output. Could you please try to get a more
> >>>>>> specific error output?
> >>>>>> Could you please share your compiler options (e.g. MkRules.local)
> >>>>>> and
> >>>>>> indeed if you made any modifications to the code?
> >>>>>>
> >>>>>> Also, as always, it is worth making sure that all code has been
> >>>>>> rebuilt
> >>>>>> using the new Rtools - e.g. delete any old package library for
> >>>>>> R-devel
> >>>>>> (4.4) you may have on your system.
> >>>>>>
> >>>>>> The testing done by myself and CRAN only covers the default
> >>>>>> compilation
> >>>>>> options as specified in the make files.
> >>>>>>
> >>>>>> Best
> >>>>>> Tomas
> >>>>>>
> >>>>>>> Thank you,
> >>>>>>>
> >>>>>>> Avi
> >>>>>>>
> >>>>>>> $ make check-devel
> >>>>>>> Testing examples for package 'base'
> >>>>>>> Testing examples for package 'tools'
> >>>>>>> comparing 'tools-Ex.Rout' to 'tools-Ex.Rout.save' ... NOTE
> >>>>>>> 1046,1047d1045
> >>>>>>> < Warning in file(con, "r") :
> >>>>>>> < file("") only supports open = "w+" and open = "w+b":
> >>>>>>> using the former
> >>>>>>> 1050,1051c1048,1049
> >>>>>>> < $ file : chr ""
> >>>>>>> < $ title : chr ""
> >>>>>>> ---
> >>>>>>> > $ file : chr "grid.Rnw"
> >>>>>>> > $ title : chr "Introduction to grid"
> >>>>>>> Testing examples for package 'utils'
> >>>>>>> Error: testing 'utils' failed
> >>>>>>> Execution halted
> >>>>>>> make[3]: *** [Makefile.win:29: test-Examples-Base] Error 1
> >>>>>>> make[2]: *** [Makefile.common:208: test-Examples] Error 2
> >>>>>>> make[1]: *** [Makefile.common:193: test-all-basics] Error 1
> >>>>>>> make: *** [Makefile:333: check-devel] Error 2
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Mar 14, 2024 at 4:45 AM Tomas Kalibera
> >>>>>>> <tomas.kalibera using gmail.com> wrote:
> >>>>>>>> Dear R Windows developers,
> >>>>>>>>
> >>>>>>>> there is now a new toolchain for R for Windows, Rtools44. It is
> >>>>>>>> now used
> >>>>>>>> by R-devel and is intended for R 4.4.0.
> >>>>>>>>
> >>>>>>>> Compared to Rtools43, it uses GCC 13 and updates other core
> >>>>>>>> components.
> >>>>>>>> See
> >>>>>>>> https://cran.r-project.org/bin/windows/Rtools/rtools44/news.html
> >>>>>>>> for
> >>>>>>>> a detailed list of changes.
> >>>>>>>>
> >>>>>>>> All users of R-devel who need to compile R packages with source
> >>>>>>>> code in
> >>>>>>>> C, C++ or Fortran should install Rtools44. From the user
> >>>>>>>> perspective,
> >>>>>>>> Rtools44 works the same way as Rtools43. See
> >>>>>>>> https://cran.r-project.org/bin/windows/base/howto-R-devel.html for
> >>>>>>>> instructions on how to build R-devel and packages for this
> >>>>>>>> version of R.
> >>>>>>>>
> >>>>>>>> It is recommended to re-install packages that need compilation
> >>>>>>>> to avoid
> >>>>>>>> potential incompatibilities with code built using Rtools43.
> >>>>>>>>
> >>>>>>>> Rtools44 also includes an experimental version for 64-bit ARM
> >>>>>>>> machines,
> >>>>>>>> using LLVM 17 (and clang, flang-new, lld, libc++) - the aarch64
> >>>>>>>> version
> >>>>>>>> of Rtools has its own installer and distribution tarballs.
> >>>>>>>>
> >>>>>>>> Best
> >>>>>>>> Tomas
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> R-SIG-windows mailing list
> >>>>>>>> R-SIG-windows using r-project.org
> >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-windows
More information about the R-SIG-windows
mailing list