[R-pkg-devel] Issue with flang-new (segfault from C stack overflow)

[Jisca Huisman]

Hello,

My package sequoia contains Fortran code, and failed to pass the 
pre-test on Debian with the new flang-new compiler. I was able to 
reproduce the issue, but strongly suspect it is an issue with 
flang-new-17 rather than with my code. However, since in the past when I 
thought the problem was not with my code I usually eventually turned out 
to be wrong, I would be grateful for a 'second opinion' and/or advise on 
further checks to figure out what the issue might be.

The error in ..._Debian_00check.log during pretest was "Error: segfault 
from C stack overflow" when running one of the examples ( 
https://win-builder.r-project.org/incoming_pretest/sequoia_2.7.5_20231209_204908/Debian/ 
)

I was able to reproduce this error on Ubuntu 22.04 with R configured as 
follows:

LIBnn=lib64 \
        CC="clang-17" \
        CXX="clang++-17" \
        FC="flang-new-17" \
        FCFLAGS="-g -O2 -mtune=native" \
        CXXFLAGS="-g -O2 -mtune=native" \
        FFLAGS="-g -O2" \
        MAIN_LDFLAGS="-pthread" \
        ./configure -C --with-valgrind-instrumentation=2 \
        --with-system-valgrind-headers --with-x=no \
        --without-recommended-packages --enable-lto=yes


I isolated the problem in a minimal working example available here: 
https://github.com/JiscaH/flang_segfault_min_example . All that does is 
pass a vector of length N*N back and forth between R and Fortran. It 
works fine for very long vectors (tested up to length 5e8), but throws a 
segfault when I reshape a large array in Fortran to a vector to pass to 
R, both when using RESHAPE() and when using loops.

During installation of the minimal package there is a warning:

** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
readelf: Warning: Unrecognized form: 0x22
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation 
path
* DONE (minWE)

and during installation of the sequoia package there are dozens of those 
warnings.

When running

R -d "valgrind --tool=memcheck --leak-check=full" --vanilla < test_example.R

where

test_example.R is:  minWE::test_fun(N=5e2); minWE::test_fun(N=5e3)

with Valgrind-3.22.0 I get no valgrind messages/warnings/errors when 
N=5e2, but a segfault and a sleuth of messages when N=5e3, see 
https://github.com/JiscaH/flang_segfault_min_example/blob/main/valgrind_output_v15.txt 
.

When configuring R with

LIBnn=lib64 \
        CC="gcc -std=gnu99" \
        CXX="g++ -fno-omit-frame-pointer" \
        FC="gfortran" \
        FCFLAGS="-g -O2 -mtune=native" \
        CXXFLAGS="-g -O2 -Wall -pedantic -mtune=native" \
        FFLAGS="-g -O2 -mtune=native" \
        MAIN_LDFLAGS="-pthread" \
        ./configure -C --with-valgrind-instrumentation=2 \
        --with-system-valgrind-headers --with-x=no \
        --without-recommended-packages --enable-lto=yes

It runs without issues.


If someone can confirm that I'm not overlooking anything, I will submit 
a bug report to https://github.com/llvm/llvm-project/issues , and hope 
that I can convince the CRAN team to accept my package despite failing 
the pre-test.


Thanks,

Jisca