[R-pkg-devel] Stack smashing detected (dynamically loaded Fortran).

[Ivan Krylov]

On Sun, 6 Feb 2022 16:32:05 +1300
Rolf Turner <r.turner using auckland.ac.nz> wrote:

> The output that I get from valgrind (see the attached file
> "smash.txt") does not enlighten me.

For whatever reason, Valgrind doesn't see the error. I think it's the
compiler-provided stack canary that complains.

I've been able to get a slightly more informative error from
AddressSanitizer, by putting the following in ~/.R/Makevars:

FFLAGS=-g -Og -fsanitize=address
FCFLAGS=-g -Og -fsanitize=address

(you'll probably want to comment that out once you're done)

then building demo.so like this:

R CMD SHLIB -o demo.so *.f -fsanitize=address

*and* starting R like this (GNU/Linux-specific):

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5 R

(The ASan version could be different on your machine.)

This gives us a line number (stack overflow in prephi.f:50). Somehow,
d2u is only of size 8 bytes (one double?) and is being overflown by
accessing its second index, despite it being declared as containing
`kstate` doubles and `k` being limited by `kstate`.

Here's how we can combine GDB and AddressSanitizer in dynamically
loaded code in R without recompililng R itself with ASan support:

R -d gdb
set environment LD_PRELOAD /usr/lib/x86_64-linux-gnu/libasan.so.5
set environment ASAN_OPTIONS abort_on_error=1
run

(GDB stops when ASan raises SIGABRT and lets us explore the program at
the time of the crash.)

Here I realised that d2u is a parameter, not a local variable. I
climbed two frames up into the getgl subroutine and found out that d2u
is defined as real(kind=8), without a dimension. This must be the
reason for the stack corruption. Same goes for d2zeta.

Unfortunately, demo.so keeps crashing R, seemingly because of memory
corruption (e.g. the function name turns out to be overwritten when
producing the error message 'array overflow in .Fortran("%s")'), but
only with certain combinations of build options, and ASan isn't helping
any more. An ASan build of R would probably help.

-- 
Best regards,
Ivan

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: asan_crash.txt
URL: <https://stat.ethz.ch/pipermail/r-package-devel/attachments/20220206/047d26f2/attachment.txt>