[R] Stack smashing again, redux.

[Rolf Turner]

On Mon, 11 Apr 2022 13:00:21 +0300
Ivan Krylov <krylov.r00t using gmail.com> wrote:

> Depending on one's dislike for different approaches, it's possible to
> use AddressSanitizer with R in at least three different ways, probably
> more. There's the Rocker project providing Docker images of R already
> built with sanitizer support [1] (but then you have to install
> Docker), there's compiling R from source with -fsanitize=address in
> CFLAGS, FFLAGS, MAIN_LDFLAGS [2] (but then you have to compile R from
> source) and there's the partially manual way I've mentioned before
> (which involves modifying one's global configuration files and
> reverting the changes later):
> 
> 1. Temporarily add the following to ~/.R/Makevars:
>    FFLAGS=-g -Og -fsanitize=address
>    FCFLAGS=-g -Og -fsanitize=address
> 2. Compile the shared object using:
>    R CMD SHLIB -o hah.so *.f -fsanitize=address
> 3. Run R as follows:
>    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5 R
> 
> With AddressSanitizer, I get the following stack buffer overflow error
> message:
> 
> > xxx <- get.gl(theta.new,sigma,X,y,cf,state,"Dbd",size,nbot,ntop)
> =================================================================
> ==716==ERROR: AddressSanitizer: stack-buffer-overflow on address
> 0x7ffea5d32458 at pc 0x7f282c38498d bp 0x7ffea5d32 000 sp
> 0x7ffea5d31ff8 WRITE of size 8 at 0x7ffea5d32458 thread T0
>     #0 0x7f282c38498c in derivfdbd_ /home/ivan/derivfdbd.f:16
>     #1 0x7f282c3852d6 in derivf_ /home/ivan/derivf.f:17
>     #2 0x7f282c385dd3 in getgl_ /home/ivan/getgl.f:19
>     #3 0x7f282c4e5eec in do_dotCode src/main/dotcode.c:1994
> 
> (skipping unrelated stack frames)
> 
> Address 0x7ffea5d32458 is located in stack of thread T0 at offset 312
> in frame #0 0x7f282c385a6f in getgl_ /home/ivan/getgl.f:2
> 
>   This frame has 7 object(s):
>     [32, 36) 'nd'
>     [96, 104) 'd2aa'
>     [160, 168) 'd2ab'
>     [224, 232) 'd2bb'
>     [288, 296) 'd2f' <== Memory access at offset 312 overflows this
> variable [352, 360) 'd2u'
>     [416, 424) 'd2zeta'
> 
> (skipping more unnecessary information)
> 
> Adding "dimension d2f(kstate,npar,npar)" to getgl.f seems to prevent
> this or any other error from happening, though I can't judge the
> calculation results; they could indicate some other problem with
> memory management.

The fact that d2f was not dimensioned in getgl was indeed the loony
that I had not spotted.  I fixed that, and another couple of
deficiencies that became apparent to me once the d2f deficiency was
drawn to my attention, and the example ran, without any stack smashing.

Thank you Ivan, from the bottom of my heart.  There is no way that I
could have got to this point without your assistance.

I would like to add that I tried your third method of applying
AddressSanitizer, and it "worked", but yet it didn't work.  (This is
the story of my life; things that work for everyone else do not work
for me. :-( ) I have attached the output from running "scr" with
AddressSanitizer enabled.  (*Before* making the necessary corrections
to getgl.)

There is nothing in the output to indicate where the problem lies.
Unlike the output that you (Ivan) observed, there is no reference at
all to getgl.  This doesn't matter now, since the problem has been
isolated and fixed, but it is an interesting demonstration of the way
that everything goes wrong for me!  See, I'm *not* paranoid!

Just as a semi-final word (and as a further demonstration of how
everything goes wrong for me --- I am *not* paranoid! :-) ) I will
mention that the real application (in contrast with my reprex) is still
not actually working.  The algorithm is not converging.  However the
stack-smashing has been eliminated so I *should* be able to track down
where things are going off the rails.

Final word:  thanks again, *HUGELY* to you, Ivan Krylov.

cheers,

Rolf

-- 
Honorary Research Fellow
Department of Statistics
University of Auckland
Phone: +64-9-373-7599 ext. 88276

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: addSanOut.txt
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20220412/84ce2d78/attachment.txt>