[R-SIG-Mac] Using sanitizers in MacOS builds

[Duncan Murdoch]

This was very helpful.  Turns out one loop decremented using i-- when it 
should have had --i.  That overwrote element "-1" of a vector, and 
caused a later crash.  One of the sanitizers caught the out-of-bounds write.

Duncan Murdoch

On 2025-12-10 9:24 a.m., Duncan Murdoch wrote:
> Thanks.  With any luck, these will show my error, and I can make some
> progress.
> 
> Duncan Murdoch
> 
> On 2025-12-10 7:29 a.m., Prof Brian Ripley wrote:
>> On 10/12/2025 11:13, Duncan Murdoch wrote:
>>> I am making some large changes to rgl, and something I've done is
>>> causing crashes which I'm having trouble tracking down.  These are
>>> likely things that one of the sanitizers could detect, so I'd like to do
>>> a build using one.
>>>
>>> The WRE manual gives partial instructions about how to enable them.  It
>>> first says to put these lines in ~/.R/Makevars:
>>>
>>>      CC = gcc -std=gnu99 -fsanitize=address -fno-omit-frame-pointer
>>>      CXX = g++ -fsanitize=address -fno-omit-frame-pointer
>>>      FC = gfortran -fsanitize=address
>>
>> Not for macOS ....
>>
>>> These don't work for me on my Mac, generating lots of compile errors
>>> that I don't see with the default settings.
>>>
>>> Later WRE gives other instructions for clang 18.  What's not clear to me
>>> is what I should do for the default compiler tools (including clang 16)
>>> from mac.r-project.org.  Can anyone suggest what I should do for this
>>> situation:
>>>
>>>     - using the default build of R from CRAN
>>
>> AFAIK you need to build R with sanitizers: it might be possible without
>> but the issues often are partially in the way the package calls back into R.
>>
>>>     - using the libs and tools from mac.r-project.org
>>
>> In principle this might needs libs built against the sanitizers but I
>> have only seen one instance over the years.  Normally you will get less
>> (but still useful) information.
>>
>> Anything built with sanitizers needs to be of the same compiler version
>> as used for R (in Linux experience, even the same patch version).
>>
>>>     - trying to build rgl with the sanitizer options
>>
>> See
>>
>> https://www.stats.ox.ac.uk/pub/bdr/M1-SAN/README.txt
>>
>> which is tested daily, but only with up-to-date macOS and tools.
>>
>> I don't think the gfortran used on macOS has sanitizer support, but it
>> any case GCC's is not compatible with LLVM's.  So this is C/C++ only.
>>
>