[R-SIG-Mac] [External] Re: problem with Rprof

[iuke-tier@ey m@iii@g oii uiow@@edu]

On Tue, 8 Nov 2022, Simon Urbanek wrote:

>
>
>> On Nov 9, 2022, at 10:03 AM, Tomas Kalibera <tomas.kalibera using gmail.com> wrote:
>>
>>
>> On 11/7/22 01:58, luke-tierney using uiowa.edu wrote:
>>> On Sun, 6 Nov 2022, Simon Urbanek wrote:
>>>
>>>> Carl,
>>>>
>>>> first, setting such low interval won't work anyway - the overhead is bigger than the sampled time, so we should really not allow it to begin with (on my machine the timer signals arrive before anything can be done so you have to kill R and you get no output).
>>>>
>>>> That said, it crashes in doprof() which is called on all threads - the main R one is ok, but one of the other threads crashes in pthread_self(). At that time R is trying to propagate the signal from all threads to the main thread which seems odd to me (since the main thread already got the signal), I'm CCing Luke in the hope that he has any ideas. This may fall in the category of "don't do this" and the fix may be to set a lower bound on the interval.
>>>
>>> I can't reproduce this on Linux or macOS.
>>>
>>> On Linux only one thread receives a signal sent to a process, but the
>>> kernel picks which one if multiple threads have the signal unblocked,
>>> so we make sure the signal gets relayed to the main thread. If macOS
>>> behaves differently then someone who knows how signals and threads
>>> interact there would have to adjust this code.
>>
>> From my reading this is the same on macOS. The profiling signal is asynchronous, sent to the process, it will be served by one thread which is picked by the OS. POSIX doesn't say which thread is preferred.
>
>
> Yes, I saw the same with extra detail that thread signal blocking doesn't seem to necessarily work on macOS.
>
>
>> While some OSes prefer the main thread (I read macOS and Linux do, but from non-authoritative sources), R may also be embedded and not run on the main thread.
>>
>> We have to do something to ensure the R thread is not running while we sample its R stack, anyway. On Windows we suspend the R thread for that. On Unix we do the relaying.  We could in principle suspend the R thread on macOS as well, but would have to use Mach calls directly.
>>
>>> Disallowing such a low interval is reasonable, but if there is a real
>>> issue on macOS then it would only mask the problem.
>>
>> Yes. The key question is why pthread_self() crashed.
>
>
> Yes, that is the main mystery. Looking at the xnu kernel sources it is equivalent to pthread_getspecific(0) [since it's just the first slot in TSD] plus a check of a magic content in there. I suspect it's that check which segfaults for whatever reason. I wanted to see if just comparing the pointer from pthread_getspecific(0) instead of pthread_self() would work since we don't care if the pthread_t is valid as we only compare it to the main thread value (not that I would propose that as a fix since it's very implementation-specific, just curious), but I didn't get that far (I cannot really reproduce it - the closest I get is a mach exception under lldb).
>
>
>
>> Otherwise, from the stack trace, the behavior looks ok. The main thread (also R thread) is serving the signal, hence the signal is blocked, but it is received again, so another thread is picked to serve it, and it is relaying it to the main thread. One more thread is picked to serve it, and it crashes while calling pthread_self(). There is also one more thread not involved in the signal handling.
>>
>> POSIX statest that pthread_self() is async-signal-safe. macOS 12.6 manuals (sigaction) however doesn't include any pthread function in the list of async-signal-functions.
>>
>> We could do some work-around (hiding the problem a bit more) like exit from the handler if the signal is being served by another thread. We could also report such situation to indicate that the interval is unreasonable. But it would be good first to know for sure what caused the problem.
>>
>
> How can you check anything if pthread functions fail? If a simple pthead_self() crashes then I don't see how you can do anything since we don't even know what thread we are, cannot call mutexes etc.
>

Some random googling seems to suggest other systems have had issues
with pthread_self not being async-signal-safe on macOS. One example is
https://bugs.openjdk.org/browse/JDK-8235962. (Not directly about
pthread_self but looks like the same issue.) They suggest a possible
mach-specific work-around. We might be able to make use of something
along those lines with a modest amount of effort.

Another option is to look at using a timing thread similar to the
Windows approach. That could probably be designed to send the signal
directly to the main thread and avoid any pthread calls in the signal
handler. There is also a wishlist item from Lionel in bugzilla to
allow a realtime timer which might also be addressed at the same time.
On the other hand, this is all a fair amount of work and not really a
priority.

Best,

luke


> Cheers,
> Simon
>
>

-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney using uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu