[Rd] segfault issue with parallel::mclapply and download.file() on Mac OS X

[Jeroen Ooms]

On Thu, Oct 4, 2018 at 6:12 PM Tomas Kalibera <tomas.kalibera using gmail.com> wrote:
>
>
> Thanks for the report, but unfortunately I cannot reproduce on my system
> (either macOS nor Linux, from the command line) to debug. Did you run
> this in the command line version of R?

It depends on which version of MacOS that you are using and
specifically which TLS back-end curl has been configured with. When
libcurl uses DarwinSSL, it may crash when opening HTTPS connections in
a fork because CoreFoundation is not fork-safe. OTOH when using
OpenSSL or LibreSSL for TLS, you usually get away with forking (though
it's still bad practice).

The standard version of libcurl that ships with MacOS was using
CoreFoundation until 10.12 but starting 10.13 they switched to
LibreSSL in order to support HTTP/2. See curl --version or
curl::curl_version() for your local config. Don't count in this
though, Apple might switch back to the fork-unsafe DarwinSSL once they
support ALPN, which is needed for HTTP/2.

As Gabor already suggested, libcurl has built-in systems for
concurrent connections. The curl package exposes this via multi_add
function. Not only is this safer than forking, it will be much faster
because it takes advantage of HTTP keep-alive and when supported it
uses HTTP2 multiplexing which allows efficiently performing thousands
of concurrent HTTPS requests over a single TCP connection.