[R-SIG-Mac] Libre SSL bug on MacOS Monterey => error in download.file()

[Petr Bouchal]

Šimone,

Thanks for this. I understand your concern about being responsible for the SSL side and can just about imagine the technical complexity of getting it done. Many thanks also for the trial build, I very much appreciate you doing this - once I can get it tested on an M1 machine, I will report back but it’s encouraging to hear that it is feasible and solves the issue.

As for Apple’s approach to SSL, my understanding from the bits of information I have been able to find is that they provide very little guarantee/stability/documentation around this, as also attested by the flip flopping you mentioned.

Thanks again.

Kind regards
Petr

> On 11. 1. 2022, at 22:12, Simon Urbanek <simon.urbanek using R-project.org> wrote:
> 
> Petře,
> 
> thanks, for the detailed analysis. It is rather curious that the issue appears only on _newer_ systems - we are more used to issues due to older CA chains and similar. It looks like an Apple bug on specific systems, so hopefully it will be fixed eventually. In general I was trying to avoid having to supply our own SSL library since that opens a whole can of worms - on one hand due the dependency issues (which libraries get compiled against what) and on the other hand we become responsible for security updates.
> 
> Thanks to Jeroen for the work-around (CURL_SSL_BACKEND=SecureTransport), using the native API is certainly preferred, there have been several issues with both OpenSSL and LibreSSL before. It seems that Apple has been flip-flopping with libcurl a lot - on El Capitan it was shipped with SecureTransport, on High-Sierra with LibreSSL, on Catalina and higher with both, but Libre the default.
> 
> I am somewhat less apprehensive to use static libcurl for R than SSL libraries as the fallout is a bit smaller. As a trial I have added static curl[2] which is close to the Apple build minus MultiSSL to big-sur nightly builds of R[3] and as expected that solves the problem. It may not be entirely unproblematic for package space, because packages often forget to prepend  --static when using static builds of libraries, and so do other dependencies that may use curl, but I'll see what comes out of it.
> 
> Cheers,
> Šimon
> 
> [1] - https://github.com/R-macos/recipes
> [2] - https://github.com/R-macos/recipes/blob/add-ons/recipes/curl
> [3] - https://mac.r-project.org/
> 
> 
> 
>> On 10/01/2022, at 11:22 PM, Petr Bouchal <pbouchal using gmail.com> wrote:
>> 
>> Dear all,
>> 
>> In brief: on Monterey, R cannot reach certain web domains due to a bug in Libre SSL - and perhaps not relying on system curl/openssl in R would be a systematic solution to this and símilar issues.
>> 
>> Specifically: on MacOS Monterey 12.1 using R 4.1.2, download.file() and other functions that rely on system-provided curl/openssl/Libre SSL (including in the curl package) have been failing on specific domains. 
>> 
>> So running 
>> 
>> download.file(“https://www.czso.cz/”, tempfile()) 
>> 
>> returns: 
>> 
>> status was ‘SSL connect error’
>> 
>> the underlying error being
>> 
>> error:06FFF089:digital envelope routines:CRYPTO_internal:bad key length.
>> 
>> This is caused by the Libre SSL bundled in MacOS Monterey and also affects several other domains, most notably https://libzip.org.
>> 
>> It is clearly an OS bug but infortunately also a situation where it affects R users because of how R relates to system libraries and is very difficult to work around.
>> 
>> It has manifested on CRAN (causing a package archival) and Github outside of R, so is not caused by a specific machine. It can be replicated on both M1 and Intel and also occurs when using curl in the system command line. 
>> 
>> The czso.cz domain is the Czech Statistical Office, which makes it quite important for a number of users, also of a package I maintain (czso) which relies on accessing this domain. I have reported this to the server admin but since the problem is in the OS, I do not expect them to be able to help. I am not an expert in web security so cannot tell if there is anything in the certificates which could be causing this. In browsers, no such issue occurs and the server is configured correctly as per ssllabs.com testing. I have also reported to Apple but it is unclear whether they will fix this given the rare nature of the issue.
>> 
>> It is difficult to work around even on individual machines as replacing the system curl/openssl requires steps beyond what a most users are comfortable with (or should be doing to begin with). Using HTTP instead of HTTPS does not work, nor does using curl —insecure and equivalents.
>> 
>> This brings back the question of whether R on MacOS should include its own openssl instead of relying on the system-provided library. This has been discussed on the r-devel list: https://stat.ethz.ch/pipermail/r-devel/2020-June/079657.html.
>> 
>> Apple also recommends against relying on shared openssl, if I understand this correctly: https://developer.apple.com/forums/thread/89051. Given Apple’s approach to openssl/Libre SSL in MacOS (the bundled Libre SSL version is 3 years old), such hard-to-handle issues are likely to reappear over time. (I don’t have in-depth knowledge of how R is compiled, so apologies for any inaccuracies; hopefully it is clear what I mean.)
>> 
>> I’d be grateful for any thoughts on how this might be handled in the specific case and perhaps generally.
>> 
>> Kind regards
>> Petr Bouchal
>> 
>> _______________________________________________
>> R-SIG-Mac mailing list
>> R-SIG-Mac using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>> 
>