[R-SIG-Mac] [Rd] Rebuilding and re-checking of downstream dependencies on CRAN Mac build machines

Simon Urbanek @|mon@urb@nek @end|ng |rom R-project@org
Thu Mar 26 23:07:17 CET 2020 

[moving to R-SIG-Mac where it belongs]

Winston,

the error in the issue 
https://github.com/rstudio/httpuv/issues/260
seems to come from libuv itself - possibly a compatibility issue with the latest Apple Security framework.

The issue in 
https://github.com/RcppCore/Rcpp/issues/1046
is genuine and a bug in Rcpp as pointed out by Kevin. My understanding is that it was merged (I was following it on Rcpp mailing list and it looked that it's taken care of).

Please note that another important factor is SDK used - the CRAN El-Capitan builds are done against the 10.11 SDK:

$ ls -l /Library/Developer/CommandLineTools/SDKs/
total 8
lrwxr-xr-x  1 root  wheel   15  6 Apr  2019 MacOSX.sdk -> MacOSX10.11.sdk
drwxr-xr-x  5 root  wheel  170  7 Aug  2017 MacOSX10.11.sdk

The R 4.0.0 builds will be done against the 10.13 SDK. You can check against later SDKs to make sure your package doesn't break on deprecated features, but generally, the CRAN builds are conservative to make sure the binaries run on older systems since not all users upgrade to the latest macOS (for a very good reason as we've seen with the carnage caused by Catalina). 

To make it very clear, CRAN's goal is not to detect incompatibilities with latest SDK, but rather to provide stable binaries, so if a package breaks outside of CRAN, it is wise for the package developers to fix such issues, but fortunately they don't affect the availability on CRAN. That is also one of the reasons why CRAN is more stable than the 3rd party package managers.

Cheers,
Simon



> On 27/03/2020, at 10:41 AM, Winston Chang <winstonchang1 using gmail.com> wrote:
> 
> Simon,
> 
> The link I provided to the httpuv issue has detailed information about
> the error:
>  https://github.com/rstudio/httpuv/issues/260
> 
> The error occurs when I run the following (although note, depending on
> which mirror you use, the recent CRAN server issues may result in
> problems downloading the packages):
>  install.packages("Rcpp")
>  install.packages("httpuv", type = "source")
> 
> Here's the full output:
>  https://gist.github.com/wch/c70b438381c9d2a8b1f917b054e0ba7e
> 
> Here's an example of the errors:
> 
> In file included from staticpath.cpp:1:
> In file included from ./staticpath.h:7:
> In file included from /Users/winston/R/3.6/BH/include/boost/optional.hpp:15:
> In file included from
> /Users/winston/R/3.6/BH/include/boost/optional/optional.hpp:28:
> In file included from
> /Users/winston/R/3.6/BH/include/boost/core/addressof.hpp:17:
> In file included from /Users/winston/R/3.6/BH/include/boost/config.hpp:57:
> In file included from
> /Users/winston/R/3.6/BH/include/boost/config/platform/macos.hpp:28:
> In file included from
> /Users/winston/R/3.6/BH/include/boost/config/detail/posix_features.hpp:18:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:664:27:
> error: unknown type name 'uuid_t'; did you mean 'uid_t'?
> int      getwgroups_np(int *, uuid_t);
>                              ^
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
> note: 'uid_t' declared here
> typedef __darwin_uid_t        uid_t;
>                              ^
> 
> This issue describes the same problem:
>  https://github.com/RcppCore/Rcpp/issues/1046
> 
> And it has been fixed in the development version of Rcpp by:
>   https://github.com/RcppCore/Rcpp/pull/1047
> 
> I know that Kevin Ushey has tried building with the CRAN-recommended
> clang 7 toolchain and encountered the same errors. I haven't done that
> yet myself, though.
> 
> -Winston
> 
> On Thu, Mar 26, 2020 at 4:00 PM Simon Urbanek
> <simon.urbanek using r-project.org> wrote:
>> 
>> Winston,
>> 
>> the Mac CRAN build builds a package only if either is true:
>> 1) the package has not passed checks
>> 2) there is a new version of the package since last successful build+check
>> 
>> The old build machine doesn't have the capacity to do full re-builds (it would take over 24h - currently the nightly build of packages takes 16-22 hours), but we're currently building a new setup for R 4.0.0 on new hardware and as a part of it we are planning to setup a "mac-builder" similar to what is currently available for Windows.
>> 
>> That said, I have run httpuv by hand on the CRAN build machine (against Rcpp 1.0.4) and I saw no issues. I have seen the discussion on Rcpp, but so far no one actually posted details of what is breaking (nor do your links include any actual details on this). I'd love to help, but the lack fo a useful report makes this impossible. If you have any actual leads, please post them. The CRAN machine uses the tools that are available on CRAN: https://cran.r-project.org/bin/macosx/tools/ (clang-7 and gfortran-6.1 for 3.6.x)
>> 
>> Cheers,
>> Simon
>> 
>> 
>>> On 27/03/2020, at 7:38 AM, Winston Chang <winstonchang1 using gmail.com> wrote:
>>> 
>>> I have two questions about the CRAN machines that build binary
>>> packages for Mac. When a new version of a package is released,
>>> (A) Do the downstream dependencies get re-checked?
>>> (B) Do the downstream dependencies get re-built?
>>> 
>>> I have heard (but do not know for sure) that the answer to (A) is no,
>>> the downstream dependencies do not get rechecked.
>>> 
>>> From publicly available information on the CRAN web server, it looks
>>> like the answer to (B) is also no, the downstream dependencies do not
>>> get rebuilt. Looking at
>>> https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
>>> the following dates for these binary packages:
>>> 
>>> - Rcpp_1.0.4.tgz: 2020-03-18
>>> - httpuv_1.5.2.tgz: 2019-09-12
>>> - dplyr_0.8.5.tgz: 2020-03-08
>>> 
>>> Rcpp was released recently, and httpuv and dplyr (which are downstream
>>> dependencies of Rcpp) have older dates, which indicates that these
>>> binary packages were not rebuilt when Rcpp was released.
>>> 
>>> In my particular case, I'm interested in the httpuv package (which I
>>> maintain). I and several others have not been able to get the CRAN
>>> version of httpuv to compile using the CRAN version of Rcpp on Mac.
>>> (It seems to compile fine on other platforms.) I have heard from
>>> maintainers of other Rcpp-dependent packages that they also can't get
>>> their packages to compile on Mac, using both the default Mac compiler
>>> toolchain and the CRAN-recommended toolchain, which uses clang 7.
>>> 
>>> For more technical details about the cause of breakage, see:
>>> https://github.com/RcppCore/Rcpp/issues/1060
>>> https://github.com/rstudio/httpuv/issues/260
>>> 
>>> If the CRAN Mac build machine is indeed able to build httpuv against
>>> the current version of Rcpp, it would be really helpful to have more
>>> information about the system configuration. If it is not able to
>>> rebuild httpuv and other packages against Rcpp, then this is a
>>> problem. Among other things, it prevents people from building their
>>> packages from source using CRAN versions of packages, and it also
>>> means that none of these packages can release a new version, because
>>> the binaries can't be built on Mac.
>>> 
>>> -Winston
>>> 
>>> ______________________________________________
>>> R-devel using r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>> 
>> 
>

More information about the R-SIG-Mac mailing list