[R-SIG-Mac] [Rd] Rebuilding and re-checking of downstream dependencies on CRAN Mac build machines

Winston Chang w|n@tonch@ng1 @end|ng |rom gm@||@com
Fri Mar 27 04:18:42 CET 2020 

You're right that the error originally reported in
https://github.com/rstudio/httpuv/issues/260 wasn't due to Rcpp. When I was
investigating the issue, I encountered a different error (the output of
which I posted to
https://gist.github.com/wch/c70b438381c9d2a8b1f917b054e0ba7e) which was
caused by Rcpp -- it's the same uuid_t/uid_t error that was seen in
https://github.com/RcppCore/Rcpp/issues/1046. More details on that below if
anyone is interested.

Regarding the bug from https://github.com/RcppCore/Rcpp/issues/1046, the
fix was merged, but it has not been released to CRAN.

It sounds like perhaps the 10.11 SDK may be a factor. I'll have to check
whether any of us who have tried to build these various packages have been
using the 10.11 SDK.

===========================

This is how I tested that Kevin's fix from
https://github.com/RcppCore/Rcpp/pull/1047 fixed my httpuv problem. I
installed the commit with the fix, and then installed httpuv from CRAN, and
that works. Then I installed the parent commit that Kevin branched from,
and then tried to install httpuv from CRAN, but that failed.

Here are the commands I used to test:
  # Installing Kevin's fix from Rcpp #1047 works
  remotes::install_github("RcppCore/Rcpp using 46bf2e2")
  install.packages("httpuv", type = "source")

  # Doesn't work (this is the parent of the 46bf2e2)
  remotes::install_github("RcppCore/Rcpp using 68a5489")
  install.packages("httpuv", type = "source")  # Results in the
uuid_t/uid_t errors

-Winston


On Thu, Mar 26, 2020 at 5:07 PM Simon Urbanek <simon.urbanek using r-project.org>
wrote:
>
> [moving to R-SIG-Mac where it belongs]
>
> Winston,
>
> the error in the issue
> https://github.com/rstudio/httpuv/issues/260
> seems to come from libuv itself - possibly a compatibility issue with the
latest Apple Security framework.
>
> The issue in
> https://github.com/RcppCore/Rcpp/issues/1046
> is genuine and a bug in Rcpp as pointed out by Kevin. My understanding is
that it was merged (I was following it on Rcpp mailing list and it looked
that it's taken care of).
>
> Please note that another important factor is SDK used - the CRAN
El-Capitan builds are done against the 10.11 SDK:
>
> $ ls -l /Library/Developer/CommandLineTools/SDKs/
> total 8
> lrwxr-xr-x  1 root  wheel   15  6 Apr  2019 MacOSX.sdk -> MacOSX10.11.sdk
> drwxr-xr-x  5 root  wheel  170  7 Aug  2017 MacOSX10.11.sdk
>
> The R 4.0.0 builds will be done against the 10.13 SDK. You can check
against later SDKs to make sure your package doesn't break on deprecated
features, but generally, the CRAN builds are conservative to make sure the
binaries run on older systems since not all users upgrade to the latest
macOS (for a very good reason as we've seen with the carnage caused by
Catalina).
>
> To make it very clear, CRAN's goal is not to detect incompatibilities
with latest SDK, but rather to provide stable binaries, so if a package
breaks outside of CRAN, it is wise for the package developers to fix such
issues, but fortunately they don't affect the availability on CRAN. That is
also one of the reasons why CRAN is more stable than the 3rd party package
managers.
>
> Cheers,
> Simon
>
>
>
> > On 27/03/2020, at 10:41 AM, Winston Chang <winstonchang1 using gmail.com>
wrote:
> >
> > Simon,
> >
> > The link I provided to the httpuv issue has detailed information about
> > the error:
> >  https://github.com/rstudio/httpuv/issues/260
> >
> > The error occurs when I run the following (although note, depending on
> > which mirror you use, the recent CRAN server issues may result in
> > problems downloading the packages):
> >  install.packages("Rcpp")
> >  install.packages("httpuv", type = "source")
> >
> > Here's the full output:
> >  https://gist.github.com/wch/c70b438381c9d2a8b1f917b054e0ba7e
> >
> > Here's an example of the errors:
> >
> > In file included from staticpath.cpp:1:
> > In file included from ./staticpath.h:7:
> > In file included from
/Users/winston/R/3.6/BH/include/boost/optional.hpp:15:
> > In file included from
> > /Users/winston/R/3.6/BH/include/boost/optional/optional.hpp:28:
> > In file included from
> > /Users/winston/R/3.6/BH/include/boost/core/addressof.hpp:17:
> > In file included from
/Users/winston/R/3.6/BH/include/boost/config.hpp:57:
> > In file included from
> > /Users/winston/R/3.6/BH/include/boost/config/platform/macos.hpp:28:
> > In file included from
> >
/Users/winston/R/3.6/BH/include/boost/config/detail/posix_features.hpp:18:
> >
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:664:27:
> > error: unknown type name 'uuid_t'; did you mean 'uid_t'?
> > int      getwgroups_np(int *, uuid_t);
> >                              ^
> >
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h:31:31:
> > note: 'uid_t' declared here
> > typedef __darwin_uid_t        uid_t;
> >                              ^
> >
> > This issue describes the same problem:
> >  https://github.com/RcppCore/Rcpp/issues/1046
> >
> > And it has been fixed in the development version of Rcpp by:
> >   https://github.com/RcppCore/Rcpp/pull/1047
> >
> > I know that Kevin Ushey has tried building with the CRAN-recommended
> > clang 7 toolchain and encountered the same errors. I haven't done that
> > yet myself, though.
> >
> > -Winston
> >
> > On Thu, Mar 26, 2020 at 4:00 PM Simon Urbanek
> > <simon.urbanek using r-project.org> wrote:
> >>
> >> Winston,
> >>
> >> the Mac CRAN build builds a package only if either is true:
> >> 1) the package has not passed checks
> >> 2) there is a new version of the package since last successful
build+check
> >>
> >> The old build machine doesn't have the capacity to do full re-builds
(it would take over 24h - currently the nightly build of packages takes
16-22 hours), but we're currently building a new setup for R 4.0.0 on new
hardware and as a part of it we are planning to setup a "mac-builder"
similar to what is currently available for Windows.
> >>
> >> That said, I have run httpuv by hand on the CRAN build machine
(against Rcpp 1.0.4) and I saw no issues. I have seen the discussion on
Rcpp, but so far no one actually posted details of what is breaking (nor do
your links include any actual details on this). I'd love to help, but the
lack fo a useful report makes this impossible. If you have any actual
leads, please post them. The CRAN machine uses the tools that are available
on CRAN: https://cran.r-project.org/bin/macosx/tools/ (clang-7 and
gfortran-6.1 for 3.6.x)
> >>
> >> Cheers,
> >> Simon
> >>
> >>
> >>> On 27/03/2020, at 7:38 AM, Winston Chang <winstonchang1 using gmail.com>
wrote:
> >>>
> >>> I have two questions about the CRAN machines that build binary
> >>> packages for Mac. When a new version of a package is released,
> >>> (A) Do the downstream dependencies get re-checked?
> >>> (B) Do the downstream dependencies get re-built?
> >>>
> >>> I have heard (but do not know for sure) that the answer to (A) is no,
> >>> the downstream dependencies do not get rechecked.
> >>>
> >>> From publicly available information on the CRAN web server, it looks
> >>> like the answer to (B) is also no, the downstream dependencies do not
> >>> get rebuilt. Looking at
> >>> https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/, I see
> >>> the following dates for these binary packages:
> >>>
> >>> - Rcpp_1.0.4.tgz: 2020-03-18
> >>> - httpuv_1.5.2.tgz: 2019-09-12
> >>> - dplyr_0.8.5.tgz: 2020-03-08
> >>>
> >>> Rcpp was released recently, and httpuv and dplyr (which are downstream
> >>> dependencies of Rcpp) have older dates, which indicates that these
> >>> binary packages were not rebuilt when Rcpp was released.
> >>>
> >>> In my particular case, I'm interested in the httpuv package (which I
> >>> maintain). I and several others have not been able to get the CRAN
> >>> version of httpuv to compile using the CRAN version of Rcpp on Mac.
> >>> (It seems to compile fine on other platforms.) I have heard from
> >>> maintainers of other Rcpp-dependent packages that they also can't get
> >>> their packages to compile on Mac, using both the default Mac compiler
> >>> toolchain and the CRAN-recommended toolchain, which uses clang 7.
> >>>
> >>> For more technical details about the cause of breakage, see:
> >>> https://github.com/RcppCore/Rcpp/issues/1060
> >>> https://github.com/rstudio/httpuv/issues/260
> >>>
> >>> If the CRAN Mac build machine is indeed able to build httpuv against
> >>> the current version of Rcpp, it would be really helpful to have more
> >>> information about the system configuration. If it is not able to
> >>> rebuild httpuv and other packages against Rcpp, then this is a
> >>> problem. Among other things, it prevents people from building their
> >>> packages from source using CRAN versions of packages, and it also
> >>> means that none of these packages can release a new version, because
> >>> the binaries can't be built on Mac.
> >>>
> >>> -Winston
> >>>
> >>> ______________________________________________
> >>> R-devel using r-project.org mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>>
> >>
> >
>

	[[alternative HTML version deleted]]

More information about the R-SIG-Mac mailing list