[R-SIG-Mac] R 4.0 toolchain

Thu Apr 2 09:34:14 CEST 2020

Thanks Simon.
Branching off here to discuss the toolchain in general.
I see the danger posting not 100% correct scripts and in blog posts in general because they might get outdated and not always reflect the official documentation.
On the other side: What is the middle way? Arguably installing a robust toolchain (on Catalina especially but for macOS in general) is not just done with the installation of the signed package.
Most normal users are not aware about all the little specialties on macOS (missing openMP enabled compiler, SDK issues) and why adjustments might be needed.

Even though the idea is to require as less manual configuration as possible, this is not always practical in the end (unless all maintainers agree on configuring their package without the need of an openMP enabled C compiler for example).
And I also think that “installing from source” is not a dev-only thing.

1. By using the Apple internal clang as the default, will maintainers be indirectly forced to remove the dependency on an openMP enabled compiler with respect to getting source installations on macOS working?
2. Can there be an official section easily accessible for users which recommends which compiler to use (and how to set the Makevars) if users really need/want to do so?
    a. Regarding openMP enabled compilers: llvm is probably the one with the least differences to Apples clang. It can be installed via brew (brew install llvm) and then linked in Makevars. Would that be an alternative?
3. SDK: I did not wanted to induce that setting the SDK vars is needed for the CRAN build chain (maybe we got each other wrong here). However, as of now, there are real issues with SDK 10.15 that result in errors that are really hard to track down or even fall under the radar when testing (if someone does not run Catalina). Therefore I am linking locally against SDK 10.13 since some days and that resolved many issues (see again the Rcpp issue and the igraph issue). Since Catalina is the latest release and many people run it there should be test/instructions somewhere on mac.rproject that deal with this/suggest not to use SDK 10.15 for R IMO. Maybe this issue deserves a special thread.

Cheers, Patrick

On 1. Apr 2020, 22:29 +0200, Simon Urbanek <simon.urbanek using r-project.org>, wrote:
> Patrick,
>
> firstly, please don't post such things - they are often wrong (as are parts of what you included it this e-mail) and it's impossible for us to track all blogs that have incorrect advice. Unfortunately, a lot of the issues we see out there are due to people finding bad advice and using it. Ideally, it should be sufficient to point to the official R documentation since that is the canonical source. If you post a blog, it should point to the official documentation. If that documentation is missing or not covering a need, please post here so we can fix it. It is more important that the official source is complete and much better for the community than to have random tidbits that may or may not be correct floating around.
>
> As for the script, please do not use R tar balls - first, you're picking the wrong one (there is no R-devel for el-capitan. R 4.0.0 pre-releases are in R-4-0-branch and they are built for high sierra). Second, the tar ball doesn't have the entitlements enabled so it doesn't match the release. Please use the signed R package. This may change, but with the complications of the Apple notarization process that's what it is today.
>
> Then there is the question of the SDK. What are you trying to setup here? What you posted here has nothing to do with the CRAN setup. You don't need 10.13 SDK to compile igraph, it compiles just fine using stock command line tools on High Sierra which is what we use on CRAN. You can, of course, pick any SDK you like, but please don't tell people that's what they should do. One of the most annoying issues is that people are recommending messing with Makevars which is incredibly fragile and will cause things to break left and right with any update. It's ok to do that for a very special purpose, but no regular user should have Makevars in their home.
>
> So if you really want to serve the community, please suggest improvements to the official documentation and post them here. If you have special needs, ask about them here.
>
> Thanks,
> Simon
>
>
>
> > On 1/04/2020, at 9:47 PM, Patrick Schratz <patrick.schratz using gmail.com> wrote:
> >
> > Thanks Simon,
> >
> > This simplifies things a lot and clears up the process. While the instructions on https://mac.r-project.org/ are more clear now I think there is still simplification needed for the install process and custom modifications that are needed.
> > This not only applies to the dev page but finally also to production end users if the SDK 10.15 issues (see below) persists.
> >
> > I’ve scripted all installation steps (R-devel, gfortran, SDK 10.13) and will probably write a blog post about it to make the R community more aware about it. Let me know if anything is wrong about it - otherwise I’ll use the code below in the blog post.
> >
> > # install R-devel
> > wget https://mac.r-project.org/el-capitan/R-devel/R-devel-el-capitan-sa-x86_64.tar.gz
> > sudo tar fvxz R*.tar.gz -C /
> > rm R-devel-el-capitan-sa-x86_64.tar.gz
> >
> > # install gfortran
> > wget https://github.com/fxcoudert/gfortran-for-macOS/releases/download/8.2/gfortran-8.2-Mojave.dmg
> > sudo hdiutil attach gfortran*.dmg
> > sudo installer -package /Volumes/gfortran*/gfortran*/gfortran*.pkg -target /
> > sudo hdiutil detach /Volumes/gfortran-8.2-Mojave
> >
> > # install SDK10.13
> > wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX10.13.sdk.tar.xz
> > tar fvxz MacOSX10.13.sdk.tar.xz
> > sudo mv MacOSX10.13.sdk /Library/Developer/CommandLineTools/SDKs/
> > rm -rf MacOSX10.13*
> >
> > In addition people need to set custom CFLAGS in ~/.R/Makevars to actually use it:
> >
> > CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> > CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> > CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> > CPPFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> >
> > In addition there really seem to be major problems regarding the 10.15 SDK. For example, {igraph} which is a package on which lots of other packages depend, cannot be installed from source with SDK 10.15 but works with SDK 10.13.
> >
> > Patrick
> > On 1. Apr 2020, 09:38 +0200, Simon Urbanek <simon.urbanek using r-project.org>, wrote:
> > > Hervé,
> > >
> > >
> > > > On 1/04/2020, at 6:19 PM, Hervé Pagès <hpages using fredhutch.org> wrote:
> > > >
> > > > Thanks Simon. A couple of days ago we've started to use the R 4.0.0 alpha build from https://mac.r-project.org/ for the Bioconductor build system. Bioconductor packages depend on thousands of CRAN packages and one thing that makes it hard for us and for our users to build/check Bioconductor packages at the moment is the absence of Mac binary packages for R 4.0.0 on CRAN. Do you have an estimate of when they will become available?
> > > >
> > >
> > > Both the packages and R are now available. I always recommend using the Mac-master https://mac.R-project.org as "mirror" since that one is most up-to-date.
> > >
> > >
> > > > Another question is where they are going to be hosted on CRAN. We need to follow CRAN's layout for Bioconductor package repositories.
> > > >
> > >
> > >
> > > We have removed the extra layer, the package type is back to "mac.binary", so bin/macosx/contrib is the location.
> > >
> > > Cheers,
> > > Simon
> > >
> > >
> > >
> > > > Thanks!
> > > > H.
> > > >
> > > >
> > > > On 3/31/20 21:27, Simon Urbanek wrote:
> > > > > Dear Mac users,
> > > > > R 4.0.0 will be using an entirely new toolchain, entirely new build system on entirely new macOS version and hardware. Therefore I would like to ask you kindly to test the binaries from
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mac.R-2Dproject.org&d=DwIFAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=-PNBcsZhHb1MLy6fZakmLP6WDLMwL-VigJVzeFtPxOI&s=FJYxweW9WiMXbNMPpMM0bHXy020Gx9P-dHMrWmt7YzE&e=
> > > > > before the release as much as you can. Raising any issues after the release is too late! So please, please, test the pre-releases. Report any issues either directly to me or this mailing list.
> > > > > The nightly builds are signed, but not necessarily notarized. However, the build fulfils Apple's conditions and is known to pass notarization (in fact the the package available for download today is actually notarized) so it should be a good test for the release which will be notarized and should work on Catalina.
> > > > > For those that want to replicate our setup - technical details: we are now building with macOS 10.13 (High Sierra) as target (i.e. the oldest supported system), regular Apple Xcode/command line tools and GNU Fortran 8.2. R builds are running on macOS 10.15 (Catalina) with Xcode 11.4 using macOS 10.13 target. Packages are built on macOS 10.13 VMs with just Apple command line tools (this should make it easy to replicate the setup using Travis, for example). All 3rd party libraries that CRAN uses are available in https://urldefense.proofpoint.com/v2/url?u=http-3A__mac.r-2Dproject.org_libs-2D4_&d=DwIFAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=-PNBcsZhHb1MLy6fZakmLP6WDLMwL-VigJVzeFtPxOI&s=3ZbcXBticg_8MfXltGKdQ9SFNqqeLqqwK_pUhEet7QY&e=
> > > > > The new R build system is in
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__svn.r-2Dproject.org_R-2Ddev-2Dweb_trunk_QA_Simon_R4&d=DwIFAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=-PNBcsZhHb1MLy6fZakmLP6WDLMwL-VigJVzeFtPxOI&s=Mmt7_pVM5aUDR3hz93-e-PtadG7wSYpaZe8Ayi5Pv4I&e=
> > > > > Packages build system has not changed and is in
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__svn.r-2Dproject.org_R-2Ddev-2Dweb_trunk_QA_Simon_packages&d=DwIFAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=-PNBcsZhHb1MLy6fZakmLP6WDLMwL-VigJVzeFtPxOI&s=eId6n08LVeIAMhMXe3jvLFKn9QGi4_by8owFreDDc84&e=
> > > > > We also plan to have a mac-builder available with similar function as the win-builder where pre-submission tests can be performed and potentially a Travis template.
> > > > > Please test R pre-releases and provide feedback!
> > > > > Thanks,
> > > > > Simon
> > > > > _______________________________________________
> > > > > R-SIG-Mac mailing list
> > > > > R-SIG-Mac using r-project.org
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dmac&d=DwIFAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=-PNBcsZhHb1MLy6fZakmLP6WDLMwL-VigJVzeFtPxOI&s=ubqivtpb1OZ2I0MEbVvGlGDK-tfti3DPtIv4LQ-Ks6A&e=
> > > >
> > > > --
> > > > Hervé Pagès
> > > >
> > > > Program in Computational Biology
> > > > Division of Public Health Sciences
> > > > Fred Hutchinson Cancer Research Center
> > > > 1100 Fairview Ave. N, M1-B514
> > > > P.O. Box 19024
> > > > Seattle, WA 98109-1024
> > > >
> > > > E-mail: hpages using fredhutch.org
> > > > Phone: (206) 667-5791
> > > > Fax: (206) 667-1319
> > > >
> > > > _______________________________________________
> > > > R-SIG-Mac mailing list
> > > > R-SIG-Mac using r-project.org
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> > >
> > > _______________________________________________
> > > R-SIG-Mac mailing list
> > > R-SIG-Mac using r-project.org
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mac
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-SIG-Mac mailing list
> > R-SIG-Mac using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>

	[[alternative HTML version deleted]]