[R-pkg-devel] Using parallel:::sendMaster &c. despite warnings

Fri Jul 23 19:28:31 CEST 2021

   Patches to parallel could certainly be *considered*, but people will 
probably want to argue about the details quite a bit.  I would suggest 
presenting/discussing your ideas for modification on r-devel before 
posting them as a 'wishlist' item to the R bug tracker. (If an R-core 
member participates in the discussion on r-devel and is convinced of the 
utility and harmlessness of your modifications the last step might be 
unnecessary).

   cheers
    Ben Bolker

On 7/23/21 1:17 PM, David Norris wrote:
> My understanding about this issue with progressr is that it is nontrivial, and will take author Henrik Bengtsson some time & thought to resolve.
> OTOH if patches to 'parallel' were welcome, I could embed my solution in a modified version of parallel::mclapply by introducing 1 or 2 new parameters (e.g., mc.prog=NULL) to support progress reporting consistently with the existing interface.
> 
> From: Duncan Murdoch <murdoch.duncan using gmail.com>
> Date: Friday, July 23, 2021 at 11:26 AM
> To: Hugh Parsonage <hugh.parsonage using gmail.com>, David Norris <david using precisionmethods.guru>
> Cc: "r-package-devel using r-project.org" <r-package-devel using r-project.org>
> Subject: Re: [R-pkg-devel] Using parallel:::sendMaster &c. despite warnings
> 
> On 23/07/2021 10:59 a.m., Hugh Parsonage wrote:
> Happily, package parallel is open-source. If you would like to make
> use of the unexported functions, you could copy the source code with
> acknowledgement into your package.
> 
> There's a good chance that wouldn't work, because parallel is a base
> package.  Base packages can work with R internals and user-written
> packages can't.
> 
> Probably David's best course of action is to submit patches where
> necessary so that he doesn't need ::: access.  The original motivation
> appeared to be avoiding inefficiency in a contributed package; if
> there's a patch that can fix that, it could be available in a few days.
>    If it needs changes to the parallel package (e.g. exposing functions
> that are currently internal), that will take much longer, and needs a
> strong argument why the current API isn't sufficient.
> 
> Alternatively, he can go ahead and use :::, but just not expect his
> package to be on CRAN.  There are other ways to distribute packages.
> 
> Duncan Murdoch
> 
>    Naturally this might be a lot of
> work and risks different behaviour between your package and parallel,
> but neither might be that terrible for you.  And certainly would be
> better than trying to pretend that the functions you want are exported
> from parallel.
> 
> 
> 
> On Fri, 23 Jul 2021 at 23:09, David Norris <mailto:david using precisionmethods.guru> wrote:
> 
> I can confirm that getFromNamespace() does indeed circumvent the WARNING.
> I might otherwise prefer ':::', however, for making its 'bad intentions' so clear.
> I was very much aware of the cautionary language under `?readChild`, but wondered whether *in practice* more liberal community policies might be in force pending the maturation of the futureverse.
> Although the sparse history of the relevant file hardly supports "might-break-at-any-time" alarmism, I do note that the last commit (in May 2020) introduced an argument sendMaster(raw.asis=TRUE) that I am actually using.
> https://github.com/wch/r-source/commits/trunk/src/library/parallel/R/unix/mcfork.R
> 
> (Also agree with Yohann that ideally one might support Windows with a warning such as data.table issues regarding my Mac's non-support for OpenMP.)
> 
> Best, David
> 
> From: Sebastian Meyer <mailto:seb.meyer using fau.de>
> Organization: Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
> Date: Friday, July 23, 2021 at 8:05 AM
> To: David Norris <mailto:david using precisionmethods.guru>, "mailto:r-package-devel using r-project.org" <mailto:r-package-devel using r-project.org>
> Subject: Re: [R-pkg-devel] Using parallel:::sendMaster &c. despite warnings
> 
> Am 23.07.21 um 13:19 schrieb David Norris:
> Because parallelized progress reporting in the futureverse.org incurs latencies too great for my application (https://github.com/HenrikBengtsson/progressr/issues/118), I have found it necessary to implement my own progress reporting using some of the non-exported functionality from `parallel`. (I do appreciate that Windows lacks the fork() system call, and will not support this. But am willing to make this an OS_type: unix-only package.)
> 
> Of course, I get a WARNING for this:
> 
> ── R CMD check results ──────────────────────────────── precautionary 0.2.6 ────
> Duration: 6m 41.8s
> 
> ❯ checking dependencies in R code ... WARNING
>        Unexported objects imported by ':::' calls:
>          ‘parallel:::readChild’ ‘parallel:::selectChildren’
>          ‘parallel:::sendMaster’
>          See the note in ?`:::` about the use of this operator.
>          Including base/recommended package(s):
>          ‘parallel’
> 
> Is this warning an absolute deal-killer on CRAN? Is there a 'correct' way to do `:::` that avoids the warning altogether?
> 
> The 'parallel' functions your package intends to access seem to be
> intentionally unexported. Their help page says: "They are not available
> on Windows, and not exported from the namespace", and "This is a very
> low-level interface for expert use only: it not regarded as part of the
> R API and subject to change without notice."
> 
> Correspondingly, the CRAN Repository Policy says
> 
> Also, ::: should not be used to access undocumented/internal objects in base packages (nor should other means of access be employed). Such usages can cause packages to break at any time, even in patched versions of R.
> 
> which kind of answers both of your questions. The policy thus implicitly
> advises against using getFromNamespace().
> 
> Best regards,
> 
>            Sebastian Meyer
> 
> 
> Kind regards,
> David Norris
> 
> ______________________________________________
> mailto:R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> 
> 
> 
> ______________________________________________
> mailto:R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> 
> ______________________________________________
> mailto:R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> 
> 
> 
> 
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>