[R-pkg-devel] Using parallel:::sendMaster &c. despite warnings

David Norris d@v|d @end|ng |rom prec|@|onmethod@@guru
Fri Jul 23 20:10:36 CEST 2021


Thanks for delineating r-devel vs this list, Ben, and for invitation to start a fresh convo over there.
I'll do that once I can exhibit a basic proposal functioning within package 'precautionary'.
Best, David

From: R-package-devel <r-package-devel-bounces using r-project.org> on behalf of Ben Bolker <bbolker using gmail.com>
Date: Friday, July 23, 2021 at 1:29 PM
To: "r-package-devel using r-project.org" <r-package-devel using r-project.org>
Subject: Re: [R-pkg-devel] Using parallel:::sendMaster &c. despite warnings

   Patches to parallel could certainly be *considered*, but people will 
probably want to argue about the details quite a bit.  I would suggest 
presenting/discussing your ideas for modification on r-devel before 
posting them as a 'wishlist' item to the R bug tracker. (If an R-core 
member participates in the discussion on r-devel and is convinced of the 
utility and harmlessness of your modifications the last step might be 
unnecessary).

   cheers
    Ben Bolker


On 7/23/21 1:17 PM, David Norris wrote:
My understanding about this issue with progressr is that it is nontrivial, and will take author Henrik Bengtsson some time & thought to resolve.
OTOH if patches to 'parallel' were welcome, I could embed my solution in a modified version of parallel::mclapply by introducing 1 or 2 new parameters (e.g., mc.prog=NULL) to support progress reporting consistently with the existing interface.

From: Duncan Murdoch <mailto:murdoch.duncan using gmail.com>
Date: Friday, July 23, 2021 at 11:26 AM
To: Hugh Parsonage <mailto:hugh.parsonage using gmail.com>, David Norris <mailto:david using precisionmethods.guru>
Cc: "mailto:r-package-devel using r-project.org" <mailto:r-package-devel using r-project.org>
Subject: Re: [R-pkg-devel] Using parallel:::sendMaster &c. despite warnings

On 23/07/2021 10:59 a.m., Hugh Parsonage wrote:
Happily, package parallel is open-source. If you would like to make
use of the unexported functions, you could copy the source code with
acknowledgement into your package.

There's a good chance that wouldn't work, because parallel is a base
package.  Base packages can work with R internals and user-written
packages can't.

Probably David's best course of action is to submit patches where
necessary so that he doesn't need ::: access.  The original motivation
appeared to be avoiding inefficiency in a contributed package; if
there's a patch that can fix that, it could be available in a few days.
    If it needs changes to the parallel package (e.g. exposing functions
that are currently internal), that will take much longer, and needs a
strong argument why the current API isn't sufficient.

Alternatively, he can go ahead and use :::, but just not expect his
package to be on CRAN.  There are other ways to distribute packages.

Duncan Murdoch

    Naturally this might be a lot of
work and risks different behaviour between your package and parallel,
but neither might be that terrible for you.  And certainly would be
better than trying to pretend that the functions you want are exported
from parallel.



On Fri, 23 Jul 2021 at 23:09, David Norris <mailto:david using precisionmethods.guru> wrote:

I can confirm that getFromNamespace() does indeed circumvent the WARNING.
I might otherwise prefer ':::', however, for making its 'bad intentions' so clear.
I was very much aware of the cautionary language under `?readChild`, but wondered whether *in practice* more liberal community policies might be in force pending the maturation of the futureverse.
Although the sparse history of the relevant file hardly supports "might-break-at-any-time" alarmism, I do note that the last commit (in May 2020) introduced an argument sendMaster(raw.asis=TRUE) that I am actually using.
https://github.com/wch/r-source/commits/trunk/src/library/parallel/R/unix/mcfork.R

(Also agree with Yohann that ideally one might support Windows with a warning such as data.table issues regarding my Mac's non-support for OpenMP.)

Best, David

From: Sebastian Meyer <mailto:seb.meyer using fau.de>
Organization: Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
Date: Friday, July 23, 2021 at 8:05 AM
To: David Norris <mailto:david using precisionmethods.guru>, "mailto:r-package-devel using r-project.org" <mailto:r-package-devel using r-project.org>
Subject: Re: [R-pkg-devel] Using parallel:::sendMaster &c. despite warnings

Am 23.07.21 um 13:19 schrieb David Norris:
Because parallelized progress reporting in the futureverse.org incurs latencies too great for my application (https://github.com/HenrikBengtsson/progressr/issues/118), I have found it necessary to implement my own progress reporting using some of the non-exported functionality from `parallel`. (I do appreciate that Windows lacks the fork() system call, and will not support this. But am willing to make this an OS_type: unix-only package.)

Of course, I get a WARNING for this:

── R CMD check results ──────────────────────────────── precautionary 0.2.6 ────
Duration: 6m 41.8s

❯ checking dependencies in R code ... WARNING
        Unexported objects imported by ':::' calls:
          ‘parallel:::readChild’ ‘parallel:::selectChildren’
          ‘parallel:::sendMaster’
          See the note in ?`:::` about the use of this operator.
          Including base/recommended package(s):
          ‘parallel’

Is this warning an absolute deal-killer on CRAN? Is there a 'correct' way to do `:::` that avoids the warning altogether?

The 'parallel' functions your package intends to access seem to be
intentionally unexported. Their help page says: "They are not available
on Windows, and not exported from the namespace", and "This is a very
low-level interface for expert use only: it not regarded as part of the
R API and subject to change without notice."

Correspondingly, the CRAN Repository Policy says

Also, ::: should not be used to access undocumented/internal objects in base packages (nor should other means of access be employed). Such usages can cause packages to break at any time, even in patched versions of R.

which kind of answers both of your questions. The policy thus implicitly
advises against using getFromNamespace().

Best regards,

            Sebastian Meyer


Kind regards,
David Norris

______________________________________________
mailto:R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel



______________________________________________
mailto:R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

______________________________________________
mailto:R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel




______________________________________________
mailto:R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


______________________________________________
mailto:R-package-devel using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel




More information about the R-package-devel mailing list