[R-pkg-devel] R package submission - too many threads error

[Stephen Abrams]

I appreciate the welcome! Also - I believe that replying to an email is the
way to respond here, but please let me know if that's not the case.

In any event - passing in a cluster context is an interesting idea. I will
think about that. Also, it seems that despite me telling myself to write
bug-free code, you have correctly identified that I don't actually make use
of the passed cores parameter - oops! This is where it would have really
helped me to have a peer reviewer. Thanks!

On Fri, Feb 14, 2025 at 3:48 PM Ivan Krylov <ikrylov using disroot.org> wrote:

> Dear Stephen Abrams,
>
> Welcome to R-package-devel!
>
> В Thu, 13 Feb 2025 22:20:50 -0500
> Stephen Abrams <stephen.abrams using gmail.com> пишет:
>
> > A secondary worry is that even if I resolve this, there might be
> > something else causing threads to spin up.
>
> Instead of using detectCores() [*] and creating cluster objects
> yourself, how about letting the user provide a cluster object for you
> as a function argument? Yes, it takes slightly more typing for the user,
> but on the other hand it lets the user:
>
>  - choose the number of cores for themselves (currently the code seems
>    to be ignoring the 'cores' argument)
>  - distribute the computation over the network by connecting to the
>    machines they know about
>  - provide a completely custom, non-PSOCK cluster object that
>    'parallel' will nevertheless will work with
>
> Since you're already using doParallel, maybe the right choice is to let
> the user call registerDoParallel()?
>
> Determining the right amount of parallelism in your code is a
> surprisingly hard problem. Especially on shared computers, a program
> naively deciding to use all (or 3/4 of all, or 1/2 of all) processors
> may end up working much worse than a purely sequential one [**].
>
> While rendering the vignette in a CRAN package, create a two-process
> cluster or set use_parallel = FALSE: CRAN needs the rest of the
> processors to check other packages in parallel with yours [***].
>
> Good luck!
>
> --
> Best regards,
> Ivan
>
> [*]
>
> https://github.com/dawdawdo/spect/blob/d48b002332f1a1c2d302afb28e08e7998f416200/R/spect.R#L388
>
> [**]
> https://mastodon.social/@henrikbengtsson/113835651303559942
>
> [***]
>
> http://contributor.r-project.org/cran-cookbook/code_issues.html#using-more-than-2-cores
>


-- 
Stephen Abrams
Divergent Blue <http://www.divergentblue.com/>

	[[alternative HTML version deleted]]