[R-pkg-devel] R CMD Check: Tests running infinite

[Henrik Bengtsson]

On Mon, Feb 6, 2017 at 12:41 AM, Gábor Csárdi <csardi.gabor at gmail.com> wrote:
> I don't think it is specific for testthat. R CMD check sets R_TESTS when it
> runs the tests, so the separate R process it starts can have some special
> startup options.
>
> The problem happens if you start another R process from your R test, and
> then R_TESTS confuses this process. (I am not sure how exactly.)

I wasn't aware of R_TESTS.  Looking at the R source code, if set, will
cause the base package to source that file _before_ and .Rprofile
scripts; it basically calls source(Sys.getenv("R_TESTS")).

Next, R CMD check runs each of the test script using
R_TESTS=startup.Rs, where startup.Rs:

    file.copy(file.path(R.home("share"), "R", "tests-startup.R"), "startup.Rs")
    if (use_gct) cat("gctorture(TRUE)" , file = "startup.Rs", append = TRUE)

Looking at the tests-startup.R file that comes with my R 3.3.2 on
Ubuntu 16.04, it seems pretty harmless:

## A custom startup file for tests
## Run as if a system Rprofile, so no packages, no assignments
options(useFancyQuotes = FALSE)

So, to me it's not clear how this could make a difference in Patrick
case.   By disabling this, i.e. Sys.setenv(R_TESTS=""), I don't see
how it affects running parallel+foreach+doParallel.  Maybe because one
of those packages are relying on fancy quotes in some protocol,
passing command-line arguments or ... something.  I would be curious
to see what file.path(R.home("share"), "R", "tests-startup.R")
contains on the system where the problem occurs.

>
> A lot of packages have to work around this:
> https://github.com/search?q=user%3Acran+R_TESTS&type=Code

I wonder if those are mostly there because of cut'n'paste behavior.

>
> Most of these use testthat, but not all of them.

My interest in this issue is because I haven't yet experienced this
myself and I run lots and lots of package tests in future that
utilizes the parallel package.  In doFuture I do similar tests, which
is on top of the foreach package.   I don't use testthat and I also
don't use doParallel in my testing.

/Henrik

>
> Gabor
>
> On Mon, Feb 6, 2017 at 2:17 AM, Henrik Bengtsson
> <henrik.bengtsson at gmail.com> wrote:
>>
>> In case someone else bumps into this later and finds this thread; can
>> you confirm that this was a problem specific to using the testthat
>> package for running the tests?
>>
>> /Henrik
>>
>> On Sun, Feb 5, 2017 at 11:28 AM, Patrick Schratz
>> <patrick.schratz at gmail.com> wrote:
>> > @gaborcsardi solved it :) See here:
>> > https://github.com/hadley/testthat/issues/567#issuecomment-277536577
>> >
>> >
>> > 2017-02-05 16:07 GMT+01:00 Patrick Schratz <patrick.schratz at gmail.com>:
>> >>
>> >> Thanks for the hint, Hendrik!
>> >> However, this change did not make a difference :/
>> >>
>> >> I tried to use all cluster closing functions I came across but tests
>> >> are
>> >> still running infinite..
>> >>
>> >> cl <- makeCluster(par.args$par.units, outfile = out.progress)
>> >> registerDoParallel(cl)
>> >>
>> >> foreach()
>> >>
>> >> parallel::stopCluster(cl)
>> >> doParallel::registerDoSEQ()
>> >> doParallel::stopImplicitCluster()
>> >>
>> >> 2017-02-05 15:04 GMT+01:00 Henrik Bengtsson
>> >> <henrik.bengtsson at gmail.com>:
>> >>>
>> >>> Use
>> >>>
>> >>>   registerDoParallel(cl)
>> >>>
>> >>> The number of parallel workers is already contained in the 'cl'
>> >>> object,
>> >>> so don't specify 'cores'!  (If you do that, I suspect you create yet
>> >>> another
>> >>> cluster (a multicore one) which is used but never closed)
>> >>>
>> >>> registerDoParallel() should ideally give an error in your case. Author
>> >>> BCC:ed.
>> >>>
>> >>> Henrik
>> >>>
>> >>> On Feb 5, 2017 03:56, "Patrick Schratz" <patrick.schratz at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Dear Uwe,
>> >>>>
>> >>>> thanks for the hint. My cluster is closed after the `foreach`call
>> >>>> using
>> >>>> `stopCluster()`.
>> >>>>
>> >>>> Before, I´ll do the following to init the cluster:
>> >>>>
>> >>>> *cl <- makeCluster(par.args$par.units, outfile = out.progress)*
>> >>>> *registerDoParallel(cl, cores = par.args$par.units)*
>> >>>>
>> >>>> *foreach()*
>> >>>>
>> >>>> *stopCluster(cl)*
>> >>>>
>> >>>>
>> >>>> Do you know of any other package which is using foreach in
>> >>>> combination
>> >>>> with
>> >>>> tests and is hosted on Github? So I could compare settings.
>> >>>>
>> >>>> Best, Patrick
>> >>>>
>> >>>> 2017-02-02 0:01 GMT+01:00 Uwe Ligges
>> >>>> <ligges at statistik.tu-dortmund.de>:
>> >>>>
>> >>>> > Check whether the parallel cluster is closed. Can it be that the
>> >>>> > cluster
>> >>>> > is still open and the check process waits for them to complete?
>> >>>> >
>> >>>> > Best,
>> >>>> > Uwe Ligges
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On 31.01.2017 13:45, Patrick Schratz wrote:
>> >>>> >
>> >>>> >> Hello,
>> >>>> >>
>> >>>> >> when running R CMD check / devtools::check, section "running
>> >>>> >> tests..." is
>> >>>> >> not finishing (40 min+).
>> >>>> >>
>> >>>> >> *Checking tests only works:*
>> >>>> >>
>> >>>> >> *==> Sourcing R files in 'tests' directory*
>> >>>> >>
>> >>>> >> *testthat results
>> >>>> >> ================================================================*
>> >>>> >> *OK: 7 SKIPPED: 0 FAILED: 0*
>> >>>> >>
>> >>>> >> *Tests complete*
>> >>>> >>
>> >>>> >>
>> >>>> >> As well as running tests line-by-line.
>> >>>> >>
>> >>>> >> How can I debug my tests to discover the problem during R CMD
>> >>>> >> check?
>> >>>> >>
>> >>>> >> *Tests are using parallelization (foreach + doParallel)*
>> >>>> >>
>> >>>> >> Best, Patrick
>> >>>> >>
>> >>>> >>         [[alternative HTML version deleted]]
>> >>>> >>
>> >>>> >> ______________________________________________
>> >>>> >> R-package-devel at r-project.org mailing list
>> >>>> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >>>> >>
>> >>>> >>
>> >>>>
>> >>>>         [[alternative HTML version deleted]]
>> >>>>
>> >>>> ______________________________________________
>> >>>> R-package-devel at r-project.org mailing list
>> >>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >>
>> >>
>> >
>>
>> ______________________________________________
>> R-package-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>
>