[R-pkg-devel] R CMD Check: Tests running infinite

Henrik Bengtsson henrik.bengtsson at gmail.com
Tue Feb 7 23:44:18 CET 2017 

On Tue, Feb 7, 2017 at 2:10 PM, Gábor Csárdi <csardi.gabor at gmail.com> wrote:
> On Tue, Feb 7, 2017 at 8:16 PM, Henrik Bengtsson
> <henrik.bengtsson at gmail.com> wrote:
> [...]
>>
>>
>> So, to me it's not clear how this could make a difference in Patrick
>> case.   By disabling this, i.e. Sys.setenv(R_TESTS=""), I don't see
>> how it affects running parallel+foreach+doParallel.  Maybe because one
>> of those packages are relying on fancy quotes in some protocol,
>> passing command-line arguments or ... something.  I would be curious
>> to see what file.path(R.home("share"), "R", "tests-startup.R")
>> contains on the system where the problem occurs.
>
>
> I investigated a bit, and the problem (or at least one problem) is that the
> R subprocess will try
> to source() startup.Rs from the working directory, and that file might not
> be there.
> For testthat, the tests run in tests/testthat, and it is typically not
> there.

Ah... that makes sence, because R CMD check passes/sets
R_TESTS=startup.Rs specifying only a relative pathname.  From runone()
part of src/library/tools/R/testing.R:

        if (WINDOWS) {
            Sys.setenv(LANGUAGE="C")
            Sys.setenv(R_TESTS="startup.Rs")
        } else
            cmd <- paste("LANGUAGE=C", "R_TESTS=startup.Rs", cmd)

Maybe a more robust version would be to use something like:

        if (WINDOWS) {
            Sys.setenv(LANGUAGE="C")
            Sys.setenv(R_TESTS=normalizePath("startup.Rs"))
        } else
            cmd <- paste("LANGUAGE=C", paste0("R_TESTS=",
normalizePath("startup.Rs")), cmd)

On the other hand, if it should be assumed that the test is run in the
same directory as startup.Rs, then testthat, and other test frameworks
that don't run under tests/, could copy startup.Rs as:

   local({
      fn <- Sys.getenv("R_TESTS")
      if (nzchar(fn) && file_test("-f", pn <- file.path("..", fn)))
file.copy(pn, fn)
   })


@Patrick,

>> By disabling this, i.e. Sys.setenv(R_TESTS="")
>
> Note that the fix from Gabor is Sys.unsetenv("R_TESTS"). What´s the difference between both?

My bad; Sys.unsetenv("R_TESTS") is the *correct* way to unset an env
variable.  My approach sets it to an empty string, which also happens
to work because the test to source the file or not is based on
nzchar(Sys.setenv("R_TESTS")) == TRUE.

/Henrik

>
> For other regular tests, RUnit, etc. AFAIK the tests run in tests/ and
> startup.Rs is
> there. Unless the tests change the working directory, in which case it is
> not.
> So another workaround is to put an empty startup.Rs file there.
>
> Maybe there are other problems as well, but just putting an empty startup.Rs
> file at the appropriate
> location fixed all cases I tried.
>
> The parallel case above in this thread is a bit unusual I think, because
> usually there
> is no freeze, the R subprocess just aborts, and this typically causes a
> testthat (or other) test
> failure.
>
> Nevertheless, putting an empty startup.Rs file in tests/testthat fixes the
> testParallel
> package as well.
>
> G
>
>>
>>
>> >
>> > A lot of packages have to work around this:
>> > https://github.com/search?q=user%3Acran+R_TESTS&type=Code
>>
>> I wonder if those are mostly there because of cut'n'paste behavior.
>>
>> >
>> > Most of these use testthat, but not all of them.
>>
>> My interest in this issue is because I haven't yet experienced this
>> myself and I run lots and lots of package tests in future that
>> utilizes the parallel package.  In doFuture I do similar tests, which
>> is on top of the foreach package.   I don't use testthat and I also
>> don't use doParallel in my testing.
>>
>> /Henrik
>>
>> >
>> > Gabor
>> >
>> > On Mon, Feb 6, 2017 at 2:17 AM, Henrik Bengtsson
>> > <henrik.bengtsson at gmail.com> wrote:
>> >>
>> >> In case someone else bumps into this later and finds this thread; can
>> >> you confirm that this was a problem specific to using the testthat
>> >> package for running the tests?
>> >>
>> >> /Henrik
>> >>
>> >> On Sun, Feb 5, 2017 at 11:28 AM, Patrick Schratz
>> >> <patrick.schratz at gmail.com> wrote:
>> >> > @gaborcsardi solved it :) See here:
>> >> > https://github.com/hadley/testthat/issues/567#issuecomment-277536577
>> >> >
>> >> >
>> >> > 2017-02-05 16:07 GMT+01:00 Patrick Schratz
>> >> > <patrick.schratz at gmail.com>:
>> >> >>
>> >> >> Thanks for the hint, Hendrik!
>> >> >> However, this change did not make a difference :/
>> >> >>
>> >> >> I tried to use all cluster closing functions I came across but tests
>> >> >> are
>> >> >> still running infinite..
>> >> >>
>> >> >> cl <- makeCluster(par.args$par.units, outfile = out.progress)
>> >> >> registerDoParallel(cl)
>> >> >>
>> >> >> foreach()
>> >> >>
>> >> >> parallel::stopCluster(cl)
>> >> >> doParallel::registerDoSEQ()
>> >> >> doParallel::stopImplicitCluster()
>> >> >>
>> >> >> 2017-02-05 15:04 GMT+01:00 Henrik Bengtsson
>> >> >> <henrik.bengtsson at gmail.com>:
>> >> >>>
>> >> >>> Use
>> >> >>>
>> >> >>>   registerDoParallel(cl)
>> >> >>>
>> >> >>> The number of parallel workers is already contained in the 'cl'
>> >> >>> object,
>> >> >>> so don't specify 'cores'!  (If you do that, I suspect you create
>> >> >>> yet
>> >> >>> another
>> >> >>> cluster (a multicore one) which is used but never closed)
>> >> >>>
>> >> >>> registerDoParallel() should ideally give an error in your case.
>> >> >>> Author
>> >> >>> BCC:ed.
>> >> >>>
>> >> >>> Henrik
>> >> >>>
>> >> >>> On Feb 5, 2017 03:56, "Patrick Schratz" <patrick.schratz at gmail.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Dear Uwe,
>> >> >>>>
>> >> >>>> thanks for the hint. My cluster is closed after the `foreach`call
>> >> >>>> using
>> >> >>>> `stopCluster()`.
>> >> >>>>
>> >> >>>> Before, I´ll do the following to init the cluster:
>> >> >>>>
>> >> >>>> *cl <- makeCluster(par.args$par.units, outfile = out.progress)*
>> >> >>>> *registerDoParallel(cl, cores = par.args$par.units)*
>> >> >>>>
>> >> >>>> *foreach()*
>> >> >>>>
>> >> >>>> *stopCluster(cl)*
>> >> >>>>
>> >> >>>>
>> >> >>>> Do you know of any other package which is using foreach in
>> >> >>>> combination
>> >> >>>> with
>> >> >>>> tests and is hosted on Github? So I could compare settings.
>> >> >>>>
>> >> >>>> Best, Patrick
>> >> >>>>
>> >> >>>> 2017-02-02 0:01 GMT+01:00 Uwe Ligges
>> >> >>>> <ligges at statistik.tu-dortmund.de>:
>> >> >>>>
>> >> >>>> > Check whether the parallel cluster is closed. Can it be that the
>> >> >>>> > cluster
>> >> >>>> > is still open and the check process waits for them to complete?
>> >> >>>> >
>> >> >>>> > Best,
>> >> >>>> > Uwe Ligges
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > On 31.01.2017 13:45, Patrick Schratz wrote:
>> >> >>>> >
>> >> >>>> >> Hello,
>> >> >>>> >>
>> >> >>>> >> when running R CMD check / devtools::check, section "running
>> >> >>>> >> tests..." is
>> >> >>>> >> not finishing (40 min+).
>> >> >>>> >>
>> >> >>>> >> *Checking tests only works:*
>> >> >>>> >>
>> >> >>>> >> *==> Sourcing R files in 'tests' directory*
>> >> >>>> >>
>> >> >>>> >> *testthat results
>> >> >>>> >>
>> >> >>>> >> ================================================================*
>> >> >>>> >> *OK: 7 SKIPPED: 0 FAILED: 0*
>> >> >>>> >>
>> >> >>>> >> *Tests complete*
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >> As well as running tests line-by-line.
>> >> >>>> >>
>> >> >>>> >> How can I debug my tests to discover the problem during R CMD
>> >> >>>> >> check?
>> >> >>>> >>
>> >> >>>> >> *Tests are using parallelization (foreach + doParallel)*
>> >> >>>> >>
>> >> >>>> >> Best, Patrick
>> >> >>>> >>
>> >> >>>> >>         [[alternative HTML version deleted]]
>> >> >>>> >>
>> >> >>>> >> ______________________________________________
>> >> >>>> >> R-package-devel at r-project.org mailing list
>> >> >>>> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >> >>>>         [[alternative HTML version deleted]]
>> >> >>>>
>> >> >>>> ______________________________________________
>> >> >>>> R-package-devel at r-project.org mailing list
>> >> >>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >> >>
>> >> >>
>> >> >
>> >>
>> >> ______________________________________________
>> >> R-package-devel at r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >
>> >
>
>

More information about the R-package-devel mailing list