[Rd] Checking package help file examples

[Bill Dunlap]

On Fri, 18 Jul 2008, Arne Henningsen wrote:

> On Friday 18 July 2008 02:19:14, Bill Dunlap wrote:
> > I am trying to figure out the sanctioned way for
> > 'R CMD check pkg' to make sure that the examples
> > in help files give the expected results.
> >
> > Writing R Extensions says that check runs the help
> > file examples (which INSTALL extracts from pkg/man/*.Rd
> > and puts into files in pkg/R-ex).  It looks like check
> > concatenates all the example files, pkg/R-ex/*.R, into one
> > file, pkg/pkg-Ex.R, which it runs to produce pkg/pkg-Ex.Rout.
> > If there is an error running these files then check aborts.
> >
> > I don't seen anything in Writing R Extensions about
> > checking that correct results were produced.  However,
> > in R 2.7.0, check will compare pkg/pkg-Ex.Rout to
> > pkg/tests/Examples/pkg/pkg-Ex.Rout.save, if the latter

oops: pkg/tests/Examples/pkg-Ex.Rout.save

> > file is in your package source, and report any differences.
> >
> > I don't see any mention of this and was wondering if this
> > method should be recommended or if there was a preferred
> > way to check help file examples for correctness.  (It would
> > be nicer if the example() function could also check for
> > correctness, and the above method doesn't allow for that.)
>
> I use *.R and corresponding *.Rout.save files in the subdirectory "tests" for
> checking packages (see e.g., packages systemfit, micEcon,
> sampleSelection, ...). I think that it is better to separate examples from
> testing, because IMHO there should be a few simple examples but tests should
> contain many different function calls with (partly) unusual arguments to
> check (nearly) the entire functionality of each function.
>
> See also the manual "Writing R Extensions"
> http://cran.r-project.org/doc/manuals/R-exts.html#Package-subdirectories

My question was about the undocumented code in R_HOME/bin/check
that compares the output from running the help file examples,
pkg.Rcheck/pkg-Ex.Rout, to the file pkg/tests/Examples/pkg-Ex.Rout.save,
if the latter file is in the package source.

        $log->checking("examples");
        ... ${pkgname}-Ex.R was created by massage-Examples.pl ...
        ... run "R < ${pkgname}-Ex.R > ${pkgname}-Ex.Rout" ...
        ## Try to compare results from running the examples to a saved
        ## previous version.
        my $Rex_out_save =
            &file_path($pkgdir, "tests", "Examples",
                       "${pkgname}-Ex.Rout.save");
        if(-f $Rex_out_save) {
            $log->checking("differences from '${pkgname}-Ex.Rout' to '${pkgname}-Ex.Rout.save'");
            $cmd = join(" ",
                        (${R::Vars::R_CMD},
                         "Rdiff",
                         &shell_quote_file_path("${pkgname}-Ex.Rout"),
                         &shell_quote_file_path("${Rex_out_save}"),
                         "0"));
            if(R_system($cmd)) {
                $log->error();
                $log->print("Comparison failed.\n");
                exit(1);
            }
            $log->result("OK");
        }
    }

A package writer noticed the pkg-Ex.Rout file created by
check and wondered if this was ever compared to a Rout.save
file as is done with other Rout files and I noticed the
above perl code that did so.  I didn't see any CRAN packages
that had a tests/Examples/pkg-Ex.Rout.save file, but some
have a pkg-Ex.Rout.save in a different place.

Should we document the current behavior (should I use this
scheme in Splus packages)?  Should we get rid of the code
or put in comments that it should not be used?

If I were writing this from scratch, I would not put all
the example output into one Rout or Rout.save file, but
put use one Rout file per help file.  Then the results
would not depend on the local collating order.

I agree that detailed tests belong in tests, but since check
runs the help file example to make sure they don't throw
errors, it seems reasonable to also check that they give
the expected results.

> Arne Henningsen
> http://www.arne-henningsen.name

----------------------------------------------------------------------------
Bill Dunlap
Insightful Corporation
bill at insightful dot com

 "All statements in this message represent the opinions of the author and do
 not necessarily reflect Insightful Corporation policy or position."