[R] Unit Testing Frameworks: summary and brief discussion

[hadley wickham]

On 5/10/07, Paul Murrell <p.murrell at auckland.ac.nz> wrote:
> Hi
>
>
> Paul Gilbert wrote:
> > Tony
> >
> > Thanks for the summary.
> >
> > My ad hoc system is pretty good for catching flagged errors, and
> > numerical errors when I have a check.  Could you (or someone else)
> > comment on how easy it would be with one of these more formal frameworks
> > to do three things I have not been able to accomplish easily:
> >
> > - My code gives error and warning messages in some situations. I want to
> > test that the errors and warnings work, but these flags are the correct
> > response to the test. In fact, it is an error if I don't get the flag.
> > How easy is it to set up automatic tests to check warning and error
> > messages work?
> >
> > - For some things it is the printed format that matters. How easy is it
> > to set up a test of the printed output? (Something like the Rout files
> > used in R CMD check.) I think this is what Tony Plate is calling
> > transcript file tests, and I guess it is not automatically available. I
> > am not really interested in something I would have to change with each
> > new release of R, and I need it to work cross-platform. I want to know
> > when something has changed, in R or my own code, without having to
> > examine the output carefully.
> >
> > - (And now the hard one.) For some things it is the plotted output that
> > matters. Is it possible to set up automatic tests of plotting? I can
> > already test that plots run. I want to know if they "look very
> > different". And no, I don't have a clue where to start on this one.
>
>
> For text-based graphics formats, you can just use diff;  for raster
> formats, you can do per pixel comparisons.  These days there is
> ImageMagick to do a compare and it will even produce an image of the
> difference.  I have an old package called graphicsQC (not on CRAN) that
> implemented some of these ideas (there was a talk at DSC 2003, see
> http://www.stat.auckland.ac.nz/~paul/index.html).  A student worked on a
> much better approach more recently, but I haven't put that up on the web
> yet.  Let me know if you'd like to take a look at the newer package (it
> would help to have somebody nagging me to get it finished off).

I'd be interested!

Hadley