[Rd] Re: package checks and tests; was: Re: A "rude" question

Wed Feb 2 15:58:01 CET 2005

[moved to R-devel]

I am a bit concerned by the fact that package developers would be 
removing all tests. The tests seem like one of the main mechanisms for 
ensuring that CRAN does not descend to the almost useless state of 
statlib: there is an automatic mechanism for removing packages that no 
longer work.

I have several tests that I don't put in packages, because they are very 
long or they don't work without locally available databases, but I think 
it is important that there should be tests that check the main 
functionality of packages.  For very long tests like monte carlo 
simulations it is usually fairly easy to devise some short version (a 
few iterations) which checks that the software actually works and gives 
results that are the same as they always have been, even though it does 
not check the more serious statistical problems one would actually want 
to do.

Perhaps there should be another directory, called something like 
longTests, that is not run daily but is run occasionally.

On r-help I suggested that users might contribute tests to package 
developers, but another possibility is that there is some mechanism for 
them to contribute suites of package tests directly to CRAN. This would 
provide a mechanism for more complete checking of packages, and perhaps 
a just system for removing packages when the developer is no longer 
active and the package is broken.

Paul Gilbert

Uwe Ligges wrote:
> Paulo Justiniano Ribeiro Jr wrote:
> 
>> One problem with distributing packages with the test sub-directory
>> is that this can overload the daily tests in the CRAN machines
>>
>> My workaround for that  is:
>> For the geoR package I run the tests in my machine but remove the tests
>> directory when submitting to CRAN.
>> Also I mantain a package web-page where I make the version
>> with tests available for downloading
> 
> 
> [moved to R-devel, chosen a more sensible subject]
> 
> If tests are running in a reasonable amount of time, it does not make 
> sense to remove them. People want to install the package on a huge 
> number of platforms you probably have not tested the package on 
> yourself. The only solution is to distribute the tests as well.
> 
> Overloading the daily tests on CRAN is another issue (strong related, 
> though). Installing and checking (in maintainer's mode, i.e. without 
> double-installing) all CRAN packages under R-release for Windows takes 8 
> hours on a (dual, but using only one) Xeon 3.06GHz machine these days.
> The checks (without isntallation) of the 15 packages with most intensive 
>  tests take ~1 hour.
> 
> Uwe Ligges
> 
> 
> 
> 
> 
>> P.J.
>>
>> On Tue, 1 Feb 2005, Spencer Graves wrote:
>>
>>
>>> Hi, Paul:
>>>
>>>      How can I access "the package tests directory" you mentioned?
>>> Only one of the 52 subdirectories of "library" in my current
>>> installation of R 2.0.1 has a "test" folder.
>>>
>>>      Thanks,
>>>      Spencer Graves
>>>
>>> Paul Gilbert wrote:
>>>
>>>
>>>> One point that did not get mentioned in this discussion, and I believe
>>>> deserves
>>>> much more publicity, is the impact of packages tests. The design of
>>>> the package
>>>> system allows package developers to put tests in packages, and these
>>>> are checked
>>>> regularly (see 
>>>> <http://cran.at.r-project.org/contrib/checkSummary.html>).
>>>>
>>>> These are intended to test the package functionality, but also give R
>>>> what is
>>>> perhaps the largest test suite of any statistical software (certainly
>>>> the most
>>>> quickly growing). While any single package's test will never guarantee
>>>> that
>>>> the package works perfectly, the ensemble goes a long way toward 
>>>> ensuring
>>>> that core R functionality behaves as intended. It seems unlikely to me
>>>> that any
>>>> commercial effort will ever be able to catch up.
>>>>
>>>> There are several ways that tests can add to our confidence that
>>>> calculations can be trusted. They can
>>>>  - check against theoretical results
>>>>  - check against published results
>>>>  - check against results from other software
>>>>  - check that calculations done in different ways give the same result
>>>>  - check that monte carlo experiments give distributions that are
>>>> consistent
>>>>     with expected results
>>>>
>>>> Some of these are relatively time consuming to set up and check the
>>>> first time,
>>>> but after that they can be automatic.
>>>>
>>>> If you have particular calculations with specific packages that you
>>>> are especially
>>>> concerned about, I encourage you to participate by devising good tests
>>>> and sending
>>>> them to the package developers. (But first check the tests they are
>>>> already doing
>>>> in the package tests directory.)
>>>>
>>>> Paul Gilbert
>>>>
>>>> msck9 at mizzou.edu wrote:
>>>>
>>>>
>>>>> Dear all, I am beginner using R. I have a question about it. When you
>>>>> use it,
>>>>> since it is written by so many authors, how do you know that the
>>>>> results are trustable?(I don't want to affend anyone, also I trust
>>>>> people). But I think this should be a question.
>>>>>
>>>>> Thanks,
>>>>> Ming
>>>>>
>>>>> ______________________________________________
>>>>> R-help at stat.math.ethz.ch mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>> PLEASE do read the posting guide!
>>>>> http://www.R-project.org/posting-guide.html
>>>>>
>>>>>
>>>>>
>>>>
>>>> ______________________________________________
>>>> R-help at stat.math.ethz.ch mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide!
>>>> http://www.R-project.org/posting-guide.html
>>>
>>>
>>> ______________________________________________
>>> R-help at stat.math.ethz.ch mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide! 
>>> http://www.R-project.org/posting-guide.html
>>>
>>>
>>
>>
>> Paulo Justiniano Ribeiro Jr
>> LEG (Laboratório de Estatística e Geoinformação)
>> Departamento de Estatística
>> Universidade Federal do Paraná
>> Caixa Postal 19.081
>> CEP 81.531-990
>> Curitiba, PR  -  Brasil
>> Tel: (+55) 41 361 3573
>> Fax: (+55) 41 361 3141
>> e-mail: paulojus at est.ufpr.br
>> http://www.est.ufpr.br/~paulojus
>>
>> ______________________________________________
>> R-help at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide! 
>> http://www.R-project.org/posting-guide.html
> 
> 
>