[Rd] How to test impact of candidate changes to package?

[Kirill Müller]

If you don't intend to keep the old business logic in the long run, 
perhaps a version control system such as Git can help you. If you use it 
in single-user mode, you can think of it as a backup system where you 
manually create each snapshot and give it a name, but it actually can do 
much more. For your use case, you can open a new *branch* where you 
implement your changes, and implement your testing logic simultaneously 
in both branches (using *merge* operations). The system handles 
switching between branches, so you can really perform invasive changes, 
and revert if you find that a particular change breaks something.

RStudio has Git support, but you probably need to use the shell to 
create a branch. On Windows or OS X the GitHub client helps you to get 
started.


Cheers

Kirill


On 09/10/2014 11:14 AM, Stephanie Locke wrote:
> I have unit tests using testthat but these are typically of these types:
> 1) Check for correct calculation for a single set of valid inputs
> 2) Check for correct calculation for a larger set of valid inputs
> 3) Check for errors when providing incorrect inputs
> 4) Check for known frailties / past issues
>
> This is more for where changes are needed to functions that apply various bits of business logic that can change over time, so there is no "one answer". A unit test (at least as I understand it) can be worked through to make sure that given inputs, the output is computationally correct. What I'd like to do is overall the impact of a potential change by testing version 1 of a function in a package for a sample, then test version 2 of a function in a package for a sample and compare the results.
>
> My difficulties encountered so far is I'm reluctantly to manually do this change invasively by overwriting the relevant files in the R directory, and then say using devtools to load it and test it with testthat as I risk producing incorrect states of my package and potentially releasing the wrong thing.  My preference would be a non-invasive method.  Currently, where I'm trying to do this non-invasively I source a new version of the function stored in a separate directory, but some of the functions dependent on it continue to reference to the package version of the functions, this means that when I'm doing test #2 I have to load lots more functions and hope I've caught them all (or do some sort of dependency hunting programmatically).
>
> I may be missing something about testthat, but what I'm doing now seems to be nowhere near optimal and I'd love to have a better solution.
>
> Cheers
>
> Stephanie Locke
> BI & Credit Risk Analyst
>
> -----Original Message-----
> From: ONKELINX, Thierry [mailto:Thierry.ONKELINX at inbo.be]
> Sent: 10 September 2014 09:30
> To: Stephanie Locke; r-devel at r-project.org
> Subject: RE: How to test impact of candidate changes to package?
>
> Dear Stephanie,
>
> Have a look at the testthat package and the related article in the R Journal.
>
> Best regards,
>
> ir. Thierry Onkelinx
> Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance Kliniekstraat 25
> 1070 Anderlecht
> Belgium
> + 32 2 525 02 51
> + 32 54 43 61 85
> Thierry.Onkelinx at inbo.be
> www.inbo.be
>
> To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of.
> ~ Sir Ronald Aylmer Fisher
>
> The plural of anecdote is not data.
> ~ Roger Brinner
>
> The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.
> ~ John Tukey
>
> -----Oorspronkelijk bericht-----
> Van: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-project.org] Namens Stephanie Locke
> Verzonden: woensdag 10 september 2014 9:55
> Aan: r-devel at r-project.org
> Onderwerp: [Rd] How to test impact of candidate changes to package?
>
> I use a package to contain simple functions that can be handled by unit tests for correctness and more complex functions that combine the simple functions with business logic.  Where there are proposals to change either the simple functions or the business logic, a sample needs to be run before the change and then after it to understand the impact of the change.
>
> I do this currently by
> 1. Using Rmarkdown documents
> 2. Loading the package as-is
> 3. Getting my sample
> 4. Running my sample through the package as-is and outputting table of results 5. sourceing new copies of functions 6. Running my sample again and outputting table of results 7. Reloading package and sourceing different copies of functions as required
>
> I really don't think this is a good way to do this as it risks missing downstream dependencies of the functions I'm trying to load into the global namespace to test.
>
> Has anyone else had to do this sort of testing before on their packages? How did you do it? Am I missing an obvious package / framework that can do this?
>
> Cheers,
> Steph
>
> --
> Stephanie Locke
> BI & Credit Risk Analyst
>
>
>
>
> The contents of this e-mail and of any attachments transmitted with it are confidential and intended solely for the use of the individual(s) to whom they are addressed. If you are not an intended recipient or the person responsible for delivering this e-mail to the intended recipient(s), you have received this e-mail in error, and access to it by you is not authorised. You may not use, copy, distribute, disclose or rely on it or any attachment, or any part of it or any attachment in any way. If you have received this e-mail in error please notify Optimum Credit Ltd by email on webmaster at optimumcredit.co.uk or phone on 03330 143125 and then delete it.
>
> All reasonable precautions have been taken to ensure that no viruses are present in E-mail. As we cannot accept responsibility for loss or damage arising from the use of E-mail or any attachment we recommend you subject these to your usual virus checking procedures prior to use. Any opinions expressed in this e-mail are those of the person sending it and not necessarily those of Optimum Credit Ltd.
>
> Optimum Credit Ltd, Haywood House South, Dumfries Place, Cardiff, CF10 3GA
>
> Authorised and regulated by the Financial Conduct Authority
>
> Optimum Credit Ltd. Registered office: Haywood House South, Dumfries Place, Cardiff, CF10 3GA. Registered in England and Wales No. 08698121
>
> Calls may be monitored or recorded for training, compliance and evidential purposes.
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> * * * * * * * * * * * * * D I S C L A I M E R * * * * * * * * * * * * * Dit bericht en eventuele bijlagen geven enkel de visie van de schrijver weer en binden het INBO onder geen enkel beding, zolang dit bericht niet bevestigd is door een geldig ondertekend document.
> The views expressed in this message and any annex are purely those of the writer and may not be regarded as stating an official position of INBO, as long as the message is not confirmed by a duly signed document.
>
>
> The contents of this e-mail and of any attachments transmitted with it are confidential and intended solely for the use of the individual(s) to whom they are addressed. If you are not an intended recipient or the person responsible for delivering this e-mail to the intended recipient(s), you have received this e-mail in error, and access to it by you is not authorised. You may not use, copy, distribute, disclose or rely on it or any attachment, or any part of it or any attachment in any way. If you have received this e-mail in error please notify Optimum Credit Ltd by email on webmaster at optimumcredit.co.uk or phone on 03330 143125 and then delete it.
>
> All reasonable precautions have been taken to ensure that no viruses are present in E-mail. As we cannot accept responsibility for loss or damage arising from the use of E-mail or any attachment we recommend you subject these to your usual virus checking procedures prior to use. Any opinions expressed in this e-mail are those of the person sending it and not necessarily those of Optimum Credit Ltd.
>
> Optimum Credit Ltd, Haywood House South, Dumfries Place, Cardiff, CF10 3GA
>
> Authorised and regulated by the Financial Conduct Authority
>
> Optimum Credit Ltd. Registered office: Haywood House South, Dumfries Place, Cardiff, CF10 3GA. Registered in England and Wales No. 08698121
>
> Calls may be monitored or recorded for training, compliance and evidential purposes.
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel