[Rd] declaring package dependencies

[Paul Gilbert]

On 13-09-14 07:20 PM, Duncan Murdoch wrote:
> On 13-09-14 12:19 PM, Paul Gilbert wrote:
>>
>>
>> On 13-09-14 09:04 AM, Duncan Murdoch wrote:
>>> On 13-09-13 12:00 PM, Dirk Eddelbuettel wrote:
>>>>
>>>> On 13 September 2013 at 11:42, Paul Gilbert wrote:
>>>> | On 13-09-13 11:02 AM, Dirk Eddelbuettel wrote:
>>>> | > It's not so much Rcpp itself or my 20-ish packages but the fact
>>>> that we (as
>>>> | > in the Rcpp authors) now stand behind an API that also has to
>>>> accomodate
>>>> | > changes in R CMD check. Case in point is current (unannounced)
>>>> change that
>>>> | > makes all Depends: Rcpp become Imports: Rcpp because of the
>>>> NAMESPACE checks.
>>>> |
>>>> | I am a bit confused by this Dirk, so maybe I am missing something. I
>>>> | think this is still a "Note" in R-devel so you do have some time to
>>>> make
>>>> | the change, at least several months, maybe more. It is not quite
>>>> what I
>>>> | think of as an "announcement", more like a shot across the bow,
>>>> but it
>>>> | is also not "unannounced".
>>>>
>>>> One package author [as in user of Rcpp and not an author of it] was
>>>> told by
>>>> CRAN this week to change his package and came to me for help -- so in
>>>> that
>>>> small way the CRAN "non-communication policy" is already creating more
>>>> work
>>>> for me, and makes me look silly as "I don't document what Rcpp-using
>>>> packages
>>>> need" as I sadly still lack the time machine or psychic powers to
>>>> infer what
>>>> may get changed this weekend.
>>>>
>>>> | More importantly, I don't think that the requirement is
>>>> necessarily to
>>>> | change Depends: Rcpp to Imports: Rcpp, the requirement is to put
>>>> | imports(Rcpp) in the NAMESPACE file. I think this is so that the
>>>> package
>>>> | continues to work even if the user does something with the search
>>>> path.
>>>> | The decision to change Depends: Rcpp to Imports: Rcpp really
>>>> depends on
>>>> | whether the package author wants Rcpp functions to be available
>>>> directly
>>>>
>>>> Rcpp is a bit of an odd-ball as you mostly need it at compile-time,
>>>> and you
>>>> require very few R-level functions (but there is package
>>>> initialization etc
>>>> pp).  We also only about two handful of functions, and those are for
>>>> functionality not all 135 packages use (eg Modules etc).
>>>>
>>>> But the focus here should not be on my hobby package. The focus needs
>>>> to be
>>>> on how four CRAN maintainers (who do a boatload of amazing work
>>>> which is
>>>> _truly_ appreciated in its thoroughness and reach) could make the
>>>> life of
>>>> authors of 4800+ packages easier by communicating and planning a tad
>>>> more.
>>>
>>> Let me paraphrase that:  "The CRAN maintainers do a lot of work, and it
>>> helps me a lot, but if they only did a little bit more work it would
>>> help me even more."
>>>
>>> I suspect they'd be more receptive to suggestions that had them doing
>>> less work, not more.
>>
>> Actually, this is one of the parts that I do not understand. It seems to
>> me that it would be a lot less work for CRAN maintainers if the
>> implications and necessary changes to packages were explained a bit more
>> clearly in a forum like R-devel that many package developers actually
>> read regularly.
>
> Then why don't you explain them?  They aren't secret.

Well, I have been trying to do that on this and related threads over the 
past few weeks. But there is a large credibility difference between my 
explanation of something I am just learning about myself and an 
explanation by a core member or CRAN maintainer of something they have 
implemented. (At least, I hope most readers of this list know there is a 
difference.)

>
> I many not fully understand how much of the response to
>> package submission gets done automatically, but I do get the sense that
>> there is a fairly large amount of actual human time spent dealing with
>> just my submissions alone. If that is representative of all developers,
>> then CRAN maintainers don't have time to do much else. (The fact that
>> they do much more suggests I may not be representative.)
>>
>> Two specific points have already been mentioned implicitly. CRAN
>> submission testing is often done at a higher/newer level using the
>> latest devel version. This results in lots of rejections for things that
>> I would fix before submission, if I knew about them.
>
> Then why don't you test against R-devel before submitting?

I have been relying on R-forge to provide that testing. One practical 
suggestion in this thread (Matthew Dowle) was to test with win-builder 
R-devel. This needs to be amplified. I had thought of win-builder as a 
mechanism to test on Windows, since I rarely work on that platform. 
Following the CRAN submission guidelines I test on win-builder if I am 
not doing the Windows testing on my own machine and the R-forge results 
are not available. (I think for a single package they are equivalent 
when R-forge is working.) But on win-builder I have usually used the 
R-release directory. Using the R-devel directory has the advantage that 
it gives an as-cran test that is almost up-to-date with the one against 
which the package is tested when it is submitted.  Another feature of 
win-builder that I had not recognized is that submitted packages are 
available in its library for a short time, so packages with version 
dependencies can be tested there, whereas they cannot be tested on R-forge.
>
>   If the tests were
>> rolled out with R, and only later incorporated into CRAN submission
>> testing, I think there would be a lot less work for the CRAN
>> maintainers.
>
> I am an R core member, not a member of CRAN.  If I make a change to R,
> I'd like to know the effect it has on the corpus of CRAN and
> Bioconductor packages, but I don't have the computing resources to run
> it against all of those in a reasonable amount of time.  We have heard
> from various corporate users of R (Revolution Computing and Google are
> two) that they would like to contribute to the project, but we don't
> actually see useful contributions from them.
>
> I don't know if you personally have influence with anyone who has such
> resources,

As far as I know I don't have any influence with anybody, but I like 
this suggestion.

Paul

> but surely some on this list do.  So why don't they write to
> me and tell me how to use their computing resources to run proposed
> changes against all published packages?  I think it's because it's
> easier to do nothing than to do something.  It's certainly easier to say
> that it's the CRAN maintainers' fault for poor communication than it is
> to identify and address the problems.
>
> Duncan Murdoch
>
>
>   (This is ignoring the possibility that CRAN submission is
>> really the testing ground for the tests, and to prove the tests requires
>> a fair amount of manual involvement. I'm happy to continue contributing
>> to this -- I've often felt my many contribution is an endless supply of
>> bugs for the checkers to catch.)
>>
>> The second point is that a facility like R-forge that runs the latest
>> checks, on many platforms, is really useful in order to reduce work for
>> both package developers and CRAN maintainers. With R-forge broken, the
>> implication for additional work for CRAN maintainers seems enormous. But
>> even with it working, not all packages are kept on R-forge, and with
>> package version dependencies R-forge does not really work. (i.e. I have
>> to get new versions of some packages onto CRAN before the new versions
>> of other packages will build on R-forge.)  Perhaps the package checking
>> part of R-forge should be separated into a pre-submission clearing house
>> to which packages are submitted. If they pass checks there then the
>> package developer could click on a submit button to do the actual
>> submission to CRAN. (Of course there needs to be a mechanism to plead
>> for the fact that the test systems do not have needed resources.)
>> Something like the daily, but with new pre-release versions of packages
>> might actually be better than the R-forge approach, for two reasons. One
>> is that package maintainers would only put packages there that they
>> think are actually working. (R-forge tries to build my svn copy even
>> when I know it is broken.) The second is that it would automatically
>> handle the version dependency problem, since package maintainers would
>> have the ability to put in place versions that should work together.
>> However, this does not need to be run daily. It only needs to be run
>> when the checks change, or for a package when the package changes.
>>
>> Paul
>>
>>>
>>> Duncan Murdoch
>>>
>>>>
>>>> | by users without them needing to specifically attach Rcpp. They are
>>>> | available with Depends but with Imports they are just used
>>>> internally in
>>>> | the package.
>>>> |
>>>> | So, one of us is confused. Usually it is me.
>>>>
>>>> No, no, I usually keep you company.
>>>>
>>>> Dirk
>>>>
>>>
>