[Rd] declaring package dependencies

[Duncan Murdoch]

On 13-09-14 12:19 PM, Paul Gilbert wrote:
>
>
> On 13-09-14 09:04 AM, Duncan Murdoch wrote:
>> On 13-09-13 12:00 PM, Dirk Eddelbuettel wrote:
>>>
>>> On 13 September 2013 at 11:42, Paul Gilbert wrote:
>>> | On 13-09-13 11:02 AM, Dirk Eddelbuettel wrote:
>>> | > It's not so much Rcpp itself or my 20-ish packages but the fact
>>> that we (as
>>> | > in the Rcpp authors) now stand behind an API that also has to
>>> accomodate
>>> | > changes in R CMD check. Case in point is current (unannounced)
>>> change that
>>> | > makes all Depends: Rcpp become Imports: Rcpp because of the
>>> NAMESPACE checks.
>>> |
>>> | I am a bit confused by this Dirk, so maybe I am missing something. I
>>> | think this is still a "Note" in R-devel so you do have some time to
>>> make
>>> | the change, at least several months, maybe more. It is not quite what I
>>> | think of as an "announcement", more like a shot across the bow, but it
>>> | is also not "unannounced".
>>>
>>> One package author [as in user of Rcpp and not an author of it] was
>>> told by
>>> CRAN this week to change his package and came to me for help -- so in
>>> that
>>> small way the CRAN "non-communication policy" is already creating more
>>> work
>>> for me, and makes me look silly as "I don't document what Rcpp-using
>>> packages
>>> need" as I sadly still lack the time machine or psychic powers to
>>> infer what
>>> may get changed this weekend.
>>>
>>> | More importantly, I don't think that the requirement is necessarily to
>>> | change Depends: Rcpp to Imports: Rcpp, the requirement is to put
>>> | imports(Rcpp) in the NAMESPACE file. I think this is so that the
>>> package
>>> | continues to work even if the user does something with the search path.
>>> | The decision to change Depends: Rcpp to Imports: Rcpp really depends on
>>> | whether the package author wants Rcpp functions to be available
>>> directly
>>>
>>> Rcpp is a bit of an odd-ball as you mostly need it at compile-time,
>>> and you
>>> require very few R-level functions (but there is package
>>> initialization etc
>>> pp).  We also only about two handful of functions, and those are for
>>> functionality not all 135 packages use (eg Modules etc).
>>>
>>> But the focus here should not be on my hobby package. The focus needs
>>> to be
>>> on how four CRAN maintainers (who do a boatload of amazing work which is
>>> _truly_ appreciated in its thoroughness and reach) could make the life of
>>> authors of 4800+ packages easier by communicating and planning a tad
>>> more.
>>
>> Let me paraphrase that:  "The CRAN maintainers do a lot of work, and it
>> helps me a lot, but if they only did a little bit more work it would
>> help me even more."
>>
>> I suspect they'd be more receptive to suggestions that had them doing
>> less work, not more.
>
> Actually, this is one of the parts that I do not understand. It seems to
> me that it would be a lot less work for CRAN maintainers if the
> implications and necessary changes to packages were explained a bit more
> clearly in a forum like R-devel that many package developers actually
> read regularly.

Then why don't you explain them?  They aren't secret.

I many not fully understand how much of the response to
> package submission gets done automatically, but I do get the sense that
> there is a fairly large amount of actual human time spent dealing with
> just my submissions alone. If that is representative of all developers,
> then CRAN maintainers don't have time to do much else. (The fact that
> they do much more suggests I may not be representative.)
>
> Two specific points have already been mentioned implicitly. CRAN
> submission testing is often done at a higher/newer level using the
> latest devel version. This results in lots of rejections for things that
> I would fix before submission, if I knew about them.

Then why don't you test against R-devel before submitting?

  If the tests were
> rolled out with R, and only later incorporated into CRAN submission
> testing, I think there would be a lot less work for the CRAN
> maintainers.

I am an R core member, not a member of CRAN.  If I make a change to R, 
I'd like to know the effect it has on the corpus of CRAN and 
Bioconductor packages, but I don't have the computing resources to run 
it against all of those in a reasonable amount of time.  We have heard 
from various corporate users of R (Revolution Computing and Google are 
two) that they would like to contribute to the project, but we don't 
actually see useful contributions from them.

I don't know if you personally have influence with anyone who has such 
resources, but surely some on this list do.  So why don't they write to 
me and tell me how to use their computing resources to run proposed 
changes against all published packages?  I think it's because it's 
easier to do nothing than to do something.  It's certainly easier to say 
that it's the CRAN maintainers' fault for poor communication than it is 
to identify and address the problems.

Duncan Murdoch


  (This is ignoring the possibility that CRAN submission is
> really the testing ground for the tests, and to prove the tests requires
> a fair amount of manual involvement. I'm happy to continue contributing
> to this -- I've often felt my many contribution is an endless supply of
> bugs for the checkers to catch.)
>
> The second point is that a facility like R-forge that runs the latest
> checks, on many platforms, is really useful in order to reduce work for
> both package developers and CRAN maintainers. With R-forge broken, the
> implication for additional work for CRAN maintainers seems enormous. But
> even with it working, not all packages are kept on R-forge, and with
> package version dependencies R-forge does not really work. (i.e. I have
> to get new versions of some packages onto CRAN before the new versions
> of other packages will build on R-forge.)  Perhaps the package checking
> part of R-forge should be separated into a pre-submission clearing house
> to which packages are submitted. If they pass checks there then the
> package developer could click on a submit button to do the actual
> submission to CRAN. (Of course there needs to be a mechanism to plead
> for the fact that the test systems do not have needed resources.)
> Something like the daily, but with new pre-release versions of packages
> might actually be better than the R-forge approach, for two reasons. One
> is that package maintainers would only put packages there that they
> think are actually working. (R-forge tries to build my svn copy even
> when I know it is broken.) The second is that it would automatically
> handle the version dependency problem, since package maintainers would
> have the ability to put in place versions that should work together.
> However, this does not need to be run daily. It only needs to be run
> when the checks change, or for a package when the package changes.
>
> Paul
>
>>
>> Duncan Murdoch
>>
>>>
>>> | by users without them needing to specifically attach Rcpp. They are
>>> | available with Depends but with Imports they are just used
>>> internally in
>>> | the package.
>>> |
>>> | So, one of us is confused. Usually it is me.
>>>
>>> No, no, I usually keep you company.
>>>
>>> Dirk
>>>
>>