[Rd] Brainstorm: Alpha and Beta testing of R versions

[Martin Maechler]

Thanks a lot, 
Andrew, for your input!

A general point about your suggestions:  You seem to assume that
bug reports are typically entered via the R-bugs web interface
(which is down at the moment and for a few more dozen hours probably},
rather than via  R's builtin  bug.report() function or the
simple e-mail to R-bugs at r-project.org  [[which will also not
properly work for the moment, as long as the bug repository is
suffering from a fiber cable cut in Kopenhagen]].

For some dinosaurs like me, having to fill a web page rather
than sending e-mail would be quite a loss of comfort, but
actually, it might not be a bad idea to require a unique
bug-entry interface -- actually we have been thinking of moving
to bugzilla -- if only Peter Dalgaard could find a smart enough
person (even to be paid) who'd port all the old bug reports into the
new format.. 


>>>>> "Andrew" == Andrew Robinson <A.Robinson at ms.unimelb.edu.au>
>>>>>     on Sun, 6 Nov 2005 11:01:30 +1100 writes:

    Andrew> Hi Martin, On Fri, Nov 04, 2005 at 09:58:47AM +0100,
    Andrew> Martin Maechler wrote:
    >> [Mainly for R-foundation members; but kept in public for
    >> general brainstorming...]

    Andrew> I'll take up the invitation to brainstorm.

good, thank 

    Andrew> As a user of R for a number of years, I'd really
    Andrew> like to perform some useful service.  I use a
    Andrew> relatively obscure platform (FreeBSD) and I can
    Andrew> compile code.  I'd like to think that I'm in the
    Andrew> target market for beta testing :).  

indeed!

    Andrew> But, I'm timid. I do not feel, in general, that R core welcomes bug
    Andrew> reports.

I think that's a partly wrong feeling; understandibly nourished
by some of our reactions about some "bug reports" that stemmed
from user misconceptions.  As you've remarked below, I've
expressed gratitude more than once for helpful bug reports.

    Andrew> I think that there are several things that could be
    Andrew> tried to encourage more, and more useful, bug
    Andrew> reports.

    Andrew> 1) Put the following text on the *front page* of the
    Andrew> tracking system, so that it is seen before the
    Andrew> reader clicks on "New Bug Report":

    Andrew> "Before submitting a bug report, please read Chapter
    Andrew> `R Bugs' of `The R FAQ'. It describes what a bug is
    Andrew> and how to report a bug.

    Andrew> If you are not sure whether you have observed a bug
    Andrew> or not, it is a good idea to ask on the mailing list
    Andrew> R-Help by sending an e-mail to
    Andrew> r-help at stat.math.ethz.ch rather than submitting a
    Andrew> bug report."

    Andrew> (BTW is this true also for alpha/beta testing?)

Yes, in principile.  The only thing to be changed would be 
   sub("-help", "-devel",  <the above text>)

    Andrew> 2) Try to use the structure of the reporting page to
    Andrew> prompt good reporting.  On the report page,
    Andrew> summarize the key points of identifying and
    Andrew> reporting a bug in a checklist format.  Maybe even
    Andrew> insist that the boxes be checked before allowing
    Andrew> submission.  Include seperate text boxes for
    Andrew> description and sample code, to suggest that sample
    Andrew> code is valued.

    Andrew> 3) On either or both pages (and in FAQ), explain
    Andrew> that thoughtful bug reports are valued and
    Andrew> appreciated.  Further, explain that bug reports that
    Andrew> do not follow the protocol are less valuable, and
    Andrew> take more time.

    Andrew> 4) Add checkboxes to the report page for alpha/beta.
    Andrew> (I suggest this for the purposes of marketing, not
    Andrew> organization.)

    Andrew> 5) On the report page, include hyperlinks to
    Andrew> archived bug reports that were good.  Do likewise
    Andrew> with some artificial bug reports that are bad.

    Andrew> 6) Add an intermediate, draft step for bug
    Andrew> submission, to allow checking.  If possible, include
    Andrew> as part of this step an automated pattern matching
    Andrew> call that identifies similarly texted bug reports,
    Andrew> provides links to the reports, and invites a
    Andrew> last-minute cross-check.


    Andrew> 7) Keep a list of people who report useful bugs in
    Andrew> alpha/beta phase on the website.  Many academics
    Andrew> could point to it as evidence of community service.

    >> In order to discourage an increased number of non-bug
    >> reports we may have to also open a "hall of shame"
    >> though...

    Andrew> 8) I'm sure that you're being ironic!  

indeed I was, partly.  The point was just that if the bug
reporting will be something like a challenge with prizes, we had
to discourage too many entries {which would be made just to try
to win (a|the) prize}.

    Andrew> 8) I'm sure that you're being ironic!  But I will
    Andrew> take the point seriously, for what it's worth.  I
    Andrew> think that humiliating submitters who haven't
    Andrew> followed the protocol is deleterious.  It seems like
    Andrew> almost every month we see someone get slapped on the
    Andrew> wrist for not doing something the right way.  Of
    Andrew> course, it's frustrating that people aren't
    Andrew> following the posting guide.  But, why is that?
    Andrew> Where is the breakdown?  It might be interesting to
    Andrew> try some follow-up (an exit interview!). If someone
    Andrew> has failed to follow the protocol, perhaps we should
    Andrew> try to find out why it was confusing, or if they
    Andrew> just ignored it.

    Andrew>    The R-core is surrounded by, and serves, a
    Andrew> community that comprises people who are not
    Andrew> sufficiently good at what R-core does to be invited
    Andrew> in to R-core. But, we're clearly interested in what
    Andrew> R-core produces.  Please don't assume that bug
    Andrew> submissions that do not follow the R protocol are
    Andrew> the consequence of deliberate malfeasance.

    Andrew>    To paraphrase Ian Fleming: Once is happenstance.
    Andrew> Twice is incompetence.  The third time, Mr. Bond, is
    Andrew> enemy action. So, ...

    Andrew> 9) Publicly thank bug reporters whether their
    Andrew> reports are useful or not.  I just googled 'R-devel
    Andrew> thank' and you figure prominently, Martin :).

Thanks again, Andrew,
for your useful input!

Martin