[Rd] Brainstorm: Alpha and Beta testing of R versions

[Henrik Bengtsson]

Andrew Robinson wrote:

>Hi Martin,
>
>On Fri, Nov 04, 2005 at 09:58:47AM +0100, Martin Maechler wrote:
>  
>
>>[Mainly for R-foundation members; but kept in public for general
>> brainstorming...]
>>    
>>
>
>I'll take up the invitation to brainstorm.
>
>As a user of R for a number of years, I'd really like to perform some
>useful service.  I use a relatively obscure platform (FreeBSD) and I
>can compile code.  I'd like to think that I'm in the target market for
>beta testing :).  But, I'm timid.  I do not feel, in general, that R
>core welcomes bug reports.
>
>I think that there are several things that could be tried to encourage
>more, and more useful, bug reports.
>
>1) Put the following text on the *front page* of the tracking system, so
>   that it is seen before the reader clicks on "New Bug Report":
>
>"Before submitting a bug report, please read Chapter `R Bugs' of `The
>R FAQ'. It describes what a bug is and how to report a bug.
>
>If you are not sure whether you have observed a bug or not, it is a
>good idea to ask on the mailing list R-Help by sending an e-mail to
>r-help at stat.math.ethz.ch rather than submitting a bug report."
>
>(BTW is this true also for alpha/beta testing?)
>
>2) Try to use the structure of the reporting page to prompt good
>   reporting.  On the report page, summarize the key points of
>   identifying and reporting a bug in a checklist format.  Maybe even
>   insist that the boxes be checked before allowing submission.
>   Include seperate text boxes for description and sample code, to
>   suggest that sample code is valued.
>  
>
...and a optional field to select one or several packages related to the 
bug.  This is a good place to clarify that problems related to 
third-party packages should not be reporter "here".  Example HTML code:

Package(s) related to the bug, if applicable:<br>
(Bugs related to packages not listed below should <em>not</em> be 
reported here. Instead, contact the package manager.)
<select name="packages" multiple>
 <option value="">- - Select one or more packages related to the bug - 
-</option>
 <option value="base">base (Base R functions)</option>
 <option value="datasets">datasets (Base R datasets)</option>
 <option value="grDevices">grDevices (Graphics devices for base and grid 
graphics)</option>
 <option value="graphics">graphics (R functions for base graphics)</option>
 <option value="grid">grid (A rewrite of the graphics layout 
capabilities, plus some support for interaction)</option>
 <option value="methods">methods (Formally defined methods and classes 
for R objects, plus other programming tools, as described in the Green 
Book)</option>
 <option value="splines">splines (Regression spline functions and 
classes)</option>
 <option value="stats">stats (R statistical functions)</option>
 <option value="stats4">stats4 (Statistical functions using S4 
classes)</option>
 <option value="tcltk">tcltk (Interface and language bindings to Tcl/Tk 
GUI elements)</option>
 <option value="tools">tools (Tools for package development and 
administration)</option>
 <option value="utils">utils (R utility functions)</option>
</select>

/Henrik

>3) On either or both pages (and in FAQ), explain that thoughtful bug
>   reports are valued and appreciated.  Further, explain that bug
>   reports that do not follow the protocol are less valuable, and take
>   more time.
>
>4) Add checkboxes to the report page for alpha/beta.  (I suggest this
>   for the purposes of marketing, not organization.)
>
>5) On the report page, include hyperlinks to archived bug reports that
>   were good.  Do likewise with some artificial bug reports that are
>   bad.
>
>6) Add an intermediate, draft step for bug submission, to allow
>   checking.  If possible, include as part of this step an automated
>   pattern matching call that identifies similarly texted bug reports,
>   provides links to the reports, and invites a last-minute cross-check.
>
>7) Keep a list of people who report useful bugs in alpha/beta phase on
>   the website.  Many academics could point to it as evidence of
>   community service.
>
>  
>
>>In order to discourage an increased number of non-bug reports we
>>may have to also open a "hall of shame" though...
>>    
>>
>
>8) I'm sure that you're being ironic!  But I will take the point
>   seriously, for what it's worth.  I think that humiliating
>   submitters who haven't followed the protocol is deleterious.  It
>   seems like almost every month we see someone get slapped on the
>   wrist for not doing something the right way.  Of course, it's
>   frustrating that people aren't following the posting guide.  But,
>   why is that?  Where is the breakdown?  It might be interesting to
>   try some follow-up (an exit interview!). If someone has failed to
>   follow the protocol, perhaps we should try to find out why it was
>   confusing, or if they just ignored it.
>
>   The R-core is surrounded by, and serves, a community that comprises
>   people who are not sufficiently good at what R-core does to be
>   invited in to R-core. But, we're clearly interested in what R-core
>   produces.  Please don't assume that bug submissions that do not
>   follow the R protocol are the consequence of deliberate
>   malfeasance.  
>
>   To paraphrase Ian Fleming: Once is happenstance.  Twice is
>   incompetence.  The third time, Mr. Bond, is enemy action. So, ...
>
>9) Publicly thank bug reporters whether their reports are useful or
>   not.  I just googled 'R-devel thank' and you figure prominently,
>   Martin :).
>
>Cheers
>
>Andrew
>  
>