[Rd] Brainstorm: Alpha and Beta testing of R versions
A.Robinson at ms.unimelb.edu.au
Sun Nov 6 01:01:30 CET 2005
On Fri, Nov 04, 2005 at 09:58:47AM +0100, Martin Maechler wrote:
> [Mainly for R-foundation members; but kept in public for general
I'll take up the invitation to brainstorm.
As a user of R for a number of years, I'd really like to perform some
useful service. I use a relatively obscure platform (FreeBSD) and I
can compile code. I'd like to think that I'm in the target market for
beta testing :). But, I'm timid. I do not feel, in general, that R
core welcomes bug reports.
I think that there are several things that could be tried to encourage
more, and more useful, bug reports.
1) Put the following text on the *front page* of the tracking system, so
that it is seen before the reader clicks on "New Bug Report":
"Before submitting a bug report, please read Chapter `R Bugs' of `The
R FAQ'. It describes what a bug is and how to report a bug.
If you are not sure whether you have observed a bug or not, it is a
good idea to ask on the mailing list R-Help by sending an e-mail to
r-help at stat.math.ethz.ch rather than submitting a bug report."
(BTW is this true also for alpha/beta testing?)
2) Try to use the structure of the reporting page to prompt good
reporting. On the report page, summarize the key points of
identifying and reporting a bug in a checklist format. Maybe even
insist that the boxes be checked before allowing submission.
Include seperate text boxes for description and sample code, to
suggest that sample code is valued.
3) On either or both pages (and in FAQ), explain that thoughtful bug
reports are valued and appreciated. Further, explain that bug
reports that do not follow the protocol are less valuable, and take
4) Add checkboxes to the report page for alpha/beta. (I suggest this
for the purposes of marketing, not organization.)
5) On the report page, include hyperlinks to archived bug reports that
were good. Do likewise with some artificial bug reports that are
6) Add an intermediate, draft step for bug submission, to allow
checking. If possible, include as part of this step an automated
pattern matching call that identifies similarly texted bug reports,
provides links to the reports, and invites a last-minute cross-check.
7) Keep a list of people who report useful bugs in alpha/beta phase on
the website. Many academics could point to it as evidence of
> In order to discourage an increased number of non-bug reports we
> may have to also open a "hall of shame" though...
8) I'm sure that you're being ironic! But I will take the point
seriously, for what it's worth. I think that humiliating
submitters who haven't followed the protocol is deleterious. It
seems like almost every month we see someone get slapped on the
wrist for not doing something the right way. Of course, it's
frustrating that people aren't following the posting guide. But,
why is that? Where is the breakdown? It might be interesting to
try some follow-up (an exit interview!). If someone has failed to
follow the protocol, perhaps we should try to find out why it was
confusing, or if they just ignored it.
The R-core is surrounded by, and serves, a community that comprises
people who are not sufficiently good at what R-core does to be
invited in to R-core. But, we're clearly interested in what R-core
produces. Please don't assume that bug submissions that do not
follow the R protocol are the consequence of deliberate
To paraphrase Ian Fleming: Once is happenstance. Twice is
incompetence. The third time, Mr. Bond, is enemy action. So, ...
9) Publicly thank bug reporters whether their reports are useful or
not. I just googled 'R-devel thank' and you figure prominently,
Senior Lecturer in Statistics Tel: +61-3-8344-9763
Department of Mathematics and Statistics Fax: +61-3-8344-4599
University of Melbourne, VIC 3010 Australia
Email: a.robinson at ms.unimelb.edu.au Website: http://www.ms.unimelb.edu.au
More information about the R-devel