[Bioc-devel] Use and Usability metrics / shields

Henrik Bengtsson henrik.bengtsson at ucsf.edu
Wed May 20 19:34:54 CEST 2015 

So, lots of things are happening in a few months: Jim Hester starts
working at Bioconductor, we get Bioc shields/badges, Jim's covr
package is released on CRAN, snare drum, ...  am I to eager if I
already now start wishing for a hi-hat as well?

/Henrik

On Tue, May 19, 2015 at 12:47 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
>
>
> ----- Original Message -----
>> From: "Leonardo Collado Torres" <lcollado at jhu.edu>
>> To: "Dan Tenenbaum" <dtenenba at fredhutch.org>
>> Cc: "Jim Hester" <james.f.hester at gmail.com>, bioc-devel at r-project.org
>> Sent: Tuesday, May 19, 2015 12:37:18 PM
>> Subject: Re: [Bioc-devel] Use and Usability metrics / shields
>>
>> Regarding the 'posts' tag, I can see that it includes a "closed
>> questions" component. For example,
>> http://www.bioconductor.org/packages/release/bioc/html/derfinder.html
>> is 3/1/9/0 right now meaning that 0 questions are closed. From
>> https://support.bioconductor.org/info/faq/, only moderators can close
>> questions. That seems like quite a bit of work for the moderators. So
>> maybe it would be best to drop the "closed questions" component. Or
>> alternatively, can the author of a package moderate the posts that
>> have a tag corresponding to their package?
>>
>
> Perhaps the wording is wrong; what 'closed' is supposed to mean is that the original poster has accepted an answer. I'll change 'closed' to 'accepted'.
>
>
>> As for 'build: warnings', it seems like it will show for some devel
>> packages all the time. For example,
>> http://www.bioconductor.org/packages/devel/bioc/html/regionReport.html
>> always has a warning in the Windows build machine due to a mismatch
>> in
>> the version of Rtools installed.
>
> This is a bug in devtools and may have already been fixed (but not yet propagated to CRAN).
> IMO this should be reflected in the build shield.
>
> Dan
>
>
>>
>> I do like these changes and the addition of shields =)
>>
>>
>> On Thu, May 14, 2015 at 10:56 AM, Dan Tenenbaum
>> <dtenenba at fredhutch.org> wrote:
>> >
>> > ----- Original Message -----
>> >> From: "Jim Hester" <james.f.hester at gmail.com>
>> >> To: "Martin Morgan" <mtmorgan at fredhutch.org>
>> >> Cc: bioc-devel at r-project.org
>> >> Sent: Thursday, May 14, 2015 7:53:03 AM
>> >> Subject: Re: [Bioc-devel] Use and Usability metrics / shields
>> >>
>> >> The common shield convention is to use blue or orange when the
>> >> information
>> >> is not qualitatively good or bad, but the color choice is just
>> >> subjective
>> >> in the end.
>> >
>> > It does seem though that we should indicate the non-changing nature
>> > of these shields with some kind of color change. Perhaps we can
>> > come up with one that works with the other design elements on the
>> > page.
>> >
>> > BTW, the 'posts' tag does change color; if there are 0 posts tagged
>> > with a package name, the shield is yellow; otherwise it's green.
>> >
>> > Dan
>> >
>> >
>> >>
>> >> On Thu, May 14, 2015 at 10:47 AM, Martin Morgan
>> >> <mtmorgan at fredhutch.org>
>> >> wrote:
>> >>
>> >> > On 05/10/2015 11:39 AM, COMMO Frederic wrote:
>> >> >
>> >> >> Dear Martin,
>> >> >>
>> >> >> All of these suggestions sound good.
>> >> >>
>> >> >> Wolfgang's suggestion regarding possible associated papers
>> >> >> might
>> >> >> be also
>> >> >> great.
>> >> >>
>> >> >> Another useful information would be to point to other
>> >> >> publications
>> >> >> where
>> >> >> a given package was used, and cited.
>> >> >> I don't know if it's technically possible, but it would be
>> >> >> greatly
>> >> >> informative to know how frequently a package is used, and how
>> >> >> it
>> >> >> performs,
>> >> >> in real contexts.
>> >> >>
>> >> >> Frederic Commo
>> >> >> Bioinformatics, U981
>> >> >> Gustave Roussy
>> >> >>
>> >> >> ________________________________________
>> >> >> De : Bioc-devel [bioc-devel-bounces at r-project.org] de la part
>> >> >> de
>> >> >> Wolfgang Huber [whuber at embl.de]
>> >> >> Date d'envoi : samedi 9 mai 2015 19:57
>> >> >> À : Martin Morgan
>> >> >> Cc: bioc-devel at r-project.org
>> >> >> Objet : Re: [Bioc-devel] Use and Usability metrics / shields
>> >> >>
>> >> >> Dear Martin
>> >> >>
>> >> >> great idea.
>> >> >> "Current build status” could perhaps be wrapped with
>> >> >> "Cross-platform
>> >> >> availability” into some sort of “Availability / Accessibility”?
>> >> >>
>> >> >> I wonder how informative it would be to make metrics such as
>> >> >> (i) citations of the associated paper
>> >> >> (ii) full-text mentions e.g. in PubmedCentral
>> >> >> actually useful. (i) could be flawed if package and paper are
>> >> >> diverged;
>> >> >> (ii) would require good disambiguation, e.g. like bioNerDS
>> >> >> http://www.biomedcentral.com/1471-2105/14/194 (or other tools?
>> >> >> not
>> >> >> my
>> >> >> expertise). Do we have someone with capabilities in this area
>> >> >> on
>> >> >> this list?
>> >> >>
>> >> >>
>> >> > Thanks for these suggestions.
>> >> >
>> >> > I like the idea of linking into the scientific literature,
>> >> > initially as
>> >> > part of the 'Citation' section on each landing page rather than
>> >> > as
>> >> > a shield
>> >> > (maybe a shield in the long term). As Wolfgang mentions it is a
>> >> > little more
>> >> > challenging than a one-liner to match the information available
>> >> > from a
>> >> > CITATION file (or automatically generated) to an appropriate
>> >> > search
>> >> > in
>> >> > PubMed, and because citations are an important formal metric it
>> >> > seems
>> >> > important to get this more-or-less right.
>> >> >
>> >> > For what it's worth the more-or-less continuous stream of papers
>> >> > citing
>> >> > 'Biocondcutor' are listed at
>> >> >
>> >> >   http://bioconductor.org/help/publications/
>> >> >
>> >> > The links in the 'Literature Search' box query various resources
>> >> > for use
>> >> > of the term 'Bioconductor'.
>> >> >
>> >> > We have so far kept the distinction between 'available' and
>> >> > 'build',
>> >> > partly because builds sometimes fail for transient (e.g.,
>> >> > connectivity)
>> >> > reasons or, in devel, because of an incomplete check-in that
>> >> > does
>> >> > not
>> >> > compromise the end-user availability and functionality of the
>> >> > version
>> >> > available via biocLite().
>> >> >
>> >> > It's kind of amusing that (a) most of the information was
>> >> > already
>> >> > available, often on the landing page (like the links to build
>> >> > reports that
>> >> > Henrik mentions, or years in bioc), so the shields are serving
>> >> > just
>> >> > to
>> >> > emphasize these; and (b) the 'green' implies some-how 'good',
>> >> > but
>> >> > many of
>> >> > the shields (e.g., years in Bioc, posts, commits, downloads) are
>> >> > actually
>> >> > never not green. Maybe these shields should be white?
>> >> >
>> >> > Thanks again for the feedback; initial response seems to be
>> >> > positive.
>> >> >
>> >> > Martin
>> >> >
>> >> >
>> >> >  PS  Martin you’ll like Fig. 2 of their paper.
>> >> >>
>> >> >> Wolfgang
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>  On May 9, 2015, at 19:15 GMT+2, Martin Morgan
>> >> >>  <mtmorgan at fredhutch.org>
>> >> >>> wrote:
>> >> >>>
>> >> >>> Bioc developers!
>> >> >>>
>> >> >>> It's important that our users be able to identify packages
>> >> >>> that
>> >> >>> are
>> >> >>> suitable for their research question. Obviously a first step
>> >> >>> is
>> >> >>> to identify
>> >> >>> packages in the appropriate research domain, for instance
>> >> >>> through
>> >> >>> biocViews.
>> >> >>>
>> >> >>>   http://bioconductor.org/packages/release/
>> >> >>>
>> >> >>> We'd like to help users further prioritize their efforts by
>> >> >>> summarizing
>> >> >>> use and usability. Metrics include:
>> >> >>>
>> >> >>> - Cross-platform availability -- biocLite()-able from all or
>> >> >>> only
>> >> >>> some
>> >> >>> platforms
>> >> >>> - Support forum activity -- questions and comments /
>> >> >>> responses, 6
>> >> >>> month
>> >> >>> window
>> >> >>> - Download percentile -- top 5, 20, 50%, or 'available'
>> >> >>> - Current build status -- errors or warnings on some or all
>> >> >>> platforms
>> >> >>> - Developer activity -- commits in the last 6 months
>> >> >>> - Historical presence -- years in Bioconductor
>> >> >>>
>> >> >>> Obviously the metrics are imperfect, so constructive feedback
>> >> >>> welcome --
>> >> >>> we think the above capture in a more-or-less objective and
>> >> >>> computable way
>> >> >>> the major axes influencing use and usability.
>> >> >>>
>> >> >>> We initially intend to prominently display 'shields' (small
>> >> >>> graphical
>> >> >>> icons) on package landing pages.
>> >> >>>
>> >> >>> Thanks in advance for your comments,
>> >> >>>
>> >> >>> Martin Morgan
>> >> >>> Bioconductor
>> >> >>> --
>> >> >>> Computational Biology / Fred Hutchinson Cancer Research Center
>> >> >>> 1100 Fairview Ave. N.
>> >> >>> PO Box 19024 Seattle, WA 98109
>> >> >>>
>> >> >>> Location: Arnold Building M1 B861
>> >> >>> Phone: (206) 667-2793
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Bioc-devel at r-project.org mailing list
>> >> >>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >> >>>
>> >> >>
>> >> >> _______________________________________________
>> >> >> Bioc-devel at r-project.org mailing list
>> >> >> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >> >>
>> >> >>
>> >> >
>> >> > --
>> >> > Computational Biology / Fred Hutchinson Cancer Research Center
>> >> > 1100 Fairview Ave. N.
>> >> > PO Box 19024 Seattle, WA 98109
>> >> >
>> >> > Location: Arnold Building M1 B861
>> >> > Phone: (206) 667-2793
>> >> >
>> >> > _______________________________________________
>> >> > Bioc-devel at r-project.org mailing list
>> >> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >> >
>> >>
>> >>       [[alternative HTML version deleted]]
>> >>
>> >> _______________________________________________
>> >> Bioc-devel at r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >>
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel

More information about the Bioc-devel mailing list