[Bioc-devel] shiny, Bioconductor, and reproducible research

Mon Jul 29 22:09:26 CEST 2013

Hi Kasper,

On Mon, Jul 29, 2013 at 12:08 PM, Kasper Daniel Hansen
<kasperdanielhansen at gmail.com> wrote:
> My perspective,
>
> Clearly we are very interested in supporting reproducible research in
> Bioconductor.  Indeed, it is a core mission of the project.  However, I
> feel that Dan is placing a very high burden of proof on shiny apps in his
> email, since it essentially says "because it is possible to potentially
> conduct non-reproducible research in shiny, we should not allow it".  The
> logical conclusion of such a statement, is that we should never allow GUIs
> nor interactive code.
>

Note that I am not setting out policy in this email, nor am I
speculating about what should or should not be allowed.

> Without having a lot of experience with shiny, I am mostly excited because
> of the potential for (semi) interactive graphics, especially for exploring
> your data (here I am like very biased since that is what we are using it
> for right now).  For such an application, there should not be any
> modification of the user's R objects.
> It is true that an interactive
> exploration of your data might be considered non-reproducible, but that is
> also the case if I inspects 100's of pdfs.

In my earlier email, I explicitly state that a shiny object that does
not modify the user session does not pose any problems with respect to
reproducibility.

>
> In my opinion we should try to get people to adhere to best practices, and
> (once we have something) we could provide example code for these practices.
>  Perhaps a starting point would be a function to tag all objects in the
> user environment and keep track of whether they are modified during the
> shiny session.  In the meantime (unless Michael and Tengfei comes up with
> an awesome solution in the next week :), I do believe we should consider
> how to host such apps, including how do we do code checking.  Ultimately,
> this would be better for the research community than to force people to
> host it on their own webpages.  If we cannot find a suitable way to do code
> checking or to package up the app, perhaps we should not do it.  But that
> would be a better reason than the potential for writing non-reproducible
> code.

I took a look at the source of shiny and it does appear that reactive
objects can be tested outside of shiny; they have some unit tests
which do this. I'd like to hear from shiny developers with their
thoughts on best practices for unit testing shiny apps (this might
make a good section in the shiny tutorial?).

I don't think anyone is interested in banning shiny apps from
Bioconductor; we like shiny. The original email was just about
thinking through the implications of shiny and reproducible research
(and also unit testing), and coming up with those best practices so
that we can get package developers to apply them to what will surely
be an increasing number of packages that use shiny.

Dan

>
> Best,
> Kasper
>
>
> On Sat, Jul 27, 2013 at 9:17 PM, Levi Waldron <lwaldron at hsph.harvard.edu>wrote:
>
>> I agree with Dan on the need to be able to "save" analysis done in a Shiny
>> Bioconductor app, allowing someone later to reload the same state, and to
>> inspect a session and see what was done.  Perhaps "save" and "load" buttons
>> should be present in the user interface of any Bioconductor Shiny app that
>> is not "read-only".
>>
>> If interactive user input can be saved (as an .RData file?), could that
>> file be used as input for unit tests either the usual console way or
>> something like an http POST?
>>
>> -Levi
>>
>>
>>
>>
>>
>> On Fri, Jul 26, 2013 at 5:42 PM, Michael Lawrence <
>> lawrence.michael at gene.com
>> > wrote:
>>
>> > An answer to both of these issues is the use of R code. It should be
>> > possible to drive shiny apps through the server, without the user's code
>> > knowing the difference. And you've already thought of recording user
>> > actions as R code. This would be an application of the command pattern,
>> as
>> > with undo/redo stacks, except each object boils down to a
>> > function/expression (Tengfei and I have a CRAN package called commandr
>> > related to this). Someone should try to make a prototype to explore these
>> > issues then we can start thinking about common infrastructure.
>> >
>> >
>> > On Fri, Jul 26, 2013 at 12:23 PM, Dan Tenenbaum <dtenenba at fhcrc.org>
>> > wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > There was a lot of talk about shiny at BioC2013 and it seems like a
>> > > lot of people are using it for lots of different cool stuff.
>> > >
>> > > Now we are at the point where some contributors want to add shiny apps
>> > > to Bioconductor as packages.
>> > >
>> > > We had some discussion internally and decided that there are a few
>> > > issues we wanted to discuss with BioC developers as well as some of
>> > > the shiny developers. Please feel free to share your thoughts.
>> > >
>> > > 1) Testing shiny apps
>> > >
>> > > Typically, bioconductor packages have man page examples, vignettes,
>> > > and (sometimes) unit tests, so when we build the packages every day on
>> > > our build system, the code in all of these is evaluated and we have
>> > > some idea of whether the package is working as it's supposed to.
>> > >
>> > > I'm not clear on how to do similar testing of a shiny application.
>> > > Since launching a shiny app takes away the R console (until the app is
>> > > closed), shiny apps should probably not be launched in example or test
>> > > code (unless interactive() is TRUE).
>> > >
>> > > Do the shiny folks (or anyone else) have thoughts on testing shiny
>> apps?
>> > >
>> > > 2) Reproducible research
>> > >
>> > > Reproducible research is really important in our community, but shiny
>> > > apps are sort of a black box as far as reproducibility is concerned.
>> > >
>> > > If shiny apps are "read-only" (that is, if they are just used to view
>> > > an object) then this is not really a problem. But shiny has the
>> > > ability now to change objects back in your R session, so we need to be
>> > > able to track what is done inside the shiny app.
>> > >
>> > > shiny apps can return some sort of object that tells you what was done
>> > > in the shiny session (insofar as it modifies any objects in the
>> > > users's session) and this object can provide a way to reproduce those
>> > > steps without the shiny app. So maybe the object would consist of
>> > > lines of code. This means that if there is a shiny app, the package
>> > > author must also provide ways to do the same transformations on
>> > > objects without shiny.
>> > >
>> > > There are probably other approaches as well. But the end goal would be
>> > > a scriptable, reproducible shiny session.
>> > >
>> > > Another thought was that a shiny app could emit some sort of state
>> > > object, and then be restarted with that object.
>> > >
>> > > Not sure if the place for such infrastructure is inside shiny itself,
>> > > or perhaps in a BioC infrastructure package.
>> > >
>> > > What are your thoughts?
>> > > Dan
>> > >
>> > > _______________________________________________
>> > > Bioc-devel at r-project.org mailing list
>> > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> > >
>> >
>> >         [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >
>>
>>
>>
>> --
>> Levi Waldron
>> Post-doctoral fellow
>> Department of Biostatistics, Harvard School of Public Health
>> Department of Biostatistics and Computational Biology, Dana-Farber Cancer
>> Institute
>> Building 1, room 412C
>> 655 Huntington Avenue
>> Boston, Massachusetts 02115
>> mobile: (617) 851-6849
>> fax: (617) 432-5619
>> http://www.hsph.harvard.edu/research/levi-waldron/
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel