[Bioc-devel] Workflows are now in git (and other Important workflow-related changes)

Fri Apr 13 20:58:39 CEST 2018

Thank you Andrzej for the info!

## Re-point (1)

Regarding my (1), I got my github repo in sync following
http://bioconductor.org/developers/how-to/git/sync-existing-repositories/
and http://bioconductor.org/developers/how-to/git/abandon-changes/#force-bioconductor--to-github-
It all worked without problems. Thanks again for the great docs!

## on.bioc auto-detection

At https://github.com/LieberInstitute/recountWorkflow/commit/b456fbbe6ddbeda297ef03b79de9e004f28c0b97#diff-a87d0071c97357f224ad6608f837fbbfR45
I'm using

on.bioc <- knitr::opts_knit$get("rmarkdown.pandoc.to") != 'latex'

to detect automatically whether I'm rendering the output for Bioc or
not. For F1000 I'll use BiocWorkflowTools::f1000_article as the output
by calling rmarkdown::render() manually. I thought this little code
would help others who want to have a single repo for both BioC and
F1000.

## Re-point (5)

>From Lori's email that she sent today, I can see at
http://bioconductor.org/checkResults/3.7/workflows-LATEST/recountWorkflow/malbec2-buildwebvig.html
how the final Bioc web version is being built. That is:

/home/biocbuild/bbs-3.7-bioc/R/bin/R -q -e
'rmarkdown::render(".buildwebvig/recountWorkflow/recount-workflow.Rmd",
output_format="BiocStyle:::html_fragment")' &&
/home/biocbuild/bbs-3.7-bioc/R/bin/R -q -e
'knitr::purl(".buildwebvig/recountWorkflow/recount-workflow.Rmd",
".buildwebvig/recountWorkflow/recount-workflow.R")'

I can see the R version being used at
http://bioconductor.org/checkResults/3.7/workflows-LATEST/index.html
although not the bioc version (that I can infer from the package
loading messages, like clusterProfiler's one). This only matters in
those 6 months when release/devel are being built on the same R
version.

Now that I have the commands and versions, if I ever see anything
weird on the rendered Bioc web version I have a way to test things
myself instead of relying on the bioc builds. Since this is a bit
different than the rest of Bioc, maybe it'd be good to document the
build commands somewhere else.

Again, let me thank all of you for your work! I'm really excited about
the new workflow build chain and how it'll simplify
maintaining/writing workflows!

Best,
Leo

On Tue, Apr 3, 2018 at 6:38 AM, Andrzej Oleś <andrzej.oles at gmail.com> wrote:
>
> Dear Leo,
>
> thanks for you inquiry. Re points 2-4: there is no need to use the token file .html_output anymore. This was an intermediate solution meant for the transition period to new output format. Now all workflow vignettes for the the website are build using this new format, regardless the `output` setting in the vignette YAML header. Note that the format set in the vignette is still used to render the document distributed with the package tarball.
>
> Cheers,
> Andrzej
>
>
> On Mon, Apr 2, 2018 at 4:32 PM, Leonardo Collado Torres <lcollado at jhu.edu> wrote:
>>
>> Hi,
>>
>> Thank you for all the work in this transition Andrzej, Lori, Valerie,
>> Nitesh and Hervé!
>>
>>
>> I have a few questions about the new system.
>>
>> 1) I already had a github repo (that I manually kept in sync with the
>> subversion one) for the recountWorkflow. Should I just follow the
>> instructions at
>> http://bioconductor.org/developers/how-to/git/sync-existing-repositories/?
>> I guess that I'll need to do this too
>> http://bioconductor.org/developers/how-to/git/abandon-changes/#force-bioconductor--to-github-
>>
>> 2) Is repo/vignettes/.html_output still needed?
>>
>> 3) Related to (2), the .html_output file used to ignore the `output`
>> line in the vignette YAML. I got an email from Lori saying that
>> output: BiocWorkflowTools::f1000_article is not working for
>> recountWorfkflow for now (no html vignette). My guess is that the
>> `output` line now matters, and in my case, that would build a pdf file
>> which is ignored due to
>> https://github.com/LieberInstitute/recountWorkflow/blob/master/.Rbuildignore#L12
>> in my .Rbuildignore.
>>
>> 4) Related to the last 2. What setup do you recommend so I can
>> maintain a single git repo that produces the files needed for
>> uploading them to F1000research? Before this transition, I had to
>> maintain 3 repos: the subversion one for Bioc, my git copy (well,
>> main), and another one with the files setup for F1000. At some point
>> in the past, Andrzej recommended that I use "output:
>> BiocWorkflowTools::f1000_article" to try to have all the files in 1
>> place. I imagine that I would still need some ifelse() statements for
>> sentences like the one I have at
>> https://github.com/LieberInstitute/recountWorkflow/blob/master/vignettes/recount-workflow.Rmd#session-information
>> I think that this question could be generalized and documented in the
>> Bioc website.
>>
>> 5) Is there anything else I should be aware of so I can reproduce any
>> errors? In the past, the Bioc workflow system would trigger a build
>> immediately after a new push, and due to the custom way workflows were
>> built (like .html_output), I relied on these builds to check if my
>> changes had worked or not: specially early on where I had issues with
>> figures, links to figures and figure captions.
>>
>> Best,
>> Leo
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 2, 2018 at 7:42 AM, Shepherd, Lori
>> <Lori.Shepherd at roswellpark.org> wrote:
>> >
>> > For now the page at
>> >
>> > https://www.bioconductor.org/help/workflows/
>> >
>> > will remain and be updated accordingly however there is discussion about having this page be removed or redirected.
>> >
>> >
>> > In the next few days I hope to have the new landing pages up and running and will make an announcement when they are live. The new changes involve having a link to the workflows on the biocViews page, that will directly link to a workflow package landing page (like the software, annotation, and experiment package do).
>> >
>> >
>> > Until the release your changes will be visible in devel which will transition over to the new release just as done for software packages -
>> >
>> > The workflow are built monday, wednesday, and friday and the new landing pages will reflect when the package is propagated like the other types of packages.
>> > http://bioconductor.org/checkResults/3.7/workflows-LATEST/
>> >
>> >
>> > Lori Shepherd
>> >
>> > Bioconductor Core Team
>> >
>> > Roswell Park Cancer Institute
>> >
>> > Department of Biostatistics & Bioinformatics
>> >
>> > Elm & Carlton Streets
>> >
>> > Buffalo, New York 14263
>> >
>> > ________________________________
>> > From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Aaron Lun <alun at wehi.edu.au>
>> > Sent: Sunday, April 1, 2018 2:17:09 PM
>> > To: Herv� Pag�s; bioc-devel at r-project.org
>> > Subject: Re: [Bioc-devel] Workflows are now in git (and other Important workflow-related changes)
>> >
>> > Thanks everybody, this is much appreciated.
>> >
>> > On that note, will the compiled workflows shown at:
>> >
>> > https://www.bioconductor.org/help/workflows/
>> >
>> > ... be updated to reflect changes in the git repositories for the
>> > workflows?
>> >
>> > Or will the workflow page just directly link to the landing page for
>> > each package? This would be very convenient, not least because it will
>> > avoid me having to make pull requests to the bioconductor.org
>> > repository every time I want to change my workflow blurb.
>> >
>> > I'm in the process of fixing and updating my various workflows, and I'm
>> > wondering when/how my changes will be visible to users.
>> >
>> > Cheers,
>> >
>> > Aaron
>> >
>> > On Fri, 2018-03-30 at 13:10 -0700, Herv� Pag�s wrote:
>> > > To the authors/maintainers of the workflows:
>> > >
>> > >
>> > > Following the svn-to-git migration of the software and data
>> > > experiment
>> > > packages last summer, we've completed the migration of the workflow
>> > > packages.
>> > >
>> > > The canonical location for the workflow source code now is
>> > > git.bioconductor.org
>> > >
>> > > Please use your git client to access/maintain your workflow the same
>> > > way you would do it for a software or data-experiment package.
>> > >
>> > > We've also migrated the workflows to our in-house build system.
>> > > Starting with Bioc 3.7, the build report for the devel versions of
>> > > the workflows can be found here:
>> > >
>> > >    https://bioconductor.org/checkResults/devel/workflows-LATEST/
>> > >
>> > > We run these builds every other day (Mondays, Wednesdays, Fridays).
>> > > Because of limited build resources, we now run the data-experiment
>> > > builds on Sundays, Tuesdays, and Thursdays only (instead of daily).
>> > >
>> > > The links to the package landing pages are not working yet. This
>> > > will be addressed in the next few days.
>> > >
>> > > Please address any error you see on the report for the workflow
>> > > you maintain.
>> > >
>> > > Note that, from now on, we're also following the same version scheme
>> > > for these packages as for the software and data-experiment packages.
>> > > That is, we're using an even y (in x.y.z) in release and an odd y in
>> > > devel. We'll take care of bumping y at release time (like we do for
>> > > software and data-experiment packages).
>> > >
>> > > After the next Bioconductor release (scheduled for May 1), we'll
>> > > start
>> > > building the release versions of the workflows in addition to the
>> > > devel versions. The build report for the release versions will be
>> > > here:
>> > >
>> > >    https://bioconductor.org/checkResults/release/workflows-LATEST/
>> > >
>> > > Finally, please note that with the latest version of BiocInstaller
>> > > (1.29.5), workflow packages can be installed with biocLite(), like
>> > > any other Bioconductor package. We'll deprecate the old mechanism
>> > > (workflowInstall()) at some point in the future.
>> > >
>> > > Thanks to Andrzej, Lori, Nitesh, and Valerie for working on this
>> > > migration.
>> > >
>> > > Let us know if you have any question about this.
>> > >
>> > > H.
>> > >
>> > >
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >
>> >
>> > This email message may contain legally privileged and/or confidential information.  If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited.  If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.
>> >         [[alternative HTML version deleted]]
>> >
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
>