[Bioc-devel] Using BioC vignettes as Jupyter Notebooks

[Neumann, Steffen]

Hi,

On Tue, 2019-07-02 at 09:50 +0000, Martin Morgan wrote:
> Do you think this approach could be re-cast in the same way that 
> https://github.com/Bioconductor/bioconductor_full has done, so that
> the container has all the software to _support_ the packages needed
> for the jupyter notebook, but that the user must do some kind of
> initialization to install the R libraries? In this way there would be
> one container instead of 1750, and it would not be heavy?

Hm, the Jupyter can not "talk" to the container that is hosting it, 
so this would have to be done upon startup. 

> One idea that we've been exploring is making _binary_ versions of
> Bioc packages available that are compatible with bioconductor_full,
> so that package installation would sometimes be much faster (a simple
> download, rather than compilation).

I like that idea to some extend. In particular, 
it'll help not having to carry the BUILD dependencies 
in the container as we currently have in e.g. release_metabolomics2.

A less known feature of Docker is to have two FROM in one Dockerfile, 
e
.g. https://github.com/ProteoWizard/container/blob/master/Dockerfile
where the second FROM can COPY from the above-built container, 
thus dropping any cruft. 

But how would the binary packages then be imported into 
the container ? The volumes work fine on the command line. 
What about a kubernetes cluster ? 

I'd like to suggest to avoid any dependency on the 
filesystem of the host, i.e. no 
-v host-site-library:/usr/local/lib/R/host-site-library

An established way to mount stuff into a container (like bioc-full)
is to use sidecar containers. Basically they share some of their 
filesystem with the "main" container (typical example: 
tomcat main container, sidecar with only the *.war file).

A challenge is duplication. If each binary with required /site-lib/content 
carries Biobase, we'd have 1750 copies of that. This would be solved if we can mount 
the required subset (=dependencies) of the 1750 binaries into site/lib/PACKAGE

Yours,
Steffen


> 
> Maritn
> 
> On 7/2/19, 10:52 AM, "Bioc-devel on behalf of Neumann, Steffen" <
> bioc-devel-bounces using r-project.org on behalf of sneumann using ipb-halle.de>
> wrote:
> 
>     Hi,
>     
>     last week over a beer some of us realized that (technically)
>     it is possible to run most[1] Rmd vignettes as Jupiter Notebook. 
>     
>     So I created https://github.com/sneumann/bioc_notebooks 
>     which takes some example packages and creates a Docker container 
>     with converted vignettes. The resulting container 
>     can be hosted locally, or on the binder system (see README).
>     
>     We found this to be fun, and it could be used 
>     as a great training resource. Only downside is 
>     that the resulting containers are not exactly light-weight.
>     
>     Yours,
>     Steffen
>     
>     [1] See the issue in the repo where it fails. 
>     
>     -- 
>     
>     IPB Halle                    AG Massenspektrometrie &
> Bioinformatik
>     Dr. Steffen Neumann          http://www.IPB-Halle.DE
>     Weinberg 3                   Tel. +49 (0) 345 5582 - 1470
>     06120 Halle                       +49 (0) 345 5582 - 0
>     sneumann(at)IPB-Halle.DE     Fax. +49 (0) 345 5582 - 1409
>     _______________________________________________
>     Bioc-devel using r-project.org mailing list
>     https://stat.ethz.ch/mailman/listinfo/bioc-devel
>     
-- 

IPB Halle                    AG Massenspektrometrie & Bioinformatik
Dr. Steffen Neumann          http://www.IPB-Halle.DE
Weinberg 3                   Tel. +49 (0) 345 5582 - 1470
06120 Halle                       +49 (0) 345 5582 - 0
sneumann(at)IPB-Halle.DE     Fax. +49 (0) 345 5582 - 1409