[Bioc-devel] Python module "tensorflow_probability" not found

Kim Philipp Jablonski k|m@ph|||pp@j@b|on@k| @end|ng |rom gm@||@com
Thu Jul 6 23:38:15 CEST 2023 

Thank you all so much for your input and the references!

@Kasper: I mostly rely on tensorflow and tensorflow-probability, so I might
somehow get rid of the keras dependency but it would require some work.

After being inspired by the lovely orthos package (thanks Charlotte!), I
decided to play around further in the basilisk direction and updated my
project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk).
The hardest part was figuring out a set of package versions which satisfy
conda's package manager (@Vincent, I feel you!).
But then it just magically worked on my local machine.

When testing with GitHub Actions, the windows runner crashes with
PackagesNotFoundError: The following packages are not available from
current channels:
  - tensorflow=2.11.1
A look at conda-forge (https://anaconda.org/conda-forge/tensorflow) reveals
that for win-64, there's only v1.14.0 available... I guess ignoring the
windows build is not an option for my bioc package?

For the ubuntu runner, my vignettes were created successfully ("* creating
vignettes ... OK"). My tests still fail, but that is expected because I
have not wrapped them with basiliskRun. Do I have to do this manually for
every function call which may interact with tensorflow (so much
boilerplate), or can I somehow implicitly use the created conda env for
every function in my package?

On Thu, Jul 6, 2023 at 2:08 PM Vincent Carey <stvjc using channing.harvard.edu>
wrote:

> That's great news.  FWIW I am finding that the advice at
> https://rstudio.github.io/reticulate/articles/python_dependencies.html
> can work to produce properly resolved python dependencies.  Just don't
> follow the example literally; the requested
> scipy version may not exist.  Version 1.11.1 does. Stay tuned.
>
> On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson <
> charlottesoneson using gmail.com>
> wrote:
>
> > Hi,
> >
> > in case it's useful: we have a package (orthos) in review (
> > https://github.com/Bioconductor/Contributions/issues/3042) which uses
> > basilisk to set up a conda environment with tensorflow and keras. It
> builds
> > and runs fine both on GitHub Actions (GitHub repo here:
> > https://github.com/fmicompbio/orthos) and on the Single Package Builder.
> > We have also tested (locally) that it will use GPUs if available, and
> that
> > the GPU configuration can be controlled from the R session (outside of
> the
> > package), e.g. by setting the CUDA_VISIBLE_DEVICES environment variable.
> >
> > Charlotte
> >
> > > On 5 Jul 2023, at 23:12, Kasper Daniel Hansen <
> > kasperdanielhansen using gmail.com> wrote:
> > >
> > > So I think Kim is interfacing to tensorflow by using the keras package
> > from
> > > CRAN (partly authored by the Rstudio people). This package leaves it to
> > the
> > > user to install tensorflow, which is a highly non-trivial installation
> > > task. There is some partly helpful instructions for using conda
> together
> > > with reticulate (see the macOS tab on
> > > https://tensorflow.rstudio.com/install/local_gpu.html). This is the
> job
> > > that basilisk handles for you. In essence, basilisk allows the
> developer
> > to
> > > specify an R-package-specific conda. Tensorflow can be run on a CPUor a
> > > GPU. Getting it to run on a user-GPU is extra complicated and I am not
> > sure
> > > basilisk can handle this.
> > >
> > > Going forward, we (Bioc) want to decide if we want to support keras on
> > our
> > > build system. This will require some work, because it is definitely not
> > > trivial to get to work (but much more possible if we limit ourselves to
> > > running on CPU). If we decide to support keras, we should try to figure
> > out
> > > how to wrap keras into a basilisk container; perhaps something like
> > > creating a keras-basilisk R package, because IF we decide to support
> > keras,
> > > this is going to be a major headache (to add to the frustration,
> > tensorflow
> > > often rearranges everything so I future issues keeping it operational).
> > >
> > > For Kim: I think you should consider if there are any alternatives to
> > > keras. Even if we get it to work on our build system, users will have
> > major
> > > headache getting this to work (I think). If there are no alternatives
> to
> > > keras, you should perhaps think about doing the keras-basilisk option I
> > > outline above (assuming that is feasible; I don't know how keras
> > interfaces
> > > with tensorflow. You might also have major headaches in your future: I
> > have
> > > found quite big differences in convergence and optimizers over time in
> TF
> > > and you're basically saying it should work with every version of
> > tensorflow
> > >> = 2.2. That's a .... strong promise considering also the difference
> > > between GPU and CPU.
> > >
> > > In the meantime, in case the core build team wants some input, I have
> > some
> > > experience with tensorflow, although so far my experience is mostly
> > > frustration.
> > >
> > > Best,
> > > Kasper
> > >
> > >
> > > On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey <
> stvjc using channing.harvard.edu
> > >
> > > wrote:
> > >
> > >> I'll try to get clearer on the basilisk situation; I forked your repo
> > >> and will plunge in soon.
> > >> It may take a while.  In the mean time I hope the BBS python stack can
> > >> be looked at
> > >> to see what the issue might be.  @Jennifer Wokaty is it possible with
> > >> the recent R upgrades that
> > >> tensorflow and tensorflow-probability might need to be
> > installed/updated?
> > >>
> > >> On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski
> > >> <kim.philipp.jablonski using gmail.com> wrote:
> > >>>
> > >>> Thanks a lot for your response!
> > >>>
> > >>> The Config/reticulate approach in DESCRIPTION looks very neat. I hope
> > we
> > >> can use it at some point.
> > >>>
> > >>> Could the issue you are facing with the conda install be, that the
> > >> package is called "tensorflow-probability" instead of
> > >> "tensorflow_probability" (
> > >> https://anaconda.org/conda-forge/tensorflow-probability)?
> > >>>
> > >>> I have followed the basilisk advice and incorporated it into my
> package
> > >> here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk.
> > >>> The basilisk docs say "Any R functions that use Python code should do
> > so
> > >> via basiliskRun()".
> > >>> This does not seem possible in my case as I am using
> > >> tensorflow(probability) wrappers instead of calling them directly.
> > >>> Building the vignettes with Github Actions fails because tensorflow
> > >> could not be found:
> > >>
> >
> https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39
> > >> (I might have to activate basilisk somehow, but using basiliskStart
> > seems
> > >> to only make sense with basiliskRun). I could also not find any advice
> > on
> > >> this elsewhere.
> > >>> Do you have a suggestion how to best handle this (or should it *just*
> > >> work)?
> > >>>
> > >>>
> > >>> On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey <
> > stvjc using channing.harvard.edu>
> > >> wrote:
> > >>>>
> > >>>> Sorry to hear about this.  Our informal outlook on python
> > >>>> interoperation in Bioconductor packages is that
> > >>>> the basilisk discipline should be used.  This allows you to pin
> > >>>> specific versions of all python dependencies
> > >>>> and use python for your R functions from an insulated conda
> > environment.
> > >>>>
> > >>>> There is also information about specifying python dependencies in
> the
> > >>>> DESCRIPTION file at
> > >>>>
> > https://rstudio.github.io/reticulate/articles/python_dependencies.html
> > >>>> I do not know if our build
> > >>>> system or R CMD install take effective advantage of that approach at
> > >> this time.
> > >>>>
> > >>>> I was surprised to see that my attempt to use
> reticulate::py_install()
> > >>>> to install tensorflow_probabiliity
> > >>>> on my laptop failed:
> > >>>>
> > >>>> '/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes'
> > >>>> '--prefix' '/home/stvjc/.local/share/r-miniconda/envs/r-reticulate'
> > >>>> '-c' 'conda-forge' 'tensorflow_probability'
> > >>>> Retrieving notices: ...working... done
> > >>>> Collecting package metadata (current_repodata.json): done
> > >>>> Solving environment: unsuccessful initial attempt using frozen
> solve.
> > >>>> Retrying with flexible solve.
> > >>>> Collecting package metadata (repodata.json): done
> > >>>> Solving environment: unsuccessful initial attempt using frozen
> solve.
> > >>>> Retrying with flexible solve.
> > >>>>
> > >>>> PackagesNotFoundError: The following packages are not available from
> > >>>> current channels:
> > >>>>
> > >>>>  - tensorflow_probability
> > >>>>
> > >>>> Current channels:
> > >>>>
> > >>>>  - https://conda.anaconda.org/conda-forge/linux-64
> > >>>>  - https://conda.anaconda.org/conda-forge/noarch
> > >>>>  - https://repo.anaconda.com/pkgs/main/linux-64
> > >>>>  - https://repo.anaconda.com/pkgs/main/noarch
> > >>>>  - https://repo.anaconda.com/pkgs/r/linux-64
> > >>>>  - https://repo.anaconda.com/pkgs/r/noarch
> > >>>>
> > >>>> To search for alternate channels that may provide the conda package
> > >> you're
> > >>>> looking for, navigate to
> > >>>>
> > >>>>    https://anaconda.org
> > >>>>
> > >>>> and use the search bar at the top of the page.
> > >>>>
> > >>>>
> > >>>> Error: one or more Python packages failed to install [error code 1]
> > >>>>
> > >>>> Enter a frame number, or 0 to exit
> > >>>>
> > >>>> 1: py_install("tensorflow_probability")
> > >>>>
> > >>>> So the "current channels" used by reticulate in my pretty stock
> > >>>> installation of R/reticulate seem flawed
> > >>>> for this purpose.  I had hoped to write a bit of code that would
> check
> > >>>> for the desired module and install
> > >>>> it if missing, that you could include in your package.
> > >>>>
> > >>>> get_pippath = function() {
> > >>>>   pypath = reticulate::conda_list() |>
> > >> (\(x)x[x$name=="r-reticulate",])()
> > >>>>   gsub("python$", "pip", pypath$python)
> > >>>> }
> > >>>>
> > >>>> check_tfp = function() {
> > >>>>   pippath = get_pippath()
> > >>>>   peek = system(paste0(pippath, " list | grep tensor"), intern=TRUE)
> > >>>>   peek = gsub(" +", " ", peek)
> > >>>>   tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x)
> > >>>> data.frame(pkg=x[1], version=x[2])))
> > >>>>   "tensorflow_probability" %in% tfdf$pkg
> > >>>> }
> > >>>>
> > >>>> install_tfp = function() {
> > >>>>   pippath = get_pippath()
> > >>>>   chk = system(paste(pippath, "install tensorflow_probability"),
> > >> intern=TRUE)
> > >>>>   chk2 = check_tfp()
> > >>>>   list(sysout=chk, tfp_installed=chk2)
> > >>>> }
> > >>>>
> > >>>> The first two functions seem to do what I want, the latter does not.
> > >>>>
> > >>>> Suggestion -- read the basilisk vignettes, use it if at all
> possible.
> > >>>> Certainly we can take
> > >>>> care of this in our build system when the holiday ends, but it would
> > >>>> be great to have the software
> > >>>> arrange the solution when necessary, and I don't see a way to
> > >>>> accomplish this at the moment.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski
> > >>>> <kim.philipp.jablonski using gmail.com> wrote:
> > >>>>>
> > >>>>> Hello,
> > >>>>>
> > >>>>> The latest builds of my package 'pareg' started failing on BioC
> 3.17
> > >> for
> > >>>>> Linux and macOS but not Windows:
> > >>>>>
> > >>
> >
> https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html
> > >>>>> .
> > >>>>> In both cases, the error messages is "Python module
> > >> tensorflow_probability
> > >>>>> was not found". This wasn't an issue in earlier releases.
> > >>>>>
> > >>>>> Do you know what the reason for this is and how I can fix it?
> > >>>>> Thanks a lot for your help!
> > >>>>>
> > >>>>> Best regards,
> > >>>>> Kim
> > >>>>>
> > >>>>>        [[alternative HTML version deleted]]
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Bioc-devel using r-project.org mailing list
> > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >>>>
> > >>>> --
> > >>>> The information in this e-mail is intended only for the person to
> whom
> > >> it
> > >>>> is
> > >>>> addressed. If you believe this e-mail was sent to you in error and
> the
> > >>>> e-mail
> > >>>> contains patient information, please contact the Partners Compliance
> > >>>> HelpLine at
> > >>>> http://www.partners.org/complianceline
> > >>>> <http://www.partners.org/complianceline> . If the e-mail was sent
> to
> > >> you in
> > >>>> error
> > >>>> but does not contain patient information, please contact the sender
> > >>>> and properly
> > >>>> dispose of the e-mail.
> > >>
> > >> --
> > >> The information in this e-mail is intended only for
> th...{{dropped:27}}
> > >
> > > _______________________________________________
> > > Bioc-devel using r-project.org mailing list
> > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
> >
>
> --
> The information in this e-mail is intended only for th...{{dropped:10}}

More information about the Bioc-devel mailing list