[Bioc-devel] Python module "tensorflow_probability" not found

Kasper Daniel Hansen k@@perd@n|e|h@n@en @end|ng |rom gm@||@com
Fri Jul 7 05:23:39 CEST 2023 

This sounds excellent Kim!

Here you can get 2.10 for Windows: https://anaconda.org/anaconda/tensorflow
although my experience is that I hate mixing channels on conda. It is also
quite interesting that this conda package also has Windows at an older
version (but just 2.10 vs. 2.12)

This really speaks to a potential need for having basilisk dependencies
being platform specific. This would also come in handy for macOS. But
AFAIK, this is not supported by basilisk currently. Might be something we
need to address.

Best,
Kasper



On Thu, Jul 6, 2023 at 5:38 PM Kim Philipp Jablonski <
kim.philipp.jablonski using gmail.com> wrote:

> Thank you all so much for your input and the references!
>
> @Kasper: I mostly rely on tensorflow and tensorflow-probability, so I might
> somehow get rid of the keras dependency but it would require some work.
>
> After being inspired by the lovely orthos package (thanks Charlotte!), I
> decided to play around further in the basilisk direction and updated my
> project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk).
> The hardest part was figuring out a set of package versions which satisfy
> conda's package manager (@Vincent, I feel you!).
> But then it just magically worked on my local machine.
>
> When testing with GitHub Actions, the windows runner crashes with
> PackagesNotFoundError: The following packages are not available from
> current channels:
>   - tensorflow=2.11.1
> A look at conda-forge (https://anaconda.org/conda-forge/tensorflow)
> reveals
> that for win-64, there's only v1.14.0 available... I guess ignoring the
> windows build is not an option for my bioc package?
>
> For the ubuntu runner, my vignettes were created successfully ("* creating
> vignettes ... OK"). My tests still fail, but that is expected because I
> have not wrapped them with basiliskRun. Do I have to do this manually for
> every function call which may interact with tensorflow (so much
> boilerplate), or can I somehow implicitly use the created conda env for
> every function in my package?
>
> On Thu, Jul 6, 2023 at 2:08 PM Vincent Carey <stvjc using channing.harvard.edu>
> wrote:
>
> > That's great news.  FWIW I am finding that the advice at
> > https://rstudio.github.io/reticulate/articles/python_dependencies.html
> > can work to produce properly resolved python dependencies.  Just don't
> > follow the example literally; the requested
> > scipy version may not exist.  Version 1.11.1 does. Stay tuned.
> >
> > On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson <
> > charlottesoneson using gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > in case it's useful: we have a package (orthos) in review (
> > > https://github.com/Bioconductor/Contributions/issues/3042) which uses
> > > basilisk to set up a conda environment with tensorflow and keras. It
> > builds
> > > and runs fine both on GitHub Actions (GitHub repo here:
> > > https://github.com/fmicompbio/orthos) and on the Single Package
> Builder.
> > > We have also tested (locally) that it will use GPUs if available, and
> > that
> > > the GPU configuration can be controlled from the R session (outside of
> > the
> > > package), e.g. by setting the CUDA_VISIBLE_DEVICES environment
> variable.
> > >
> > > Charlotte
> > >
> > > > On 5 Jul 2023, at 23:12, Kasper Daniel Hansen <
> > > kasperdanielhansen using gmail.com> wrote:
> > > >
> > > > So I think Kim is interfacing to tensorflow by using the keras
> package
> > > from
> > > > CRAN (partly authored by the Rstudio people). This package leaves it
> to
> > > the
> > > > user to install tensorflow, which is a highly non-trivial
> installation
> > > > task. There is some partly helpful instructions for using conda
> > together
> > > > with reticulate (see the macOS tab on
> > > > https://tensorflow.rstudio.com/install/local_gpu.html). This is the
> > job
> > > > that basilisk handles for you. In essence, basilisk allows the
> > developer
> > > to
> > > > specify an R-package-specific conda. Tensorflow can be run on a
> CPUor a
> > > > GPU. Getting it to run on a user-GPU is extra complicated and I am
> not
> > > sure
> > > > basilisk can handle this.
> > > >
> > > > Going forward, we (Bioc) want to decide if we want to support keras
> on
> > > our
> > > > build system. This will require some work, because it is definitely
> not
> > > > trivial to get to work (but much more possible if we limit ourselves
> to
> > > > running on CPU). If we decide to support keras, we should try to
> figure
> > > out
> > > > how to wrap keras into a basilisk container; perhaps something like
> > > > creating a keras-basilisk R package, because IF we decide to support
> > > keras,
> > > > this is going to be a major headache (to add to the frustration,
> > > tensorflow
> > > > often rearranges everything so I future issues keeping it
> operational).
> > > >
> > > > For Kim: I think you should consider if there are any alternatives to
> > > > keras. Even if we get it to work on our build system, users will have
> > > major
> > > > headache getting this to work (I think). If there are no alternatives
> > to
> > > > keras, you should perhaps think about doing the keras-basilisk
> option I
> > > > outline above (assuming that is feasible; I don't know how keras
> > > interfaces
> > > > with tensorflow. You might also have major headaches in your future:
> I
> > > have
> > > > found quite big differences in convergence and optimizers over time
> in
> > TF
> > > > and you're basically saying it should work with every version of
> > > tensorflow
> > > >> = 2.2. That's a .... strong promise considering also the difference
> > > > between GPU and CPU.
> > > >
> > > > In the meantime, in case the core build team wants some input, I have
> > > some
> > > > experience with tensorflow, although so far my experience is mostly
> > > > frustration.
> > > >
> > > > Best,
> > > > Kasper
> > > >
> > > >
> > > > On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey <
> > stvjc using channing.harvard.edu
> > > >
> > > > wrote:
> > > >
> > > >> I'll try to get clearer on the basilisk situation; I forked your
> repo
> > > >> and will plunge in soon.
> > > >> It may take a while.  In the mean time I hope the BBS python stack
> can
> > > >> be looked at
> > > >> to see what the issue might be.  @Jennifer Wokaty is it possible
> with
> > > >> the recent R upgrades that
> > > >> tensorflow and tensorflow-probability might need to be
> > > installed/updated?
> > > >>
> > > >> On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski
> > > >> <kim.philipp.jablonski using gmail.com> wrote:
> > > >>>
> > > >>> Thanks a lot for your response!
> > > >>>
> > > >>> The Config/reticulate approach in DESCRIPTION looks very neat. I
> hope
> > > we
> > > >> can use it at some point.
> > > >>>
> > > >>> Could the issue you are facing with the conda install be, that the
> > > >> package is called "tensorflow-probability" instead of
> > > >> "tensorflow_probability" (
> > > >> https://anaconda.org/conda-forge/tensorflow-probability)?
> > > >>>
> > > >>> I have followed the basilisk advice and incorporated it into my
> > package
> > > >> here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk.
> > > >>> The basilisk docs say "Any R functions that use Python code should
> do
> > > so
> > > >> via basiliskRun()".
> > > >>> This does not seem possible in my case as I am using
> > > >> tensorflow(probability) wrappers instead of calling them directly.
> > > >>> Building the vignettes with Github Actions fails because tensorflow
> > > >> could not be found:
> > > >>
> > >
> >
> https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39
> > > >> (I might have to activate basilisk somehow, but using basiliskStart
> > > seems
> > > >> to only make sense with basiliskRun). I could also not find any
> advice
> > > on
> > > >> this elsewhere.
> > > >>> Do you have a suggestion how to best handle this (or should it
> *just*
> > > >> work)?
> > > >>>
> > > >>>
> > > >>> On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey <
> > > stvjc using channing.harvard.edu>
> > > >> wrote:
> > > >>>>
> > > >>>> Sorry to hear about this.  Our informal outlook on python
> > > >>>> interoperation in Bioconductor packages is that
> > > >>>> the basilisk discipline should be used.  This allows you to pin
> > > >>>> specific versions of all python dependencies
> > > >>>> and use python for your R functions from an insulated conda
> > > environment.
> > > >>>>
> > > >>>> There is also information about specifying python dependencies in
> > the
> > > >>>> DESCRIPTION file at
> > > >>>>
> > > https://rstudio.github.io/reticulate/articles/python_dependencies.html
> > > >>>> I do not know if our build
> > > >>>> system or R CMD install take effective advantage of that approach
> at
> > > >> this time.
> > > >>>>
> > > >>>> I was surprised to see that my attempt to use
> > reticulate::py_install()
> > > >>>> to install tensorflow_probabiliity
> > > >>>> on my laptop failed:
> > > >>>>
> > > >>>> '/home/stvjc/.local/share/r-miniconda/bin/conda' 'install' '--yes'
> > > >>>> '--prefix'
> '/home/stvjc/.local/share/r-miniconda/envs/r-reticulate'
> > > >>>> '-c' 'conda-forge' 'tensorflow_probability'
> > > >>>> Retrieving notices: ...working... done
> > > >>>> Collecting package metadata (current_repodata.json): done
> > > >>>> Solving environment: unsuccessful initial attempt using frozen
> > solve.
> > > >>>> Retrying with flexible solve.
> > > >>>> Collecting package metadata (repodata.json): done
> > > >>>> Solving environment: unsuccessful initial attempt using frozen
> > solve.
> > > >>>> Retrying with flexible solve.
> > > >>>>
> > > >>>> PackagesNotFoundError: The following packages are not available
> from
> > > >>>> current channels:
> > > >>>>
> > > >>>>  - tensorflow_probability
> > > >>>>
> > > >>>> Current channels:
> > > >>>>
> > > >>>>  - https://conda.anaconda.org/conda-forge/linux-64
> > > >>>>  - https://conda.anaconda.org/conda-forge/noarch
> > > >>>>  - https://repo.anaconda.com/pkgs/main/linux-64
> > > >>>>  - https://repo.anaconda.com/pkgs/main/noarch
> > > >>>>  - https://repo.anaconda.com/pkgs/r/linux-64
> > > >>>>  - https://repo.anaconda.com/pkgs/r/noarch
> > > >>>>
> > > >>>> To search for alternate channels that may provide the conda
> package
> > > >> you're
> > > >>>> looking for, navigate to
> > > >>>>
> > > >>>>    https://anaconda.org
> > > >>>>
> > > >>>> and use the search bar at the top of the page.
> > > >>>>
> > > >>>>
> > > >>>> Error: one or more Python packages failed to install [error code
> 1]
> > > >>>>
> > > >>>> Enter a frame number, or 0 to exit
> > > >>>>
> > > >>>> 1: py_install("tensorflow_probability")
> > > >>>>
> > > >>>> So the "current channels" used by reticulate in my pretty stock
> > > >>>> installation of R/reticulate seem flawed
> > > >>>> for this purpose.  I had hoped to write a bit of code that would
> > check
> > > >>>> for the desired module and install
> > > >>>> it if missing, that you could include in your package.
> > > >>>>
> > > >>>> get_pippath = function() {
> > > >>>>   pypath = reticulate::conda_list() |>
> > > >> (\(x)x[x$name=="r-reticulate",])()
> > > >>>>   gsub("python$", "pip", pypath$python)
> > > >>>> }
> > > >>>>
> > > >>>> check_tfp = function() {
> > > >>>>   pippath = get_pippath()
> > > >>>>   peek = system(paste0(pippath, " list | grep tensor"),
> intern=TRUE)
> > > >>>>   peek = gsub(" +", " ", peek)
> > > >>>>   tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x)
> > > >>>> data.frame(pkg=x[1], version=x[2])))
> > > >>>>   "tensorflow_probability" %in% tfdf$pkg
> > > >>>> }
> > > >>>>
> > > >>>> install_tfp = function() {
> > > >>>>   pippath = get_pippath()
> > > >>>>   chk = system(paste(pippath, "install tensorflow_probability"),
> > > >> intern=TRUE)
> > > >>>>   chk2 = check_tfp()
> > > >>>>   list(sysout=chk, tfp_installed=chk2)
> > > >>>> }
> > > >>>>
> > > >>>> The first two functions seem to do what I want, the latter does
> not.
> > > >>>>
> > > >>>> Suggestion -- read the basilisk vignettes, use it if at all
> > possible.
> > > >>>> Certainly we can take
> > > >>>> care of this in our build system when the holiday ends, but it
> would
> > > >>>> be great to have the software
> > > >>>> arrange the solution when necessary, and I don't see a way to
> > > >>>> accomplish this at the moment.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski
> > > >>>> <kim.philipp.jablonski using gmail.com> wrote:
> > > >>>>>
> > > >>>>> Hello,
> > > >>>>>
> > > >>>>> The latest builds of my package 'pareg' started failing on BioC
> > 3.17
> > > >> for
> > > >>>>> Linux and macOS but not Windows:
> > > >>>>>
> > > >>
> > >
> >
> https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html
> > > >>>>> .
> > > >>>>> In both cases, the error messages is "Python module
> > > >> tensorflow_probability
> > > >>>>> was not found". This wasn't an issue in earlier releases.
> > > >>>>>
> > > >>>>> Do you know what the reason for this is and how I can fix it?
> > > >>>>> Thanks a lot for your help!
> > > >>>>>
> > > >>>>> Best regards,
> > > >>>>> Kim
> > > >>>>>
> > > >>>>>        [[alternative HTML version deleted]]
> > > >>>>>
> > > >>>>> _______________________________________________
> > > >>>>> Bioc-devel using r-project.org mailing list
> > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > > >>>>
> > > >>>> --
> > > >>>> The information in this e-mail is intended only for the person to
> > whom
> > > >> it
> > > >>>> is
> > > >>>> addressed. If you believe this e-mail was sent to you in error and
> > the
> > > >>>> e-mail
> > > >>>> contains patient information, please contact the Partners
> Compliance
> > > >>>> HelpLine at
> > > >>>> http://www.partners.org/complianceline
> > > >>>> <http://www.partners.org/complianceline> . If the e-mail was sent
> > to
> > > >> you in
> > > >>>> error
> > > >>>> but does not contain patient information, please contact the
> sender
> > > >>>> and properly
> > > >>>> dispose of the e-mail.
> > > >>
> > > >> --
> > > >> The information in this e-mail is intended only for
> > th...{{dropped:27}}
> > > >
> > > > _______________________________________________
> > > > Bioc-devel using r-project.org mailing list
> > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >
> > >
> >
> > --
> > The information in this e-mail is intended only for th...{{dropped:10}}
>
> _______________________________________________
> Bioc-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>


-- 
Best,
Kasper

	[[alternative HTML version deleted]]

More information about the Bioc-devel mailing list