[Bioc-devel] Python module "tensorflow_probability" not found

Kim Philipp Jablonski k|m@ph|||pp@j@b|on@k| @end|ng |rom gm@||@com
Sat Jul 8 11:19:14 CEST 2023 

Thanks! I now only used the anaconda channel to install both tensorflow and
tensorflow-probability.
This works fine for Linux and Windows, but now macOS crashes with "Intel
MKL FATAL ERROR: Cannot load libmkl_intel_thread.1.dylib" (
https://github.com/cbg-ethz/pareg/actions/runs/5489390187/jobs/10003486720#step:14:1034)
when building the environment. There are multiple suggested fixes, and I
went with adding the "nomkl" package to the conda env.

I have now reached an interesting situation: the windows checks pass (
https://github.com/cbg-ethz/pareg/actions/runs/5493258256/jobs/10011315040),
but they time out for Linux and macOS (
https://github.com/cbg-ethz/pareg/actions/runs/5489978270/jobs/10004818116#step:20:181)
after over 5 hours of running tests.

I have nonetheless pushed my changes to the devel branch on bioc'
repository and hope for the best.

On Fri, Jul 7, 2023 at 5:23 AM Kasper Daniel Hansen <
kasperdanielhansen using gmail.com> wrote:

> This sounds excellent Kim!
>
> Here you can get 2.10 for Windows:
> https://anaconda.org/anaconda/tensorflow although my experience is that I
> hate mixing channels on conda. It is also quite interesting that this conda
> package also has Windows at an older version (but just 2.10 vs. 2.12)
>
> This really speaks to a potential need for having basilisk dependencies
> being platform specific. This would also come in handy for macOS. But
> AFAIK, this is not supported by basilisk currently. Might be something we
> need to address.
>
> Best,
> Kasper
>
>
>
> On Thu, Jul 6, 2023 at 5:38 PM Kim Philipp Jablonski <
> kim.philipp.jablonski using gmail.com> wrote:
>
>> Thank you all so much for your input and the references!
>>
>> @Kasper: I mostly rely on tensorflow and tensorflow-probability, so I
>> might
>> somehow get rid of the keras dependency but it would require some work.
>>
>> After being inspired by the lovely orthos package (thanks Charlotte!), I
>> decided to play around further in the basilisk direction and updated my
>> project (https://github.com/cbg-ethz/pareg/tree/feature-basilisk).
>> The hardest part was figuring out a set of package versions which satisfy
>> conda's package manager (@Vincent, I feel you!).
>> But then it just magically worked on my local machine.
>>
>> When testing with GitHub Actions, the windows runner crashes with
>> PackagesNotFoundError: The following packages are not available from
>> current channels:
>>   - tensorflow=2.11.1
>> A look at conda-forge (https://anaconda.org/conda-forge/tensorflow)
>> reveals
>> that for win-64, there's only v1.14.0 available... I guess ignoring the
>> windows build is not an option for my bioc package?
>>
>> For the ubuntu runner, my vignettes were created successfully ("* creating
>> vignettes ... OK"). My tests still fail, but that is expected because I
>> have not wrapped them with basiliskRun. Do I have to do this manually for
>> every function call which may interact with tensorflow (so much
>> boilerplate), or can I somehow implicitly use the created conda env for
>> every function in my package?
>>
>> On Thu, Jul 6, 2023 at 2:08 PM Vincent Carey <stvjc using channing.harvard.edu>
>> wrote:
>>
>> > That's great news.  FWIW I am finding that the advice at
>> > https://rstudio.github.io/reticulate/articles/python_dependencies.html
>> > can work to produce properly resolved python dependencies.  Just don't
>> > follow the example literally; the requested
>> > scipy version may not exist.  Version 1.11.1 does. Stay tuned.
>> >
>> > On Thu, Jul 6, 2023 at 7:43 AM Charlotte Soneson <
>> > charlottesoneson using gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > in case it's useful: we have a package (orthos) in review (
>> > > https://github.com/Bioconductor/Contributions/issues/3042) which uses
>> > > basilisk to set up a conda environment with tensorflow and keras. It
>> > builds
>> > > and runs fine both on GitHub Actions (GitHub repo here:
>> > > https://github.com/fmicompbio/orthos) and on the Single Package
>> Builder.
>> > > We have also tested (locally) that it will use GPUs if available, and
>> > that
>> > > the GPU configuration can be controlled from the R session (outside of
>> > the
>> > > package), e.g. by setting the CUDA_VISIBLE_DEVICES environment
>> variable.
>> > >
>> > > Charlotte
>> > >
>> > > > On 5 Jul 2023, at 23:12, Kasper Daniel Hansen <
>> > > kasperdanielhansen using gmail.com> wrote:
>> > > >
>> > > > So I think Kim is interfacing to tensorflow by using the keras
>> package
>> > > from
>> > > > CRAN (partly authored by the Rstudio people). This package leaves
>> it to
>> > > the
>> > > > user to install tensorflow, which is a highly non-trivial
>> installation
>> > > > task. There is some partly helpful instructions for using conda
>> > together
>> > > > with reticulate (see the macOS tab on
>> > > > https://tensorflow.rstudio.com/install/local_gpu.html). This is the
>> > job
>> > > > that basilisk handles for you. In essence, basilisk allows the
>> > developer
>> > > to
>> > > > specify an R-package-specific conda. Tensorflow can be run on a
>> CPUor a
>> > > > GPU. Getting it to run on a user-GPU is extra complicated and I am
>> not
>> > > sure
>> > > > basilisk can handle this.
>> > > >
>> > > > Going forward, we (Bioc) want to decide if we want to support keras
>> on
>> > > our
>> > > > build system. This will require some work, because it is definitely
>> not
>> > > > trivial to get to work (but much more possible if we limit
>> ourselves to
>> > > > running on CPU). If we decide to support keras, we should try to
>> figure
>> > > out
>> > > > how to wrap keras into a basilisk container; perhaps something like
>> > > > creating a keras-basilisk R package, because IF we decide to support
>> > > keras,
>> > > > this is going to be a major headache (to add to the frustration,
>> > > tensorflow
>> > > > often rearranges everything so I future issues keeping it
>> operational).
>> > > >
>> > > > For Kim: I think you should consider if there are any alternatives
>> to
>> > > > keras. Even if we get it to work on our build system, users will
>> have
>> > > major
>> > > > headache getting this to work (I think). If there are no
>> alternatives
>> > to
>> > > > keras, you should perhaps think about doing the keras-basilisk
>> option I
>> > > > outline above (assuming that is feasible; I don't know how keras
>> > > interfaces
>> > > > with tensorflow. You might also have major headaches in your
>> future: I
>> > > have
>> > > > found quite big differences in convergence and optimizers over time
>> in
>> > TF
>> > > > and you're basically saying it should work with every version of
>> > > tensorflow
>> > > >> = 2.2. That's a .... strong promise considering also the difference
>> > > > between GPU and CPU.
>> > > >
>> > > > In the meantime, in case the core build team wants some input, I
>> have
>> > > some
>> > > > experience with tensorflow, although so far my experience is mostly
>> > > > frustration.
>> > > >
>> > > > Best,
>> > > > Kasper
>> > > >
>> > > >
>> > > > On Wed, Jul 5, 2023 at 3:12 PM Vincent Carey <
>> > stvjc using channing.harvard.edu
>> > > >
>> > > > wrote:
>> > > >
>> > > >> I'll try to get clearer on the basilisk situation; I forked your
>> repo
>> > > >> and will plunge in soon.
>> > > >> It may take a while.  In the mean time I hope the BBS python stack
>> can
>> > > >> be looked at
>> > > >> to see what the issue might be.  @Jennifer Wokaty is it possible
>> with
>> > > >> the recent R upgrades that
>> > > >> tensorflow and tensorflow-probability might need to be
>> > > installed/updated?
>> > > >>
>> > > >> On Wed, Jul 5, 2023 at 3:01 PM Kim Philipp Jablonski
>> > > >> <kim.philipp.jablonski using gmail.com> wrote:
>> > > >>>
>> > > >>> Thanks a lot for your response!
>> > > >>>
>> > > >>> The Config/reticulate approach in DESCRIPTION looks very neat. I
>> hope
>> > > we
>> > > >> can use it at some point.
>> > > >>>
>> > > >>> Could the issue you are facing with the conda install be, that the
>> > > >> package is called "tensorflow-probability" instead of
>> > > >> "tensorflow_probability" (
>> > > >> https://anaconda.org/conda-forge/tensorflow-probability)?
>> > > >>>
>> > > >>> I have followed the basilisk advice and incorporated it into my
>> > package
>> > > >> here: https://github.com/cbg-ethz/pareg/tree/feature-basilisk.
>> > > >>> The basilisk docs say "Any R functions that use Python code
>> should do
>> > > so
>> > > >> via basiliskRun()".
>> > > >>> This does not seem possible in my case as I am using
>> > > >> tensorflow(probability) wrappers instead of calling them directly.
>> > > >>> Building the vignettes with Github Actions fails because
>> tensorflow
>> > > >> could not be found:
>> > > >>
>> > >
>> >
>> https://github.com/cbg-ethz/pareg/actions/runs/5466980003/jobs/9952690137#step:15:39
>> > > >> (I might have to activate basilisk somehow, but using basiliskStart
>> > > seems
>> > > >> to only make sense with basiliskRun). I could also not find any
>> advice
>> > > on
>> > > >> this elsewhere.
>> > > >>> Do you have a suggestion how to best handle this (or should it
>> *just*
>> > > >> work)?
>> > > >>>
>> > > >>>
>> > > >>> On Tue, Jul 4, 2023 at 1:04 PM Vincent Carey <
>> > > stvjc using channing.harvard.edu>
>> > > >> wrote:
>> > > >>>>
>> > > >>>> Sorry to hear about this.  Our informal outlook on python
>> > > >>>> interoperation in Bioconductor packages is that
>> > > >>>> the basilisk discipline should be used.  This allows you to pin
>> > > >>>> specific versions of all python dependencies
>> > > >>>> and use python for your R functions from an insulated conda
>> > > environment.
>> > > >>>>
>> > > >>>> There is also information about specifying python dependencies in
>> > the
>> > > >>>> DESCRIPTION file at
>> > > >>>>
>> > >
>> https://rstudio.github.io/reticulate/articles/python_dependencies.html
>> > > >>>> I do not know if our build
>> > > >>>> system or R CMD install take effective advantage of that
>> approach at
>> > > >> this time.
>> > > >>>>
>> > > >>>> I was surprised to see that my attempt to use
>> > reticulate::py_install()
>> > > >>>> to install tensorflow_probabiliity
>> > > >>>> on my laptop failed:
>> > > >>>>
>> > > >>>> '/home/stvjc/.local/share/r-miniconda/bin/conda' 'install'
>> '--yes'
>> > > >>>> '--prefix'
>> '/home/stvjc/.local/share/r-miniconda/envs/r-reticulate'
>> > > >>>> '-c' 'conda-forge' 'tensorflow_probability'
>> > > >>>> Retrieving notices: ...working... done
>> > > >>>> Collecting package metadata (current_repodata.json): done
>> > > >>>> Solving environment: unsuccessful initial attempt using frozen
>> > solve.
>> > > >>>> Retrying with flexible solve.
>> > > >>>> Collecting package metadata (repodata.json): done
>> > > >>>> Solving environment: unsuccessful initial attempt using frozen
>> > solve.
>> > > >>>> Retrying with flexible solve.
>> > > >>>>
>> > > >>>> PackagesNotFoundError: The following packages are not available
>> from
>> > > >>>> current channels:
>> > > >>>>
>> > > >>>>  - tensorflow_probability
>> > > >>>>
>> > > >>>> Current channels:
>> > > >>>>
>> > > >>>>  - https://conda.anaconda.org/conda-forge/linux-64
>> > > >>>>  - https://conda.anaconda.org/conda-forge/noarch
>> > > >>>>  - https://repo.anaconda.com/pkgs/main/linux-64
>> > > >>>>  - https://repo.anaconda.com/pkgs/main/noarch
>> > > >>>>  - https://repo.anaconda.com/pkgs/r/linux-64
>> > > >>>>  - https://repo.anaconda.com/pkgs/r/noarch
>> > > >>>>
>> > > >>>> To search for alternate channels that may provide the conda
>> package
>> > > >> you're
>> > > >>>> looking for, navigate to
>> > > >>>>
>> > > >>>>    https://anaconda.org
>> > > >>>>
>> > > >>>> and use the search bar at the top of the page.
>> > > >>>>
>> > > >>>>
>> > > >>>> Error: one or more Python packages failed to install [error code
>> 1]
>> > > >>>>
>> > > >>>> Enter a frame number, or 0 to exit
>> > > >>>>
>> > > >>>> 1: py_install("tensorflow_probability")
>> > > >>>>
>> > > >>>> So the "current channels" used by reticulate in my pretty stock
>> > > >>>> installation of R/reticulate seem flawed
>> > > >>>> for this purpose.  I had hoped to write a bit of code that would
>> > check
>> > > >>>> for the desired module and install
>> > > >>>> it if missing, that you could include in your package.
>> > > >>>>
>> > > >>>> get_pippath = function() {
>> > > >>>>   pypath = reticulate::conda_list() |>
>> > > >> (\(x)x[x$name=="r-reticulate",])()
>> > > >>>>   gsub("python$", "pip", pypath$python)
>> > > >>>> }
>> > > >>>>
>> > > >>>> check_tfp = function() {
>> > > >>>>   pippath = get_pippath()
>> > > >>>>   peek = system(paste0(pippath, " list | grep tensor"),
>> intern=TRUE)
>> > > >>>>   peek = gsub(" +", " ", peek)
>> > > >>>>   tfdf = do.call(rbind, lapply(strsplit(peek, " "), function(x)
>> > > >>>> data.frame(pkg=x[1], version=x[2])))
>> > > >>>>   "tensorflow_probability" %in% tfdf$pkg
>> > > >>>> }
>> > > >>>>
>> > > >>>> install_tfp = function() {
>> > > >>>>   pippath = get_pippath()
>> > > >>>>   chk = system(paste(pippath, "install tensorflow_probability"),
>> > > >> intern=TRUE)
>> > > >>>>   chk2 = check_tfp()
>> > > >>>>   list(sysout=chk, tfp_installed=chk2)
>> > > >>>> }
>> > > >>>>
>> > > >>>> The first two functions seem to do what I want, the latter does
>> not.
>> > > >>>>
>> > > >>>> Suggestion -- read the basilisk vignettes, use it if at all
>> > possible.
>> > > >>>> Certainly we can take
>> > > >>>> care of this in our build system when the holiday ends, but it
>> would
>> > > >>>> be great to have the software
>> > > >>>> arrange the solution when necessary, and I don't see a way to
>> > > >>>> accomplish this at the moment.
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> On Tue, Jul 4, 2023 at 5:32 AM Kim Philipp Jablonski
>> > > >>>> <kim.philipp.jablonski using gmail.com> wrote:
>> > > >>>>>
>> > > >>>>> Hello,
>> > > >>>>>
>> > > >>>>> The latest builds of my package 'pareg' started failing on BioC
>> > 3.17
>> > > >> for
>> > > >>>>> Linux and macOS but not Windows:
>> > > >>>>>
>> > > >>
>> > >
>> >
>> https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pareg/nebbiolo1-buildsrc.html
>> > > >>>>> .
>> > > >>>>> In both cases, the error messages is "Python module
>> > > >> tensorflow_probability
>> > > >>>>> was not found". This wasn't an issue in earlier releases.
>> > > >>>>>
>> > > >>>>> Do you know what the reason for this is and how I can fix it?
>> > > >>>>> Thanks a lot for your help!
>> > > >>>>>
>> > > >>>>> Best regards,
>> > > >>>>> Kim
>> > > >>>>>
>> > > >>>>>        [[alternative HTML version deleted]]
>> > > >>>>>
>> > > >>>>> _______________________________________________
>> > > >>>>> Bioc-devel using r-project.org mailing list
>> > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> > > >>>>
>> > > >>>> --
>> > > >>>> The information in this e-mail is intended only for the person to
>> > whom
>> > > >> it
>> > > >>>> is
>> > > >>>> addressed. If you believe this e-mail was sent to you in error
>> and
>> > the
>> > > >>>> e-mail
>> > > >>>> contains patient information, please contact the Partners
>> Compliance
>> > > >>>> HelpLine at
>> > > >>>> http://www.partners.org/complianceline
>> > > >>>> <http://www.partners.org/complianceline> . If the e-mail was
>> sent
>> > to
>> > > >> you in
>> > > >>>> error
>> > > >>>> but does not contain patient information, please contact the
>> sender
>> > > >>>> and properly
>> > > >>>> dispose of the e-mail.
>> > > >>
>> > > >> --
>> > > >> The information in this e-mail is intended only for
>> > th...{{dropped:27}}
>> > > >
>> > > > _______________________________________________
>> > > > Bioc-devel using r-project.org mailing list
>> > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> > >
>> > >
>> >
>> > --
>> > The information in this e-mail is intended only for th...{{dropped:10}}
>>
>> _______________________________________________
>> Bioc-devel using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
>
> --
> Best,
> Kasper
>

	[[alternative HTML version deleted]]

More information about the Bioc-devel mailing list