[Bioc-devel] python function in Basilisk never finishes

[Clemens Kohl]

Hello everyone,

I am one of the developers of the package APL 
(https://github.com/VingronLab/APL) for the visualization and analysis 
of single cell transcriptomics data.

To speed up the singular value decomposition of large matrices we use 
torch.svd from pytorch and are trying to simplify the installation of 
the python dependencies through the Bioconductor package basilisk 
(http://bioconductor.org/packages/release/bioc/html/basilisk.html).

Our current implementation with basilisk can be found here: 
https://github.com/VingronLab/APL/tree/dev

However we run into a problem we are so far unable to figure out, and my 
hope is that someone with more experience in using basilisk could point 
us into the right direction.

When running run_cacomp(matrix, python = TRUE)  (the function that 
ultimately calls torch.svd within basiliskRun) it finishes for very 
small matrices (an example for such a small dataset can be found in 
/tests/testthat/testdata/countries.rda), but for larger datasets it 
keeps running without any errors while continuously using up CPU 
resources. An example that does not finish can be found when running 
run_cacomp on the darmanis dataset in our vignette. When we previously 
only used reticulate we did not observe such a behaviour.

I assume this is due to some behaviour of basilisk that I do not 
understand yet. Is there a parameter I can set that would prevent this 
from happening?

A less urgent problem is that programs that run within the basilisk 
conda environment seem to sometimes use shared system libraries instead 
of using the libraries installed in the conda environment. On one of our 
test machines it failed because it used an older gcc library outside the 
conda environment. When explicitly instructed to preferentially use the 
conda libraries by setting

export LD_LIBRARY_PATH=||/path/to/temp/basilisk/conda/environment/

we can make it use the correct libraries, but this would not be a 
solution suitable for most users.


Many thanks in advance!


Best regards,

Clemens Kohl


	[[alternative HTML version deleted]]