[Rd] [External] svd For Large Matrix

[Dirk Eddelbuettel]

Dario,

On 14 August 2021 at 00:00, Dario Strbenac via R-devel wrote:
| Good day,
| 
| Ah, I was confident it wouldn't be environment-specific but it is. My environment is
| 
| R version 4.1.0 (2021-05-18)
| Platform: x86_64-pc-linux-gnu (64-bit)
| Running under: Debian GNU/Linux 10 (buster)
| 
| Matrix products: default
| BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
| LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3
| 
| It crashes at about 180 GB RAM usage. The server has 1024 GB physical RAM in it. Modestly downsampling to 30 million cells avoids the segmentation fault. The segmentation fault originates from BLAS
| 
| Program received signal SIGSEGV, Segmentation fault.
| 0x00007ffff7649c10 in ATL_dgecopy () from /usr/lib/x86_64-linux-gnu/libblas.so.3

This would allow to do what was suggested: trying different BLAS. On Debian
and alike you can just install Atlas (as you have), or OpenBLAS, or the
reference BLAS (and there my script and post from a few years ago to use MKL
but depending on which release you use may actually be directly accessible),
or now also BLIS, or ...

In short, we see a bug when using Atlas. I would at least try OpenBLAS.

Here is what I see in a Docker container using testing/unstable -- you will
see a shorter list but you *will* have the three different openblas versions
at a minimum.

root using somedocker:~# apt-cache search libblas | grep -- -dev
libatlas-base-dev - Automatically Tuned Linear Algebra Software, generic static
libblis-openmp-dev - BLAS-like Library Instantiation Software Framework (dev,32bit,openmp)
libblis-pthread-dev - BLAS-like Library Instantiation Software Framework (dev,32bit,pthread)
libblis-serial-dev - BLAS-like Library Instantiation Software Framework (dev,32bit,serial)
libblis64-openmp-dev - BLAS-like Library Instantiation Software Framework (dev,64bit,openmp)
libblis64-pthread-dev - BLAS-like Library Instantiation Software Framework (dev,64bit,pthread)
libblis64-serial-dev - BLAS-like Library Instantiation Software Framework (dev,64bit,serial)
libblas-dev - Basic Linear Algebra Subroutines 3, static library
libblas64-dev - Basic Linear Algebra Subroutines 3, static library (64bit-index)
libopenblas-openmp-dev - Optimized BLAS (linear algebra) library (dev, openmp)
libopenblas-pthread-dev - Optimized BLAS (linear algebra) library (dev, pthread)
libopenblas-serial-dev - Optimized BLAS (linear algebra) library (dev, serial)
libopenblas64-openmp-dev - Optimized BLAS (linear algebra) library (dev, 64bit, openmp)
libopenblas64-pthread-dev - Optimized BLAS (linear algebra) library (dev, 64bit, pthread)
libopenblas64-serial-dev - Optimized BLAS (linear algebra) library (dev, 64bit, serial)
libblasr-dev - tools for aligning PacBio reads to target sequences (development files)
root using somedocker:~# 

Dirk

-- 
https://dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org