[R-sig-Debian] custom built R will not change BLAS/LAPACK with update-alternatives

Ramon Diaz-Uriarte rd|@z02 @end|ng |rom gm@||@com
Sat Dec 30 19:51:58 CET 2023


Thanks. I answer below.

And now I realize I must have read this not paying enough attention: "You can specify a particular BLAS library via a value for the configuration option --with-blas. If this is given with no =, its value is taken from the environment variable BLAS_LIBS, set for example in config.site. If neither the option nor the environment variable supply a value, a search is made for a suitable59 BLAS." (https://cran.r-project.org/doc/manuals/r-devel/R-admin.html#BLAS ), and footnote 59 says: "The search order is currently OpenBLAS, BLIS, ATLAS, platform-specific choices (see below) and finally a generic libblas." It is in the configure script of R: line 46396 searches for OpenBLAS, on 46453 it looks for BLIS, later (46510) for ATLAS, etc.

Indeed, that is what I see looking more carefully at the output.  If I have OpenBLAS, then it locates it immediately:

configure: searching for an external BLAS
configure: searching for BLAS in default libraries
checking for dgemm_... no
configure: searching for OpenBLAS
checking for sgemm_ in -lopenblas... yes
checking whether double complex BLAS can be used... yes


If I don't, but I  have ATLAS, it does this (OpenBLAS, then BLIS, then ATLAS)

configure: searching for an external BLAS
configure: searching for BLAS in default libraries
checking for dgemm_... no
configure: searching for OpenBLAS
checking for sgemm_ in -lopenblas... no
configure: searching for BLIS
checking for sgemm_ in -lblis... no
configure: searching for ATLAS
checking for ATL_xerbla in -latlas... yes


So I think what I am seeing is expected. But this would suggest your build machine did not have OpenBLAS, but it seems it did (from your ldd output); or else, maybe your machine had a config.site or BLAS_LIBS? 


On Sat, 30-December-2023, at 19:16:42, Dirk Eddelbuettel <edd using debian.org> wrote:
> If you match the exact build-deps I use and the exact options from
> debian/rules you should end up with the exact same output. As that is whay
> you desire but do not get we need to drill down into where it deviates. FWIW
> last time I build the Debian the (relevant) part (of the large) configure log
> was below.  What do you get for BLAS/LAPACK?
>
>
> R is now configured for x86_64-pc-linux-gnu
>
>   Source directory:            .
>   Installation directory:      /usr
>
>   C compiler:                  gcc  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   Fortran fixed-form compiler: gfortran  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -fcf-protection
>
>   Default C++ compiler:        g++ -std=gnu++17  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   C++11 compiler:              g++ -std=gnu++11  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   C++14 compiler:              g++ -std=gnu++14  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   C++17 compiler:              g++ -std=gnu++17  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   C++20 compiler:              g++ -std=gnu++20  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   C++23 compiler:              g++ -std=gnu++23  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
>   Fortran free-form compiler:  gfortran  -g -O2 -ffile-prefix-map=/build/r-base-4.3.2=. -fstack-protector-strong -fstack-clash-protection -fcf-protection
>   Obj-C compiler:	        
>
>   Interfaces supported:        X11, tcltk
>   External libraries:          pcre2, readline, BLAS(generic), LAPACK(generic), curl
>   Additional capabilities:     PNG, JPEG, TIFF, NLS, cairo, ICU
>   Options enabled:             shared R library, R profiling, memory profiling
>
>   Capabilities skipped:        
>   Options not enabled:         shared BLAS
>
>   Recommended packages:        no
>
>



My configure log (from my "for real" builds)
==============================================

R is now configured for x86_64-pc-linux-gnu

  Source directory:            .
  Installation directory:      /usr/local

  C compiler:                  gcc  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  Fortran fixed-form compiler: gfortran  -g -O2 -march=native -fstack-protector-strong -fstack-clash-protection -fcf-protection

  Default C++ compiler:        g++ -std=gnu++17  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  C++11 compiler:              g++ -std=gnu++11  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  C++14 compiler:              g++ -std=gnu++14  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  C++17 compiler:              g++ -std=gnu++17  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  C++20 compiler:              g++ -std=gnu++20  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  C++23 compiler:              g++ -std=gnu++23  -O2 -g -march=native -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  Fortran free-form compiler:  gfortran  -g -O2 -march=native -fstack-protector-strong -fstack-clash-protection -fcf-protection
  Obj-C compiler:	        

  Interfaces supported:        X11, tcltk
  External libraries:          pcre2, readline, BLAS(OpenBLAS), LAPACK(in blas), curl
  Additional capabilities:     PNG, JPEG, TIFF, NLS, cairo, ICU
  Options enabled:             shared R library, R profiling

  Capabilities skipped:        
  Options not enabled:         shared BLAS, memory profiling

  Recommended packages:        yes


Key differences:
================

- I am not using "-ffile-prefix-map" (but I can't imagine that being the cause of the issue)
- I am configuring without memory profiling (but enabling memory profiling did not change things)
- I am using "march=native" (but this is not the cause, since removing it gives me the same issues)
- And the OpenBLAS, of course.


> Despite the display of 'shared BLAS' not enabled, I still have
>
> root using fb498fe53700:/# ldd /usr/lib/R/modules/lapack.so 
>         linux-vdso.so.1 (0x00007ffcb09c7000)
>         libR.so => /lib/libR.so (0x00007ff7dd7f1000)
>         liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007ff7dd0f6000)
>         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff7dd017000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff7dce36000)
>         libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007ff7dcdd3000)
>         libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007ff7dcd7b000)
>         libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007ff7dccdf000)
>         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff7dccb0000)
>         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007ff7dcc9d000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff7dcc7e000)
>         libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007ff7dcc50000)
>         libicuuc.so.72 => /lib/x86_64-linux-gnu/libicuuc.so.72 (0x00007ff7dca52000)
>         libicui18n.so.72 => /lib/x86_64-linux-gnu/libicui18n.so.72 (0x00007ff7dc726000)
>         libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ff7dc6de000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007ff7ddc74000)
>         libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007ff7da3f0000)
>         libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007ff7da126000)
>         libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007ff7da0f3000)
>         libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007ff7da0a1000)
>         libicudata.so.72 => /lib/x86_64-linux-gnu/libicudata.so.72 (0x00007ff7d82cf000)
>         libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff7d80b5000)
>         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff7d8095000)
>         libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ff7d804e000)
>         libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007ff7d7f74000)
>         libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007ff7d7f47000)
>         libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007ff7d7f41000)
>         libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007ff7d7f33000)
>         libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007ff7d7f2c000)
>         libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ff7d7f1b000)
> root using fb498fe53700:/# ls -l /lib/x86_64-linux-gnu/liblapack.so.3
> lrwxrwxrwx 1 root root 49 Nov 27  2022 /lib/x86_64-linux-gnu/liblapack.so.3 -> /etc/alternatives/liblapack.so.3-x86_64-linux-gnu
> root using fb498fe53700:/# ls -l /lib/x86_64-linux-gnu/libblas.so.3
> lrwxrwxrwx 1 root root 47 Nov 27  2022 /lib/x86_64-linux-gnu/libblas.so.3 -> /etc/alternatives/libblas.so.3-x86_64-linux-gnu
> root using fb498fe53700:/# 

~/tmp/R4/R-4.3.2-85742/modules $ ldd lapack.so 
	linux-vdso.so.1 (0x00007ffd96786000)
	libR.so => /usr/lib/libR.so (0x00007f9d9d174000)
	libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f9d9ace0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9d9ac01000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9d9aa1f000)
	libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f9d9a9a8000)
	libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f9d9a951000)
	libpcre2-8.so.0 => /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9d9a8b9000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9d9a891000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f9d9a87e000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9d9a861000)
	libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007f9d9a832000)
	libicuuc.so.72 => /usr/lib/x86_64-linux-gnu/libicuuc.so.72 (0x00007f9d9a62b000)
	libicui18n.so.72 => /usr/lib/x86_64-linux-gnu/libicui18n.so.72 (0x00007f9d9a2ef000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f9d9a29e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9d9d60d000)
	libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f9d99fcb000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f9d99f9a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9d99f93000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f9d99f40000)
	libicudata.so.72 => /usr/lib/x86_64-linux-gnu/libicudata.so.72 (0x00007f9d9816e000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9d97f18000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9d97ef4000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f9d97e17000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f9d97de7000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f9d97de1000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f9d97dd2000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f9d97dcb000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f9d97db8000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9d97db3000)

Interesting, you also have libopenblas.so.0 (in a different position).

>
> the /etc/alternatives mechanism you are after. (That was from rocker/r-base,
> a quick way for me to get to clean and current Debian build.)

ls -l /usr/lib/x86_64-linux-gnu/liblapack.so.3
lrwxrwxrwx 1 root root 49 Mar 28  2022 /usr/lib/x86_64-linux-gnu/liblapack.so.3 -> /etc/alternatives/liblapack.so.3-x86_64-linux-gnu

And that is the one from OpenBLAS
ls -l /etc/alternatives/liblapack.so.3-x86_64-linux-gnu
lrwxrwxrwx 1 root root 56 Dec 30 20:44 /etc/alternatives/liblapack.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3


For libblas:
ls -l /usr/lib/x86_64-linux-gnu/libblas.so.3
lrwxrwxrwx 1 root root 47 Mar 28  2022 /usr/lib/x86_64-linux-gnu/libblas.so.3 -> /etc/alternatives/libblas.so.3-x86_64-linux-gnu

ls -l /etc/alternatives/libblas.so.3-x86_64-linux-gnu
lrwxrwxrwx 1 root root 54 Dec 30 20:44 /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3



To further try to understand, I've uninstalled openblas, but left atlas:


  Selection    Path                                          Priority   Status
------------------------------------------------------------
* 0            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3   35        auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3   35        manual mode
  2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3    10        manual mode


And then, configure shows its looking for this. 
configure: searching for an external BLAS
configure: searching for BLAS in default libraries
checking for dgemm_... no
configure: searching for OpenBLAS
checking for sgemm_ in -lopenblas... no
configure: searching for BLIS
checking for sgemm_ in -lblis... no
configure: searching for ATLAS
checking for ATL_xerbla in -latlas... yes

And


External libraries:          pcre2, readline, BLAS(ATLAS), LAPACK(generic), curl


ldd now shows

~/tmp/Rpt/R-patched/modules $ ldd lapack.so 
	linux-vdso.so.1 (0x00007ffd83045000)
	libR.so => /usr/lib/libR.so (0x00007f89196e9000)
	liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f8918fb5000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8918ed6000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8918cf4000)
	libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f8918cb2000)
	libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f8918c59000)
	libpcre2-8.so.0 => /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f8918bc1000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8918b99000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f8918b86000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8918b69000)
	libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007f8918b3a000)
	libicuuc.so.72 => /usr/lib/x86_64-linux-gnu/libicuuc.so.72 (0x00007f8918933000)
	libicui18n.so.72 => /usr/lib/x86_64-linux-gnu/libicui18n.so.72 (0x00007f89185f7000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f89185a6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8919b82000)
	libatlas.so.3 => /usr/lib/x86_64-linux-gnu/libatlas.so.3 (0x00007f8918226000)
	libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f8917f53000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8917f2d000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f8917efc000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8917ef7000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f8917ea4000)
	libicudata.so.72 => /usr/lib/x86_64-linux-gnu/libicudata.so.72 (0x00007f89160d2000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8915e7a000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f8915d9f000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f8915d6f000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f8915d69000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f8915d5a000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f8915d51000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f8915d40000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8915d3b000)


Now we see libatlas.

R.


>
> Dirk


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-31
Facultad de Medicina 
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02 using gmail.com
       r.diaz using uam.es
       ramon.diaz using iib.uam.es

https://ligarto.org/rdiaz




More information about the R-SIG-Debian mailing list