[R-SIG-Mac] using optimized BLAS [Was: problem installing rjags package]
Michael Braun
braunm at mit.edu
Thu Oct 14 23:03:55 CEST 2010
OK, here's what I did, and it seems to work:
1. I started with a clean R installation, using only the CRAN binary.
2. I ran the following script:
#!/bin/bash
export BL=/opt/intel/Compiler/11.1/089/Frameworks/mkl/Libraries/em64t
export GF=~/Install_Files/MKL_gfortran_interface/lib/em64t
export IOMP=/opt/intel/Compiler/11.1/089/lib
ld -dylib -arch x86_64 -L$BL -L$GF -reexport-lmkl_intel_lp64 -reexport-lmkl_sequential -reexport-lmkl_core -reexport-lpthread -o libmklblas_seq.dylib
ld -dylib -arch x86_64 -L$BL -L$GF -L$IOMP -reexport-lmkl_intel_lp64 -reexport-lmkl_intel_thread -reexport-lmkl_core -reexport-liomp5 -reexport-lpthread -o libmklblas_par.dylib
This gave me two Intel MKL libraries. I then pointed the libRblas.dylib symbolic link to one or the other, depending on if I want a sequential or threaded.
The performance of the threaded MKL is comparable to ATLAS (sometimes better, sometimes worse), but is much, much better than vecLib.
So this gives me the BLAS I want, and keeps my life very simple by letting me use the CRAN binary.
But as researchers, we know that every answer spawns new questions.
1. Can I do something similar with LAPACK (say, if I want a threaded Cholesky decomposition or solving a system of equations)?
2. Although it is true that one can adjust the number of threads that the MKL BLAS uses by setting the MKL_NUM_THREADS environment variable, it appears that this variable must be set before the BLAS is loaded. Since BLAS is loaded at the same time as R, it would appear that this kind of adaptation "on the fly" is not possible. Is there a way to "reset" the BLAS at a suitable point in an R script, so I could unload BLAS, change the variable, and then reload it?
The *specific* case I'm talking about is what would happen if I were to call multithreaded BLAS from a function that is running via multicore, foreach and/or plyr? I'm thinking I want threading turned on when I'm not in a foreach loop, but off when I am.
3. Do I have to worry about the MKL Fortran interfaces to BLAS or LAPACK at all? (I'm talking about the libmkl_blas95_lp64.a and libmkl_lapack95_lp64.a libraries). I needed to link to them to compile R from source, but if I omit them from today's new creation, will that mess me up down the road?
4. As I mentioned before, I'm learning a lot this as I go, but from what I've read, the newer Intel processors support a number of vectorization instructions that are not supported by earlier chips (this SSE4.2 stuff). I have tried compiling other packages (e.g., GSL) with and without SSE4.2, and find that adding the appropriate flag in the Intel compiler gives me 10-20 percent improvement in some applications (can't remember which--did it a while ago). So that's why I add those flags when I compile R from source. But I recognize that the CRAN binary needs to have some backwards compatibility. The Intel compilers offer such an option (e.g., -axsse4.2 will compile to use the best possible set of instructions up to sse4.2). Is the CRAN binary compiled in the same way? Put another way, is it compiled for the "lowest" common instruction set, or will it use the most advanced instruction set if it's available?
As before, I am happy to split these questions into separate discussion threads. But now the questions are more of the "this is interesting--I'd like to learn more," variety, instead of the "I have a problem, please help me" type. And of course, thanks so much for pointing me in the right direction, and for all the work you do in developing and supporting R.
On Oct 12, 2010, at 11:44 AM, Simon Urbanek wrote:
>
> On Oct 12, 2010, at 11:19 AM, Michael Braun wrote:
>
>> Thanks, Simon:
>>
>> I do have a Nehalem Mac Pro, and I've found MKL to be much faster than vecLib.
>
> Yes, because vecLib has a serious bug and doesn't spawn threads on Nehalem, but ATLAS is faster than MKL on Nehalem Mac Pro -- here are the tcrossprod timings (on 2.66GHz Mac Pro):
>
> vecLib 6.43
> ATLAS serial 4.80
> MKL serial 4.30
> MKL parallel 0.90
> ATLAS parallel 0.71
>
>
>> I saw in one of the manuals that one could point R to another BLAS. But I wasn't sure what to do with MKL because of their "layering" approach to linking. So if I point libRblas to something else, how to I tell the link which of the multiple libraries I need?
>>
>
> MKL is picking them at run time so you don't have to point it to the machine-specific libraries. You can simply create one library (ld -dylib) re-exporting exactly what you would have done when linking it in R (i.e. use -reexport-l instead of -l).
>
> For ATLAS it's much easier since you can compile it statically with PIC and create one library -- e.g. see
> http://r.research.att.com/libatlas-MacPro.dylib
> (Note that one is 10.6 only but includes both 32-bit and 64-bit ATLAS 3.9.25)
>
> Cheers,
> Simon
>
>
>> And since we are now straying away from the original topic about JAGS, let me know if you prefer that I start another thread. I do try to follow the posting guide as best I can.
>>
>> Michael
>>
>>
>>
>>
>>
>> On Oct 12, 2010, at 9:10 AM, Simon Urbanek wrote:
>>
>>> Michael,
>>>
>>> On Oct 9, 2010, at 5:56 PM, Michael Braun wrote:
>>>
>>>> Hi!
>>>>
>>>> I am trying to install rjags on my Mac Pro running OSX 10.6.4, and R 2.11.1 (which I compiled myself so I could link to the Intel MKL blas, which I prefer to other options).
>>>
>>>
>>> Just FWIW: CRAN R comes with shared BLAS option enabled and allows easy plug-in of MKL BLAS without the need to re-compile (just use the libRblas.dylib symlink). Note that MKL BLAS is often slower than ATLAS so if your main goal is speed I suggest using ATLAS (or vecLib unless you have Nehalem-type Mac Pro) - it is also easier to handle (both from technical and licensing point of view).
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>>
>>>> I was successfully able to install JAGS 2.1.0 through both the binary installer, and compiling from source (I get the same problem either way).
>>>>
>>>> But I would like to use the rjags package. When I do
>>>>
>>>> install.packages("rjags")
>>>>
>>>>
>>>> I get the folliowing output:
>>>>
>>>> * installing *source* package ‘rjags’ ...
>>>> checking for prefix by checking for jags... /usr/local/bin/jags
>>>> checking for g++... g++
>>>> checking for C++ compiler default output file name... a.out
>>>> checking whether the C++ compiler works... yes
>>>> checking whether we are cross compiling... no
>>>> checking for suffix of executables...
>>>> checking for suffix of object files... o
>>>> checking whether we are using the GNU C++ compiler... yes
>>>> checking whether g++ accepts -g... yes
>>>> checking how to run the C++ preprocessor... g++ -E
>>>> checking for grep that handles long lines and -e... /usr/bin/grep
>>>> checking for egrep... /usr/bin/grep -E
>>>> checking for ANSI C header files... yes
>>>> checking for sys/types.h... yes
>>>> checking for sys/stat.h... yes
>>>> checking for stdlib.h... yes
>>>> checking for string.h... yes
>>>> checking for memory.h... yes
>>>> checking for strings.h... yes
>>>> checking for inttypes.h... yes
>>>> checking for stdint.h... yes
>>>> checking for unistd.h... yes
>>>> checking Console.h usability... yes
>>>> checking Console.h presence... yes
>>>> checking for Console.h... yes
>>>> checking for gcc... gcc
>>>> checking whether we are using the GNU C compiler... yes
>>>> checking whether gcc accepts -g... yes
>>>> checking for gcc option to accept ISO C89... none needed
>>>> checking for jags_version in -ljags... yes
>>>> configure: creating ./config.status
>>>> config.status: creating src/Makevars
>>>> configure: creating ./config.status
>>>> config.status: creating src/Makevars
>>>> config.status: creating R/unix/zzz.R
>>>> ** libs
>>>> *** arch - x86_64
>>>> g++ -arch x86_64 -I/Library/Frameworks/R.framework/Resources/include -I/Library/Frameworks/R.framework/Resources/include/x86_64 -I/usr/local/include/JAGS -I/opt/intel/Compiler/11.1/089/Frameworks/mkl/Headers -I/Install_Files/MKL_gfortran_interface/include/em64t/lp64 -fPIC -m64 -msse4.2 -g -O2 -c jags.cc -o jags.o
>>>> g++ -arch x86_64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/intel/Compiler/11.1/089/Frameworks/mkl/Libraries/em64t -L/Users/braunm/Install_Files/MKL_gfortran_interface/lib/em64t -o rjags.so jags.o -lgsl -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
>>>> installing to /Users/braunm/.R/Library/rjags/libs/x86_64
>>>> ** R
>>>> ** data
>>>> ** preparing package for lazy loading
>>>> ** help
>>>> *** installing help indices
>>>> ** building package indices ...
>>>> Error : .onLoad failed in loadNamespace() for 'rjags', details:
>>>> call: dyn.load(file, DLLpath = DLLpath, ...)
>>>> error: unable to load shared library '/Users/braunm/.R/Library/rjags/libs/x86_64/rjags.so':
>>>> dlopen(/Users/braunm/.R/Library/rjags/libs/x86_64/rjags.so, 10): Symbol not found: _JAGS_NA
>>>> Referenced from: /Users/braunm/.R/Library/rjags/libs/x86_64/rjags.so
>>>> Expected in: flat namespace
>>>> in /Users/braunm/.R/Library/rjags/libs/x86_64/rjags.so
>>>> ERROR: installing package indices failed
>>>> * removing ‘/Users/braunm/.R/Library/rjags’
>>>>
>>>>
>>>> However, when I go to look at the symbols in the jags library, I see
>>>>
>>>> braunm4: /usr/local/lib $ nm libjags.dylib |grep JAGS
>>>> 0000000000098650 S _JAGS_NA
>>>> 0000000000110338 S _JAGS_NAN
>>>> 0000000000110348 S _JAGS_NEGINF
>>>> 0000000000110340 S _JAGS_POSINF
>>>> 000000000000133c t __GLOBAL__I_JAGS_NA
>>>>
>>>> and when I try
>>>>
>>>> braunm4: /usr/local/lib $ nm libjags.a |grep JAGS
>>>> U _JAGS_NA
>>>> U _JAGS_NEGINF
>>>> U _JAGS_NA
>>>> U _JAGS_NA
>>>> 0000000000000060 S _JAGS_NA
>>>> 00000000000000b8 B _JAGS_NAN
>>>> 00000000000000c8 B _JAGS_NEGINF
>>>> 00000000000000c0 B _JAGS_POSINF
>>>> U _JAGS_NA
>>>> U _JAGS_NEGINF
>>>> U _JAGS_POSINF
>>>> U _JAGS_NEGINF
>>>> U _JAGS_POSINF
>>>> U _JAGS_NEGINF
>>>> U _JAGS_NA
>>>> U _JAGS_NEGINF
>>>> U _JAGS_NEGINF
>>>> U _JAGS_NA
>>>> U _JAGS_NEGINF
>>>> U _JAGS_POSINF
>>>>
>>>>
>>>> So the jags libraries (both static and dynamic) are in the standard place, and it looks like the symbols rjags.so needs are there as well. So I'm not sure what the rjags installer is looking for. Is there perhaps an environment variable that I am missing?
>>>>
>>>> Here is my sessionInfo():
>>>>
>>>>> sessionInfo()
>>>> R version 2.11.1 (2010-05-31)
>>>> x86_64-apple-darwin10.4.2
>>>>
>>>> locale:
>>>> [1] en_US.UTF-8/en_US.UTF-8/C/C/en_US.UTF-8/en_US.UTF-8
>>>>
>>>> attached base packages:
>>>> [1] stats grDevices datasets graphics utils methods base
>>>>
>>>> other attached packages:
>>>> [1] xtable_1.5-6 doMC_1.2.1 multicore_0.1-3 foreach_1.3.0 codetools_0.2-2 iterators_1.0.3 coda_0.13-5 matrixcalc_1.0-1 lattice_0.18-8 reshape_0.8.3 plyr_1.2.1
>>>> [12] mvtnorm_0.9-92
>>>>
>>>> loaded via a namespace (and not attached):
>>>> [1] grid_2.11.1 tcltk_2.11.1 tools_2.11.1
>>>>
>>>>
>>>> Thanks in advance for your help.
>>>>
>>>> Michael
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------
>>>> Michael Braun
>>>> Homer A. Burnell (1928) Career Development Professor,
>>>> and Assistant Professor of Management Science (Marketing Group)
>>>> MIT Sloan School of Management
>>>> 100 Main St.., E62-535
>>>> Cambridge, MA 02139
>>>> braunm at mit.edu
>>>> 617-253-3436
>>>>
>>>> _______________________________________________
>>>> R-SIG-Mac mailing list
>>>> R-SIG-Mac at stat.math.ethz.ch
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>>>>
>>>>
>>>
>>
>> -------------------------------------------
>> Michael Braun
>> Homer A. Burnell (1928) Career Development Professor,
>> and Assistant Professor of Management Science (Marketing Group)
>> MIT Sloan School of Management
>> 100 Main St.., E62-535
>> Cambridge, MA 02139
>> braunm at mit.edu
>> 617-253-3436
>>
>>
>>
>
-------------------------------------------
Michael Braun
Homer A. Burnell (1928) Career Development Professor,
and Assistant Professor of Management Science (Marketing Group)
MIT Sloan School of Management
100 Main St.., E62-535
Cambridge, MA 02139
braunm at mit.edu
617-253-3436
More information about the R-SIG-Mac
mailing list