[R-sig-hpc] Disabling implicit parallelism of multithreaded BLAS from an R package
Roger Koenker
rkoenker at illinois.edu
Wed Aug 13 17:40:40 CEST 2014
I hope it won't be viewed as highjacking this thread if I mention that
disabling implicit parallelism can be an issue in a variety of other contexts
as well. I've recently experienced this on our cluster with simulations that
used foreach and doMPI in an embarrassingly parallel way, but use Rmosek/Mosek
in the innermost loop to solve convex optimization problems. By default Mosek
tries to use multicore capabilities of the cluster nodes and this results in rather
poor performance as different instances of the foreach loop fight with one another
for cpu resources. Fortunately, this can be controlled in Rmosek so the foreach
instances are single threaded as illustrated in the new release of my REBayes
package. How common such "features" are, I have no clue, but it can be
significant. In my application it cut elapsed time of my simulations from about
14 hours to about 5.
url: www.econ.uiuc.edu/~roger Roger Koenker
email rkoenker at uiuc.edu Department of Economics
vox: 217-333-4558 University of Illinois
fax: 217-244-6678 Urbana, IL 61801
On Aug 13, 2014, at 9:28 AM, Ei-ji Nakama <nakama at ki.rim.or.jp> wrote:
> hello,
> RhpcBLASctl is a package in the state of having started making it....
>
> Example..
> $ R -q
>> library(RhpcBLASctl)
>> omp_get_max_threads()
> [1] 8
>> omp_set_num_threads(1)
>> omp_get_max_threads()
> [1] 1
>> blas_get_num_procs()
> [1] 4
>> blas_set_num_threads(2)
>> blas_get_num_procs()
> [1] 2
>> blas_set_num_threads(get_num_cores())
>> blas_get_num_procs()
> [1] 4
>
> Its need to think independently because there is no relativity in the
> control of OpenMP and BLAS.
> If Goto or OpenBLAS is done in the build, the instruction of NO_WARMUP
> and NO_AFFINITY might be gentle to the user.
>
>
> 2014-08-13 21:02 GMT+09:00 Scott Ritchie <sritchie73 at gmail.com>:
>> Hi again Dirk,
>>
>> If its just an issue with the particular setup I'm using, I'm happy to
>> stick `export OPENBLAS_NUM_THREADS=1` in my bash profile.
>>
>> Good to know about Goto and MKL. I'd just seen them mentioned in the gcbd
>> vignette you suggested I read, so assumed they would have the same issue. I
>> had originally hope to just set the relevant environment variables for all
>> of them (even if they weren't relevant to the user).
>>
>> I'll construct a minimal example to use as a test case and send it to the
>> mailing list.
>>
>> Cheers,
>>
>> Scott
>>
>>
>> On 13 August 2014 21:33, Dirk Eddelbuettel <edd at debian.org> wrote:
>>
>>>
>>> Scott,
>>>
>>> On 13 August 2014 at 20:08, Scott Ritchie wrote:
>>> | Hi Claudia,
>>> |
>>> | The system I tested it on is running Ubuntu 12.04, so it should
>>> | theoretically work.
>>>
>>> I keep forgetting you run an outdated OS version. As I recall, the
>>> OpenBLAS
>>> in that release is older / was lagging. This may work better on newer
>>> ones.
>>> Worst case, Consider rebuilding the OpenBLAS package from source on your
>>> box to create a
>>> local backp. (Been there, done that, need to document the process ...)
>>>
>>> Do you have a small / simple testcase?
>>>
>>> | I'd also like a way of disabling implicit parallelism if R has been
>>> | compiled with Goto BLAS and MKL, since I assume those users will have the
>>> | same problem.
>>>
>>> You may need to change their APIs... MKL, as I recall, listen to the same
>>> (Intel-given) variable. GotoBLAS is a dead project which was succeeded by
>>> OpenBLAS, so changing Goto may be impossible.
>>>
>>> Thanks also to Claudia for that reminder. I had forgotten this Linux-only
>>> aspect.
>>>
>>> Cheers, Dirk
>>>
>>>
>>>
>>> | On 13 August 2014 18:21, beleites,claudia <claudia.beleites at ipht-jena.de
>>>>
>>> | wrote:
>>> |
>>> | > Hi Scott
>>> | >
>>> | > You may need to contact the developers of openblas about
>>> | > openblas_set_num_threads. 2yrs ago when Simon was writing that wrapper
>>> it
>>> | > was only implemented for Linux. All other is just had a non- working
>>> dummy.
>>> | >
>>> | > Hth
>>> | >
>>> | > Claudia
>>> | >
>>> | > --
>>> | > Claudia Beleites, Chemist
>>> | > Spectroscopy/Imaging
>>> | > Leibniz Institute of Photonic Technology
>>> | > Albert-Einstein-Str. 9
>>> | > 07745 Jena
>>> | > Germany
>>> | >
>>> | > email: claudia.beleites at ipht-jena.de
>>> | > phone: +49 3641 206-133
>>> | > fax: +49 2641 206-399
>>> | >
>>> | >
>>> | > ________________________________________
>>> | > Von: r-sig-hpc-bounces at r-project.org [r-sig-hpc-bounces at r-project.org
>>> ]"
>>> | > im Auftrag von "Scott Ritchie [sritchie73 at gmail.com]
>>> | > Gesendet: Mittwoch, 13. August 2014 06:40
>>> | > An: r-sig-hpc at r-project.org
>>> | > Betreff: [R-sig-hpc] Disabling implicit parallelism of multithreaded
>>> BLAS
>>> | > from an R package
>>> | >
>>> | > Hi all,
>>> | >
>>> | > I'm developing a package in which I explicitly handle parallelism, and
>>> in
>>> | > each parallel R session, running some SVD calculations through Rcpp +
>>> | > Armadillo.
>>> | >
>>> | > On one of my test machines, R 3.1.0 has been compiled against
>>> OpenBLAS, so
>>> | > the calls to SVD are implicitly parallel.
>>> | >
>>> | > I can disable this by exporting the appropriate environment variable
>>> before
>>> | > calling R (OPENBLAS_NUM_THREADS=1), but I would ideally like to
>>> abstract
>>> | > that detail away from the user (as some of my intended target audience
>>> may
>>> | > be biologists without a strong computational background, but
>>> nevertheless
>>> | > operating on a cluster someone else has set up).
>>> | >
>>> | > I've tried a couple of things to disable the implicit parallelism
>>> without
>>> | > any success:
>>> | > - Sys.setenv doesn't have any effect, even on the new R sessions
>>> spawned
>>> | > through `foreach` with the `doMC` parallel backend (this surprises me,
>>> is
>>> | > my mental model of parallelism in R correct?)
>>> | > - exporting the environment variables using `cstdlib::putenv` in the
>>> C++
>>> | > code directly before the call to `svd`.
>>> | > - The solution I found through the r-sig-hpc archives:
>>> | > https://stat.ethz.ch/pipermail/r-sig-hpc/2012-July/001432.html.
>>> Directly
>>> | > calling openblas_set_num_threads(1); in the C++ code has no effect.
>>> | >
>>> | > I've also tried explicitly disabling BLAS in the hope that Armadillo
>>> will
>>> | > call the LAPACK routines directly (although I may have done this
>>> | > incorrectly) by doing the following:
>>> | > - removing the $(BLAS_LIBS) from the PKG_LIBS variable in the
>>> | > Makevars/Makevars.win files
>>> | > - Setting #define ARMA_DONT_USE_BLAS and #define ARMA_DONT_USE_WRAPPER
>>> | > before including the <RcppArmadillo.h> header file in my source code.
>>> | >
>>> | > Ideally I'm looking for a solution that allows me to set the maximum
>>> number
>>> | > of threads to 1 for all the multithreaded BLAS libraries in my package
>>> to
>>> | > prevent conflicts between the implicit and explicit parallelism. Is
>>> this
>>> | > possible?
>>> | >
>>> | > Regards,
>>> | >
>>> | > Scott Ritchie
>>> | >
>>> | > [[alternative HTML version deleted]]
>>> | >
>>> | > _______________________________________________
>>> | > R-sig-hpc mailing list
>>> | > R-sig-hpc at r-project.org
>>> | > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>> |
>>> | [[alternative HTML version deleted]]
>>> |
>>> | _______________________________________________
>>> | R-sig-hpc mailing list
>>> | R-sig-hpc at r-project.org
>>> | https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>
>>> --
>>> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>>>
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-hpc mailing list
>> R-sig-hpc at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
>
>
> --
> Best Regards,
> --
> EI-JI Nakama <nakama (a) ki.rim.or.jp>
> "\u4e2d\u9593\u6804\u6cbb" <nakama (a) ki.rim.or.jp>
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
More information about the R-sig-hpc
mailing list