[R-sig-hpc] socket cluster/gotoblas2 configuration confusion

[Stephen Weston]

This is really a shot in the dark, but you could try executing:

  clusterEvalQ(cl, readLines(sprintf('/proc/%d/status', Sys.getpid())))

and look for the lines that mention "Cpus_allowed".  It's
conceivable that your snow workers have been restricted to
execute on a subset of the node's cores.  But that seems
rather unlikely since you're using a socket cluster.

- Steve


On Wed, Nov 23, 2011 at 11:09 AM, Claudia Beleites
<claudia.beleites at ipht-jena.de> wrote:
> Steve,
>
>> You don't say how you set GOTO_NUM_THREADS to 6.
>
> sorry, I forgot to tell you all:
>
> I used
> export GOTO_NUM_THREADS=6
> in the shell before starting R.
>
> and I did check by
>>      clusterEvalQ(cl, system ("echo $GOTO_NUM_THREADS"))
> which gave me 6 for both workers.
>
> so does:
>>   clusterEvalQ(cl, Sys.getenv('GOTO_NUM_THREADS'))
> [[1]]
> [1] "6"
>
> [[2]]
> [1] "6"
>
> I did not know the Sys.getenv/Sys.setenv functions, though.
>
> Thanks,
>
> Claudia
>
>> You
>> might want to verify that it did get set in each of the snow worker
>> processes by using the command:
>>
>>     clusterEvalQ(cl, Sys.getenv('GOTO_NUM_THREADS'))
>>
>> If it returns any empty strings in the resulting list, then the
>> environment variable is not set in the corresponding worker.
>>
>> You probably should set this variable through an appropriate
>> shell startup file, but you could at least temporarily use:
>>
>>     clusterEvalQ(cl, Sys.setenv(GOTO_NUM_THREADS=6))
>
>
>>
>> - Steve
>>
>>
>> On Wed, Nov 23, 2011 at 9:44 AM, Claudia Beleites
>> <claudia.beleites at ipht-jena.de>  wrote:
>>>
>>> Dear all,
>>>
>>> I'm just doing my first steps with parallelized calculations and got
>>> quite
>>> confused.
>>>
>>> Here's what I want, what I have and what I did:
>>>
>>> - I want to parallelize calculations on a Centos server with 2 x 6 cores
>>> and
>>> 8 GB RAM (it is actually part of a cluster, but I have access only to
>>> this
>>> node, and the other nodes do not (yet) have R installed).
>>>
>>> - My Data is too large to work with in one piece.
>>> But it comes in separate files of suitable size: I can work nicely with 2
>>> to
>>> 3 samples in memory at the same time.
>>>
>>> - So my idea was to start up a snow socket cluster with 2 or 3 workers.
>>>
>>> - In addition I want to use an optimized and blas. Linear algebra is only
>>> a
>>> small part of the analysis so it does make sense to have the socket
>>> cluster
>>> with as many workers as possible and have the linear algebra parts use up
>>> to
>>> n / nworkers cores.
>>>
>>> So I built R 2.14.0 using gotoblas2 and set $GOTO_NUM_THREADS to 6.
>>> Matrix
>>> multiplication in a fresh R session now is much faster and CPU usage
>>> shows
>>> the expected 6 cores working:
>>>
>>>> system.time ({m<- matrix (1:9e6, 3e3); m%*%m; NULL})
>>>
>>>       User      System verstrichen
>>>      5.219       0.126       1.111
>>>
>>> However, the socket clusters seem not to use the GOTO_NUM_THREADS:
>>>>
>>>> library (snow)
>>>> cl<- makeCluster(2,type="SOCK")
>>>> tm<- snow.time(clusterEvalQ(cl, {m<- matrix (1:9e6, 3e3); m%*%m; NULL}))
>>>> tm
>>>
>>> elapsed    send receive  node 1  node 2
>>>  9.553   0.001   0.010   9.510   9.543
>>>>
>>>> tm$data
>>>
>>> [[1]]
>>>     send_start send_end recv_start recv_end exec
>>> [1,]          0    0.001      9.511    9.512 9.51
>>>
>>> [[2]]
>>>     send_start send_end recv_start recv_end  exec
>>> [1,]      0.001    0.001      9.544    9.553 9.543
>>>
>>>> tm$elapsed
>>>
>>> elapsed
>>>  9.553
>>>>
>>>
>>> CPU usage shows 2 cores working, and the times correspond to that.
>>>
>>> What configuration do I need to do in order to make the blas use more
>>> threads for the worker processes? Anything else I should do differently?
>>>
>>>
>>>> sessionInfo ()
>>>
>>> R version 2.14.0 (2011-10-31)
>>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>>
>>> locale:
>>>  [1] LC_CTYPE=de_DE.UTF-8       LC_NUMERIC=C
>>>  [3] LC_TIME=de_DE.UTF-8        LC_COLLATE=de_DE.UTF-8
>>>  [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=de_DE.UTF-8
>>>  [7] LC_PAPER=C                 LC_NAME=C
>>>  [9] LC_ADDRESS=C               LC_TELEPHONE=C
>>> [11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C
>>>
>>> attached base packages:
>>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>>
>>> other attached packages:
>>> [1] snow_0.3-8
>>>
>>> Additional questions:
>>> - Is there some command like sessionInfo () that yields information about
>>> the blas (particularly NUM_THREADS)?
>>> - Is there some command that I can use to tell the blas how many threads
>>> to
>>> use during an R session? Can I set environment variables from within R?
>>> Searching didn't help as I got only info about R environments...) Would
>>> that
>>> actually help here?
>>>
>>> Thanks a lot for your help.
>>>
>>> Claudia
>>>
>>> --
>>> Claudia Beleites
>>> Spectroscopy/Imaging
>>> Institute of Photonic Technology
>>> Albert-Einstein-Str. 9
>>> 07745 Jena
>>> Germany
>>>
>>> email: claudia.beleites at ipht-jena.de
>>> phone: +49 3641 206-133
>>> fax:   +49 2641 206-399
>>>
>>> _______________________________________________
>>> R-sig-hpc mailing list
>>> R-sig-hpc at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>
>
>
> --
> Claudia Beleites
> Spectroscopy/Imaging
> Institute of Photonic Technology
> Albert-Einstein-Str. 9
> 07745 Jena
> Germany
>
> email: claudia.beleites at ipht-jena.de
> phone: +49 3641 206-133
> fax:   +49 2641 206-399
>
>
>
>
>