[R-sig-hpc] Creating different working directories for each node?

Novack-Gottshall, Philip M. pnovack-gottshall at ben.edu
Tue Aug 12 16:56:55 CEST 2014


The tempfile option might work, as might using Sys.info()["nodename"] to
specify individual subdirectories.

Are you aware of any functions that can be used to identify which CPU a
process has been spawned to? The package 'snowfall' has sfGetCluster(),
but that doesn't seem to be able to be called *within* a cluster function.

Thanks again,
Phil


On 8/11/2014 4:10 PM, Brian G. Peterson wrote:
> You didn't tell us which clusterning mechanism you're using, but most 
> will allow you to retrieve a unique node id or node number, which you 
> could use to create a subdirectory.
>
> Also, 'tempfile' in R will create a vector of unique temporary file 
> names, which you could pass to the nodes, or append with a node id.
>
> Brian
>
> On 08/11/2014 03:59 PM, Novack-Gottshall, Philip M. wrote:
>> Greetings,
>>
>> I'm trying to run some code on a cluster in which an internal qhull
>> convex-hull function repeatedly writes then scans a dummy file. (Very
>> inefficient, I know, but c'est la vie.) The problem is that all nodes
>> share the same default working directory, and get confused because they
>> are all trying to read/write the same dummy file. (So far as I know,
>> there is no way to specify unique file names for the dummy files in the
>> internal .C function, especially with my lack of C prowess.)
>>
>> I've been playing around with having my function (called using
>> snowfall's 'sfClusterApply') specify unique working directories (using
>> 'setwd'), say one wd for each CPU, so that the individual dummy files
>> are set within unique directories. This seems a plausible wrap-around so
>> long as I confirm that I'm matching up the wd with the correct CPU process.
>>
>> I wanted to check whether anyone has any other recommendations before I
>> waste my time on further troubleshooting.
>>
>> If it's relevant, I'm running my code using management package snowfall
>> on a CentOS/OpenMPI Intel cluster with 2 hyperthreaded 6-cores in each
>> of 16 nodes (allowing 382 functional CPUs/unique working directories for
>> the job).
>>
>> Thanks,
>> Phil
>>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>
>

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Phil Novack-Gottshall
 Associate Professor
 Department of Biological Sciences
 Benedictine University
 5700 College Road 
 Lisle, IL 60532

 pnovack-gottshall at ben.edu
 Phone: 630-829-6514
 Fax: 630-829-6547
 Office: 332 Birck Hall
 Lab: 107 Birck Hall
 http://www1.ben.edu/faculty/pnovack-gottshall
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




More information about the R-sig-hpc mailing list