[R-sig-hpc] multicore with functions calling an exe/.sh file

[Stephen Weston]

If you're executing an external program from R using mclapply, and that
program is creating output files, then you need to arrange for that
program to create the output files with different names, or to create them
in different directories.  Usually that would be done with a command line
argument to the program.  If it doesn't have such an argument, the best
thing to do is add one.  If you can't do that, you could try executing them
from different directories by executing setwd in the parallel function
before executing the external program.  Unfortunately, that could cause
other problems, especially if the program is reading data files from the
current working directory.  But hopefully, the program supports an option
to change the output file name or directory.

- Steve

P.S.  Why is a program running on a Linux machine using a file with
the path "Z:\home\zambrhe\S090-test\file.cio"?


On Fri, Jun 24, 2011 at 5:12 AM, Mauricio Zambrano-Bigiarini
<mauricio.zambrano at jrc.ec.europa.eu> wrote:
> Dear List,
>
> I'm just doing my first trials with HPC, and I would like to ask your
> opinion regarding the following issue.
>
> In R 2.13.0, I have an optimization algorithm for hydrological models, which
> internally runs the .exe/.sh file of the model, and then computes and writes
> into a file the results of different parameter values.
>
> So far this algorithm runs only in a sequential mode, i.e., trying different
> parameter values one after another, and I would like to parallelize it.
>
> My first attempt was using the multicore library, and changing the existing
> 'lapply' loop for an 'mclapply' one. However, when I run the optimization
> algorithm, I got several error messages:
>
> forrtl: Sharing violation
> forrtl: Sharing violation
> forrtl: Sharing violation
>
> forrtl: severe (30): open failure, unit 1, file
> Z:\home\zambrhe\S090-test\file.cio
>
> which are due to the fact that all the 4 process are trying to access the
> same .exe/.sh file and to modify the same input files at the same time.
>
>
> Is there any way to use multicore for parallelizing this type of
> optimization function or should I move to some master/slave option ?
>
>
> Thanks in advance for any help.
>
>
> Mauricio Zambrano-Bigiarini
>
>
> PS,
> SesionInfo():
> R version 2.13.0 (2011-04-13)
> Platform: i386-redhat-linux-gnu (32-bit)
>
> --
> =======================================================
> FLOODS Action
> Land Management and Natural Hazards Unit
> Institute for Environment and Sustainability (IES)
> European Commission, Joint Research Centre (JRC)
> TP 261, Via Enrico Fermi 2749, 21027 Ispra (VA), Italy
> webinfo    : http://floods.jrc.ec.europa.eu/
> work-phone : (+39)-(0332)-789588
> work-fax   : (+39)-(0332)-786653
> =======================================================
> DISCLAIMER:\ "The views expressed are purely those of th...{{dropped:11}}
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>