[R-sig-hpc] Rmpi and ncdf4

Paul Johnson pauljohn32 at gmail.com
Tue Sep 14 20:30:05 CEST 2010


On Fri, Sep 10, 2010 at 11:12 AM, clement <clement.tisseuil at gmail.com> wrote:
> Dear members,

>
>> I am using the ncdf4 package to work on General Circulation Model (GCM)
>> data (NetCDF file format) and I would like to parallelize some calculations
>> using Rmpi. Does anyone have an experience or advices in using Rmpi and
>> ncdf4 packages?
>>

Dear Clement:

Thanks for posting your code. It really helps me to learn when I can
read through what other people try.

I found myself wondering "how much" of the whole data set is used by
each slave.  Supposing the slave needs only a smaller piece,   I think
your problem would work more efficiently if you have the master load
the data one time and have it send the separate pieces to the slaves
for the work.  Well, that's what I would do because I've had very bad
experience when lots of nodes try to access the same file on NFS.  (It
causes something like a traffic jam as the processes fight over each
other).

Instead of doing system(rm ...), I'd suggest you clean files with the
file.remove function (see ?files) .  That will work across platforms,
so even people who use Windows might someday be able to run your code.

pj
-- 
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas



More information about the R-sig-hpc mailing list