Hello,
 
I'm attempting to use doMPI to process a directory of data files, returning NULL to the master. The slaves take off running something but there appears to be a lot of communication going on. The master's RAM gradually fills up and then the whole thing falls over.  
 
I can run the process on a single machine and it uses less than 1GB of RAM. I have 4 available on each machine so I'm sure I have the resources. I can run sincMPI.R from Steve Weston's excellent "Introduction to doMPI" so I confident R is running on the cluster. I have included a NULL statement at the end of my foreach loop that (I think) ensures that nothing is passed back to the master.  
My available machines are on a VMware Virtual Platform. The operating system is opensuse linux 11.1. 
 
Can anyone show me the correct syntax for such a problem?
 At the command line I'm using the following:
    orterun -hostfile mpihosts.hosts -n 1 R --slave -f GenVol.R
 
My R script (GenVol.R) contains the following:
    library(doMPI)
    cl <- startMPIcluster() 
    registerDoMPI(cl)
    filenames <- dir("~/P/lidar/YEAR1_LAS")
    foreach(i = filenames, .packages = c("rgdal", "ForestSensing4")) %dopar% { 
      setwd("~/P/lidar/WorkingFolder/testMPI/") # can't specify path in GDAL filenames so setwd()
      source("~/P/lidar/WorkingFolder/Rscripts/readlasNosizelimit.R") # readlas for files with gt 5 million points
      load("~/P/lidar/WorkingFolder/YEAR1_bil/volmodel.Rdata") # current model 
      allas <- readlas(paste("~/P/lidar/YEAR1_LAS/", substr(i, 1, 15), sep = ""), HeadersOnly = TRUE)
      if(as.numeric(allas@PublicHeaderBlock[19,1]) > 500){  # check for enough points
        allas <- readlas(paste("~/P/lidar/YEAR1_LAS/", substr(i, 1, 15), sep = ""), xyzicsOnly = TRUE)
        if(2 %in% allas@data$Classification){ # check for enough ground points
          allas <- vegetation(allas)
          qs <- quantilesurface(object = allas)
          qs[["tvol"]] <- predict(vol.model, newdat = qs@data, type = "response")
          filename <- paste(substr(i, 1, 11), ".bil", sep = "")
          write(NULL, file = filename)  
          withRestarts(writeGDAL(dataset = qs, fname = filename, drivername = "EHdr"), abort = function(){})  
          rm(allas, qs)
          gc()
          }
        }
      NULL ### return NULL to the master
      } 
    closeCluster(cl)
    mpi.quit()
     
 
 
Dr Robert Musk
Forest Biometrician
Management Research
Forestry Tasmania
79 Melville St Hobart Australia 7000
Phone: +61 3 6235 8127
Fax: +61 3 6235 8292
Email: Rob.Musk@forestrytas.com.au 
 
 

---------------------------------------------------------------------------------------------------------------------------------------------
This transmission is intended solely for the person or organisation to whom it is addressed.
It is confidential and may contain legally privileged information.
If you have received this transmission in error, you may not use, copy or distribute it.
Please advise us by return e-mail or by phoning 61 3 6235 8333 and immediately delete the transmission in its entirety.
We will meet your reasonable expenses of notifying us.
Despite our use of anti-virus software, Forestry Tasmania cannot guarantee that this transmission is virus-free.

	[[alternative HTML version deleted]]

