Robert,
Thanks for your reply. I have a couple of questions about your comments, but before that, an update:
 
Converting to rasters native format resulted in *huge* improvements. All else remaining equal, running a predict with a stack pointed to native rasters took about 5 hrs, 10 minutes, compared to over 10 hours as noted below. Watching CPU/RAM activity I can even see the gaps in activity when (I'm assuming) file I/O is occurring and these gaps were clearly much shorter when using the native raster format. Perhaps this is because it doesn't have to go through GDAL?  No matter what the reason, I am thrilled. 
 
I've also had a chance to increase chunk size just a bit (chunksize =  2e+08), and this did not create any noticeable improvements. I'm trying now with a small increase in maxmemory.
 
Thanks for the pointer on raster::predict requiring lots of overhead. I'll explore hardwiring a simple function to hopefully reduce the overhead there. Would examples in one of your recent vignettes be a good place to start?
 
I am using classification, but I appreciate your comment on using clusters. I had explored this in the past, via snow, and I don't think I saw much improvement. I'll try it again. Are you saying classification may not work, as in it will break, or there just won't be speed improvements?
 
Best, 
Tim

>>> "Robert J. Hijmans" <r.hijmans@gmail.com> 5/17/2012 9:50 PM >>>
Thanks Tim, 

That is good to know. The settings are somewhat conservative to avoid virtual memory use on macs (which slows things down a lot on that platform). Still, even the default settings should beat scanning ascii files, so there might be room for improvement. One reason might be that the raster::predict function tries to accommodate many different models/situations, so there is more overhead. 

I do not know if the native file format helps here. My guess is that it does not, but that it might if you used a single multi-layer file (RasterBrick) in the native format.

In your case, the most limiting factor is not the file I/O but the randomForest prediction. With 8 cores I see a reduction in run time of up to 50%. If you are doing regression (classification may not work) all you need to do is run 

beginCluster() 

Robert



On Thu, May 17, 2012 at 5:05 AM, Tim Howard <tghoward@gw.dec.state.ny.us> wrote:


Some updates for the list archives:

Using
setOptions(maxmemory = 1e+09, chunksize=1e+08)
reduced predict time to about 10 hours 50 min on my system. Good! I also spent the night converting all the GRIDS to native raster format and am trying the next run with that setup to see if read times differ significantly.

I'll report back, and then also see if I can increase chunksize even further.

The raster package truly simplifies things -- I love it!

> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: x86_64-pc-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] rgdal_0.7-8 randomForest_4.6-6 raster_1.9-82 sp_0.9-98

loaded via a namespace (and not attached):
[1] grid_2.15.0 lattice_0.20-6 tools_2.15.0


>>> Tim Howard 5/16/2012 12:58 PM >>>
all -

I'm trying to tweak my setup of the raster package to increase the speed of a predict on a randomForest object.

My system: Windows 7 64-bit with 12GB RAM using R x64 2.15.0, raster version 1.9-82

I'm running the predict on a rasterStack of about 45 layers.

Question #1. Does chunksize refer to the chunk of a single layer in the stack or the entire stack? To rephrase, do I have to be sure there is room to take a chunk consisting of chunksize from every layer and thus need room for chunksize * 45?

The default is 1e+07, which I read as 10MB. Is that right?

Question #2. Is there a rule of thumb for how much buffer to leave between chunksize and maxmemory? The default is 1+e08, which I read as 100MB.

I just tried

setOptions(maxmemory = 4e+09, chunksize=1e+09)

and the predict never got past 0% completion before bailing with a memory error.

Dropping it down to

setOptions(maxmemory = 1e+09, chunksize=1e+08)

is looking like it has promise, with RGui using about 1-2GB.

Using the default settings took about 13 hours for the predict to complete, which is actually slower than my old method of using scan on ASCII files (reading then writing two rows at a time), which generally took 11 hours.

Other than trial and error, is there a way to estimate how far I can push these settings?

Question #3. Tangential question - right now the stack is reading ArcGIS GRID files. If there is a significant speed increase in converting all my grids to native raster package format, I'll do that. I can't find any discussion on the relative read speeds of the different raster formats. Does anyone have a feel for this?

Any other suggestions or direction welcome!

Thanks so much!

Tim

[[alternative HTML version deleted]]

_______________________________________________
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo



	[[alternative HTML version deleted]]

