Some updates for the list archives:
 
Using  
setOptions(maxmemory = 1e+09, chunksize=1e+08)
reduced predict time to about 10 hours 50 min on my system. Good!  I also spent the night converting all the GRIDS to native raster format and am trying the next run with that setup to see if read times differ significantly. 
 
I'll report back, and then also see if I can increase chunksize even further. 
 
The raster package truly simplifies things  -- I love it!
 
> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: x86_64-pc-mingw32/x64 (64-bit)
 
locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    
 
attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     
 
other attached packages:
[1] rgdal_0.7-8        randomForest_4.6-6 raster_1.9-82      sp_0.9-98         
 
loaded via a namespace (and not attached):
[1] grid_2.15.0    lattice_0.20-6 tools_2.15.0  


>>> Tim Howard 5/16/2012 12:58 PM >>>
all - 
 
I'm trying to tweak my setup of the raster package to increase the speed of a predict on a randomForest object. 
 
My system: Windows 7 64-bit with 12GB RAM using R x64 2.15.0, raster version 1.9-82
 
I'm running the predict on a rasterStack of about 45 layers. 
 
Question #1. Does chunksize refer to the chunk of a single layer in the stack or the entire stack? To rephrase, do I have to be sure there is room to take a chunk consisting of chunksize from every layer and thus need room for chunksize * 45?
 
The default is 1e+07, which I read as 10MB.  Is that right?
 
Question #2. Is there a rule of thumb for how much buffer to leave between chunksize and maxmemory? The default is 1+e08, which I read as 100MB.
 
I just tried 
 
setOptions(maxmemory = 4e+09, chunksize=1e+09)
 
and the predict never got past 0% completion before bailing with a memory error. 
 
Dropping it down to
 
setOptions(maxmemory = 1e+09, chunksize=1e+08)
 
is looking like it has promise, with RGui using about 1-2GB. 
 
Using the default settings took about 13 hours for the predict to complete, which is actually slower than my old method of using scan on ASCII files (reading then writing two rows at a time), which generally took 11 hours.  
 
Other than trial and error, is there a way to estimate how far I can push these settings?
 
Question #3. Tangential question - right now the stack is reading ArcGIS GRID files. If there is a significant speed increase in converting all my grids to native raster package format, I'll do that. I can't find any discussion on the relative read speeds of the different raster formats. Does anyone have a feel for this?
 
Any other suggestions or direction welcome!
 
Thanks so much!
 
Tim

	[[alternative HTML version deleted]]

