[R-sig-Geo] Raster math on a stack of large rasters

[Gregovich, Dave P (DFG)]

I apologize for not including my sessionInfo() previously:
> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] raster_2.5-8 sp_1.2-4

loaded via a namespace (and not attached):
[1] rgdal_1.2-5     tools_3.3.3     Rcpp_0.12.9     grid_3.3.3      lattice_0.20-34







From: Gregovich, Dave P (DFG)
Sent: Thursday, April 06, 2017 9:27 AM
To: 'r-sig-geo at r-project.org'
Subject: Raster math on a stack of large rasters

Hi,
I am performing a math operation on a stack of large rasters. The code below uses smaller files for illustration and reproducibility.
Any alternative way of performing this task that does not create huge temporary files, and perhaps cuts down on processing time, would be greatly appreciated. The 'calc'  process creates a couple of temporary raster files that are huge. The first one is 142 GB, and I don't have hard drive space for that one and the second one that begins writing during the process.
Thanks kindly for any advice!
Dave.

#create raster stack and coefficients...
library(raster)
mod.coefs <- rnorm(10)
s <- stack()
r <- raster(nrow = 100, ncol = 100)
#actual rasters I am working with are about 40000, pixels square, with each GeoTiff raster in the stack taking about 2.5 GB on disk

for(i in 1:10){
  r[] <- rnorm(10000)
  s <- addLayer(s, r)
}

#attempt to perform raster math...
out.file <- 'C:/ out.rast.tif'
out.rast <- calc(s, function(x){exp(sum(mod.coefs * x))}, filename = out.file)

#at this point, the temporary files in the \AppData\Local\Temp\RtmpqQYzfS\raster folder eventually become quite large,
#with one .GRI file reaching 142 GB, and another now growing to 8 GB before I ended the process
#the file 'out.file' has not been created at that point.
#_____________end____________________________________________________________________

	[[alternative HTML version deleted]]